spiega deploy

After many testing we came to this setup:

LLM

anik-jha/Qwen3.6-35B-A3B-coding-reap50-GGUF : 43 token/s, really capable thanks to the removal of experts moep, looping sometimes, too long context damage efficiency and start looping

options

KV cache: 8bit, no vision, multi token prediction

provider

unsloth (less featured than lemonade but really performant)

assistant

pi_coding (uses less context)

sandbox

neither the provider nor the assistant can reach sensitive data. Only folders in the context

Serving local LLMs is tricky, I expected the tools to be more advanced in managing resources but most of the options are not really optimized and there is a lot to test to have a real advantage in serving models. If a model is too slow or too irresponsible there is improvement.

speed

should be at least 10 token/s

coding

coding assistants use tools, sometimes too many, sometimes too few

thinking

some models do overthink

hardware

some providers don’t optimize CPU, GPU, NPU, RAM, VRAM

reliability

the assistant to operate destroying changes

compromise

as I learned with LaTeX sometimes is better to give up with wishes and do what the tool is best at

Select the right hardware is crucial for LLM. We see that computing power is not as important as:

bandwidth

billions of weights get move per token

byte/token

bytes per token

RAM/VRAM/storage

how much you move around

context

how much is needed per task, how much you can compress the kv cache.

thinking

to be balanced properly

It is really difficult to come out with a clear metric about quality and it depends a lot on the task

graphics

many models without screenshots

consistent

how much that particular model creates logical results

We need to balance:

tools

what LLMs should use

instructions

instructions should be generic enough to overcome lack of common sense by LLM

ownership

the assistant to be autonomous enough to progress but not too much to create no-sense

testing

too many tests are unproductive, too few risky

productivity

a clear added value

clean

over-engineered code is hard to maintain and fix

reliable

no hidden consequences

Regarding the models we can download we need to consider few options to make sure we use the best tool for our scope

Ideally we don’t use a single LLM for everything but we can mix dedicated models to optimize system capabilities. Orchestration on a laptop is tough, it is hard to have multiple models running at the same time and loading/unloading models and context takes time. Still we want to create a pipeline where we can use different models to increase efficiency.

We can as well use python to switch between models or comfyUI.

Context takes a lot of RAM. We need to understand what tasks we can perform for the given contexts because too often the model interrupts before finishing the task. Really important is during each session to ask the model to keep one README.html per folder updated so we can resume the task without re-loading the session and consume tokens.

Different UI provide options to load models but they don’t suggest the best fitting for our purpose and hardware.

I have 32GB RAM and 8GB VRAM which is difficult for coding purposes

Each of this parameters can bring a lot of boost in performance

It is really tricky to provide instructions to agents so they don’t

token

coding assistant consuming too much

context

provide too much information

misunderstanding

agents interpret instruction differently from common sense

power

it is better to limit agents in sandbox and let them experiment more than overthink

This is the current implementation after many, many trials.

Scope: integrate emacs and org files to create infographics.
You live inside docker ~/lav/src/spiega/deploy/mcp_server/docker-compose.yml and mount project folders only. I'm on host.
You can use sudo in your container but can't use docker.
Backup before editing in ~/lav/src/spiega/tmp/ -> ask if test is passed -> delete.
Don't overthink, tasks are straightforward.
When we start a session ask first what you want to do before executing so I make sure you understand the scope.
Write and update the folder README.html as you go about the scope we are working on.

Everything here is containerized:

safe

I exactly know what the agents can see

scale

I can reproduce easily the same configuration on other hardware

performant

I decide how many resources I want to allocate

admin

I can kill a container anytime

storage

I sacrifice a bit of storage but gain a lot in control

image

it took a lot of time to find the best image to serve multiple services on nvidia, amd and arm

We want to use a simple configuration, we have models hosted on a simple gaming laptop and an IoT device jetson. What we use should work with AMD and ARM. ARM is particularly tricky since the compatibility between cuda-toolkit and pytorch requires a lot of trials between versions and compiling from source. ARM lacks a lot of deb packages and I tried to install blender 5 from source for example but didn’t succeeded.

The host has few tools, the idea is that the client side is minimal and the heavy calculation will be performed on the containers which can be easily moved. Decreasing the context and tools for the coding assistants increases security and improves productivity.

---
title: knowledge parsing
fontSize: 10
darkMode: True
theme: neo-dark
---
flowchart LR
KN["`
source code
knowledge
personal
`"]
DOC@{ shape: docs, label: "Knowledge"}
LLM@{ shape: procs, label: "LLMs"}
EM@{ shape: lin-cyl, label: "emacs" }

EM -- writes --> DOC
LLM -- reads --> DOC
LLM -- writes --> KN

We have tested multiple LLMs and want to note down quality

anik-jha/Qwen3.6-35B-A3B-coding-reap50-GGUF

43 token/s, really capable thanks to the removal of experts moep, looping sometimes, too long context damage efficiency

unsloth/Qwen3.5-9B-GGUF

for simple tasks, hard to get clear understanding, could sit on VRAM with basic contextx

What we have tested so far ¹ :

ollama

fully featured and integrated it is really easy to bind it to software like editors or coding assistants but performances are pretty poor and it’s the token/s are not enough

unsloth studio

really performant (llama_cpp level), fully featured but still in beta. It needs a lot of work around with endpoints

[ ] llama_cpp

performant but not user friendly. It requires a lot of configuration, even finding and downloading a useful model is painful

[ ] vllm

provided a configuration but I still couldn’t test it

[ ] lm studio

good performances but I want to simplify the stack

We test jev as well

cd ~/Downloads/
git clone https://github.com/browser-use/jev-ultrafast.git
cd jev-ultrafast
uv sync
cp .env.example .env
uv run jev

In order for LLM to be helpful we need the functionalities of coding assistants so they can indeed test what they claim and reduce hallucinations.

IDE tools

they include IDE tools like syntax checks, linters, unit tests so the model can test the output of the produced code before exiting

search

web search for specific API documentation (which I cache)

Coding assistants can be really dangerous

rules

nothing grants they get followed, coding agents often overcome them

filesystem

they can access the entire filesystem and they often start looking at files outside of the context

sensitive information

they can access passwords, keys and sensitive data

delete

they can delete a single file or the entire project without backuping the data

execute

they execute commands outside of the context and rules

super user

access to docker is really dangerous

computing

they get stuck in loops and keep consuming resources

We worked and tested different coding assistants

pi coding

pi coding, nicely integrated with ollama. Less tools and more productive

opencode

over featured, not too efficient

[ ] aider

UI not really useful

[ ] cursor

not a real assistant, produces code which you can’t edit without explaining what you did and why and hoping it understands the context

After many many trials I tested different images to be able to select a good configuration for LLM serving and coding agent. Ollama has a simple configuration but is not performant so I had to create new containers to link the other services. It is pretty straightforward to launch a standalone service but to integrate them it gets pretty complicated. I tried agentic supports but they were always find non integrated solutions which make portability hard.

The tricky part is to let an user inside docker modify the local files on the host without changing the permissions. This costed me a lot of time to configure and I still find the setup not optimal. Docker decided to create a non root user as ubuntu which makes difficult to replicate the same permissions. The workaround I found uses a conf in Dockerfile and to link the ubuntu user to username to match my host user and add a bunch of groups and permissions. The /home directory is still mounted as root so the user struggles to create conf files in home. I don’t want to mount the entire home directory but only the folder containing the source code.

RUN usermod -l username ubuntu
RUN usermod -d /home/username/ username
RUN usermod -s /bin/bash username
RUN groupadd username
RUN usermod -aG ubuntu username
RUN usermod -aG username username

export DOCKER_USER="$(id -u):$(id -g)"

and in docker-compose.yml to mount the folder as current user

user: "${UID-1000}:${GID-1000}"

In case we don’t want to type, we can use dictation software to transpose our voice into text. Here we test different dictation tools.

Voxtype

sudo apt install cargo libasound2-dev libclang-dev cmake pkg-config
cargo build --release
sudo apt install wtype
./target/release/voxtype setup --download
./target/release/voxtype

Hyprland (~/.config/hypr/hyprland.conf):
bind = SUPER, V, exec, voxtype record start
bindr = SUPER, V, exec, voxtype record stop

speedofsound

#sudo apt install voxtype
#flatpak install flathub net.mkiol.SpeechNote
sudo snap install speedofsound

(keymap-global-set "s-z" #'speedofsound)

CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                  PORTS                                                                          NAMES
ca8d2804a6e5   ghcr.io/k3d-io/k3d-proxy:5.9.0   "/bin/sh -c nginx-pr…"   13 minutes ago   Up 13 minutes           80/tcp, 0.0.0.0:37877->6443/tcp                                                k3d-spiega-serverlb
056c497f5a54   rancher/k3s:v1.35.5-k3s1         "/bin/k3d-entrypoint…"   14 minutes ago   Up 13 minutes                                                                                          k3d-spiega-server-0
4a16eed24c3a   websearch-api                    "/entrypoint.sh"         2 hours ago      Up 2 hours              0.0.0.0:8092->8092/tcp, [::]:8092->8092/tcp                                    orio-search-api
d07ec672c3e3   redis:7-alpine                   "docker-entrypoint.s…"   2 hours ago      Up 2 hours (healthy)    0.0.0.0:6379->6379/tcp, [::]:6379->6379/tcp                                    orio-search-redis
578c40f81b15   searxng/searxng:latest           "/usr/local/searxng/…"   2 hours ago      Up 2 hours (healthy)    0.0.0.0:8091->8080/tcp, [::]:8091->8080/tcp                                    orio-search-searxng
49b9a2e4dbff   alpine/socat:latest              "socat TCP-LISTEN:91…"   3 days ago       Up 3 days                                                                                              blender-mcp-bridge
60962c6c4fb6   alpine/socat:latest              "socat TCP-LISTEN:13…"   3 days ago       Up 3 days                                                                                              lemonade-mcp-bridge
4f5e8494e64a   mcp_server-mcp_server            "bash /app/run.sh"       13 days ago      Up 4 days (unhealthy)   127.0.0.1:8171-8172->8171-8172/tcp                                             mcp_server
17e093fb3202   mariadb                          "docker-entrypoint.s…"   2 months ago     Up 4 days               3306/tcp                                                                       webserver-mysql-1
d0bfb6f96a58   webserver-nginx                  "nginx -g 'daemon of…"   2 months ago     Up 4 days               0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   nginx

nerd dictation

pip3 install vosk
git clone https://github.com/ideasman42/nerd-dictation.git
cd nerd-dictation
wget https://alphacephei.com/kaldi/models/vosk-model-small-en-us-0.15.zip
unzip vosk-model-small-en-us-0.15.zip
mv vosk-model-small-en-us-0.15 model

whisper model

pip install -U openai-whisper

Unsloth was originally developed as tool for training. Here we want

custom

a custom model that learns from my way of programming while I use a coding agent

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
helm install grafana grafana/grafana

"grafana" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "grafana" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: grafana
LAST DEPLOYED: Wed Sep 23 16:42:00 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.default.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace default port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################

# Install K3s (works on ARM, AMD, NVIDIA)
curl -sfL https://get.k3s.io | sudo INSTALL_MODE=1 sh -
# Configure GPU support in containerd (requires NVIDIA container runtime)
# Add to /etc/k3s/agent.env for worker nodes
# K3S_EXEC="--node-label gpu=true --node-role=worker"

• Server Configuration (k3s.yaml)

# This is for the server startup args, usually passed to the server process
# server:
#   service-port: 6443
#   agent-port: 10000
#   cluster-init: false
#   token: "mytoken"
#   data-store-endpoint: /var/lib/rancher/k3s/server/db
#   disable:
#     traefik: false  # Only if you handle ingress manually
#
# To enable GPU support at runtime, ensure the 'nvidia-container-toolkit'
# is installed on the host and the runtime is set in containerd.

• Deployment (Ollama + HostPath)

#+BEGIN_EXAMPLE

This defines the LLM service with hostPath for source editing.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          command:
            - ollama
            - serve
          ports:
            - containerPort: 11434
          securityContext:
            runAsUser: 1000  # Match host UID
            fsGroup: 1000     # Match host GID for hostPath
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
            - name: workspace
              mountPath: /workspace
      volumes:
        - name: models
          hostPath:
            path: /root/.ollama
            type: DirectoryOrCreate
        - name: workspace
          hostPath:
            path: ~/projects/spiega  # Adjust to your host path
            type: DirectoryOrCreate