spiega deploy reveal config

spiega deploy

This documentation project needs some software to be able to produce rich documentation out of text files. Here we list the software and the configuration used. We don’t want to use public models to avoid information leak but optimize the output for tiny local models.

How we deploy this project

LLMs

covered in agent calling

LLM serving

we test and benchmark different LLM serving tools

LLM with coding tools

present

covered in present documentation

container

Everything here is containerized:

safe

I exactly know what the agents can see

scale

I can reproduce easily the same configuration on other hardware

performant

I decide how many resources I want to allocate

admin

I can kill a container anytime

storage

I sacrifice a bit of storage but gain a lot in control

image

it took a lot of time to find the best image to serve multiple services on nvidia, amd and arm

hardware

We want to use a simple configuration, we have models hosted on a simple gaming laptop and an IoT device . What we use should work with AMD and ARM. ARM is particularly tricky since the compatibility between cuda-toolkit and pytorch requires a lot of trials between versions and compiling from source. ARM lacks a lot of deb packages and I tried to install blender 5 from source for example but didn’t succeeded.

architecture

The host has few tools, the idea is that the client side is minimal and the heavy calculation will be performed on the containers which can be easily moved. Decreasing the context and tools for the coding assistants increases security and improves productivity.

---
title: knowledge parsing
fontSize: 10
darkMode: True
theme: neo-dark
---
flowchart LR
KN["`
source code
knowledge
personal
`"]
DOC@{ shape: docs, label: "Knowledge"}
LLM@{ shape: procs, label: "LLMs"}
EM@{ shape: lin-cyl, label: "emacs" }

EM -- writes --> DOC
LLM -- reads --> DOC
LLM -- writes --> KN

LLM serving

What we have tested so far ¹ :

ollama

fully featured and integrated it is really easy to bind it to software like editors or coding assistants but performances are pretty poor and it’s the token/s are not enough

unsloth studio

really performant (llama_cpp level), fully featured but still in beta. It needs a lot of work around with endpoints

[ ] llama_cpp

performant but not user friendly. It requires a lot of configuration, even finding and downloading a useful model is painful

[ ] vllm

provided a configuration but I still couldn’t test it

[ ] lm studio

good performances but I want to simplify the stack

jev serving

We test jev as well

cd ~/Downloads/
git clone https://github.com/browser-use/jev-ultrafast.git
cd jev-ultrafast
uv sync
cp .env.example .env
uv run jev

coding assistant support

In order for LLM to be helpful we need the functionalities of coding assistants so they can indeed test what they claim and reduce hallucinations.

IDE tools

they include IDE tools like syntax checks, linters, unit tests so the model can test the output of the produced code before exiting

search

web search for specific API documentation (which I cache)

Coding assistants can be really dangerous

rules

nothing grants they get followed, coding agents often overcome them

filesystem

they can access the entire filesystem and they often start looking at files outside of the context

sensitive information

they can access passwords, keys and sensitive data

delete

they can delete a single file or the entire project without backuping the data

execute

they execute commands outside of the context and rules

super user

access to docker is really dangerous

computing

they get stuck in loops and keep consuming resources

coding assistant types

We worked and tested different coding assistants

pi coding

pi coding, nicely integrated with ollama. Less tools and more productive

opencode

over featured, not too efficient

[ ] aider

UI not really useful

[ ] cursor

not a real assistant, produces code which you can’t edit without explaining what you did and why and hoping it understands the context

implementation

We currently have the following services which we can deploy locally or on server

[ ] webserver

nginx, php, mariadb, certbot

docker containers

After many many trials I tested different images to be able to select a good configuration for LLM serving and coding agent. Ollama has a simple configuration but is not performant so I had to create new containers to link the other services. It is pretty straightforward to launch a standalone service but to integrate them it gets pretty complicated. I tried agentic supports but they were always find non integrated solutions which make portability hard.

docker user

The tricky part is to let an user inside docker modify the local files on the host without changing the permissions. This costed me a lot of time to configure and I still find the setup not optimal. Docker decided to create a non root user as ubuntu which makes difficult to replicate the same permissions. The workaround I found uses a conf in Dockerfile and to link the ubuntu user to username to match my host user and add a bunch of groups and permissions. The /home directory is still mounted as root so the user struggles to create conf files in home. I don’t want to mount the entire home directory but only the folder containing the source code.

RUN usermod -l username ubuntu
RUN usermod -d /home/username/ username
RUN usermod -s /bin/bash username
RUN groupadd username
RUN usermod -aG ubuntu username
RUN usermod -aG username username

export DOCKER_USER="$(id -u):$(id -g)"

and in docker-compose.yml to mount the folder as current user

user: "${UID-1000}:${GID-1000}"

further development

dictation

In case we don’t want to type, we can use dictation software to transpose our voice into text. Here we test different dictation tools.

Voxtype

sudo apt install cargo libasound2-dev libclang-dev cmake pkg-config
cargo build --release
sudo apt install wtype
./target/release/voxtype setup --download
./target/release/voxtype

Hyprland (~/.config/hypr/hyprland.conf):
bind = SUPER, V, exec, voxtype record start
bindr = SUPER, V, exec, voxtype record stop

speedofsound

#sudo apt install voxtype
#flatpak install flathub net.mkiol.SpeechNote
sudo snap install speedofsound

(keymap-global-set "s-z" #'speedofsound)

CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                  PORTS                                                                          NAMES
ca8d2804a6e5   ghcr.io/k3d-io/k3d-proxy:5.9.0   "/bin/sh -c nginx-pr…"   13 minutes ago   Up 13 minutes           80/tcp, 0.0.0.0:37877->6443/tcp                                                k3d-spiega-serverlb
056c497f5a54   rancher/k3s:v1.35.5-k3s1         "/bin/k3d-entrypoint…"   14 minutes ago   Up 13 minutes                                                                                          k3d-spiega-server-0
4a16eed24c3a   websearch-api                    "/entrypoint.sh"         2 hours ago      Up 2 hours              0.0.0.0:8092->8092/tcp, [::]:8092->8092/tcp                                    orio-search-api
d07ec672c3e3   redis:7-alpine                   "docker-entrypoint.s…"   2 hours ago      Up 2 hours (healthy)    0.0.0.0:6379->6379/tcp, [::]:6379->6379/tcp                                    orio-search-redis
578c40f81b15   searxng/searxng:latest           "/usr/local/searxng/…"   2 hours ago      Up 2 hours (healthy)    0.0.0.0:8091->8080/tcp, [::]:8091->8080/tcp                                    orio-search-searxng
49b9a2e4dbff   alpine/socat:latest              "socat TCP-LISTEN:91…"   3 days ago       Up 3 days                                                                                              blender-mcp-bridge
60962c6c4fb6   alpine/socat:latest              "socat TCP-LISTEN:13…"   3 days ago       Up 3 days                                                                                              lemonade-mcp-bridge
4f5e8494e64a   mcp_server-mcp_server            "bash /app/run.sh"       13 days ago      Up 4 days (unhealthy)   127.0.0.1:8171-8172->8171-8172/tcp                                             mcp_server
17e093fb3202   mariadb                          "docker-entrypoint.s…"   2 months ago     Up 4 days               3306/tcp                                                                       webserver-mysql-1
d0bfb6f96a58   webserver-nginx                  "nginx -g 'daemon of…"   2 months ago     Up 4 days               0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   nginx

nerd dictation

pip3 install vosk
git clone https://github.com/ideasman42/nerd-dictation.git
cd nerd-dictation
wget https://alphacephei.com/kaldi/models/vosk-model-small-en-us-0.15.zip
unzip vosk-model-small-en-us-0.15.zip
mv vosk-model-small-en-us-0.15 model

whisper model

pip install -U openai-whisper

TODO training

Unsloth was originally developed as tool for training. Here we want

custom

a custom model that learns from my way of programming while I use a coding agent

orchestration

We start with a new cluster

#wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
#curl -sfL https://get.k3s.io | sh -
cd ~/lav/src/spiega/deploy/k3s
k3d cluster create spiega

k3d cluster list
k3d node list

NAME     SERVERS   AGENTS   LOADBALANCER
spiega   1/1       0/0      true
NAME                  ROLE           CLUSTER   STATUS
k3d-spiega-server-0   server         spiega    running
k3d-spiega-serverlb   loadbalancer   spiega    running

kubectl get pods --all-namespaces    

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   17m

kubectl get nodes

NAME                  STATUS   ROLES           AGE   VERSION
k3d-spiega-server-0   Ready    control-plane   13m   v1.35.5+k3s1

docker ps

kubectl cluster-info

Kubernetes control plane is running at https://0.0.0.0:37877
CoreDNS is running at https://0.0.0.0:37877/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:37877/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

kubectl -n kube-system get all 

NAME                                          READY   STATUS      RESTARTS   AGE
pod/coredns-8db54c48d-kq29f                   1/1     Running     0          14m
pod/helm-install-traefik-44bzt                0/1     Completed   1          14m
pod/helm-install-traefik-crd-q2ncc            0/1     Completed   0          14m
pod/local-path-provisioner-5d9d9885bc-2xmk7   1/1     Running     0          14m
pod/metrics-server-786d997795-plfbx           1/1     Running     0          14m
pod/svclb-traefik-66a097a0-n7dkj              2/2     Running     0          13m
pod/traefik-9bcdbbd9-b2785                    1/1     Running     0          13m

NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/kube-dns         ClusterIP      10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP       14m
service/metrics-server   ClusterIP      10.43.78.182   <none>        443/TCP                      14m
service/traefik          LoadBalancer   10.43.88.161   172.19.0.2    80:31923/TCP,443:30552/TCP   13m

NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-traefik-66a097a0   1         1         1       1            1           <none>          13m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns                  1/1     1            1           14m
deployment.apps/local-path-provisioner   1/1     1            1           14m
deployment.apps/metrics-server           1/1     1            1           14m
deployment.apps/traefik                  1/1     1            1           13m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-8db54c48d                   1         1         1       14m
replicaset.apps/local-path-provisioner-5d9d9885bc   1         1         1       14m
replicaset.apps/metrics-server-786d997795           1         1         1       14m
replicaset.apps/traefik-9bcdbbd9                    1         1         1       13m

NAME                                 STATUS     COMPLETIONS   DURATION   AGE
job.batch/helm-install-traefik       Complete   1/1           30s        14m
job.batch/helm-install-traefik-crd   Complete   1/1           28s        14m

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=LoadBalancer
kubectl get deployment

kubectl get pods
kubectl get svc

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   17m

helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
helm install grafana grafana/grafana

"grafana" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "grafana" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: grafana
LAST DEPLOYED: Wed Sep 23 16:42:00 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.default.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace default port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################

testing

model suggestion

For a local, heterogeneous environment (ARM, AMD, NVIDIA), avoid vanilla Kubernetes. Use a lightweight distribution to match your “simple configuration” requirement.

1. Distribution: Use K3s (single binary, ARM-native) or Kind. K3s supports device plugins for NVIDIA (Jetson) and AMD (ROCm) GPUs out of the box.

2. Permissions (Fixing Host Access): Instead of the usermod workaround in the Dockerfile, use HostPath volumes restricted by RBAC.

   ;; hostPath allows editing without root inside the pod
   volumeMounts:
     - name: workspace
       mountPath: /app/code
   volumes:
     - name: workspace
       hostPath:
         path: /home/youruser/projects
   

   Configure the Pod Security Context to allow the specific node user (matching ubuntu) to access the path without full root access.

3. GPU Integration: Deploy the nvidia-device-plugin (for NVIDIA) or equivalent for AMD to dynamically allocate GPU resources to the LLM pods.
4. Services:
   • Ingress: Use nginx to expose the LLM and API endpoints securely.
   • Storage: Keep models in ConfigMaps or local PVCs for portability. Keep source code in hostPath for speed.
5. Migration: Convert docker-compose.yml to Helm Charts or Kustomize to manage the stack declaratively.

This setup reduces context (as desired for security) while maintaining the flexibility of containerization on ARM/AI hardware.

Write here in org-mode the snippet to create the k3s configuration suggested above:

Installation

# Install K3s (works on ARM, AMD, NVIDIA)
curl -sfL https://get.k3s.io | sudo INSTALL_MODE=1 sh -
# Configure GPU support in containerd (requires NVIDIA container runtime)
# Add to /etc/k3s/agent.env for worker nodes
# K3S_EXEC="--node-label gpu=true --node-role=worker"

• Server Configuration (k3s.yaml)

# This is for the server startup args, usually passed to the server process
# server:
#   service-port: 6443
#   agent-port: 10000
#   cluster-init: false
#   token: "mytoken"
#   data-store-endpoint: /var/lib/rancher/k3s/server/db
#   disable:
#     traefik: false  # Only if you handle ingress manually
#
# To enable GPU support at runtime, ensure the 'nvidia-container-toolkit'
# is installed on the host and the runtime is set in containerd.

• Deployment (Ollama + HostPath)

#+BEGIN_EXAMPLE

This deployment addresses the permission issue:

1. Uses ’hostPath’ to access the code without root inside the pod.

2. Matches host user/group via ’securityContext’.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      # Use host PID for better file permissions
      containers:
        - name: ollama
          image: ollama/ollama
          ports:
            - containerPort: 11434
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
            - name: code
              mountPath: /app
          # Request GPU if present
          resources:
            limits:
              nvidia.com/gpu: 1
      volumes:
        # Access host directory (must be writable by the UID)
        - name: code
          hostPath:
            path: /home/youruser/projects/source
            type: DirectoryOrCreate
        - name: models
          hostPath:
            path: /home/youruser/.ollama
            type: DirectoryOrCreate

#+END_EXAMPLE

• Note on Permissions

;; Ensure K3s containerd respects the host user for file operations
;; Edit /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

• Prerequisites

This setup assumes a K3s server is installed. Replace the command in the DaemonSet with the correct K3s command if not running via script.

• Configuration Files

k3s_server.yaml

# Install K3s on server
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.0+ex1.sh

ollama_service.yaml

This defines the LLM service with hostPath for source editing.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          command:
            - ollama
            - serve
          ports:
            - containerPort: 11434
          securityContext:
            runAsUser: 1000  # Match host UID
            fsGroup: 1000     # Match host GID for hostPath
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
            - name: workspace
              mountPath: /workspace
      volumes:
        - name: models
          hostPath:
            path: /root/.ollama
            type: DirectoryOrCreate
        - name: workspace
          hostPath:
            path: ~/projects/spiega  # Adjust to your host path
            type: DirectoryOrCreate

nvidia_gpu.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: llm
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
  namespace: kube-system
  labels:
    app: nvidia-device-plugin
spec:
  selector:
    matchLabels:
      app: nvidia-device-plugin
  template:
    metadata:
      labels:
        app: nvidia-device-plugin
    spec:
      serviceAccountName: nvidia-device-plugin
      containers:
        - name: device-plugin
          image: nvidia/k8s-device-plugin:1.0
          volumeMounts:
            - name: nvidia-device-plugin-socket
              mountPath: /var/run/nvidia-device-plugin.sock
              name: nvidia-device-plugin-socket
      volumes:
        - name: nvidia-device-plugin-socket
          hostPath:
            path: /var/run/nvidia-device-plugin.sock

• Usage
• Apply the server config (already run).

• Apply GPU support:

  kubectl apply -f k3s/gpu-daemonset.yaml
  

• Apply LLM service:

  kubectl apply -f k3s/ollama-deploy.yaml
  

• Port Forward for access:

  kubectl port-forward svc/ollama 11434:11434 -n llm
  

• Edit source directly on host, accessible via mounted volume at /workspace inside the pod.