Created: 2026-08-21 Fri 13:30
We are looking for local agents to support our development. We need to create a simple system to make sure:
Agents support with multiple tasks but can get quickly out of control. The setting we build here has the following features:
Following is a visual representation of the differences between two extremes:
v The estimate comes from the following considerations:
| conf | consistency | control | privacy | proficiency | integration | overview | versatility |
|---|---|---|---|---|---|---|---|
| vibe+public | 3.7 | 2.5 | 0.5 | 5.0 | 3.7 | 2.7 | 2.5 |
| emacs@local | 4.7 | 4.9 | 5.0 | 3.7 | 4.8 | 4.8 | 5.0 |
Single tools can improve some features but make other less effective:
Figure 1: Evaluation of local vs cloud alternatives
Contribution of single tools
| conf | consistency | control | safety | boost | integration | overview | enhancement |
|---|---|---|---|---|---|---|---|
| +gptel | 0 | -0.2 | -0.2 | 0.4 | 0.3 | 0.8 | 0.2 |
| +ellama | 0 | -0.2 | -0.2 | 0.3 | 0.2 | 0.3 | 0.3 |
| +mcp | 0 | -0.2 | -0.2 | 0.2 | 1.0 | 0 | 0.5 |
| +org | 1.0 | 0.3 | 0 | 0.4 | 0.4 | 1.0 | 0.4 |
| +repl | 1.0 | 0.3 | 0 | 0.2 | 0.7 | 0.3 | 0.6 |
| +pi-code | 0 | -0.5 | -0.4 | 0.5 | 0.2 | -0.2 | 0.3 |
Figure 2: Contribution of the single tools to the project
quadrantChart
title implementation benefits
x-axis Low help --> High assistance
y-axis Low control --> High ownership
quadrant-1 control and productivity
quadrant-2 control but useless
quadrant-3 unpredictable
quadrant-4 high costs/low ownership
current setup: [0.6, 0.7]
public models: [0.9, 0.23]
no coding agent: [0.30, 0.69]
no docker: [0.6, 0.34]
no .org file: [0.40, 0.34]
Figure 3: Quadrant representation of the project and its goals
Agents can fulfill many tasks. The work has shifted from the paradigm of executing tasks (and often not document them) to document first what you want to do and plan (or let plan) the agents to execute. The most important task is to test and quality check the results which means to own the pipeline and put enough logs and monitoring tools to allow traceability (know what agent did what).
We need an interface which is usually an IDE or a terminal to plan and orchestrate everything.
Here the choice is:
Here is a sketch of the project
---
title: implementation sketch
---
flowchart LR
KN["`
script
knowledge
agenda
links
tasks
`"]
DOC@{ shape: docs, label: "Knowledge"}
MC@{ shape: procs, label: "mcp server"}
DT@{ shape: lin-cyl, label: "storage" }
EL@{ shape: notch-pent, label: "ellama" }
GP@{ shape: notch-pent, label: "gptel" }
PI@{ img: "/home/sabeiro/lav/src/spiega/icon/dev.svg", label: "pi-coding", pos: "c", w: 60, h: 60, constraint: "off" }
%%A@{ icon: "fa:user", form: "square", label: "User Icon", pos: "t", h: 60 }
E(emacs) --> GP
E --> EL
E -- ask --> EL
EL -- connect --> OL[\ollama\]
OL -- answer --> EL
EL -- insert --> E
GP -- connects --> MC
MC -- summarize --> DOC
GP -- decide --> JT[\vllm\]
JT -- elaborate --> GP
GP -- insert --> E
E -- debug --> PI
PI -- write --> DT
PI -- edit --> E
E -- edit --> OR[org-file]
OR -- contain --> KN
Figure 4: diagram of the implementation
The documentation is generated inside this .org file snippet
graph LR A[emacs] --> B[gptel] A[emacs] --> C[ellama]
Code for generating the documentation.
emacs has a lot of publishing options but in some cases I need a really custom design and therefore I strip header and footer from the html and append my own. We have a bash file for that convert_slide.sh. I need as well to convert the mermaid svg graph into a format blender can import which means first of all to convert the plain text into path which is accomplished by text2svg. This script tries as well to rename path ids to allow easier grouping into blender.
cd $HOME/lav/src/blender_twin/docs/
#bash ./script/convert_slide.sh
bash ./script/convert_single.sh ./plan/agent_call.org
#bash ./script/text2svg.sh
static/agent_call.html
Here we explore different options to deploy local models, the only common denominator is docker because we want to control what those models can access and reduce context. Many services have docker options but the pre-built images are difficult to integrate in our workflow and the proposed images are heavy and hard to tune. To find a performant solution we decided to start from a “yeast” image following simple principles:
The build time takes 10min to 1h so we really need to separate the static components from the dynamic one. We want to standardize the process because unfortunately every service has its own: folder structure, endpoints, configuration files, ports, enviroment… Here I force all the different project to be reachable from the host in a similar way to avoid re-writing patches and integration tools. Additionally part of the hardware we use runs on ARM so we need to build our own images.
The “yeast” start from a ubuntu image and installs basic libraries
FROM ubuntu:latestRUN apt-get update && apt-get install -y localesRUN apt install -y curl libatomic1 python3-pip bash git#&& rm -rf /var/lib/apt/lists/*#RUN localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8ENV LANG en_US.utf8
We copy the app files into the container
COPY ./app /appRUN chmod +x /app/run.shWORKDIR /app
We need then to allow the container user to edit the files keeping the same permission as the host user to allow coding assistants to edit the files keeping the same consistency. Unfortunately this step is not easy in docker for reasons I don’t understand but I found a workaround.
ARG UID=1000ARG GID=1000RUN usermod -l $USER ubuntuRUN usermod -aG ubuntu $USERRUN usermod -aG $USER $USERRUN usermod -d /home/$USER/ $USERRUN usermod -s /bin/bash $USERRUN groupadd $USER
On top of this image we install some applications
RUN curl -fsSL https://openclaw.ai/install.sh | bash -s -- --no-prompt --no-onboardRUN curl -fsSL https://pi.dev/install.sh | bash -s -- --no-prompt --no-onboardRUN curl -fsSL https://opencode.ai/install | bash -s -- --no-prompt --no-onboardRUN curl -LsSf https://aider.chat/install.sh | bash -s --RUN curl -fsSL https://unsloth.ai/install.sh | sh
And we build the image
docker compose up --build
We then test the configuration using bash inside the container
docker compose exec -it ubuntu bash
Docker compose here starts the container built in the previous step
services:ubuntu_base:build:context: ./dockerfile: Dockerfilecontainer_name: ubuntu_basedeploy:resources:reservations:devices:- driver: nvidiacount: allcapabilities: [gpu]runtime: "nvidia"ports:- "8083:8083"restart: unless-stoppedstdin_open: truetty: trueworking_dir: /app/entrypoint: ["bash","/app/run.sh"]healthcheck:test: ["CMD", "cat", "/app/run.sh"]interval: 30stimeout: 30sretries: 3volumes:- ${HOME}/Downloads/llm_model/:${HOME}/models/- ${HOME}/lav/src/:${HOME}/lav/src/- ${HOME}/log/ubuntu/:/var/log/- ./app/:/app/- /var/run/docker.sock:/var/run/docker.socknetworks:- webserver-netnetworks:webserver-net:name: webserver_webserver-netdriver: bridgeexternal: true
Network and webserver are on a separate container
services:nginx:container_name: nginxbuild:context: nginx/restart: unless-stoppedtty: truecpus: 0.2environment:SERVICE_NAME: appSERVICE_TAGS: devvolumes:- ./nginx/conf.d:/etc/nginx/conf.d- ${HOME}/lav/siti:/var/www/html/- ${HOME}/log:/var/log/ports:- "80:80"- "443:443"networks:- webserver-net
We use GPUs in this project
deploy:resources:reservations:devices:- driver: nvidiacount: allcapabilities: [gpu]runtime: "nvidia"
We can as well limit RAM and CPU resources which we tune restarting the containers.
Volumes are managed by docker-compose, we use a standard structure:
We use a webserver-net network to connect the different containers. This network is managed from the nginx container.
To allow the agents to work on the host system we need different workarounds:
All the LLM services like ollama, llama.cpp, unsloth, lmstudio… have docker versions we can use but those implementations can be barely used outside of testing since the real added value of LLM is the combination with a coding assistant. Ollama is by far the most easy to integrate tool, their APIs are included in many packages and you can integrate
A few tools for project management using org and agents.
A Gantt representation of the project
---
displayMode: compact
title: Ignored if specified in chart
config:
gantt:
useWidth: 800
rightPadding: 0
topAxis: true #false
numberSectionStyles: 2
---
gantt
dateFormat <YYYY-MM-DD>
title Knowledge base action plan
excludes weekends
review : vert, v1, <2026-06-22>, 1d
section local models
deploy LLMs :done, deploy, <2026-05-20>, 7d
coding agent :done, deploy, <2026-05-27>, 14d
section list & summarize
parse knowledge :done, dev, <2026-06-01>, 5d
create graphs :done, dev, <2026-06-07>, 7d
section hierarchy
hierarchy :done, distil, <2026-06-16>, 9d
visualization :crit, distil, <2026-06-20>, 4d
section video
sceencast :active, create, <2026-06-12>, 5d
animate text :active, create, <2026-06-17>, 9d
section publish
content :active, share, <2026-06-22>, 10d
publication :milestone, share, <2026-06-27>, 5d
Figure 5: Gantt representation of the project
We can as well integrate our workflow into a Kanban
---
config:
kanban:
ticketBaseUrl: 'https://mermaidchart.atlassian.net/browse/#TICKET#'
---
kanban
Todo
[compare model serve]
docs[benchmark vllm, llama.cpp and ollama]
[In progress]
id6[blog posts about the local implementation ]
id9[Ready for deploy]
id8[cloud webdav]@{ assigned: 'bot1' }
id10[Ready for test]
id4[Create parsing tests]@{ ticket: MC-2038, assigned: 'K.Sveidqvist', priority: 'High' }
id66[last item]@{ priority: 'Very Low', assigned: 'knsv' }
id11[Done]
id5[agent confs, org files]
id2[local deployment]@{ ticket: MC-2036, priority: 'Very High'}
id3[graph integration]@{ ticket: MC-2037, assigned: bot1, priority: 'High' }
Figure 6: Gantt representation of the project
We can show sankey diagrams too
---
config:
sankey:
showValues: false
---
sankey
LLM cloud,LLM local,95
cursor,pi-coding,60
cursor,aider,20
cursor,open-code,20
Figure 7: Sankey representation of the project
We can present as well a timeline regarding the development of this project
timeline
title blender twin development
2025-12-12 : nvidia jetson CV application
2026-01-12 : camera controller with pose estimate
2026-02-12 : physics optimization engine in js
2026-03-12 : moving from cloud to local LLMs
2026-04-12 : new laptop to run local + jetson
2026-05-12 : .org files and productivity workflow
2026-06-12 : knowledge graph
Figure 8: Timeline
Architectural diagram for the project:
flowchart TD
subgraph Sensors ["IoT Sensor Layer"]
A[Temperature Sensors] -->|MQTT| B[Messaging Bus]
C[Humidity Sensors] -->|MQTT| B
D[Pressure Sensors] -->|MQTT| B
end
subgraph Processing ["Processing Layer"]
B -->|Ingest| E[Data Stream Processor]
E -->|Normalize| F[Time Series DB]
E -->|Transform| G[Feature Extractor]
G -->|Analyze| H[Anomaly Detector]
end
subgraph Intelligence ["Intelligence Layer"]
H -->|Alerts| I[Predictive ML Model]
I -->|Predictions| J[Decision Engine]
F -->|Historical Data| K[Knowledge Graph]
end
subgraph Visualization ["Visualization Layer"]
J -->|Commands| L[Control Actions]
K -->|Entity Relations| M[Graph Database]
E -->|State Updates| Q[3D WebGL Viewer]
Q -->|Render| R[Blender Scene]
end
subgraph Twin ["Digital Twin Model"]
R -->|Sync| V[Physics Simulation]
V -->|Thermal Analysis| W[Heat FEM Solver]
W -->|Results| Q
end
style Sensors fill:#e1f5fe
style Processing fill:#fff3e0
style Intelligence fill:#e8f5e9
style Visualization fill:#f3e5f5
style Twin fill:#ffe0b2
linkStyle default stroke:#333,stroke-width:2px
Figure 9: result of mermaid plot
Using mainly ollama (with llama cpp took a lot of time for configuration) to serve LLMs. Ollama has a nice interface for python which basically removed all my external dependencies from langchain and llamaindex (which keeps on changing APIs and packaging and my code needs to be re-written over and over). I currently manage to run:
We tested as well other LLM serving toos: The current LLM serving tools tested
The current coding assistant is on pi-agent while opencode, aider and cursor where also tested.
The current configuration uses pi-coding and the configuration is specified in:
Emacs is omnipresent in my developments . Emacs integrates bash commands with macros and program outputs, I can connect language models with mcp servers. I currently use emacs.el as current init file and gptel_tools.el for connecting with mcp. gptel_tools_allowed commands is an additional list of bash commands which the tool allows
Features:
The most efficient way to integrate LLMs into the workflow is having a integration with all the available tools.
sequenceDiagram
emacs-->ellama: prompt
emacs-->gptel: tools
emacs-->pi_agent : instruction
pi_agent-->ollama: prompt
gptel-->mcp_server: prompt
mcp_server-->llama.cpp: instruction
gptel-->emacs: code
pi_agent-->emacs: code
Figure 10: result of mermaid plot
The most efficient way to integrate LLMs is to use and their capabilities. Org files contain many different tags which are interpreted as multiple entities.
LLMs should be able to correctly find and update those tags and automate the work of logging what the user is doing.
I currently have the following mcp servers:
The microcontollers are the eyes and ears of the LLMs. We need to extend our MCP network to access and link the different devices so we can enhance our capabilities to the physical world. Here is the list of the we use. For that we need:
Ellama is an Emacs frontend for chatting with large language models via llm-ollama. Models run locally through Ollama – no cloud API needed.
(http://127.0.0.1:11434). Unlike gptel, it does not route through the MCP server (the bottino/mcp FastAPI agent), so it has no access to the MCP tools (ollama_list_models, camera_describe_scene, etc.) or the MCP tool-call loop.
However, emacs.el now discovers models dynamically by querying Ollama’s /api/tags endpoint (through the same nginx that proxies MCP).Set MY_OLLAMA_URL env var to route discovery through the MCP server, e.g. http://bottino:11434. No more hardcoded model list — when you pull a new model in Ollama, it appears in M-x ellama-switch-provider automatically
(after a 3-second idle delay at startup, or instantly via M-x my-ellama-refresh-providers).
| Key | Command | Description |
|---|---|---|
C-c e |
ellama |
Start a new chat session |
C-c C-c |
ellama-chat-send-last-message |
Send last message in chat buffer |
Defined in emacs.el lines 332-404:
Default chat provider (ellama-provider)
qwen2.5:3bnomic-embed-text
Coding provider (ellama-coding-provider)
qwen2.5-coder:3bellama-code or C-c e c
Summarization provider (ellama-summarization-provider)
qwen2.5:3b
Translation provider (ellama-translation-provider)
qwen2.5:3b
Extraction provider (ellama-extraction-provider)
qwen2.5-coder:7b-instruct-q8_0
Naming provider (ellama-naming-provider)
qwen2.5:3bellama-generate-name-by-llm)Predefined providers for interactive switching
| Provider | Chat model | Embedding model |
|---|---|---|
zephyr |
zephyr:7b-beta-q6_K |
zephyr:7b-beta-q6_K |
mistral |
mistral:7b-instruct-v0.2-q6_K |
mistral:7b-instruct-v0.2-q6_K |
mixtral |
mixtral:8x7b-instruct-v0.1-q3_K_M-4k |
mixtral:8x7b-instruct-v0.1-q3_K_M-4k |
Switch interactively with M-x ellama-switch-provider.
Start a chat
C-c e or M-x ellama
This opens a chat buffer in full frame. Type your message, then C-c C-c
to send.
Coding assistant
M-x ellama-code or select the coding provider interactively.
Ask for code, refactoring, or debugging help. The coding provider uses
qwen2.5-coder:3b with 32K context – suitable for whole functions or
small files.
Summarize text
Select a region, then M-x ellama-summarize or M-x ellama-summarize-region.
Ellama will summarize the selected text using the summarization provider.
Translate text
M-x ellama-translate – prompts for target language, then translates the
current region or buffer.
Extract structured data
M-x ellama-extract – extracts structured information from text (e.g.
names, dates, quantities from free-form text).
Use a different model
M-x ellama-switch-provider – pick from zephyr, mistral, mixtral,
or any other provider you define.
Display behaviour
display-buffer-full-frame).display-buffer-at-bottom).ellama-context-header-line-global-mode,
ellama-session-header-line-global-mode).Requirements
emacs-major-version in emacs.el)http://127.0.0.1:11434)ollama list):
qwen2.5:3bqwen2.5-coder:3bqwen2.5-coder:7b-instruct-q8_0nomic-embed-textzephyr:7b-beta-q6_K, mistral:7b-instruct-v0.2-q6_K,
mixtral:8x7b-instruct-v0.1-q3_K_M-4k| Symptom | Likely cause | Fix |
|---|---|---|
No provider available |
Ollama not running | systemctl start ollama |
Connection refused |
Wrong host/port | Check OLLAMA_HOST env or ollama serve |
| Model not found | Model not pulled | ollama pull qwen2.5:3b |
| Model exists in Ollama but not in ellama | Need to refresh provider list | M-x my-ellama-refresh-providers or wait for idle timer |
| Slow responses | Model too large for hardware | Use qwen2.5:3b instead of 7B variants |
| Context truncated | num_ctx too small |
Increase num_ctx in provider config |
Ellama now discovers models dynamically at startup (with a 3-second idle delay so init isn’t blocked). When you pull a new model in Ollama, it appears automatically after restarting Emacs, or you can refresh on demand:
M-x my-ellama-refresh-providers
This queries Ollama’s /api/tags endpoint, filters out embedding-only
models (nomic-embed-text, all-minilm), and rebuilds the
ellama-providers alist with a make-llm-ollama entry per model.
ollama list
or from Emacs:
M-x shell-command ollama list
By default, discovery hits http://127.0.0.1:11434/api/tags. To route
through the MCP server’s nginx instead, set:
MY_OLLAMA_URL=http://bottino:11434
or in Emacs:
M-x set-variable my-ollama-url "http://bottino:11434"
M-x my-ellama-refresh-providers
ollama pull deepseek-coder:6.7b
Then in Emacs:
M-x my-ellama-refresh-providers
M-x ellama-switch-provider → select deepseek-coder:6.7b
Useful commands