agent calling

motivation

Agents support with multiple tasks but can get quickly out of control. The setting we build here has the following features:

local

all the knowledge sits inside this network and internet is hardly required

litte resources

the network is nothing but a and

automation

the creation of the knowledge base and graphs is automated

human tasks

to ideate and break down tasks, the tools do the rest

interconnection

knowledge base is connected with task description, connected with effort estimate, connected with action plan, connected with agenda

few dependencies

the project rely on few software: an IDE, python and LLM serving software

safety

the LLMs don’t have access to confidential information, the coding agent can see only restricted folders

cheap

use the power of your local hardware, no subscription, no fees

structure

start thinking at planning first, execution is the least effort

document

the single source of truth create a vast documentation to further define tasks

present

the same files used for planning create slides and technical documentation

organized

access easily all the important configurations in your project

setup

Following is a visual representation of the differences between two extremes:

vibe+public

you use a software like cursor and you pay a subscription to a public mode

emacs@local

you have full control of the deployment and you know exactly what the models can see and edit

v The estimate comes from the following considerations:

public models

are much more advanced than local, open-source models

client IDE

access all your files

agents

it’s really hard to give precise instructions and make sure they respect the boundaries

coding agent

each one has different policies but often they overdo and it’s time consuming to get maintainable code

integration

the ease of adding tools and capabilities which of course is in favor of open-source

enhancement

is the difference between boost in productivity and overview work, sometimes the balance is negative

local tools

Single tools can improve some features but make other less effective:

agents

provide added knowledge but can develop the project in a direction where is difficult to gain control

coding assistant

can write useful code and add unknown libraries but can break everything

skills

can be dangerous without proper control

knowledge base

is essential to maintain consistency but files need to be updated and linked

REPL

test on the fly is the best option to avoid never ending testing

Contribution of single tools

quadrantChart
    title implementation benefits
    x-axis Low help --> High assistance
    y-axis Low control --> High ownership
    quadrant-1 control and productivity 
    quadrant-2 control but useless
    quadrant-3 unpredictable
    quadrant-4 high costs/low ownership
    current setup: [0.6, 0.7]
    public models: [0.9, 0.23]
    no coding agent: [0.30, 0.69]
    no docker: [0.6, 0.34]
    no .org file: [0.40, 0.34]

agentic components   executive

Agents can fulfill many tasks. The work has shifted from the paradigm of executing tasks (and often not document them) to document first what you want to do and plan (or let plan) the agents to execute. The most important task is to test and quality check the results which means to own the pipeline and put enough logs and monitoring tools to allow traceability (know what agent did what).

We need an interface which is usually an IDE or a terminal to plan and orchestrate everything.

Here the choice is:

IDE

emacs because you focus on typing and don’t get distracted by useless information

plan

.org files: they contain agenda, trees, graphs, attributes, source code… you can basically write a complete software within a file

LLM

ollama because it’s important to own the processes and limit spending

coding agent

currently on pi-agent, opencode and aider where not efficient with local models. Aider UX is limited (no autocompletion, messed up my repo)

MCP

language models work at best with tools, I keep building MCP servers to ease LLM tasks

graphs

all the results are outputted in nodes and links to be visualized and navigated. I use neo4j to store the data

programming

mainly in python using REPL functionalities in emacs to test and execute every single line of code and check the data results

deploy

all the resources with so I can easily move them around

hardware

local: laptop and jetson (desc: )

Here is a sketch of the project

---
title: implementation sketch
---
flowchart LR
KN["`
script
knowledge
agenda
links
tasks
`"]
DOC@{ shape: docs, label: "Knowledge"}
MC@{ shape: procs, label: "mcp server"}
DT@{ shape: lin-cyl, label: "storage" }
EL@{ shape: notch-pent, label: "ellama" }
GP@{ shape: notch-pent, label: "gptel" }
PI@{ img: "/home/sabeiro/lav/src/spiega/icon/dev.svg", label: "pi-coding", pos: "c", w: 60, h: 60, constraint: "off" }
%%A@{ icon: "fa:user", form: "square", label: "User Icon", pos: "t", h: 60 }

E(emacs) --> GP
E --> EL
E -- ask --> EL
EL -- connect --> OL[\ollama\]
OL -- answer --> EL
EL -- insert --> E
GP -- connects --> MC
MC -- summarize --> DOC
GP -- decide --> JT[\vllm\]
JT -- elaborate --> GP
GP -- insert --> E
E -- debug --> PI
PI -- write --> DT
PI -- edit --> E
E -- edit --> OR[org-file]
OR -- contain --> KN

The documentation is generated inside this

active plan   management

A brief representation of the implementation plan

graph LR
A[emacs] --> B[gptel]
A[emacs] --> C[ellama]

Code for generating the documentation.

active deploy

Here we explore different options to deploy local models, the only common denominator is docker because we want to control what those models can access and reduce context. Many services have docker options but the pre-built images are difficult to integrate in our workflow and the proposed images are heavy and hard to tune. To find a performant solution we decided to start from a “yeast” image following simple principles:

Dockerfile

static starts from the same ubuntu image, install the dependencies, adds the /app folder and grant permissions. Building a container takes time and we want only the static information to be in the image

docker-compose.yml

dynamic includes the network, health-check, volumes, environment variables, launches the app. We need a quick way to test and change the configuration.

The build time takes 10min to 1h so we really need to separate the static components from the dynamic one. We want to standardize the process because unfortunately every service has its own: folder structure, endpoints, configuration files, ports, enviroment… Here I force all the different project to be reachable from the host in a similar way to avoid re-writing patches and integration tools. Additionally part of the hardware we use runs on ARM so we need to build our own images.

Dockerfile

The “yeast” start from a ubuntu image and installs basic libraries

FROM ubuntu:latest
RUN apt-get update && apt-get install -y locales
RUN apt install -y curl libatomic1 python3-pip bash git  
#&& rm -rf /var/lib/apt/lists/* 
#RUN localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8

We copy the app files into the container

COPY ./app /app
RUN chmod +x  /app/run.sh
WORKDIR /app

We need then to allow the container user to edit the files keeping the same permission as the host user to allow coding assistants to edit the files keeping the same consistency. Unfortunately this step is not easy in docker for reasons I don’t understand but I found a workaround ref1.

ARG UID=1000
ARG GID=1000
RUN usermod -l $USER ubuntu
RUN usermod -aG ubuntu $USER
RUN usermod -aG $USER $USER
RUN usermod -d /home/$USER/ $USER
RUN usermod -s /bin/bash $USER
RUN groupadd $USER

We grant sudo and no password:

RUN passwd --delete sabeiro
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
RUN adduser sabeiro sudo
RUN mkdir -p /home/sabeiro/ && chown -R ${UID}:${GID} /home/sabeiro/
USER sabeiro
RUN sudo ls

We need to add to docker-compose.yml to mount the volume as non-root

user: "${UID}:${GID}" 

On top of this image we install some applications

RUN curl -fsSL https://openclaw.ai/install.sh | bash -s -- --no-prompt --no-onboard
RUN curl -fsSL https://pi.dev/install.sh |  bash -s -- --no-prompt --no-onboard
RUN curl -fsSL https://opencode.ai/install | bash -s -- --no-prompt --no-onboard
RUN curl -LsSf https://aider.chat/install.sh | bash -s --
RUN curl -fsSL https://unsloth.ai/install.sh | sh

And we build the image

docker compose up --build

We then test the configuration using bash inside the container

docker compose exec -it ubuntu bash

docker-compose.yml

Docker compose here starts the container built in the previous step

services:
  ubuntu_base:
    build:
      context: ./
      dockerfile: Dockerfile
    container_name: ubuntu_base
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    runtime: "nvidia"
    ports:
      - "8083:8083"
    restart: unless-stopped
    stdin_open: true
    tty: true
    working_dir: /app/
    entrypoint: ["bash","/app/run.sh"]
    healthcheck:
      test: ["CMD", "cat", "/app/run.sh"]
      interval: 30s
      timeout: 30s
      retries: 3
    volumes:
      - ${HOME}/Downloads/llm_model/:${HOME}/models/
      - ${HOME}/lav/src/:${HOME}/lav/src/
      - ${HOME}/log/ubuntu/:/var/log/
      - ./app/:/app/
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - webserver-net

networks:
  webserver-net:
    name: webserver_webserver-net
    driver: bridge
    external: true

Network and webserver are on a separate container

services:
  nginx:
    container_name: nginx
    build:
      context: nginx/
    restart: unless-stopped
    tty: true
    cpus: 0.2
    environment:
      SERVICE_NAME: app
      SERVICE_TAGS: dev
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d
      - ${HOME}/lav/siti:/var/www/html/
      - ${HOME}/log:/var/log/
    ports:
      - "80:80"
      - "443:443"
    networks:
      - webserver-net

devices

We use GPUs in this project

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]
runtime: "nvidia"

We can as well limit RAM and CPU resources which we tune restarting the containers.

volumes

Volumes are managed by docker-compose, we use a standard structure:

app

all the files we need to run the service

log

mount a /log directory where all services can write logs separated from the repo

edit

we mount the source files the agents should be able to access with the same permissions as the host user

conf

all the configuration files we need to export to the app in the guest

static

a folder outside of the repo where we download and share big files like LLM models

network

We use a webserver-net network to connect the different containers. This network is managed from the nginx container. To let the host and the container to communicate we could use socat.

services:
  blender-mcp-bridge:
    image: alpine/socat:latest
    container_name: blender-mcp-bridge
    network_mode: "host"
    command: TCP-LISTEN:9192,fork,reuseaddr TCP:127.0.0.1:9191
    restart: unless-stopped

  lemonade-mcp-bridge:
    image: alpine/socat:latest
    container_name: lemonade-mcp-bridge
    network_mode: "host"
    command: TCP-LISTEN:13306,fork,reuseaddr TCP:127.0.0.1:13305
    restart: unless-stopped

agents

To allow the agents to work on the host system we need different workarounds:

tty

we need to allow console interactions

permissions

we use the same username as the host

network bridge

to use external programs on host like blender

deploy on docker

All the LLM services like ollama, llama.cpp, unsloth, lmstudio… have docker versions we can use but those implementations can be barely used outside of testing since the real added value of LLM is the combination with a coding assistant. Ollama is by far the most easy to integrate tool, their APIs are included in many packages and you can integrate

hardware

This is the but we want to deploy our local agents on other devices as well. Those are the main specs.

backends

We have different backends to explore:

ROCm

Open software stack that includes programming models, tools, compilers, libraries, and runtimes for AI and HPC solution development on AMD GPUs

CPU

cuda

allows software to use certain types of graphics processing units for accelerated general-purpose processing

vulkan

next generation graphics and compute API that provides high-efficiency, cross-platform access to modern GPUs used in PCs, consoles, mobile phones and embedded platforms.

todo CPU

Generally the GPU is a better option for LLM but the CPU can access 32GB of memory.

todo GPU

To use the nvidia graphic cards I need to install many packages

ROCm

From linuxvox.

sudo apt update && sudo apt upgrade
sudo mkdir --parents --mode=0755 /etc/apt/keyrings
wget https://repo.amd.com/rocm/packages-multi-arch/gpg/rocm.gpg -O - | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/amdrocm.gpg > /dev/null
 
sudo tee /etc/apt/sources.list.d/rocm.list << EOF
deb [arch=amd64 signed-by=/etc/apt/keyrings/amdrocm.gpg] https://repo.amd.com/rocm/packages-multi-arch/ubuntu2404 stable main
EOF
sudo apt update
sudo apt install rocm
sudo usermod -a -G render,video $LOGNAME
pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm7.2/

done NPU

Nowadays laptops have NPU (neural processing units) but it is generally difficult to use them the same way they are used in cell phones. The recommended size for LLM application is 40 tops but I should be able to use my 16 tops for light tasks like autocompletion. The idea is to use fast flow lm with this implementation Let’s try to follow this guide and the official documentation. Need to download the drivers.

Ubuntu 26 has the package lemonade: run LLMs and Whisper on the AMD XDNA 2 NPU. This solution is made up of:

• Upstream NPU driver in the Linux 7.0+ kernel (with backports for 6.xx kernels).
• AMD IRON compiler for XDNA NPUs.
• FastFlowLM, a lightweight LLM runtime for AMD NPUs, which today has added Linux support.
• Lemonade, ties everything together with a streamlined user experience.

sudo apt update
sudo apt install python3.12 python3.12-venv libboost-filesystem1.74.0 dkms
sudo apt update && sudo apt install linux-oem-24.04c
cd /home/sabeiro/Elŝutujo
unzip RAI_1.8_Linux_NPU_XRT.zip
sudo apt install --fix-broken -y ./xrt_202620.2.25.37_24.04-amd64-base.deb
sudo apt install --fix-broken -y ./xrt_202620.2.25.37_24.04-amd64-base-dev.deb
sudo apt install --fix-broken -y ./xrt_202620.2.25.37_24.04-amd64-npu.deb
sudo apt install --fix-broken -y ./xrt_plugin.2.25.260102.56.release_24.04-amd64-amdxdna.deb

source /opt/xilinx/xrt/setup.sh
xrt-smi examine

Need to download ryzen ai.

cd /home/sabeiro/Elŝutujo
mkdir ryzen_ai-1.8.0
cp ryzen_ai-1.8.0.tgz ryzen_ai-1.8.0
cd ryzen_ai-1.8.0
tar -xvzf ryzen_ai-1.8.0.tgz

Dependencies for docker git clone https://github.com/hpenedones/fastflowlm-docker.git cd fastflowlm-docker docker build -t fastflowlm .

sudo add-apt-repository ppa:amd-team/xrt
sudo apt update && sudo apt install libxrt-npu2
# Set memlock to unlimited (needs reboot)
echo -e "* soft memlock unlimited\n* hard memlock unlimited" | sudo tee -a /etc/security/limits.conf
sudo reboot

git clone https://github.com/hpenedones/fastflowlm-docker.git
cd fastflowlm-docker
docker build -t fastflowlm .

lemonade

Between ollama, llama.cpp, unsloth, vllm the best experience was with lemonade. Lemonade has a simple UI able to manage:

llama.cpp

for serving LLM

vllm

for serving LLM

whisper

talk2text

stable diffusion

image generation

ROCm

for amd

cuda

nvidia drivers

onnx

custom models

vulkan

cross platform API for 3D (like openGL)

todo benchmark

Now let’s test the configuration and check what performs at best. We have tested:

ollama

11434

unsloth

8889

lmstudio

llama.cpp

curl localhost:11434/api/tags | jq | grep \"model\" | awk -F " " '{print $2}' 

profiling

Performances are crucial for a real added value and we need to make sure the system is well-configured. Coding agents are not effective, they keep trying until tests pass so speed is crucial.

1. Two llama-server instances running on ports 57569 and 54001 (both serving the same model)
2. Swap is actively being used (3GB + 2.9GB = 5.9GB out of 8GB available)
3. GPU utilization at 0% - The servers are not using GPU efficiently

configuration analysis

Based on the system monitoring, here are the findings:

Hardware Resources

• GPU: NVIDIA GeForce RTX 5060 Laptop GPU (8151 MiB VRAM)
• RAM: 30 GB total, ~5.4 GB available
• CPU: AMD Ryzen 7 260 (16 cores)
• Model: Qwen3.5-9B-GGUF (~9 GB compressed, needs offloading to RAM)

Performance Bottlenecks Identified

1. Swap Memory Active ⚠️ Critical
   • 3GB of /swapfile + 2.9GB of /swap.img in use
   • Swap swapping kills LLM performance by orders of magnitude
   • Swappiness is set to 60 (too high for LLM workloads)
2. Redundant Unsloth Instances ⚠️ High Waste
   • Instance 1: Port 8888, `–parallel 1`, low utilization
   • Instance 2: Port 54001, `–parallel 2`, active but suboptimal flags
   • Both serving the same model unnecessarily
3. Context Length Mismatch
   • Server running with `-c 22528` (half of max)
   • Could benefit from higher batch size for throughput

optimization steps

Step 1

Disable Swap Memory: Swap memory is the #1 killer of LLM performance. When RAM fills up, data moves to disk at ~20-100 MB/s vs GPU VRAM’s 500 GB/s.

Temporary fix:

sudo swapoff -a

Permanent fix: Add to `/etc/rc.local` or create `/etc/systemd/system/disable-swap.service`:

[Unit]
Description=Disable Swap for LLM Workloads

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'swapoff -a'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Reduce swappiness (optional):

sudo sysctl vm.swappiness=10
# Make permanent in /etc/sysctl.conf:
vm.swappiness=10
vm.vfs_cache_pressure=50

Step 2

Stop Redundant Unsloth Server

Find and kill the port 8888 server:

pkill -f "unsloth run.*port 8888"
# or
kill $(pgrep -f "unsloth run.*--port=8888")

Step 3

optimized configuration file optimized

Step 4: Adjust for Your Hardware (RTX 5060 Laptop)

Your GPU has 8 GB VRAM but Qwen3.5-9B needs more than that. Here’s the optimal configuration:

For Qwen3.5-9B-GGUF:

python unsloth run \
  -H 127.0.0.1 \
  -p 8889 \
  --model unsloth/Qwen3.5-9B-GGUF \
  --context-length 32768 \
  --parallel 4 \
  --max-batch-size 4 \
  --flash-attn on \
  -c 32768 \
  -ngl -1 \
  --kv-unified off \
  --metrics

Key optimizations:

• `–kv-unified off`: Disables KV cache unification (better for single GPU)
• `–parallel 4`: Enable concurrent requests (you have RAM headroom)
• `–max-batch-size 4`: Process multiple tokens in parallel
• `-c 32768`: Full context utilization

### Step 5: Monitor and Tune

Check performance metrics:

# Watch GPU usage
watch -n 1 nvidia-smi

# Check for swap activity
watch -n 1 "free -h && swapon --show"

# Monitor Unsloth server
curl http://localhost:8889/metrics | grep -E "tokens|time|iters"

Expected improvements:

• Tokens/second: Should increase from ~15 to ~25-30 tokens/sec
• Latency: First token should drop from 500ms to ~200ms
• Throughput: Sustained throughput could reach 40-60 tokens/sec with `–parallel 4`

### Step 6: Alternative - Use Ollama Instead

If Unsloth continues to be slow, consider switching to Ollama which is better optimized for single-GPU laptop setups:

# Stop Unsloth
pkill -f unsloth

# Start Ollama (already configured in your system)
ollama serve

# Pull quantized model (less VRAM usage)
ollama pull qwen3.5:9b

Ollama benefits:

• Better single-GPU optimization
• Lower overhead (~10-15% faster for single user)
• Less memory footprint
• Simpler configuration

### Quick Performance Checklist

• VRAM sufficient (8 GB for 9B model) ⚠️ Model will partially offload to RAM
• [ ] Swap disabled CRITICAL
• [ ] Redundant servers stopped
• [ ] `–parallel` increased to match workload
• [ ] Batch size tuned for throughput
• [ ] Flash Attention enabled ✓
• [ ] Context length optimized ✓

### Model Recommendations for Your Hardware

Best recommendation: Switch to `qwen2.5-coder:7b` or `codegemma:7b` for 30% faster performance with same quality for coding tasks.

### Emergency Commands

# Stop all Unsloth instances
pkill -f unsloth

# Check current GPU memory usage
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

# Restart with optimized settings
~/unsloth_optimized.sh &

# Force stop if frozen
sudo fuser -k 8889/tcp

# Monitor swap (should be empty)
watch -n 1 "free -h | grep Swap"

project management

A few tools for project management using org and agents.

gantt for the project

A Gantt representation of the project

---
displayMode: compact
title: Ignored if specified in chart
config:
  gantt:
      useWidth: 800
      rightPadding: 0
      topAxis: true  #false
      numberSectionStyles: 2
---
gantt
    dateFormat  <YYYY-MM-DD>
    title  Knowledge base action plan
    excludes  weekends

    review : vert, v1, <2026-06-22>, 1d

    section local models
        deploy LLMs          :done,  deploy, <2026-05-20>, 7d
        coding agent         :done,  deploy, <2026-05-27>, 14d

    section list & summarize
        parse knowledge         :done,  dev, <2026-06-01>, 5d
        create graphs           :done,  dev, <2026-06-07>, 7d

    section hierarchy
        hierarchy     :done, distil, <2026-06-16>, 9d
        visualization :crit, distil, <2026-06-20>, 4d

    section video
        sceencast    :active, create, <2026-06-12>, 5d
        animate text :active, create, <2026-06-17>, 9d

    section publish
        content     :active, share, <2026-06-22>, 10d
        publication :milestone, share, <2026-06-27>, 5d

Kanban

We can as well integrate our workflow into a Kanban

---
config:
  kanban:
    ticketBaseUrl: 'https://mermaidchart.atlassian.net/browse/#TICKET#'
---
kanban
  Todo
    [compare model serve]
    docs[benchmark vllm, llama.cpp and ollama]
  [In progress]
    id6[blog posts about the local implementation ]
  id9[Ready for deploy]
    id8[cloud webdav]@{ assigned: 'bot1' }
  id10[Ready for test]
    id4[Create parsing tests]@{ ticket: MC-2038, assigned: 'K.Sveidqvist', priority: 'High' }
    id66[last item]@{ priority: 'Very Low', assigned: 'knsv' }
  id11[Done]
    id5[agent confs, org files]
    id2[local deployment]@{ ticket: MC-2036, priority: 'Very High'}
    id3[graph integration]@{ ticket: MC-2037, assigned: bot1, priority: 'High' }

sankey

We can show sankey diagrams too

---
config:
  sankey:
    showValues: false
---
sankey

LLM cloud,LLM local,95
cursor,pi-coding,60
cursor,aider,20
cursor,open-code,20

timeline

We can present as well a timeline regarding the development of this project

timeline
    title blender twin development
    2025-12-12 : nvidia jetson CV application
    2026-01-12 : camera controller with pose estimate
    2026-02-12 : physics optimization engine in js
    2026-03-12 : moving from cloud to local LLMs
    2026-04-12 : new laptop to run local + jetson
    2026-05-12 : .org files and productivity workflow
    2026-06-12 : knowledge graph

done architecture   architecture

Architectural diagram for the project:

flowchart TD
    subgraph Sensors ["IoT Sensor Layer"]
        A[Temperature Sensors] -->|MQTT| B[Messaging Bus]
        C[Humidity Sensors] -->|MQTT| B
        D[Pressure Sensors] -->|MQTT| B
    end

    subgraph Processing ["Processing Layer"]
        B -->|Ingest| E[Data Stream Processor]
        E -->|Normalize| F[Time Series DB]
        E -->|Transform| G[Feature Extractor]
        G -->|Analyze| H[Anomaly Detector]
    end

    subgraph Intelligence ["Intelligence Layer"]
        H -->|Alerts| I[Predictive ML Model]
        I -->|Predictions| J[Decision Engine]
        F -->|Historical Data| K[Knowledge Graph]
    end

    subgraph Visualization ["Visualization Layer"]
        J -->|Commands| L[Control Actions]
        K -->|Entity Relations| M[Graph Database]
        E -->|State Updates| Q[3D WebGL Viewer]
        Q -->|Render| R[Blender Scene]
    end

    subgraph Twin ["Digital Twin Model"]
        R -->|Sync| V[Physics Simulation]
        V -->|Thermal Analysis| W[Heat FEM Solver]
        W -->|Results| Q
    end

    style Sensors fill:#e1f5fe
    style Processing fill:#fff3e0
    style Intelligence fill:#e8f5e9
    style Visualization fill:#f3e5f5
    style Twin fill:#ffe0b2
    linkStyle default stroke:#333,stroke-width:2px

done coding assistant   present dev

The current coding assistant is on pi-agent while opencode, aider and cursor where also tested.

pi-agent

currently the running options because it’s more efficient for local models. It reduces the load and unload of files and the number of tokens.

cursor

is not opensource and I can’t integrated in my workflow. Cursor itself said that with using python REPL emacs it’s the best option

aider

I find it less invasive than opencode but struggles to perform simple operations and creates a lot of junk in the folder

opecode

a bit too invasive for my taste, it takes too many autonomous decisions making it difficult to control

The current configuration uses pi-coding and the configuration is specified in:

rules

how the agent should behave in this project

AGENTS.html

what types of agents we need

SYSTEM.html

what is the system configuration

SYSTEM_APPEND.html

further agent defitions

SKILLS.html

skillset for agents

commands

ESC to stop the current operation, /q to quit

security

pi runs inside the same container as ollama with the same user as the host system. Mounted is only the src directory so pi cannot access critical information like passwords, keys… and cannot run any docker command which can easily get to sudo rights

done emacs   present

Emacs is omnipresent in my developments . Emacs integrates bash commands with macros and program outputs, I can connect language models with mcp servers. I currently use emacs.el as current init file and gptel_tools.el for connecting with mcp. gptel_tools_allowed commands is an additional list of bash commands which the tool allows

Features:

LLMs

I use different packages to interact with the local models

ellama

for reasoning and structuring the project and for code examples in any buffer

gptel

for the integration with mcp

gptel base

base package

mcp.el

start the hub

custom gptel tools

user defined tools

gptel-mcp

integration between the packages

mcp

for adding my own tools and external mcp

aideremacs

connect aider with emacs but was creating too much trash in the repository

pi agent

connect pi-agent with emacs

The most efficient way to integrate LLMs into the workflow is having a integration with all the available tools.

sequenceDiagram
 emacs-->ellama: prompt
 emacs-->gptel: tools
 emacs-->pi_agent : instruction
 pi_agent-->ollama: prompt
 gptel-->mcp_server: prompt
 mcp_server-->llama.cpp: instruction
 gptel-->emacs: code
 pi_agent-->emacs: code

.org files LLM integration

The most efficient way to integrate LLMs is to use and their capabilities. Org files contain many different tags which are interpreted as multiple entities.

dates

<2026-06-19 Fri> as agenda entries

clock

[C-c C-x TAB] to start logging the start of an activity

nodes

ID and ROAM_ALIASES to tag any section or file

agenda

to schedule a task or to set an alert [C-c a]

tags

to specify meta information for that section [C-c C-q]

status

whether an action is done or pending [Space - left/right arrow]

trees

explain hierarchical structures [::]

code

define code to execute `#+begin_src bash` `#+end_src`

plots

plot data with gnuplot

link

link to anything: files, websites, buffers, images…

local org files

open new buffers from shell

LLMs should be able to correctly find and update those tags and automate the work of logging what the user is doing.

active mcp for code dev   present dev

I currently have the following mcp servers:

blender-mcp

to use blender from emacs

proj desc

alternative

proj desc 2

alternative

proj desc 3

alternative

blender mcp with client

alternative

integration llama.cpp

alternative

jetson-mcp

a custom mcp server running on a nvidia jetson using computer vision capabilities with pytorch

bottino-mcp

a custom mcp server running locally on this laptop

mcp-hub

additional gptel tools administrated by mcp_server

active microcontroller   embedded

The microcontollers are the eyes and ears of the LLMs. We need to extend our MCP network to access and link the different devices so we can enhance our capabilities to the physical world. Here is the list of the we use. For that we need:

microcontrollers

ESP32, arduino, teensy

sensors

motion, temperature

camera

ESP32, jetson raspy HD

webserver

ESP32 to expose an endpoint our MCP server can call

interface

BLE midi is really versatile

knwoledge ellama   dev

Ellama is an Emacs frontend for chatting with large language models via llm-ollama. Models run locally through Ollama – no cloud API needed.

• jdormit/emacs-opencode (https://github.com/jdormit/emacs-opencode) — native client: M-x opencode for sessions, opencode-ask for prompts, session management, and deep Emacs integration via emacs-mcp-server (read buffers, eval elisp, run compile commands)
• colobas/opencode.el (https://github.com/colobas/opencode.el) — port of my tools into gptel with LSP integration, diagnostics, symbol navigation, and agent presets
• rogs/opencode.el (https://git.rogs.me/rogs/opencode.el) — lightweight terminal-based integration using vterm or shell-mode with a transient menu
• xenodium/agent-shell (https://github.com/xenodium/agent-shell) — ACP-based native Emacs buffer for LLM agents, supports OpenCode as a backend
• tninja/ai-code-interface.el (https://github.com/tninja/ai-code-interface.el) — unified interface wrapping multiple CLI agents including OpenCode

emacs.el loads the whole ellama setup lazily via ellama-load-and-call (emacs.el lines 810-815), which adds user-emacs-directory to the load path and pulls in ellama_setup (=~/.emacs.d/ellama_setup.el). All providers and helpers below are defined there, not in emacs.el.

Ellama connects directly to the LLM backend

Chat providers are built with make-llm-ollama (default http://127.0.0.1:11434). Unlike gptel, it does not route through the MCP server (the bottino/mcp FastAPI agent), so it has no access to the MCP tools (ollama_list_models, camera_describe_scene, etc.) or the MCP tool-call loop.

ellama_setup.el discovers providers dynamically at startup. The codepath used is my-unsloth-list-models — it queries Unsloth’s {UNSLOTH_URL}/api/inference/models and requires UNSLOTH_TOKEN (sent as a Bearer token; Unsloth rejects unauthenticated model-list requests). Set UNSLOTH_URL (default http://127.0.0.1:8889) to point at your Unsloth server. my-ollama-list-models (Ollama’s /api/tags via OLLAMA_URL, default http://127.0.0.1:11434) is present but currently not used — the refresh helper was switched to the Unsloth source.

Listed model names appear in M-x ellama-switch-provider automatically after a 3-second idle delay at startup (or instantly via M-x my-ellama-refresh-providers), with a qwen2.5:7b fallback provider if discovery fails.

Note: discovery lists Unsloth model names but builds make-llm-ollama providers — the resolved names must be reachable through your chosen backend.

Key bindings

Ellama starts via the global keymap defined in ellama_setup.el:

C-x e is deliberately left unbound by ellama so it keeps its stock meaning kmacro-end-and-call-macro (macro execute); the ellama prefix uses C-c e.

Providers (models)

Defined in =~/.emacs.d/ellama_setup.el:

Default chat provider (ellama-provider)

• Model: qwen3.5:9b
• Embedding: nomic-embed-text
• Context: 8192 tokens (num_ctx)
• Used for: general conversation, questions

Coding provider (ellama-coding-provider)

• Model: qwen2.5-coder:7b
• Embedding: nomic-embed-text
• Context: 32768 tokens
• Used for: code generation, debugging, refactoring
• Activated via M-x ellama-code (there is no dedicated prefix for it, but the coding provider can also be selected with M-x ellama-switch-provider)

Summarization provider (ellama-summarization-provider)

• Model: llama3.2:latest
• Embedding: nomic-embed-text
• Context: 32768 tokens
• Used for: summarizing regions, buffers, or files

Translation provider (ellama-translation-provider)

• Model: qwen2.5:7b
• Embedding: nomic-embed-text
• Context: 32768 tokens

Extraction provider (ellama-extraction-provider)

• Model: qwen2.5-coder:7b-instruct-q8_0
• Embedding: nomic-embed-text
• Context: 32768 tokens
• Used for: structured data extraction from text

Naming provider (ellama-naming-provider)

• Model: qwen2.5:7b
• Used for: auto-naming new chat sessions (via ellama-generate-name-by-llm)
• Uses a stop non-standard param of a newline

Dynamic providers for interactive switching

ellama-providers is (re)built by my-ellama-refresh-providers from the Unsloth model list at startup (my-unsloth-list-models), falling back to a single qwen2.5:7b provider when discovery fails. Each entry embeds qwen2.5:7b-style make-llm-ollama settings with nomic-embed-text and 32768-token context, so newly pulled models appear automatically in M-x ellama-switch-provider.

interaction

Switch interactively with M-x ellama-switch-provider.

Start a chat

C-c e or M-x ellama

This opens a chat buffer in full frame. Type your message, then C-c C-c to send.

Coding assistant

M-x ellama-code or select the coding provider interactively.

Ask for code, refactoring, or debugging help. The coding provider uses qwen2.5-coder:7b with 32K context – suitable for whole functions or small files.

Summarize text

Select a region, then M-x ellama-summarize or M-x ellama-summarize-region. Ellama will summarize the selected text using the summarization provider.

Translate text

M-x ellama-translate – prompts for target language, then translates the current region or buffer.

Extract structured data

M-x ellama-extract – extracts structured information from text (e.g. names, dates, quantities from free-form text).

Use a different model

M-x ellama-switch-provider – pick from the dynamically-discovered providers, or any other provider you define.

Display behaviour

• Chat buffer opens in full frame (display-buffer-full-frame) — note that ellama_setup.el overrides the chat/instant display action to display-buffer-no-window at load (lines 5-6), so the interactive prompts stay unobtrusive.
• Header line shows ellama context and session ID in all buffers (ellama-context-header-line-global-mode, ellama-session-header-line-global-mode).

Requirements

• Emacs ≥ 30
• An LLM backend reachable by ellama_setup.el — either:
  • Unsloth, for dynamic model discovery (UNSLOTH_URL, default http://127.0.0.1:8889; needs UNSLOTH_TOKEN); or
  • Ollama (OLLAMA_URL, default http://127.0.0.1:11434) serving the chat models below
• Models expected (check with ollama list):
  • qwen3.5:9b (default chat)
  • qwen2.5-coder:7b (coding)
  • llama3.2:latest (summarization)
  • qwen2.5:7b (naming, translation, fallback)
  • qwen2.5-coder:7b-instruct-q8_0 (extraction)
  • nomic-embed-text (embeddings)

Troubleshooting

Syncing models with Ollama   dev

Ellama now discovers models dynamically at startup (with a 3-second idle delay so init isn’t blocked). When you pull a new model in Ollama, it appears automatically after restarting Emacs, or you can refresh on demand:

M-x my-ellama-refresh-providers

This queries Ollama’s /api/tags endpoint, filters out embedding-only models (nomic-embed-text, all-minilm), and rebuilds the ellama-providers alist with a make-llm-ollama entry per model.

Check which models are actually available

ollama list

or from Emacs:

M-x shell-command ollama list

Change the Ollama endpoint

By default, discovery hits http://127.0.0.1:11434/api/tags. To route through the MCP server’s nginx instead, set:

MY_OLLAMA_URL=http://bottino:11434

or in Emacs:

M-x set-variable my-ollama-url "http://bottino:11434" M-x my-ellama-refresh-providers

Refresh on pulling a new model

ollama pull deepseek-coder:6.7b

Then in Emacs:

M-x my-ellama-refresh-providers M-x ellama-switch-provider → select deepseek-coder:6.7b

aider assistant

Useful commands

/add

<files> * Add files to the chat (Aider can edit them).

/drop

[files] * Remove files from the chat. No args drops everything.

/ls

List files currently in the chat.

/read-only

<files> * Add files for reference but block edits.

/clear

Clear the visible chat history.

/reset

Drop all files and clear history.

/tokens

* Print token usage for the current chat.

/cost

Show the running session cost.

/code

* Switch to code mode (or run one prompt in code mode).

/architect

* Switch to architect mode.

/ask

* Switch to ask mode (read-only).

/chat-mode

<name> Generic mode switcher.

/context

Enter context mode.

/model

<name> * Switch the main model on the fly.

/models

<query> Search the available models list.

/editor-model

<name> Switch the editor model.

/weak-model

<name> Switch the weak model.

/think-tokens

N Set thinking-token budget per turn.

/reasoning-effort

LEVEL Set reasoning effort for compatible models.

/diff

* Show the diff Aider just wrote.

/undo

* Revert the last commit Aider made.

/git

<cmd> Run a git command without leaving the agent.

/commit

[msg] Commit dirty changes manually.

/run

<cmd> * Run a shell command and feed the output into the chat. Alias: !.

/test

[cmd] Run the configured test command. Aider sees the output.

/lint

Run the configured lint command.

/web

<url> Fetch a URL and add the rendered text to the chat.

/voice

Voice input via Whisper. Requires portaudio.

/paste

Paste image or text from the clipboard.

/copy

Copy the last assistant reply to the clipboard.

/copy-context

Copy chat context as markdown.

/load

PATH Run a saved /command script.

/save

PATH Save the current session as a replayable script.

/multiline-mode

or /ml Toggle multi-line input.

/vim

Toggle Vim key bindings.

/edit

or /editor [cmd] Open $EDITOR for a long prompt.

/map

Print the current repo map.

/map-refresh

Force a repo-map refresh now.

/settings

Print effective settings.

/ok

Shorthand approval (used in Architect mode).

/report

Open a pre-filled GitHub issue with anonymised diagnostics.

/help

Aider answers questions about itself.

/quit

or /exit Exit cleanly.