llm provider

sabeiro

2026-09-05 Sat 00:00

Created: 2026-10-08 Thu 11:09

LLMs providers

We have tested and benchmarked multiples LLMs providers because we need to find a setup which is actually useful for productivity. After many trials I finally got lemonade running Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf with 35 token/s

implementation

We have tested:

ollama: 11434 Never managed to have acceptable speed
unslot: 8880 I didn’t manage to efficiently manage multiple models, performances were suffering
lemonade: 13305 best option on all aspects
vllm: many manual options
llama.cpp: difficult to adapt to load
lmstudio: little advantages to llama.cpp

provider	pros	cons
ollama	really reach in API (force json output)	really slow
unsloth	useful for fine tuning	issues with parallel requests
lemonade	can distribute models across NPU, CPU and GPU	short context size
vllm	performant	a lot of manual tests
llama.cpp	the basis for everything else	doesn’t adapt easily to load

They should all use openAI/ollama compatible endpoints so we define here the port of the service we want to test:

13305

ollama API

Standard ollama API endpoints (most of the providers are compatible with).

curl http://localhost:$PORT/api/tags | grep model

modelManagerWidth: 280,

curl http://localhost:$PORT/api/ps

{"models":[{"details":{"families":["thinksound"],"family":"thinksound","format":"gguf","parameter_size":"","parent_model":"","quantization_level":""},"digest":"sha256:0000000000000000000000000000000000000000000000000000000000000000","expires_at":"2099-01-01T00:00:00Z","model":"ThinkSound-SFX:latest","name":"ThinkSound-SFX:latest","size":0,"size_vram":0},{"details":{"families":["llamacpp"],"family":"llamacpp","format":"gguf","parameter_size":"9B","parent_model":"","quantization_level":"Q4_K_XL"},"digest":"sha256:0000000000000000000000000000000000000000000000000000000000000000","expires_at":"2099-01-01T00:00:00Z","model":"Qwen3.5-9B-GGUF:latest","name":"Qwen3.5-9B-GGUF:latest","size":0,"size_vram":0}]}

lemonade

Lemonde can manage

hardware: CPU, GPU, NPU
backend: llama.cpp, vulkan, whisper,
models: onnx, cuda, ROCm,
media: stable diffusion, audio, vision

manage system

We need to mange the system to not overload resources.

curl localhost:13305/live

{"status":"ok"}

curl localhost:13305/metrics | grep tokens_predicted

# HELP lemonade_llamacpp_tokens_predicted_total Number of generation tokens processed
# TYPE lemonade_llamacpp_tokens_predicted_total counter
lemonade_llamacpp_tokens_predicted_total{checkpoint="unsloth/Qwen3.5-9B-GGUF:Qwen3.5-9B-UD-Q4_K_XL.gguf",device="gpu",model_name="Qwen3.5-9B-GGUF",recipe="llamacpp",type="llm"} 2933
# HELP lemonade_llamacpp_tokens_predicted_seconds_total Total time spent generating tokens
# TYPE lemonade_llamacpp_tokens_predicted_seconds_total counter
lemonade_llamacpp_tokens_predicted_seconds_total{checkpoint="unsloth/Qwen3.5-9B-GGUF:Qwen3.5-9B-UD-Q4_K_XL.gguf",device="gpu",model_name="Qwen3.5-9B-GGUF",recipe="llamacpp",type="llm"} 61.7785

#curl localhost:13305/v1/health -H "Content-Type: application/json"  | jq | grep \"model\" | awk -F " " '{print $2}'
curl localhost:13305/v1/health -H "Content-Type: application/json"  | jq | grep model_name

"model_name": "Qwen3.5-9B-GGUF",

Load mode to lemonade:

curl localhost:13305/api/v1/load -H "Content-Type: application/json" -d '{"model_name": "Qwen3.5-9B-GGUF","ctx_size": 8192,"llamacpp_backend":"vulkan","pinned": true}'

{“checkpoint”:“unsloth/Qwen3.5-9B-GGUF:Qwen3.5-9B-UD-Q4_K_XL.gguf”

model_name:“Qwen3.5-9B-GGUF”

recipe:“llamacpp”

status:“success”}

Unload mode from lemonade:

curl localhost:13305/api/v1/unload -H "Content-Type: application/json" -d '{"model_name": "Qwen3.5-9B-GGUF","ctx_size": 8192,"llamacpp_backend":"vulkan","pinned": true}'

{“checkpoint”:“unsloth/Qwen3.5-9B-GGUF:Qwen3.5-9B-UD-Q4_K_XL.gguf”

model_name:“Qwen3.5-9B-GGUF”

recipe:“llamacpp”

status:“success”}

services

Lemonade can be found as

sudo service lemond status

We look for all backends

lemonade backends

Recipe              Backend     Status          Message/Version                               Action
----------------------------------------------------------------------------------------------------------------------------------------------------
acestep             cuda        installed       v0.1.1                                         -
                    rocm        installed       v0.1.1                                         -
                    vulkan      installed       v0.1.1                                         -
kokoro              cpu         installable     Backend is supported but not installed.        lemonade backends install kokoro:cpu
llamacpp            cpu         installed       b10825                                         -
                    cuda        installed       b10820                                         -
                    rocm        installed       b10820                                         -
                    vulkan      installed       b10825                                         -
moonshine           cpu         installed       moonshine0.0.62                                -
onnxruntime         cpu         installed       0.3.7                                          -
openmoss            cuda        installable     Backend is supported but not installed.        lemonade backends install openmoss:cuda
                    vulkan      installable     Backend is supported but not installed.        lemonade backends install openmoss:vulkan
sd-cpp              cpu         installable     Backend is supported but not installed.        lemonade backends install sd-cpp:cpu
                    cuda        update_required Backend update is required before use.         lemonade backends install sd-cpp:cuda
                    rocm        installable     Backend is supported but not installed.        lemonade backends install sd-cpp:rocm
                    vulkan      installable     Backend is supported but not installed.        lemonade backends install sd-cpp:vulkan
thenoise            rocm        installable     Backend is supported but not installed.        lemonade backends install thenoise:rocm
thinksound          cuda        installed       v0.1.2                                         -
                    rocm        installed       v0.1.2                                         -
                    vulkan      installable     Backend is supported but not installed.        lemonade backends install thinksound:vulkan
trellis             cuda        installable     Backend is supported but not installed.        lemonade backends install trellis:cuda
                    rocm        installable     Backend is supported but not installed.        lemonade backends install trellis:rocm
                    vulkan      installable     Backend is supported but not installed.        lemonade backends install trellis:vulkan
vllm                rocm        installable     Backend is supported but not installed.        lemonade backends install vllm:rocm
whispercpp          cpu         installable     Backend is supported but not installed.        lemonade backends install whispercpp:cpu
                    rocm        installed       v1.8.4                                         -
                    vulkan      installable     Backend is supported but not installed.        lemonade backends install whispercpp:vulkan
----------------------------------------------------------------------------------------------------------------------------------------------------

capabilities

classify (like phishing).

curl -X POST http://localhost:$PORT/v1/classify   -H "Content-Type: application/json"   -d '{"model": "Phishing-Email-Detection-ONNX", "input": "Please verify your account at http://secure-login.example now."}'

Evaluate a routing policy document against a prompt and return the decision the engine would make

curl -X POST http://localhost:$PORT/api/v1/routing/validate \
     -H "Content-Type: application/json" \
     -d '{
           "policy": {
             "version": "1",
             "recipe": "collection.router",
             "components": ["Qwen3-8B-GGUF", "vllm.qwen3-32b"],
             "routing": {
               "candidates": ["Qwen3-8B-GGUF", "vllm.qwen3-32b"],
               "default_model": "Qwen3-8B-GGUF",
               "rules": [
                 {
                   "id": "code-to-big",
                   "match": {"keywords_any": ["def ", "function", "compile"]},
                   "route_to": "vllm.qwen3-32b"
                 }
               ]
             }
           },
           "prompt": "please write a def to reverse a list"
         }'

{"decision":{"default_used":false,"matched_rule":"code-to-big","outputs":{},"route_to":"vllm.qwen3-32b","trace":[{"condition":"keywords_any","result":true}],"version":"1"},"normalized_policy":{"components":["Qwen3-8B-GGUF","vllm.qwen3-32b"],"recipe":"collection.router","routing":{"candidates":["Qwen3-8B-GGUF","vllm.qwen3-32b"],"default_model":"Qwen3-8B-GGUF","rules":[{"id":"code-to-big","match":{"keywords_any":["def ","function","compile"]},"route_to":"vllm.qwen3-32b"}]},"version":"1"}}

generate

audio

cd ~/lav/viudi/seq/samples/
curl -X POST http://localhost:$PORT/v1/audio/generations \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ThinkSound-SFX",
        "prompt": "glass shattering on a stone floor",
        "duration": 5,
        "seed": 42
      }' \
  --output clip.wav

{"models":[{"details":{"families":["thinksound"],"family":"thinksound","format":"gguf","parameter_size":"","parent_model":"","quantization_level":""},"digest":"sha256:0000000000000000000000000000000000000000000000000000000000000000","expires_at":"2099-01-01T00:00:00Z","model":"ThinkSound-SFX:latest","name":"ThinkSound-SFX:latest","size":0,"size_vram":0},{"details":{"families":["llamacpp"],"family":"llamacpp","format":"gguf","parameter_size":"9B","parent_model":"","quantization_level":"Q4_K_XL"},"digest":"sha256:0000000000000000000000000000000000000000000000000000000000000000","expires_at":"2099-01-01T00:00:00Z","model":"Qwen3.5-9B-GGUF:latest","name":"Qwen3.5-9B-GGUF:latest","size":0,"size_vram":0}]}

music

cd ~/lav/viudi/seq/samples/
curl -X POST http://localhost:$PORT/v1/audio/generations \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ACE-Step-Music",
        "prompt": "warm acoustic folk ballad, fingerpicked guitar, gentle female vocals",
        "lyrics": "[verse]\nMoonlight spills across the floor\nShadows dancing by the door\n\n[chorus]\nWe sing until the morning light\nCarried on the wind tonight",
        "duration": 60
      }' \
  --output song.wav

3d mesh

curl -X POST http://localhost:$PORT/v1/3d/generations \
  -H "Content-Type: application/json" \
  -d "{
        \"model\": \"TRELLIS-3D\",
        \"image\": \"$(base64 -w0 input.png)\",
        \"resolution\": 512,
        \"seed\": 42
      }" \
  --output model.glb

manage resources

curl http://localhost:$PORT/v1/system-stats -H "Content-Type: application/json" | grep token

curl http://localhost:$PORT/v1/system-info -H "Content-Type: application/json" | grep token

curl http://localhost:$PORT/v1/stats -H "Content-Type: application/json" | grep token

,#+begin_src bash :exports both :results output replace :var PORT=service_port curl http://localhost:$port/v1/models/Qwen3.9-9B-GGUF/files | grep port #+end_src

<!doctype html><html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; media-src 'self' blob: data:; img-src 'self' data: blob: https:; connect-src 'self' blob: http: https: ws: wss: ipc: tauri:; frame-src 'self' https://lemonade-server.ai file:;"><title>Lemonade App</title><link rel="icon" type="image/x-icon" href="favicon.ico"><script defer="defer" src="renderer.bundle.js"></script>
    getServerPort: () => parseInt(window.location.port) || 13305,