The EULLM Inference Engine: What It Actually Does, Explained Simply

When people talk about artificial intelligence, they almost always picture something like ChatGPT: you type a message into a website, a distant server (often in the United States) processes it and replies. EULLM Engine was built to break that pattern, and it's an inference engine designed and built in Italy by I3K. It's a program that runs large language models (LLMs) directly on your own computer, your company's server, or European hardware, without a single word you write ever having to leave your machine. Here's what it can actually do today, concretely and without inflated promises.

One single program, no complicated install

EULLM Engine is a single binary. No Python to install, no Docker, no external libraries to configure by hand. You download one executable file (for Linux, macOS or Windows, on both Intel/AMD and ARM processors) and run it. That's deliberate: the goal is that anyone, a small law firm, a hospital, a solo developer, can have a local AI engine running without hiring a DevOps team just to get it started.

Ollama-compatible, without being a fork

One of its most practical strengths is compatibility with the Ollama API, the most widely used open-source project for running LLMs locally. That means any application, plugin, or script already written to talk to Ollama works with EULLM without changes, only the address it points to changes. On top of that, EULLM also exposes an OpenAI-compatible API (/v1/chat/completions), so it works with anything built with ChatGPT in mind too. It isn't a fork of Ollama's code: it's an implementation written from scratch in Italy, in Rust, designed from day one for auditability and data sovereignty.

Built-in chat, ready to use

You don't need any external client to start chatting with a model. When you launch EULLM in run mode, the engine automatically opens a chat interface in your browser, on a dedicated port. If you'd rather stay in the terminal, a flag (--cli) gives you a classic command-line interface, handy for people working over SSH or on machines without a graphical environment. If you only want EULLM as an engine, with no interface at all, say, inside your own application, another flag (--no-ui) makes it fully headless.

There's also an interactive picker: launch the program without specifying anything, and it offers to choose an already-downloaded model, one from the catalog, or to paste a custom path or URL. Useful for anyone who doesn't want to memorize commands.

Built-in web browsing

One of the most interesting features, and one that directly answers a frequent question ("can the model search for information online?"), is the --web flag. When it's active, if the message you write contains a URL, the engine downloads it server-side, extracts its text content, and inserts it into the prompt before generating a reply. In practice, the model can "read" a web page you point it to, with no external plugins or third-party services involved. It's still a young feature and should be used with awareness (any system that downloads content from a user-supplied URL needs security attention), but it's already usable and useful for summarizing articles, technical documentation, or reference pages.

Managing models: download, switch, free up memory

EULLM treats models as "packages" managed with simple commands:

• pull: downloads a model from the EULLM catalog, from Hugging Face, or from a direct URL.
• run: loads it and runs it immediately, in interactive mode.
• serve: starts the engine as a persistent server (daemon), loading models on demand. This is the mode built for continuous "service" use, for example behind a business application.
• list / show: list downloaded models and show their details.
• rm: removes a model from disk.
• unload: frees the video memory (VRAM) used by a model without having to restart the whole server. Handy when you want to reclaim memory for another process without interrupting the service.
• import-ollama: if you already have models downloaded with Ollama, this command migrates them into EULLM without re-downloading them from scratch, saving time and bandwidth.

A particularly useful feature for anyone running EULLM as a service is hot model-switching: just specify a different model in the model field of an HTTP request, and the server loads it on the fly, no restart needed. That makes EULLM well suited to scenarios where multiple applications share the same engine but use different models depending on the task.

Performance: continuous batching and smart caching

Under the hood, EULLM uses a continuous-batching scheduler: when multiple requests arrive at the same time, the engine processes them together efficiently instead of queuing them one after another, making better use of the available GPU or CPU.

There's also a cache-reuse mechanism (the so-called KV cache) that, in multi-turn conversations, recognizes how much of the previous context has already been "computed" and avoids recalculating it from scratch every time. In tests on real hardware this leads to near-100% reuse in multi-turn conversations, with a direct impact on response speed.

For anyone with limited memory available, the KV cache can also be compressed (quantized) into different formats, trading a small amount of precision for lower RAM or VRAM usage. It's a configurable choice, not a forced one.

Runs everywhere: GPU, CPU, European hardware

EULLM supports several underlying compute backends: NVIDIA cards (CUDA, the most tested and mature combination), Apple Silicon (Metal, validated by the community), and, currently build-from-source only and not yet shipped as a ready binary, AMD (ROCm) and Vulkan for other GPUs.

But perhaps the most significant point for EULLM's mission is that the engine also runs well on CPU alone, with no dedicated GPU. This was demonstrated concretely on a Radxa Orion O6 board (an ARM CIX P1 SoC, with no dedicated GPU), running a 35-billion-parameter MoE (mixture-of-experts) model entirely on CPU, at 9-11 tokens per second. That's a meaningful data point for the European digital sovereignty mission EULLM, an engine born and built in Italy, is pursuing: you don't need expensive hardware or imports from outside Europe to have a usable local AI.

For MoE-type models, there's also an option (--cpu-moe / --n-cpu-moe) that keeps the model's "experts" in system RAM while the rest (attention, cache) stays on the GPU. A smart compromise for running huge models on GPUs with limited video memory, and one that works with any GPU backend, not just CUDA.

Baseline security and transparency

By default, EULLM only listens on localhost and applies an allowed-IP list even when configured to be reachable from outside: a basic safeguard against accidentally exposing the service on the network.

The engine also keeps a local audit trail in JSONL format: for every request it logs the model used, request type, token count, duration, and timestamp. Importantly, it logs metadata only, never prompt or response content, and this file stays on the machine, it's never sent to any external server. It's built for anyone who needs to meet traceability obligations (for instance under the EU AI Act) without giving up content confidentiality.

Multimodality, still experimental

Anyone who builds EULLM with the optional multimodal flag also gets support for image input (and, down the line, audio), via a --image command-line flag or the images field in the chat API. It's still a young feature (MVP), meant for people who want to experiment now with models that can "see," but it isn't yet part of the standard binary.

What it does not do yet, said honestly

For completeness, and to avoid promising more than EULLM delivers today: it doesn't yet have OpenAI-style tool-calling, doesn't yet have full structured JSON schemas (response_format: json_schema), doesn't yet do speculative decoding, doesn't yet serve in-process embedding or reranking, and doesn't yet handle LoRA adapters at runtime. These are all on the roadmap, not in today's engine. It's better to say so clearly than to sell something that isn't there yet.

A compact engine, built in Italy

EULLM Engine is, today, a compact inference engine designed and built in Italy, compatible with the most widely used standards (Ollama and OpenAI), capable of running language models on both GPU and plain CPU, even on inexpensive ARM hardware, with a built-in chat ready to use, the ability to have the model read web pages, flexible model management, and a real focus on security and local traceability. All of this while staying a single binary, with no hidden dependencies and without a single byte of your data ever having to leave your machine: a concrete, European, Italian alternative to inference engines built elsewhere.