The story behind Projekt ULTIA

It all started on the evening of 19 September 2026 with an idea: a personal JARVIS. The model was the digital butler from the Iron Man films, an assistant who knows the whole computer, listens and acts on its own.

The starting point was modest. A laptop running Windows 11 with an AMD Ryzen 5 5500U, 16 GB of soldered memory and the built-in Radeon graphics. The laptop’s display was broken, so it has been hooked up to the TV ever since. Internet came over mobile data, because the landline router had not been delivered yet.

Local instead of cloud. The first attempt used the Hermes agent framework and an online model via OpenRouter. It failed that same evening, first on a malformed request, then on missing credit. The consequence: “Screw the online connection.” Since then the language model has run locally, at first through Ollama. The choice fell on an uncensored model, because the safety rules were to be set by the owner, not by the manufacturer.

A painful download. The model file of about 6.5 GB came in over mobile data, at times at 350 KB/s. On the first night the download crashed the router. Only after several attempts and a detour through a phone as a hotspot was the model there on the evening of 20.09.

JARVIS becomes ULTIA. On 20.09. the first dashboard was built: dark, futuristic, with a large glowing core. After trying orange-gold and midnight purple, it stayed with the toxic green ULTIA still wears today. Instead of the male film voice it was to be a female one. At 6:28 pm the name was set: ULTIA. As an extra safeguard, Asimov’s three laws of robotics became part of how she works.

The first wishes from that day are partly still on the list: connecting e-mail, calendar, WhatsApp and Discord, giving ULTIA access to screen and input, protecting sensitive data with the fingerprint reader, listening for a voice command and muting her with a button or a call.

On 21 September ULTIA ran for the first time: the local model as the brain, the dashboard as the face and a note about her personality. It said: confident, dry humour, short sentences, no empty phrases. That note later became the personality file that still shapes ULTIA today.

• 21.09.: the first working day with ULTIA. After she deleted a file without asking on day one, she now asks before every action. She gets her own user account with admin rights and an Obsidian vault as her memory.
• 22.09.: a voice service with wake word and speech recognition (faster-whisper). In the dashboard ULTIA gets her shape made of dots. The first draft “looked like Minecraft” and became finer and more delicate.
• 24.09.: a detailed design of the figure in DAZ Studio.
• 25.09.: A click on the figure’s chest shows processor and network values straight from the system.
• 26.09.: The language model now runs on llama.cpp instead of Ollama, faster and more direct.
• 06.10.: the first real ULTIA program: its own window instead of a browser, conversations with the language model, first tools for system values, commands and web search. She starts with Windows.
• 08.10.: a fixed folder plan, a guide for a clean rebuild, speed tests, a first voice with Piper.

Between 6 and 8 October alone, the new program was started 34 times. The main problem was speed: the language model shared the graphics chip with everything else, and every answer took many seconds.

On 9 October ULTIA moved to her own Linux partition. Under Ubuntu the AMD graphics could be pushed much further: with more shared graphics memory, the space for the model grew from 6.5 to about 11 GB. The same day ULTIA got full access to the system and a fullscreen mode. For the first time it showed that she announces things instead of doing them. That was the start of a long series of improvements to her reliability.

The brain. The default is a Gemma 4 model, with Qwen 3.5 and Hermes 3 as alternatives. A saved starting point makes sure ULTIA answers within seconds after booting. On 23.10. the working memory for conversations was tripled.

The memory. Notes, personality, tasks and a log of every action live in an Obsidian vault. Since 20.10. the conversation history also survives a restart and a model switch.

The tools. By 14.10. these were added: writing files, searching folders, running her own code and operating mouse and keyboard herself. Later came screenshots, image analysis, code search and file editing.

Do more, talk less. Small models like to announce instead of act. So since 25.10. ULTIA notices when she is only talking or claims success although a tool refused, and follows up on her own. Since 27.10. she breaks out of endless loops by herself.

ULTIA is a female figure made of thousands of glowing dots. Her colour and glow show her mood. A click on her head or chest shows system values.

The figure. From 9 to 11.10. it was day after day about weighting and animation: the right hand, the left elbow, poses that went through her own body, hair that looked “stapled to the head”. On 11.10. a completely rebuilt model was created: barefoot, flat on the floor, left hand on the hip.

She moves herself. With short commands in her answer ULTIA moves every part of her body. While speaking she moves to match each sentence; when idle she stretches, yawns or does a few squats.

Real motion instead of formulas. Since 12.10. a motion engine picks the next fitting movement from real full-body recordings and blends smoothly. Today almost 2,000 freely usable recordings are available, plus more than 30 exercises she can demonstrate.

A workplace. When ULTIA works, she types on a holo keyboard with ten fingers, exactly the text she is writing. While she waits for a tool, she plays with a ball.

The holo screen. Next to her floats a small screen for lists, results and pictures. Since late October it shows all kinds of media, from animations like the rain radar to videos and PDFs, fullscreen on request.

Voice. On 9.10. all German female voices from Piper were compared, none convinced. The winner was a version of Chatterbox that runs on the graphics chip at 4 bits. Since 11.10. ULTIA speaks with it, based on a real speaker. The emphasis follows her mood, and the text appears sentence by sentence at speaking pace.

Hearing. A dedicated service listens all the time and transcribes with Whisper, locally on the laptop. ULTIA reacts to her wake word and recognises her owner’s voice.

Sight. Say “have a look” and ULTIA gets the current camera image. Since 19.10. her language model analyses images itself and can zoom in. She knows the time and the position of the sun and uses them to interpret the light in the room.

Sense of space. Since 20.10. ULTIA knows a 3D overview of the flat and can tell through the camera where her owner is standing. Via Bluetooth on phone and tablet she knows whether her owner is at home, and she can watch the room while nobody is there.

In two nights, on 19 and 20.10., an Android app of her own was built. By 27.10. 16 versions had followed.

Own network. Phone and tablet are connected to the laptop through a private VPN, paired by QR code. On the road the app finds its own way in.

The real dashboard in your hand. The app shows ULTIA’s dashboard from the laptop, everything runs in sync. Add to that chat, voice input and a photo button.

Always reachable. The phone listens for the wake word and checks the voice. When it is locked, ULTIA still answers, but full rights there only come with a reliably recognised voice.

The phone as a sense organ. Every 30 seconds the device reports location, motion, light, noise, battery and more. In return ULTIA controls the phone: lock, ring, torch, alarm, notification, photo.

The goal behind ULTIA is a digital office assistant: she should get tasks done on her own, several at once, not just answer.

Two work lanes. Since 19.10. longer jobs keep running on a side lane while ULTIA carries on talking up front.

Routines and triggers. ULTIA sets up fixed routines and triggers, such as motion in the picture or a device leaving the flat, and then works through the job without asking.

Her own tools. If ULTIA lacks a skill, since 23.10. she builds herself a tool, tests it and uses it right away. Customers get this ability too.

Connections. Since 25.10.: e-mail, calendar, contacts, tasks, cloud files, Telegram, Discord and read-only bank access.

Improving herself. ULTIA can apply changes to herself on trial. If she crashes afterwards, the change is rolled back automatically.

Almost every security rule in ULTIA came from a real incident.

Her personality is protected. ULTIA may read her personality file but never change it herself. Every write attempt opens a login prompt for the owner.

Fingerprint for anything sensitive. Risky actions need approval by fingerprint, on the laptop or, since 23.10., on the phone. There a key that never leaves the device signs the approval.

Never again a restart at night. On 20.10. at 2:28 am the laptop restarted because a command triggered it. Since then ULTIA can no longer restart the computer by command.

Double approval for updates. Every update needs two confirmations, first on the laptop, then on the phone. Before that, a review window shows every changed line, with anything suspicious highlighted.

On 20.10. the decision was made to sell ULTIA. Since then there are five editions, an installer, a dedicated licence server and an update system that reaches every part.

Editions. Demo, Lite and Full, Business and the owner’s own Master version. The phone app and device control come with Full and Business.

Installer and Windows. An installer for Linux and Windows sets up ULTIA with a licence key. On 21 and 22.10. the Windows version caught up.

Licence server. A dedicated server manages keys, customers and updates. ULTIA has her own tightly limited access to it.

Website and mail server. From 27.10. this website was built, in eight languages, with customer accounts, orders and downloads. On 28.10. it went online under its own domain. A dedicated mail server is prepared.

Update system. Every computer carries a build number made of system, graphics card and memory. Updates can target matching computers only; security fixes and bug fixes always go to everyone.

The last week of October was about fixing teething troubles, almost all of them from real everyday life with ULTIA.

• Room watch with the wrong picture: ULTIA was supposed to photograph the room and saved the screen instead. Now she takes real camera photos, and she may no longer misuse unsuitable tools.
• Window screenshots: ULTIA photographs single windows, even hidden ones in the background.
• All media on the holo screen: pictures, animations, videos, sound and PDFs, plus a fullscreen mode.
• 20 minutes for a rain radar: Instead of showing the radar, ULTIA loaded the image file as text into the conversation. Now such data never gets in, and she knows right away how to show things.
• More space: The ULTIA partition grew from 69 to 107 GB.

The next big step is an external graphics card. With it ULTIA gets a stronger language model and more speed.

• an open beta, then the app in the Play Store and an iPhone app
• her own motion recordings and a male avatar
• recognising sounds, not just speech
• WhatsApp and more business features
• a backup system
• making the Windows version fully fit for everyday use