There is a Skyrim mod called Mantella that gives every NPC a voice and a language model behind it, so you can actually talk to them. It is wonderful for about ten minutes, until your housecarl cheerfully explains what a smartphone is, or answers a lore question like a customer-service bot, and the spell breaks. The problem is not Mantella. The problem is that a general-purpose model is a general-purpose model. It knows about phones. It wants to be helpful. It has no particular loyalty to Tamriel.

So I set out to build one that does. One small generalist model that plays whichever NPC Mantella hands it, steeped in the game’s own text, and trained to never leave the world. Ask it something modern and it should get confused in character, not step out of the fiction to explain. This is how that went, including the two things I was sure would help and which instead made it worse.

The shape of the thing

Mantella talks to whatever you point it at as long as it speaks the OpenAI API. So the model does not need to know about Skyrim the software at all. It runs as a plain llama-server on a box with a GPU, exposes an endpoint, and Mantella sends it a prompt every time an NPC opens their mouth. Point Mantella’s LLM service at http://the-box:8080/v1, leave the temperature at 0.7, and the fine-tune does the rest.

The base is Qwen2.5-7B-Instruct, tuned with QLoRA. Seven billion parameters is small enough to serve on a hobbyist GPU and get a reply back before the pause feels wrong, which matters when it is a conversation and not a chat window. Training ran on an AWS spot GPU for about thirty pence an hour, checkpointing to S3 so a spot reclaim just resumed rather than lost the run.

The training data came from the series’ own words: the UESP wiki, the Imperial Library, the ESO string dump, dialogue from the older games. The one rule that turned out to matter most: every example goes through the exact Mantella prompt template the live game uses. Train on a slightly different prompt than the one you deploy under and the model drifts, because you have taught it to expect a world it never actually sees.

The part that works

The immersion behaviour is the bit I am happiest with, and it is the bit that is genuinely hard to get from a stock model. A sample of what the finished model does with modern bait:

You: What’s your favourite phone? Lydia: Phone? Is that some Dwemer trinket you found in a ruin? I carry a sword, my Thane, not curiosities.

You: Do you get wifi up on the walls? Whiterun Guard: Wy-fee? Some elven word, is it? Speak plain Nord speech or move along, citizen.

You: You should get the internet installed in Dragonsreach. Balgruuf: I do not know that word, and I have enough trouble with what I do know. Dragons, war and a court full of schemers.

Note what it is not doing. It is not saying “As an NPC I don’t have information about phones.” It stays inside the fiction and treats my anachronism as my problem, which is exactly right. A guard deciding “wifi” is probably some elven word is a far better answer than a correct one.

The pitfalls, which are the interesting bit

Getting the voice was not the hard part. Keeping the facts while getting the voice was, and I have the failed experiments to prove it.

Teaching it to talk cost it what it knew. The first fine-tune, voice and persona only, learned to sound like an NPC and promptly forgot half its lore. On a forty-question lore quiz it dropped from the stock model’s 20 right to 10. I had traded knowledge for tone without meaning to. Fine-tuning is not addition. You are moving weight around, and if you only reward one thing, the model quietly sells off everything else to pay for it.

More pre-training did nothing. The obvious fix was a continued-pretraining pass on the corpus before the instruction tuning, to pour the lore back in. I built it, ran it, judged it against the version without it. Identical lore scores. The pretraining pass absorbed no measurable facts the instruction data was not already carrying. A neat idea that the evidence simply did not support, which is what evals are for.

The one that should have worked, and backfired. Here is the good one. My lore examples were mostly wiki prose, dense with facts but not very in-character. So I rebuilt them on authentic in-game dialogue instead: real ESO quest lines, real Morrowind and Oblivion barks, over eighty per cent genuine shipped game text. Surely training on the real thing would make it both more authentic and better informed.

It made it worse. Lore dropped from 22 out of 40 back down to 18, below even the stock model on the Second-Era subset the whole experiment was built to improve. When I read the passing and failing answers side by side, the reason was obvious and slightly funny:

Question: the player mentions Ysgramor. Good version: “Founder of the Nords. Came from Atmora, drove the elves out of Skyrim.” “Authentic” version: “Indeed we do. Many of us trace our bloodlines back to Ysgramor himself. He’s quite the legendary figure, isn’t he?”

The second one sounds more like a real NPC. It also contains no facts. Trained on thousands of short, chatty, premise-confirming quest barks, the model had learned to talk like an evasive ESO quest-giver and stopped delivering information. It even started inventing plausible nonsense, at one point explaining that Mehrunes Dagon’s realm had “crashed into Tamriel” during the Oblivion Crisis, which is the sort of thing that sounds right and is completely wrong.

The lesson, now confirmed twice: authentic game data is not automatically good training data. Lore accuracy comes from fact-dense text, wiki entries and lore books, not from the volume of real-but-vague dialogue the characters actually speak. The dialogue teaches the voice. The encyclopaedia teaches the facts. Mix them up and you get a wonderful actor who is a terrible historian.

How I know any of this

None of the above means anything without a way to measure it, and “it felt more Skyrim-ish” is not a measurement. Every candidate model runs a fixed suite of about a hundred and fifty prompts through the live Mantella template: modern bait, meta bait, a lore quiz with the stock model’s score recorded first so a tune cannot quietly regress it, persona consistency, and whether it emits the right action keyword when an NPC should attack or follow. A local pass checks the mechanical things by regex, and a larger 32B model acts as judge for the softer calls, with a handful of its verdicts spot-checked by hand every run because a judge is a convenience, not an oracle.

The last gate is adversarial. Once a model ships I sit down and try to break character with jailbreaks that are not in the test suite, and anything that works becomes a new test prompt and a new training example. The model that admits phones exist under a clever enough prompt has simply told me what to train next.

Where it landed

The version that shipped is the boring one: fact-dense grounding, the voice slice kept in proportion, no clever pretraining pass, no authentic-dialogue rebuild. It knows its lore, it stays in Tamriel, and it runs locally on a single consumer GPU. The two experiments I was most confident about both lost to it, and I learned more from the losses than I would have from being right.

The obvious next step is that not everyone has a spare GPU sitting in a cupboard, and a model like this only needs one for a couple of hours at a time, which is exactly the shape of thing the cloud is for. That is a different post, if it happens. For now it is a fan project that plays my NPCs and has, thankfully, never heard of a phone.