LLM host thyself

Putting together your own self-hosted models is easier than I thought it would be.

However, the challenges traditional businesses face when doing something like this can be non-trivial.

Since the beginning of the LLM/AI boom/craze/adoption event I've worked with several systems including ChatGPT, MS CoPilot, and most recently Claude Code.

In the decades leading up to the AI era, my work as a translator was closely associated with neural networks and machine learning. Mostly it was SDL Trados (now RWS) and setting a system like that up correctly is something like setting up an LLM in the current environment, but the preliminary procedures are of a different nature. Translation memories are a special kind of SQLite database, and termbases constrain the specific terminology the customer wants.

The analogs for that in today's world are markdown files, hooks, and harness engineering. It's all about the mise en place, setup is what governs the success or failure of the outputs in these endeavors. Same as neural machine translation.

In the past six months I've also spun up two Postgres databases to use in my repositories, this was an exercise in "let's see if I can do it" and the experiment has been successful so far. Ultimately only useful on a very narrow bandwidth, but that could develop into a use case for my business.

Self-hosting LLMs has recently become both more and less possible. Much easier to implement, especially on Apple systems, but much more expensive in terms of hardware.

I'm going to go out on a limb and predict that the current memory chip situation will have a positive effect for self-hosting in the long term, the signs are already there. Devs were compiling models that would work on laptop and mobile platforms already, but now more attention is being given to tailoring models that can run on a "normal" amount of memory. Necessity is the mother of resource optimization, if you will.

I'm just a guy who has been fascinated by the idea of thinking machines my entire life. And my work has often ranged from data adjacent to data intense, I've been a user in ERP systems like Oracle, I've worked with translation engines, and towards the tail end of the Covid lockdowns I decided to take a Python course.

Putting my little toe back into the water of programming opened up a new world of possibilities for me, because it helped remind me what I learned years ago (DOS, Basic & Co.) about how these systems work.

More importantly it gave me the confidence to try things I hadn't done in while, outside of hacking the registry in Windows, which was done out of necessity not to make or improve something.

So, I downloaded Ollama, spun up some models, asked the questions and waited. I figured out how to enforce a short term and long term memory by using OpenWebUI and got to the point where it could do low-risk, mechanical tasks fairly well. Still needs some tweaking, but serviceable for now.