Expert Systems to Generative AI — tiny steps that caused giant leaps in productivity

In the beginning, we wrote the rules by hand. The expert systems of the 1980s and 90s were magnificent and exhausting. You found the best person in the building — the diagnostician, the underwriter, the network guru — and you sat a knowledge engineer across from them for months, extracting their judgment one IF-THEN at a time. Thousands of rules, curated into a knowledge base, executed by a deterministic inference engine that never hallucinated, never improvised, never surprised you. The execution was flawless. The problem was everything else: knowledge engineering cost a fortune, the expert could never quite articulate the intuition they actually used, and the systems were brittle in a way that bordered on comic — one step outside the rulebook and the magnificent machine went silent, or confidently applied a rule that did not fit. Every new product version meant another expensive rule-surgery project. The support costs compounded, and expert systems became a cautionary tale: perfect execution, starved by its own knowledge.

The generative AI era inverted exactly one thing: where the rules come from. Instead of interviewing the expert, you curate the raw record of their work — every decision, every correction, every recovery — and let an autoregressive probabilistic model learn the patterns from the data. The rules are no longer written; they are learned. And here is the part people get wrong about how these systems run in production: the model does not execute anything. It fills in a carefully formatted card — a JSON form, a tool call — and hands it to a deterministic harness that reads the card and does the work. It reminded me of nothing so much as the mainframe punchcard. Judgment happens in the model; execution happens in code; the card is the boundary. The expert system is back, but the knowledge base is learned, and the inference engine is split in two: a probabilistic form-filler in front of a deterministic executor.

Talk is cheap, so I ran the experiment on myself. I took a small open-weights model — free to download — and trained a LoRA adapter on three months of my own agent sessions: every failure, every recovery, curated into a few thousand clean examples. The adapter training cost me under $50 in GPU time. The model itself was free, so the total cost of my custom model is $50. For this price nowadays you can get dinner in a good restaurant. But I have a model and it works. I ran it in ollama, served it locally as an endpoint, and pointed my harnesses at it — Interpreter, VSCode+RooCode. I could not tell the difference.

Your mileage may vary. But the direction is set: the cost of a custom model is now the cost of a dinner, the data is the work you already did, and the harness does not care whose card it reads

Which task actually needs AI?

AI earns its place at the workplace when a task is a workflow, not a single computable step: data collected from scattered places, analyzed, and turned into a conclusion someone can act on. That workflow exists in every industry — but I am only familiar with IT, so the table below shows how it applies in IT. The pattern I find: AI collects, analyzes, and drafts the conclusion; the human owns it.

Task / Activity	Human value add	Where AI earns / helps
Report generation from data spread across enterprise databases and files	Knows which questions matter and what the numbers mean for the business	Collects the scattered data (retrieval), analyzes it, drafts the report in the house format
Infra cost-saving initiatives (find waste, rightsize, decommission)	Owns the risk call, approves destructive actions, adjudicates exceptions	Finds waste candidates, sizes the savings, drafts the justification a manager reads
Architecture evaluation against best practices	Judgment under tradeoffs; the final call and the accountability	Checks designs against conventions, cites the violated principle, drafts the review
Incident triage & error recovery	Novel failure modes, escalation judgment, the 3 a.m. call	Recognizes known failure modes and proposes the known fix instantly (my $50 specialist)
Capacity planning	Business context — launches, seasonality, risk appetite	Crunches the utilization series, narrates trends and the outlook
Compliance & audit evidence	Interprets gray areas, signs the attestation	Gathers evidence across systems, drafts the control narratives

Read the columns as a split of labor: the middle column is judgment and accountability; the last column is collection, analysis, and drafting. That is the whole claim. The tasks where AI helps most are not the ones with the most data or the fanciest math — they are the ones where the conclusion must be composed, in a specific voice, from scattered inputs, repeatedly.

Two consequences:

1. The human column is not ceremonial. Every row ends in a person who owns the outcome — the architect's call, the manager's approval, the attestation signature. Not caution theater: the model drafts, the human decides. The day a row loses its human value-add, it stops being an AI task and becomes a cron job — and that is fine too. Automation is the final stage of a well-understood task, not a failure of AI.

2. The AI column is smaller and cheaper than advertised. Judgment + your format + repetition is exactly the profile where a small model trained on your own data beats a frontier subscription. You do not need the model that writes poetry. You need the one that knows your loop.

Kimi-K3 shift focus to Memory

AI inference economics is getting even more difficult. Newer models are compute efficient but memory heavy. This is shifting the economics in favor of hyperscalers and away from DIY labs like I have. There is no way you can build a lab in your garage to serve kimi-k3 (after they release their openweights next week). Forget ollama pull kimi-k3... 

If you are a financier of hyperscaler (buying their debt), this is good news because DIY and bottom feeders just discoved a BTE (barrier to entry). But if you are a lab want to further research, you are SOL'ed. 

Kimi-k3 has 2.8T parameters that all need to be in memory during inferencing. That memory requirement is approximately 1.7TB of DRAM. 

It's not just the weights — context eats memory too

That 1.7TB is just the model weights sitting in memory. There is a second memory cost that scales with your traffic: the context window. Every conversation with the model lives in memory as the KV cache — key and value vectors computed for every token, at every layer, that must stay resident in memory for the entire session so the model can attend back to them.

This means your memory bill grows with (a) context length and (b) concurrent users. One long-context session can chew through tens of GB of KV cache. Multiply by thousands of simultaneous users and the KV cache can rival the weights themselves.

So the DIY math is even worse than the table suggests. You don't just need enough memory to hold the model — you need enough to hold every user's conversation at the same time. Hyperscalers spread that cost across a fleet; your garage lab eats it all at once.

Precision Format	Memory Size	Platform
MXFP4 (Kimi's own FP4)	~1.5-2 TB	Multigpu cluster (single rack at least)
Q4 Quantization	~1.0-1.2 TB	Single Enterprise Server with multiple GPUs
Aggressive 2-bit	~1TB	Might fit in a high end laptop, but performance is abysmal and lots of hallucinations
BF16/BF32	~6-12TB	Multirack solution - Only option is hosted

The Sparsity Opportunity

• Current consumer and even most enterprise GPUs have limited hardware support for sparse operations.

• Sparse models often require custom kernels and careful integration with inference engines.

• Quality can vary dramatically depending on how aggressively sparsity is applied.

A 8GPU host costs around 700K today, to host the high precision model, we will need 3-5 racks of these hosts (assuming 2 per rack) to get 12-16TB in the cluster. The cost of that is approximately $6M. Add to that cost of power and cooling and we are talking $7-8M. Now imagine you have a great new model optimization, you will need to raise 50M+ just in seed round to get started to support a multi million burn rate. 

There is one technical path that could meaningfully change this equation: sparsity.

Sparsity techniques prune weights to zero during or after training, then use specialized kernels that skip calculations and storage for those zeros. In theory, a 2.8T parameter model with 70–90% sparsity could reduce its memory footprint by 3–5× while preserving most of its capability.

If the Kimi-k3 weights are released with high-quality sparsity patterns (or if the community rapidly develops them), the effective memory requirement could drop from 1.7 TB down to the 400–600 GB range. That would bring the model back into reach for well-equipped single-server labs instead of requiring an entire rack.

However, significant challenges remain:

The lab that cracks efficient, high-sparsity inference at this scale will enjoy a massive cost advantage. Until then, the economic headwinds strongly favor those who can amortize millions of dollars of infrastructure across vast workloads — the hyperscalers.

Prompt is the new Config, Workflow is the new Product

 Prompt are dynamic configurations that can adapt to incoming requests. Today configurations are static files which may miss a lot of corner cases. Enter an LLM and now we can reason with APIs and dynamically workaround blocks. This one ability of the LLMs can create panic among all gateways to enterprise. A gateway allows/deny based on a static policy. What if we embed this intelligence in them so they adapt continuously to incoming traffic. This is atleast one reason why firewalls/cybersec products are receiving scrutiny from CIOs. A prompt injection can completely change the security posture of an enterprise. 

When you have a wild and powerful entity on the loose, you need to build harness to channel this energy. That is the workflow. More and more startups are now focusing on workflows instead of just functionality. They called this harness a wrapper around LLM. These wrappers are the new gateways into the enterprise. 

Token Pricing: Reasoning Tax,, GPU Utilization & GPU Recency

 According to newly published TPI (Token Price Index), average cost of 1M tokens is just over $2. It rose over 75% in 1 year. So does a model serving provider make money off this pricing? Yes and No. The marginal cost of power + infra + facilities ranges between 6 cents to 1.60 cents per million token. The key factors driving costs higher are (a) low utilization of GPU (currently around 15%), (b) reasoning tax (the token generated inside to support customer tokens (c) GPU recency. 

On the latest GPU, the cost of generating token is lower creating an incentive for providers to move up to the latest GPU. But this also means they pay for supply constraint which will likely stay forever because the GPUs that are one generation behind are not used. So the amortization on those GPUs which assume 100% consumption for the full 6 years is just too low i.e. margins are artificially high. 

The GPU utilization relies on many techniques at deployment time but batching is the one which makes a meaningful impact. Assuming 100% of the input is batched is euphoric. We are average 15-25% on recent GPU and almost 1-2% on older GPUs. No one wants to run on legacy GPUs. Net net this factor is also artificially inflating margins. 

The reasoning tax comes from ordinary queries but especially from agentic AI. Agents generate output that is sent back to the model which then starts generating reasoning tokens which are currently not charged to the customer. These reasoning token cost is eaten by the provider and is referred to as "Reasoning Tax". 

OpenClaw is MicroSaaS

 This OpenClaw has a annoyingly long setup, but after all the effort, it is worth it. I set it up to routinely check my gmail. The process is well documented on openclaw but it does require account with Google Cloud, Installation of gcloud SDK, Go, GoG. And of course a lot of knowledge of how Linux works. I did it on WSL on windows. 

Finally, it gave me all my emails which is not the point. You can ask it questions which you can't in a email client. Check this out. 

Question: Who keeps sending me email? In the last week who has sent me the most emails? 

Question: Can you summarize all the emails from Federal Reserve Board?

Question: Can you summarize the April 8th FOMC Minutes? 

Ok, so all of this is pretty straightforward. But note that I haven't used any tokens or have I?  What about monetization? Who is making money with all this? 

It turns out LLM providers like Anthropic, Gemini are charging fees for tokens usage. Cloud hosting services charge for any managed instances (I am using my laptop). ClawHub skills for specific vertical (read custom integration) get paid. Skills vendors are making some money (not enough to quit your job yet but good for Happy Meals from McD). 

Local LLMs does not cut it

 After using ollama on a local LLMs, I found it is not very useful and I was constantly going to online versions of the same model or proprietary models. My local setup was under $1K and the results from the various queries over last 6 months were not any better than a web search. There are two problems with local inferencing (I am going to skip the most obvious one i.e. the hardware is not as powerful). 

The first one is distribution shift where the data on which the LLM is trained is not the same and sometimes differs significantly to the prompt query. This results in incomplete or superficial answers to queries where online models provide significantly more detail. This is kind of like talking to a person who is not in the know but knows enough to sound dangerous. 

The second is overfitting. This is where the weights have decided to converge on an answer for any set of prompts that are similar. It does not find accurate answers that fit the prompt so it serves the generalized version. This is kind of like thinking in stereotypes.

The key point is the real slim shady (yes eminem) here is the data itself. If you train on prior exams and then take an exam that was set by some external body, you will find yourself not as prepared to take the exam as you thought you were. The content and theories have not changed, just the way to test is different. Failure to detect and correct these shifts can be disastrous because sometime it means the model has to be trained from scratch. Yep that $150M used to train the model was flushed down the toilet. Hopefully, in 2026, we focus more on the data and how to partition and reassemble weights and not as much on GPU/VRAM. The latter is really a commodity and in abundant supply. 

BTW, B200 only supports certain versions of PyTorch. If this continues we will get fragmentation in jobs as with every new release of GPU, all software would have to be upgraded. In the end GenAI is not much different from Natural AI. 

Happy Holidays.