Think Inside the Box© : Local LLMs Are the New RDBMS

Boring is not the insult. Boring is the destination.

Somewhere during a recent build, I stopped thinking about the LLM as the “AI part” of the application.

Local models were performing different roles. The application retrieved evidence, supplied the relevant context, invoked inference, applied deterministic logic where determinism mattered, and sent uncertain conclusions for review.

There was surprisingly little ceremony around the model itself.

That caught my attention.

I had started the exercise to answer a practical Enterprise AI question firsthand. As with other “What the bit saw” exercises, the approach was simple: when opinion gets ahead of evidence, build enough to find out.

The original question was whether locally hosted inference could do serious analytical work over real, changing information.

Promisingly, it could.

But once the system was working, another thought became difficult to ignore.

What if inference is heading toward the same architectural status that relational databases eventually achieved?

Not the same technology.

The same ordinariness.

From breakthrough to plumbing

Edgar Codd proposed the relational model in 1970. Commercial relational databases followed. SQL became standardised. Over time, something that had once represented a fundamentally new approach to computing became so normal that applications simply assumed a database would be there.

Nobody launches an enterprise application today by proudly announcing that it uses a relational database.

That would be a fairly short launch event.

The database succeeded partly by becoming invisible.

The interesting work moved elsewhere: data modelling, keys, transactions, indexing, access control, recovery, availability and operations.

The exciting technology became infrastructure.

That is the sense in which I think local LLMs are heading toward becoming the new RDBMS.

The analogy is about architectural status, not behaviour.

Inference will not replace relational databases. An LLM is not a database. And not every application will need locally hosted inference any more than every workload needs the same database architecture.

The claim is narrower:

Inference is beginning to move from something applications advertise to something serious applications may simply assume is available.

If that happens, the implications go much further than choosing a model.

What the bit saw: the model was not the system

The experiment made this particularly clear.

Before a useful answer could appear, documents had to be acquired, associated with the right entity, extracted, classified and divided into usable evidence.

Some information worked well with lexical search. Some benefited from vector retrieval. Metadata mattered. Recency mattered. Diversity and reranking mattered. Provenance mattered.

Then inference happened.

And after inference, other things still mattered.

Calculations that should produce one correct answer remained deterministic. Outputs carrying material uncertainty needed review. Citations had to lead back to evidence. Operational behaviour had to survive something rather less glamorous than an AI benchmark: restarting the machine.

The model was important.

But the model was not the system.

That experience changed the questions I now ask first when looking at an Enterprise AI architecture.

What can the model actually see, where did that evidence come from, and how current is it?

What happens when retrieval finds nothing, and what information must never reach inference?

Which outputs can remain probabilistic, where must deterministic software take over, and can we explain later why the system produced the answer it did?

Those questions usually tell me more about the maturity of an AI system than its model leaderboard position.

Retrieval modelling may become the new schema work

This is the part of the experiment I find most architecturally interesting.

Relational databases created more than a storage technology.

They created a discipline around shaping information for use.

Entities. Relationships. Primary keys. Normalisation. Cardinality. Indexes.

Generations of developers learnt that database performance and correctness depended heavily on how the information had been modelled before the query arrived.

Inference systems are creating a parallel discipline.

I think of it as retrieval modelling.

Which information should become part of the evidence base?

Where should a document be divided?

Should a table and its explanatory paragraph remain together?

Which metadata must travel with every chunk?

What should be embedded, and what should remain available to exact lexical search?

How should recency influence retrieval?

Should diversity be enforced before or after reranking?

How much context improves an answer — and when does more context simply become more noise?

These can sound like implementation details of RAG.

I don’t think they are.

They are becoming information-modelling decisions for systems that reason over evidence.

That makes them architectural.

One lesson from the build illustrates why.

An identifier that looked more authoritative on paper turned out to perform worse against the live data. Symbol matching reached 98% on the measured sample; ISIN matching reached 79%.

That was not a model problem.

It was an information-modelling assumption meeting reality.

Relational systems taught us not to treat schema as clerical work.

Intelligent systems may teach us the same thing about retrieval.

You cannot prompt your way out of weak information architecture.

The cleverest model is blind to evidence it never received

This also puts model capability in perspective.

A foundation model can be extraordinarily capable and still know nothing about the document that entered your organisation this morning.

It may not know your latest internal policy, customer interaction, research note or contractual clause.

That is not a failure of intelligence.

It is an evidence boundary.

For many enterprise use cases, the useful capability is therefore not simply:

give me a powerful model

but:

connect an appropriate model to current, trusted, governed evidence.

That changes where enterprise value sits.

Model capability matters. So do the corpus, source quality, permissions, retrieval design, provenance, freshness and review loop.

As inference becomes easier to obtain, those surrounding disciplines become relatively more important.

There is a familiar pattern here.

Cheaper software authoring makes clarity and verification more valuable.

More accessible inference makes trustworthy evidence more valuable.

Technology removes one constraint.

Engineering discovers where the constraint moved.

Why “local” matters

There is an obvious challenge to the title.

Why local LLMs?

Why not simply inference?

Because for some enterprise workloads, where inference happens is itself an architectural requirement.

I am not arguing that every organisation should run every model on-premises.

That would turn locality into ideology, and ideology is rarely a useful architecture pattern.

Managed external inference can make perfect sense where the information, economics, latency and operating requirements support it.

But there are workloads where data sovereignty, confidentiality, network isolation, resilience or control change the equation.

In those environments, “we have an agreement with the model provider” and “this information cannot leave this boundary” are fundamentally different requirements.

In the experiment, the local-only rule was therefore not left as prose in a policy document.

The application validated the configured model endpoint and rejected one outside the permitted local boundary.

That is a much more interesting form of governance.

A policy says:

Don’t do this.

Architecture says:

The system is not designed to let you do this accidentally.

Relational systems normalised this instinct long ago.

We don’t merely ask developers not to violate referential integrity. We build constraints.

We don’t hope transactions happen atomically. We engineer for it.

Inference infrastructure will need the same instinct.

Where a rule is consequential and technically enforceable, turn the rule into a boundary.

The analogy breaks in one important place

Relational databases trained us to expect a mostly deterministic contract.

Ask the same deterministic query over the same state and we expect the same result.

Inference does not give us that contract.

Some run-to-run variation can be legitimate.

And the experiment uncovered something more dangerous: consistency that looked reassuring but was still wrong.

A fifteen-question benchmark scored 12/15.

A later run scored 11/15.

Two fairly similar numbers.

Then a configuration problem was found: reranking had not actually been operating as intended.

Once fixed, the correctly reranked result was 8/15.

The two earlier measurements had agreed closely enough to create confidence.

Both were measuring the same broken path.

A broken ruler can be wonderfully consistent.

That is where database-era operational thinking stops being sufficient.

Inference infrastructure needs its own discipline around evaluation, provenance, fallback behaviour, context inspection, uncertainty, observability and bounded variability.

A healthy model does not guarantee a healthy inference system any more than a healthy database guarantees a good application.

The whole path matters.

Data Bits to Business Bytes

If inference does become ordinary infrastructure, the first consequence is that architecture work moves.

We will spend less time asking whether an application “has AI” and more time deciding where reasoning belongs, what evidence it may use and what constraints surround it.

Retrieval and evidence design therefore stop being specialist RAG concerns and become part of core information architecture.

Schema design did not disappear when databases became easier to use.

It became foundational.

I expect something similar around source governance, metadata, chunking, retrieval, ranking, provenance and freshness.

And governance has to expand with it.

The meaningful control surface is not just the model. It is the entire reasoning chain: acquisition, permissions, retrieval, inference, deterministic checks, review, logging and downstream action.

Putting a policy document next to an inference endpoint is not governance.

Designing important boundaries so that they are visible, testable and difficult to bypass gets much closer.

Boring will be a milestone

Technology industries tend to celebrate visible progress.

Bigger models. Higher benchmarks. Longer context windows. Impressive demos.

Enterprise maturity usually looks different.

It arrives when people stop talking about the technology and start depending on it.

Databases became foundational when nobody felt compelled to put “database-powered” on the front page.

Browsers became infrastructure when “web-enabled” stopped sounding impressive.

Cloud became ordinary when deployment location became an architectural decision rather than an identity.

Inference may be heading toward the same fate.

Good.

Boring is not the insult. Boring is the destination.

The most interesting milestone in Enterprise AI may be the day an architect puts an inference service on a system diagram with roughly the same excitement as drawing a database.

Because by then, the conversation will have moved to the questions that actually matter.

What information does it reason over?

What boundaries constrain it?

How do we know when it is wrong?

What happens when it fails?

What does the business gain because it is there?

That is where engineering gets interesting again.

The experiment that prompted this piece started as a question about whether local inference could do serious analytical work.

Promisingly, it could.

What stayed with me afterwards was how quickly the models themselves began to feel ordinary.

The difficult work moved into the architecture around them.

I suspect that is a preview of where Enterprise AI is heading.

Not toward a world where every application loudly announces that it contains intelligence.

Toward one where intelligence is simply another engineered capability — designed, governed, measured and expected to work.

If you’re already deciding where inference belongs in your architecture — or whether it belongs there at all — I’d be interested in comparing notes.

Where useful, I’d rather build enough to see what the bit says than debate the architecture indefinitely.

#ThinkInsideTheBox #EnterpriseAI#LocalAI #AIArchitecture #LLM #DataArchitecture #SystemsThinking