Change by Design.© When Code Stops Being the Bottleneck

Test Early. Learn Fast. What firsthand work with local inference taught me about finding the real constraint.

A business leader from the RegTech world recently reached out for inputs local inference.

The question wasn’t whether a model could run locally. That part is increasingly uninteresting.

The real question was whether locally hosted inference could handle serious enterprise work: continuously changing information, large documents, analytical reasoning, traceability, reports, alerts and the operational discipline expected of a real system.

There was plenty we could have debated — architectures, benchmarks, model sizes, GPUs.

But seeing is believing.

And when in doubt, let the bit decide.

That is increasingly how I like to approach emerging-technology questions. If the opinions are plentiful but the evidence is thin, build enough to find out.

I think of these exercises as What the bit saw.

Make the proposition explicit. Build enough of the real thing. Put real data through it. Then pay attention to what actually happens.

The original RegTech environment contained information far too confidential to become experimental material, so I chose a different product premise using publicly available company information: regulatory filings, corporate disclosures, credit-rating rationales, earnings-call transcripts, news and other evidence.

The relationship between that premise and the original conversation ends there. One supplied the question. The other gave me a safe place to investigate it.

And, as I tend to do with these experiments, I used evenings and weekends rather than the office day. I wanted to stay close enough to the architecture, data, runtime behaviour and failures to form a view firsthand — not assemble one from somebody else’s slides.

First, the answer we came looking for

Can locally hosted inference perform serious analytical work over real, continuously changing information?

Promisingly, yes.

Over roughly two weeks, the experiment went from product discovery and architecture to a functioning system operating on real company information.

It acquired and processed documents from multiple sources. It created a searchable evidence repository. It combined lexical and vector retrieval. Local models handled different reasoning roles. The system supported evidence-grounded chat, analytical reports, alerts, contradiction detection and forensic findings. Deterministic calculations remained deterministic rather than being handed to a model merely because one was available.

Most importantly, it began doing analytical work of the kind that can otherwise consume hours of human effort — and could keep doing it as new information arrived.

That does not prove that every enterprise workload should move to local inference.

It does answer the question we set out to investigate.

The proposition survived contact with a real system.

That is what I wanted from the PoC.

Then things became more interesting.

The system answered questions we hadn’t asked

Once a working system existed, it started exposing assumptions sitting around the model.

Some were about data.

Some about identifiers.

Some about evaluation.

Some about operations.

That distinction turned out to matter.

The experiment did not produce one binary verdict called “AI worked” or “AI failed.”

It produced a much more useful picture:

Inference capability — promising.

Public-data availability — inconsistent.

Identifier assumptions — sometimes wrong.

Evaluation machinery — capable of creating false confidence.

Operational controls — needed refinement.

That, for me, is what serious Enterprise AI experimentation should reveal.

The model is one layer of the system.

It shouldn’t get blamed for every problem around it — or credited for every success.

The 80% that became 33%

One acceptance target was to onboard at least 80% of the pilot companies automatically.

The real result was 33%.

If that number is reported without explanation, it sounds like a failed technology experiment.

It wasn’t.

The inference system had not failed to analyse company information.

The information required to automate onboarding simply wasn’t consistently available from the public sources we had expected to rely upon.

A company directory could omit a usable website address even for a major issuer. An investor-relations site might not contain all the expected material. Different companies exposed information in different ways. Some sources behaved exactly as hoped; others did not exist in the form the initial architecture assumed.

The important question therefore wasn’t:

Why did the AI achieve only 33%?

It was:

What actually constrained the other 67%?

Once we asked that, the constraint moved from inference to information availability.

Different problem.

Different remedy.

And a far more useful PoC result.

A technology can be viable while one of its dependencies is not.

Knowing which is which matters before anyone commits serious money or operating dependency to it.

The identifier that looked more authoritative

Another assumption concerned company mapping.

ISIN seemed the obvious identifier to prefer over a trading symbol. It is formal, structured and theoretically more authoritative.

Then we measured both against live data.

Symbol matching achieved 98%.

ISIN achieved 79%.

The exchange announcement data could carry an ISIN that was stale relative to the company master. The theoretically stronger identifier was practically weaker for that particular workflow.

That doesn’t mean symbols are universally better than ISINs.

It means the assumption had a context.

Real data exposed it.

This is why I like asking what the bit saw.

Architecture gives us a reasoned hypothesis.

Evidence gets the final vote.

Two wrong answers can agree beautifully

The sharpest lesson came from evaluation.

A configuration mismatch meant the reranking component was silently falling back rather than operating as intended.

The benchmark scored 12 out of 15.

A later run scored 11 out of 15.

Two close results. Plausible variation. Reassuring.

Then we found the configuration problem.

Once reranking was genuinely operating, the score was 8 out of 15.

Both earlier measurements had been wrong.

Worse, they had agreed with each other.

A broken ruler can be wonderfully consistent.

Repeatability tells us whether a result can be reproduced. It does not prove that we measured what we thought we measured.

For AI systems, that distinction deserves more attention because some outcome variance is expected. Two similar numbers can easily feel like evidence that the evaluation mechanism is healthy.

Sometimes they are.

Sometimes they are two readings from the same defect.

So evaluation needs to ask more than what score did we get?

Did retrieval actually execute?

Did reranking run?

Which context reached inference?

Which model answered?

Did anything silently fall back?

A metric without evidence about the path that produced it is weaker than it looks.

Test Early. Learn Fast.

These examples changed the way I think about the economics of a PoC.

For years, building enough software to test an architectural proposition properly was expensive. That encouraged long debates before implementation because engineering capacity itself was scarce.

Agentic engineering changes part of that equation.

It can now be surprisingly inexpensive to build enough of a serious system to test assumptions that might once have survived weeks of meetings.

That gives us a better operating principle than “fail fast.”

Test Early. Learn Fast.

The goal isn’t failure.

The goal is cheap validation.

Test the proposition while changing direction is inexpensive.

Find the bad source assumption before the entire data architecture depends on it.

Find the identifier problem before it propagates downstream.

Find the evaluation defect before months of decisions are based on its numbers.

The opportunity isn’t merely to write software faster.

It is to shorten the distance between an assumption and the evidence that can validate or overturn it.

That is a much more interesting use of speed.

Fast code still scales missing assumptions

Not every lesson was specific to AI.

One test could downgrade the development database to its base state. Under the wrong invocation, it wiped real acquired data.

A safeguard followed.

Then a different test path reached the same destructive operation and wiped it again.

It would be convenient to call that an agent-safety problem.

It wasn’t.

A CI pipeline, shell script or human could have caused the same incident.

The issue was that the first safeguard prevented a known command rather than the underlying class of destructive behaviour.

We had automated an incomplete understanding.

And automation executes omissions just as efficiently as instructions.

The same principle surfaced elsewhere when tests stayed green while a real application path contained a dormant defect that only appeared when a user actually exercised it.

Faster implementation therefore needs faster reality checks alongside it.

Automated verification.

Real-data verification.

Operational verification.

They answer different questions.

More code does not remove that distinction.

It makes the distinction more important.

A PoC should tell us which assumption failed

This may be the most useful lesson for leaders evaluating Enterprise AI.

A PoC isn’t one number.

And it shouldn’t produce one vague verdict.

A real system has layers:

model capability, data availability, data quality, retrieval, integration, operational reliability, governance, user usefulness and economics.

One can work while another does not.

In this experiment, local inference looked promising while automated public-source onboarding remained constrained.

Retrieval could work while a chosen identifier performed worse than expected.

The models could be healthy while the evaluation machinery produced misleading confidence.

Those require completely different interventions.

So instead of asking only:

Did the PoC work?

I increasingly prefer:

Which parts of the proposition survived? Which assumptions didn’t? And what would have to change before we scaled it?

That is a better investment question.

And, usually, a better engineering question too.

Data Bits to Business Bytes

Three implications stand out for me.

First, diagnose the layer before prescribing the remedy.

A data-availability problem is not automatically a model problem. A retrieval problem isn’t solved by buying a larger model. An evaluation defect isn’t evidence that the underlying architecture is poor.

“AI worked” and “AI failed” are increasingly inadequate descriptions of complex systems.

Second, use PoCs to reduce uncertainty, not manufacture confidence.

The purpose is not to produce an impressive demonstration. It is to find out what is true while the cost of being wrong is still manageable.

Sometimes the useful answer is yes.

Sometimes it is no.

Often, as here, it is: yes — provided we understand these other constraints.

Third, spend the new engineering velocity on learning.

Agentic tooling can shorten the cycle from question to implementation dramatically.

The valuable loop becomes:

proposition → implementation → evidence → diagnosis → revision

Code sits inside that loop.

It no longer necessarily dominates it.

That changes the engineering operating model.

Why I wanted to do this firsthand

There was a simpler reason I spent those evenings and weekends on the exercise.

I wanted an answer I had earned.

Enterprise AI is moving too quickly for every important question to be answered secondhand.

Research matters.

Architecture matters.

Experience from others matters.

But some questions become much clearer once you have worked close enough to the data, the models and the runtime behaviour to see where reality disagrees with the diagram.

The original question here was whether local inference could support a serious analytical workload.

The answer was promisingly yes.

Then the system added some footnotes.

Public information wasn’t as uniformly available as expected.

Formal identifiers weren’t always the best operational identifiers.

A benchmark could agree with itself and still be wrong.

Automation could scale an omitted constraint as efficiently as a carefully designed one.

Those weren’t reasons to retreat.

They were things worth knowing before the experiment became an enterprise commitment.

That is the larger opportunity I see in agentic engineering.

Not simply cheaper software.

Cheaper evidence. Earlier validation. Faster learning.

When the answer to an Enterprise AI question isn’t obvious, I increasingly prefer not to debate it indefinitely.

Test early.

Learn fast.

Ask what the bit saw.

Then decide.

And if you’re wrestling with a proposition where the opinions are plentiful but the evidence isn’t, I’d be glad to compare notes — and, where it makes sense, build enough together to let the bit decide.

#ChangeByDesign #EnterpriseAI #AgenticEngineering #LocalAI #AIArchitecture #EngineeringLeadership #SystemsThinking