All guides

Where AI earns its keep, and where it quietly costs you

AI7 min read

Stanford’s RegLab and Human-Centered AI Institute tested how often general-purpose language models produce false legal content. On specific legal queries they found hallucination rates between 69% and 88%.

A follow-up study looked at the specialist tools built to solve exactly that problem, with retrieval systems and curated legal databases behind them. Lexis+ AI hallucinated on 17% of queries. Westlaw’s AI-Assisted Research came in at 33%. GPT-4 hit 43%.

Purpose-built legal AI, sold to law firms, wrong on between one in six and one in three answers. That is the state of the art on a task where accuracy is the entire product.

What that means for your business

It does not mean avoid the technology. It means being precise about which jobs you hand it.

The useful split is between tasks where a wrong answer is obvious and tasks where a wrong answer looks exactly like a right one. Ask a model to draft five subject lines and a bad one is visible instantly. Ask it for the compliance requirements in your sector and a fabricated regulation reads identically to a real one, complete with a plausible section number.

Sort your candidate use cases along that line before you build anything. The first group is nearly free money. The second group needs verification machinery, and the verification usually costs more than the generation.

Where it works now

Drafting first versions is the strongest case. A model that gets you to a rough draft in thirty seconds saves the worst part of writing, which is the blank page. You still edit, and you would have edited a human first draft too.

Summarising material you already have works well, because the source is in front of you and errors are checkable. Meeting notes into actions, a long thread into a decision, fifty support tickets into five themes.

Classification and extraction are underrated and very reliable. Routing enquiries by topic, tagging content, pulling structured fields out of unstructured text. Narrow tasks, checkable outputs. That is where most of the quiet operational value sits, and it rarely makes it into anyone’s AI strategy deck because it photographs badly.

Code assistance genuinely accelerates development, with the caveat that it accelerates producing code rather than producing correct code. The review burden shifts rather than disappearing.

Where it fails expensively

Anything stated as fact to a customer without a human reading it first. Pricing, availability, specifications, policy, legal or medical claims. A chatbot that invents a refund policy has made a commitment somebody has to honour or explain away.

Anything where the model needs current information it was not given. Models do not know your stock levels, this week’s prices, or what changed in your terms last month unless you feed that in at request time.

Anything where confident tone substitutes for accuracy. The Stanford numbers exist because models produce fluent, well-formatted, entirely wrong answers, and fluency is precisely what makes the errors hard to spot.

Build the checking in

If you do put a model somewhere that matters, design the verification as part of the system rather than as a policy asking people to be careful.

Ground answers in your own documents and require the output to cite which one it used, then check that the cited source exists and actually says that. Constrain the output shape with a schema so downstream code fails loudly on a malformed response instead of passing garbage along. Route low-confidence cases to a person by default. Log every input and output so you can audit what happened when somebody complains.

That last one gets skipped constantly and it is the one you will want at three in the morning.

The cost of this checking is the real cost of the feature. A demo takes an afternoon. The version you can put in front of customers takes considerably longer, and teams that budget for the demo are the ones who quietly abandon the project six weeks later.

Model choice is a cost decision

Capability and price vary by more than an order of magnitude across models, and most tasks do not need the most capable one. Classification and extraction run fine on small fast models. Reasoning over a long document does not.

Build so the model is swappable. Providers ship new versions constantly, prices move, and a system with the model name hardcoded in fifteen places is a system nobody wants to upgrade. Route through a single layer and make the choice configuration rather than code.

Watch the token spend from day one. A feature that costs pennies in testing can cost real money at production volume, and the surprise usually arrives as a bill rather than a graph.

Start with something boring

The instinct is to build the customer-facing chatbot, because that is the visible one. It is also the one with the worst risk profile and the highest expectations.

Pick an internal task instead. Something where the output is checked by the person who asked for it, where a mistake costs a minute rather than a customer, and where you will find out quickly whether the thing helps. Get that working, learn what verification your context needs, then move outward.

That is how we scope AI solutions work, and the answer is often that a smaller piece of automation does the job without a model at all. Some of what we have built this way is on the projects page. If you want a straight opinion on whether AI helps your particular process, get in touch.

References