Write the eval before you write the model.
A model with no eval is a demo. It looks good in the room, and nobody can tell when it gets worse.
We start with fifty scored cases from your own material. That answers, cheaply, whether the task is one a model can do at all.
Email us
Data preparation is most of the job
Training on your data means getting your data into a state worth training on. Deduplication, labelling, and splitting carefully enough that the test set isn't quietly leaking into the training set.
Projects overrun here, and it's the part that gets left out of the proposal.
-
Cases come from your own material, not a public benchmark that flatters everyone equally.
-
Scoring is written down, including who breaks the tie when two reviewers disagree.
-
Evals run on every change, so a regression shows up in your pipeline instead of in a complaint.
What we do here
Getting documents, records and transcripts into a state something can actually be trained or grounded on.
Fine tuning where the task is narrow and repeated. Often the answer is a smaller model on better data.
Scored test sets and regression runs, so quality is a number you can show a committee.
Running it in your environment, monitored, with the model swappable when a better one appears.
What we'll talk you out of
Four ideas come up constantly and we argue against all of them:
-
A chatbot over documents nobody maintains. It will answer confidently from a 2019 policy, and a customer will find out first.
-
Automated decisions people can appeal, unless a human signs off and can see why the model scored it that way.
-
Fine tuning before prompting and retrieval have been tried properly. Slower, costlier, usually unnecessary.
-
Any pilot without a named owner in the business. Those get demoed once and quietly abandoned.
Practical questions
Three worth settling in the first conversation.
-
Which models do you use?
Whichever scores best on your eval, and that changes every few months. Building so the model is swappable costs almost nothing at the start and saves a rewrite later.
-
How long does a first engagement take?
Two to four weeks to produce an eval, a data readiness assessment and a recommendation. Deliberately short, because it's the piece that tells you whether to spend the rest.
-
What if the answer is that a model won't help?
Then we say so in writing, with the reason. That's a planned outcome of the first engagement rather than a failure of it, and you keep a document that's useful to the next person who proposes the idea.