AI Evaluations: How Startups Build a Defensible Learning Loop
— By The Global Intelligencer
Build AI evaluations around customer workflows, observed failures and reliable grading. Turn each incident into evidence that improves the next release.
Key Implications
- Useful evaluation cases describe customer success and failure rather than generic model intelligence.
- Deterministic checks should handle outcomes that can be verified through ordinary code.
- Evaluation sets need fresh cases and holdouts so improvements remain meaningful.
- A defensible learning loop depends on lawful data access, disciplined labels and actual product changes.
Editorial note: This is AI-assisted editorial analysis, not original news reporting. Factual claims were checked against the linked primary sources on October 17, 2026. Recommendations are analysis, hypothetical examples are labeled, and photographs are illustrative.
A startup does not gain a durable advantage merely by collecting many prompts or reporting a high score on a convenient test set. The useful advantage comes from learning which customer tasks fail, understanding why they fail and turning that knowledge into product changes that survive the next release.
AI evaluations can support that process when they represent the work customers actually need. This article explains how to build a disciplined evaluation loop. It does not claim that a particular startup has achieved a competitive moat or that any evaluation method guarantees commercial success. The examples are hypothetical product situations.
Define evaluation around the task
Anthropic's Demystifying evals for AI agents discusses tasks, trials and graders, and recommends deterministic grading where it fits, with model and human assessment where needed. Its guidance offers an implementation perspective, not independent proof of a startup's product quality.
Our analysis is that the most valuable proprietary knowledge concerns a customer's definition of success. A general benchmark may help compare broad capabilities, but it cannot automatically specify whether a product completed a particular workflow correctly. Start with that local definition before choosing a grading technique or building an evaluation dashboard.
Write an acceptance criterion a customer would recognize
Describe what the task should accomplish and what would make it unacceptable. A document-extraction product might need specific fields, evidence for each field and a clear missing-value state. A scheduling assistant might need the right participants, permitted time window and confirmation before invitations are sent.
Include legitimate non-completion. If a necessary record is missing, a correct system may request more information. If an action is outside the user's authority, a correct system may refuse it. An evaluation that rewards completion in every case encourages the product to hide uncertainty or cross a boundary.
Keep the criteria separate from one model's preferred wording. Customers often care about an accurate result or correct state change rather than a particular phrase. Overly literal text matching can penalize acceptable variation while missing a serious behavioral failure. Match the grading method to the requirement being evaluated.
Choose graders according to what can be verified
Use ordinary code for properties that can be checked reliably that way. A required field can be present or absent. A proposed identifier can exist or not exist in a test dataset. A tool can create the expected synthetic record or fail to do so. These checks can provide clear evidence without asking another model to interpret everything.
Some properties require judgment, such as whether an explanation adequately addresses a nuanced question. A model-based grader may help scale that review, but it needs a rubric and calibration. Compare its judgments with informed human assessment on representative cases. Disagreement should lead to investigation rather than an assumption that either side is automatically correct.
Reserve human review for cases where it adds useful information. Reviewers need the task, evidence and criteria, not just an isolated answer. A preference score without context may reveal style differences instead of product correctness. Record why a result failed so the team can decide what to change.
Build a dataset with known rights and provenance
Start with synthetic cases, appropriately licensed material or customer data the business is authorized to use for evaluation. Document where each case came from and what restrictions apply. Access to production data does not automatically establish unrestricted rights to retain or reuse it across development environments.
Remove unnecessary sensitive information while preserving the behavior under test. A regression case about a missing permission can often use synthetic users and documents. A case about document structure may not require the original customer's names or account details. The goal is a useful test, not a comprehensive archive of customer activity.
Keep labels and expected outcomes reviewable. If the supposed correct answer is wrong, the evaluation can reward a regression and reject an improvement. Give cases owners and a way to flag ambiguity. Treat disputed labels as work to resolve rather than noise that can be averaged away.
Use failures to expand coverage
When a customer reports a failure, identify the underlying behavior. Was retrieval incomplete? Did the model choose an inappropriate tool? Was the output unsupported? Did a retry create a duplicate action? The answer determines what a meaningful regression case should preserve.
Avoid copying only the final message into the test set. An agent failure may depend on tool state, permissions or the sequence of events. Recreate the relevant environment with controlled fixtures where possible. Otherwise, the test may pass while no longer exercising the condition that caused the incident.
Keep the original task and expected boundary explicit. The product should still complete legitimate work while handling the failure condition correctly. A change that prevents every action may stop one incident but destroy the feature. Regression cases should help maintain both useful capability and the intended limits.
Protect a holdout from routine tuning
A team can become very good at passing the cases it repeatedly inspects. That is useful for fixing known problems, but it is weaker evidence of broader improvement. Preserve cases that are not used for day-to-day prompt adjustment and compare behavior on fresh examples as the product evolves.
Separate development feedback from release evidence. Engineers need detailed failure examples to improve the system. A release decision also needs evidence that the improvement extends beyond those examples. The appropriate split depends on dataset size and task diversity; no fixed ratio is universally correct.
Refresh coverage when the product or customer population changes. A new document type, language, tool or access model can create a different workload. A stable score on an old dataset does not prove that the new feature is ready. Record which responsibilities each evaluation actually covers.
Include operating behavior, not just output quality
An acceptable answer that takes too long or requires excessive human repair may not serve the intended product. Track completion time, resource use and intervention alongside correctness. Connect those measurements to cost per successful task rather than evaluating each model call in isolation.
For agents, inspect consequential tool effects. A model can say that an operation succeeded while the downstream record remains unchanged. It can also repeat an operation after an uncertain response. Controlled state checks make those differences visible. Text-only evaluation misses part of the product's behavior.
Evaluate recovery from tool errors and incomplete inputs. A robust product should preserve enough state to explain what completed and what remains pending. It should not fabricate success simply to produce a tidy answer. These cases often reveal whether the architecture supports reliable operation under real conditions.
Make release decisions from the pattern of failures
Averages can hide consequential regressions. A change that improves many easy cases while breaking a small class of high-impact tasks deserves scrutiny. Report performance by meaningful task category and identify failures that should block release regardless of the overall score.
Write the release criteria before running the comparison where practical. Otherwise, a team can reinterpret the threshold to accommodate the result it wants. Allow exceptions through an explicit decision process that records the limitation and the user-facing consequence. An exception should be visible rather than silently folded into a success claim.
Preserve the configuration used for the evaluation: model, instructions, tools and relevant data versions. This helps explain later changes and reproduce failures. The purpose is not bureaucratic completeness. It is the ability to tell whether an apparent improvement came from a deliberate product change or an unnoticed environment difference.
Where defensibility can emerge
Evaluation work becomes strategically valuable when it connects lawful task data, reliable labels, observed failures and rapid correction. A competitor may access a similar foundation model without having the same understanding of a customer's workflow. That difference can matter if it produces consistently better service.
It is not automatic. A dataset that is stale, mislabeled or detached from release decisions offers little protection. Nor does a private benchmark prove customer demand. The learning loop needs to improve something customers value, and the business still needs distribution, trust and a viable economic model.
Treat evaluation as an operating capability rather than a marketing number. Its strongest evidence is a product change that fixes a known problem without breaking the rest of the workflow. Over time, a disciplined collection of those improvements may support a defensible position.
Frequently asked questions
How many evaluation cases does a startup need?
Enough to cover its important tasks and failure modes with useful evidence. A small, well-understood set can be more actionable than a large collection of ambiguous labels. Expand coverage as incidents and new features reveal gaps, and state what remains untested.
Can one AI model grade another?
Yes, but the grader needs a clear rubric and calibration against appropriate human judgment. Use deterministic checks for properties that do not require interpretation. A model-generated score should not be mistaken for an independent guarantee of correctness.
Does a benchmark improvement establish a moat?
No. It establishes an improvement on the tested benchmark under the recorded conditions. A competitive advantage also requires useful customer outcomes and an ability to keep learning. Claims should stay within the evidence actually measured.
The founder's next step
Choose one important workflow, write its acceptance criteria and build a reviewable set of representative cases. Add observed failures, preserve a holdout and connect results to release decisions. That is a concrete learning loop a startup can improve.
Sources and related analysis
- Anthropic: Demystifying evals for AI agents.
- AI cost per successful task.
- AI agent observability.
- AI vendor exit planning.
The photograph is illustrative and does not depict a company evaluated here. Image supplied through Unsplash.