The Readiness Bar: How We Score Context Before Anyone Uses It

The "Context Layer"

The analytics stack has shifted. Where a dashboard was previously the terminal output, agentic systems support conversational querying, proactive detection of anomalies, and capabilities that were until recently theoretical. That capability depends on a prerequisite: an explicit context layer. Directing an agent at a warehouse without one produces a false sense of precision, since the system responds with equal confidence whether or not the question was resolved to the correct entities and definitions. Context is what makes raw data interpretable to an agentic workflow, reliably and at scale, rather than reducing it to a static figure on a screen. Lumi’s knowledge base is where that context is captured and governed: the metric definitions, business rules, exclusions, and terminology a business runs on, recorded once and retrieved as each question requires, so the agentic workflow draws on the definitions relevant to what was asked. To establish where a client’s knowledge base sits on a context maturity curve, Lumi’s Value Engineering team puts the knowledge base through a series of evals that separate the context components proven reliable from those still requiring remediation.

Constructing a context layer is only half the challenge. The other half is demonstrating that the agentic system can use the context layer in a manner that reliably and consistently produces accurate results. Unlike a sales demo which is a single trial under favorable conditions, an enterprise deployment is a population of trials, run daily by everyone who touches the system. The evaluative question therefore shifts from capability to reliability: not whether the knowledge base can produce a correct answer, but whether it does so consistently under repeated use outside controlled conditions. The operating criterion can be stated as an invariance condition: the same question should return the same answer irrespective of who asks it or how it is phrased. The eval framework below is how the Lumi AI Value Engineering team tests that condition before a knowledge base is released to the everyday business users across an enterprise.

In Lumi, the knowledge base is the context layer positioned between an organization’s data and the agentic harness built on top of it. It supplies the operational definitions a given business runs on. To take one illustrative example: “average order value” might be computed only over paid, customer-facing orders, excluding wholesale and internal orders that would otherwise bias the result, while “active customer” resolves to a business-specific rule rather than a generic one. The specific definitions vary by business; what generalizes is the requirement that they be stated explicitly rather than inferred, since a system compelled to infer them will infer something plausible but incorrect. Knowledge base evals therefore target two distinct properties: whether the agentic workflow uses that context to produce a correct answer, and whether it can produce that same answer regardless of who is asking the question or how it is asked. In this framework, one eval is a single question intent, run repeatedly across n-phrasings and scored as a set rather than as one answer.

Testing more phrasings per eval sharpens the readiness signal, at a proportional cost in tokens:

Accuracy Is the easy part. Consistency is the goal.

A single correct answer establishes that the system can succeed once, which is a sufficient result for a sales demo. It does not establish reliability in production at large, complex enterprises. In practice, five colleagues will request the same quantity in multiple ways: “average order value,” “AOV,” “what’s the typical basket size.” All five colleagues expect an identical figure in return.

Accordingly, the unit of measurement is not one question graded once. Each eval holds the underlying intent fixed and varies the phrasing, asking it several different ways, then quantifies how much the answer moves across runs. Coverage is not limited to one intent: each eval session applies this phrasing-variance test across a representative set of evals, so the resulting score characterizes performance across the distribution of questions a team actually asks rather than a single curated example.

Consider a single eval built on one intent, average order value, run across five independent phrasings, each graded for accuracy:

Across the five runs, the generated query logic and the key figures are each compared as a set, producing one Approach Consistency score and one Output Consistency score for the eval. Those combine with the average Accuracy score into a single Composite Score.

Three dimensions, one composite score

Every eval is scored on three dimensions, which are then combined into a single Composite Score on a 0 to 100 scale:

Approach Consistency. Does the system reason about the question the same way each time? The generated query logic is compared across all the phrasings tested in the eval using Levenshtein string-edit-distance similarity, after normalization, so semantically equivalent variations are not penalized.

Output Consistency. Do the key figures agree across phrasings? Scored with Jaccard similarity, this is the highest-signal measure in the framework, because the quantity is directly observable. If a figure shifts with the surface form of the question, the context layer contains an ambiguity.

Accuracy. Assessed independently of consistency: is the answer correct when validated against the source data definitions?

How consistency is measured, and where measurement gives way to judgment

The first two dimensions are computed, not judged. No reviewer inspects the answers and decides whether they “feel” similar. Every run in an eval is scored with the same similarity models under the same parameters, so the result is reproducible and independent of any individual reviewer.

Accuracy is treated differently by design. A similarity model can establish that two queries correspond; it cannot establish that the logic or formula underlying them was the correct one, because correctness is defined relative to the business rather than to the text. That’s why Accuracy is graded by an analyst for every eval: where a validated answer key exists, each run is checked against it, and where none exists, the analyst establishes the expected result from the source data before grading anything against it. In either case the run is rated in isolation, blind to whether the other runs agree. Two answers can be mathematically similar and both incorrect, a condition consistency scoring cannot detect on its own.

How an eval session runs: the n-phrasings-variance test

Each eval session applies the following protocol across a representative set of evals rather than a single curated question type.

1. Define the eval. Each eval begins from a question an actual stakeholder would ask: one with a single correct answer and many plausible formulations.
2. Vary the phrasing. Run each formulation as an independent run of the same eval, varying vocabulary, register, and structure as different colleagues would, while holding the underlying intent constant.
3. Record all outputs. For every run, the generated query logic, the natural-language answer, and the key figures the answer relies on are captured verbatim, so nothing is normalized away before scoring.
4. Score logic and output consistency. Automated comparison across all runs in the eval, with no human judgment introduced at this stage.
5. Validate accuracy against the source of truth. An analyst cross-references the answer set for the eval against a validated answer key where one is available, and derives the expected result directly from the source data where one is not. Runs are rated in isolation, independent of whether the remaining runs agree.
6. Aggregate into a Composite Score. The three dimensions combine into one score per eval, and per-eval scores aggregate into a session-level eval report that can be tracked over time.

What the score tells you, and what to do about it

A composite score is only useful if its interpretation is shared, and only actionable if it points somewhere. The tier tells you whether a domain is ready to release. The pattern of failure underneath it tells you what to change, and in practice the pattern is the more useful output.

The tier sets the decision. The failure pattern sets the work. Five remediation classes account for most of the movement we see between tiers.

1. Define default filter conditions explicitly. Most business metrics carry unstated exclusions: internal orders, test accounts, cancelled transactions, wholesale channels. A system with no instruction will sometimes include them and sometimes not, which produces exactly the kind of phrasing-dependent variance these evals are built to catch. The exclusions have to be stated as defaults in the context layer, not left to be inferred per question.

2. Give date and time fields enough context to join and sort. When failures cluster on time-qualified phrasings ("last quarter", "by month", "for the period") while unqualified phrasings score well, the gap is almost never in the metric definition. It is in the date fields. Common causes are ambiguous grain, several plausible date columns with no guidance on which is authoritative, and dates stored as strings, which degrades sorting and comparison. Converting string dates to native date or numeric representations, and stating which date column is canonical for which question type, is a general best practice that tends to lift several evals at once.

3. Bind each filter to the right part of the calculation. The signature here is distinctive: outputs cluster into two or three distinct values, each internally consistent, rather than scattering. That usually means a filter is being applied at the wrong level of a computation rather than being missed entirely. At one enterprise hardware client, average unit price was defined so that a server-only filter was intended to constrain the unit-count denominator alone, while the numerator carried total order value. The knowledge base definition was ambiguous about scope, so the system sometimes applied the filter to the whole calculation, dropped non-server value out of the numerator, and understated the result. Restating the definition so the filter binds only to the unit count moved three affected question types out of the bottom tier and into the acceptable range. No data model change was required.

4. Name fields for their function. Field names carry more weight than most teams expect. The most common gain comes from making one distinction explicit: which fields are labels, meant for display and grouping, and which are functional, meant for sorting, filtering, and joining. Where that distinction is implicit, the same intent gets resolved through different join paths on different phrasings, which shows up as low approach consistency even when outputs happen to agree. A naming convention that encodes the distinction removes the ambiguity at the source.

5. Write field context as instruction rather than prohibition. Field descriptions framed around what not to do consistently underperform descriptions framed around intended use. "Use for month-over-month comparison" outperforms "do not use for trending." The same holds for surface conventions: standardized capitalization and consistent phrasing across field descriptions measurably reduce approach variance, because the model is reading those descriptions as signal about intended use.

Where context tuning is not enough, revisit the model. The five remediations above operate on the context layer and are comparatively cheap. When low scores persist across unrelated topics rather than concentrating in a few, the constraint is usually structural: too many equally plausible join paths, or entity relationships that require reconstruction on every query. The fix is a data model legible to a reader, typically a star or snowflake schema, where the grain of each table and the path between them is unambiguous. This is a larger investment, and the eval scores are what justify it. A domain that plateaus in the Acceptable tier through three rounds of context tuning is making a data model argument, not a context argument.

Then re-run against the identical phrasing set. Remediation is only demonstrated if the test does not change. The same evals, the same phrasings, the same scoring parameters. A score that improves because the phrasing set was softened is not evidence of anything, and holding the set fixed is what makes the before and after comparable.

Why this matters

Most evaluation approaches ask “did the AI get it right?” once, on a curated sales demo question, and stop there. That answers whether a system can work, not whether it works reliably when a hundred users ask a hundred slightly different versions of the same question.

That distinction is why this eval framework exists: the real question is whether the agentic workflow can be relied on in production. A gap identified under controlled testing is materially cheaper to remediate than the same gap surfaced by business users.

The payoff also compounds downstream: an agentic workflow that answers consistently regardless of phrasing earns user trust faster, and that trust is what turns a one-time sales demo into daily, habitual use, the behaviour that ultimately drives adoption at scale.

See where your context layer stands

This context commissioning process is conducted by Lumi’s Value Engineering team for every Enterprise client during onboarding. The resulting eval report identifies which components of the client’s knowledge base meet the reliability threshold and which require remediation before Lumi is rolled out to the client’s wider user base. The threshold is deliberately high: a domain is not released to the wider team until its evals clear it, which forces remediation to occur before users encounter the failure. The procedure is demanding by design, and the resulting evidence is a substantial part of why clients trust the deployment.

Get in touch about a readiness assessment.

‍

‍