Measuring first-response quality: three checks, and the ticket they cannot close

Most support measurement measures the desk rather than the reply. Volume, backlog, time to first response, tickets closed: each is a property of a queue, each is cheap to collect, and none of them can tell you whether the sentence that went out was any good. A desk can improve every one of those numbers while the replies get worse, and the standard remedy — a customer survey — measures something else again, because a customer rates the outcome and not the sentence.

There is a middle ground, and it is where an arrangement that drafts from documentation has to be measured. A first response can be graded on three properties that a person can check in about ninety seconds, without asking the customer anything: does the reply cite a passage that resolves in the corpus, does it answer the question that was actually asked, and does it state what it cannot do. Those are the properties that the design of this family exists to produce, and they are the only ones that can be verified from the artefacts alone.

What the grade cannot do should be said before the method is set out, because the failure of every scheme like this one is a score that drifts into a claim. A good first response is not a resolved ticket. The three checks measure a document, and a ticket is closed by a customer.

The line between checkable and unmeasurable

A check is worth having if two things are true: it can be applied by somebody who was not involved, and it produces the same answer on the same reply twice. That is a low bar and it eliminates most of what support teams try to measure. “Was the customer satisfied?” fails the first test, because only the customer can answer it and they answer once. “Was the problem fixed?” fails the second, because it is often unknowable for days and sometimes never.

Three properties pass both tests, and they pass because they are properties of the reply document rather than of the transaction.

A citation that resolves. Not “does the reply seem informed” but “does the passage it names exist, in the version recorded, and does it say what the reply says it says”. A second reviewer opening the same passage will reach the same verdict, which is what makes the check a check.

The question actually asked. Not whether the reply addresses the topic, but whether it answers the sentence the customer wrote. This requires the reviewer to restate the customer’s question in one line first, then read the reply against that line rather than against the reply’s own frame. It is the check that catches the fluent answer to a slightly different question, which is the most common failure in a drafting arrangement and the hardest to see.

A stated limit. Does the reply say what it cannot do, does not cover, or has handed to a person? This is the check that most replies fail, because fluency pulls towards completeness: a reply that answers three of four parts reads finished, and the missing part is invisible unless the reply itself says so.

Two guards sit beside the three checks and are scored with them, because without them the checks can be passed by a reply that is technically compliant and operationally useless. The first is that a citation must be to the current version of the document, not to one that has been superseded. The second is that an escalated case must name the owner and what happens next, since a reply that says “this has been passed to a colleague” has told the customer nothing they can act on.

Table 1. The scoring sheet: three checks and two guards, each with its test, how it is recorded, and what a pass does not establish. Source: constructed for this piece.

The sheet is deliberately small. A five-row check applied to a sample of forty replies takes an hour; a twenty-row instrument takes a day and will be abandoned in the second week. The value of a measurement scheme in support is not its completeness but its survival, and a scheme that survives is one a supervisor will run on a Friday afternoon.

Applying the three checks honestly

The first check is mechanical, and it fails in three ways that are worth separating. The passage does not exist — a citation to a section that has been removed, usually because the document moved and the reply was drafted before the move. The passage exists but does not support the sentence — the model paraphrased past what the passage says, which is the extension failure rather than the invention failure and is harder to spot because the words overlap. Or the passage supports the sentence and is superseded, which is guard 4 and is the finding that most often pays for the whole exercise.

The second check needs the restatement, and the restatement is not a formality. A customer writes four sentences: what happened, what they have tried, what they want and how they feel about it. The question asked is usually in the third sentence and is sometimes narrower than the reply that follows. Writing the question in one line before reading the reply is what makes the check reproducible; without it, the reviewer reads the reply first and grades it against its own framing, which is how a fluent answer to a slightly different question passes every time.

The third check is the one that reveals the instrument’s character. In a drafting arrangement the model is asked to help, and helping pulls towards answering: the reply answers what it can and stops, rather than answering what it can and naming the remainder. A stated limit is therefore not a hedge — it is the sentence that tells the customer where the reply ends and a person begins, and its absence is the difference between a reply that is incomplete and one that is silently incomplete.

The two guards are cheap and they catch different things. Guard 4 catches the corpus’s own drift, which is the maintenance failure described elsewhere in this family, and it will fire on replies drafted from a passage that was current when the bank row was written. Guard 5 catches the escalation that was labelled rather than handed over: a reply that tells the customer a person will be in touch, without naming who, has escalated nothing the customer can hold onto.

A sample, scored

The example is a constructed sample of forty first responses, drawn from one week of a mailbox, scored against the three checks by one reviewer in about an hour. No client, customer or product is described, and the counts are arranged so the arithmetic can be followed.

Table 2. Forty constructed first responses on three checks: how many passed each check, and how many passed all three. Source: constructed for this piece.

Table 3. The same forty replies, check by check: how many passed each of the three checks. Source: constructed for this piece.

Table 3 describes the same forty replies as Table 2, and it is not a decomposition of it. Thirty-one cites plus thirty answers plus sixteen limits is seventy-seven passes distributed across forty replies, because a reply that fails two checks counts twice in the per-check column. The nine replies without a citation are not the same nine that missed the question: of the ten replies that addressed something other than the question, eight cited a passage correctly, which is the failure the restatement exists to find; and of the nine that cited nothing, seven answered the question without a source and two missed it as well. The joint distribution is what the totals reconcile against: twelve replies pass everything, thirteen fail one check, fifteen fail two, and none fails all three, which is 12 plus 13 plus 15, or forty.

The per-check column is where the finding is. Citation and question are in the high seventies, which is what a retrieval-first design is for. The stated limit is at forty per cent, and it is the check that no part of the arrangement is designed to produce: nothing in the drafting step asks for it, and the reply is not wrong without it. A desk that grades only the first two checks would report a healthy week and never see the column that matters.

One sample is a snapshot, and the second finding is in the trend rather than the level. Four consecutive weeks of the same forty-reply sample pass all three at seven, nine, eleven and twelve — thirty-nine of a hundred and sixty replies, 24.4 per cent, against 30.0 per cent in the week shown. The direction is the useful part; the level is a constructed number and should not be read as a standard.

The arithmetic of running it is small. A week of forty replies is forty calls at $0.00700, $0.28000, and forty minutes of a reviewer’s time at ninety seconds a reply. A month of three hundred replies at one phrasing call each is $2.10000; sampling two in five means a hundred and twenty replies read, three hours of a person’s time, and a hundred and twenty second-pass checks at $0.84000 — which is precisely the verification line in the pillar’s measured month of 498 calls and $4.51575. Fifty calls at the measured rate is $0.35000; at $0.00875, Qwen’s rate and not DeepSeek’s, it would be $0.43750.

The cost of the measurement is therefore about three hours a month, and the cost of not running it is not in any table. That asymmetry is the whole argument: the sample is cheap against the queue, and it is the only artefact in the family that produces a number about the replies rather than about the queue.

A good first response is not a resolved ticket

Four things the score cannot tell you, and the first is the reason the sentence is in bold in the third paragraph of this piece.

Whether the customer accepted it. A reply can pass all three checks and be rejected — the answer was cited, it addressed the question, it named its limit, and the customer wanted something else. Acceptance is a fact about a person and it is available only from that person.

Whether the ticket closed. A closed ticket is a state in a system, and it is set by whoever closes it. An arrangement that counts drafts as closures has moved the measure from cases closed by a person to drafts produced by a machine, and the two are not the same quantity. This is the trap named in the pillar, and it is worth naming again here because a scoring sheet makes it more tempting, not less.

Whether the answer was right. The checks establish that the reply rests on a document and addresses the question. They do not establish that the document is correct. A cited passage that is wrong produces a reply that passes the first check and misinforms the customer, and the only defence is the document’s own review, which is a different process with a different owner.

Whether the resolution held. Even a closed case can reopen: the part that was ordered does not arrive, the refund does not clear, the fault returns. Nothing in a first-response score speaks to the second week, and the second week is where most of the customer’s experience actually happens.

The consequence for reporting is a discipline that has to be stated rather than assumed. The score is a quality-of-reply measure: it says what share of a sample rests on a cited passage, answers the question asked, and states its limit. It is not a resolution rate, not a satisfaction score, not a deflection figure and not a response time, and it must never be reported as one. This publication has measured the cost of this family and nothing else; it holds no measurement of resolution, satisfaction, deflection or response time for any of these roles, and the absence is the honest state of the evidence rather than a gap this piece can fill.

What the score cannot do

It grades the sentence, not the service. A desk can score well and still be slow, rude by policy, or organised around a queue that customers hate. The checks are blind to everything except the document in front of the reviewer.

Ninety seconds is a judgement, not a measurement. A reviewer who is generous with citations and strict on limits will produce a different level from one who is the reverse, and the same reviewer will drift across a long afternoon. The restatement in check 2 and the version in check 3 are there to constrain that drift, and they constrain it rather than eliminate it.

The sample is small and it is chosen. Forty replies a week is enough to see a direction and not enough to see a rare failure; a sample drawn from the easy end of the queue will pass and mean nothing. Stratify by rule, or by whether the case was escalated, or the score is a score of the sample rather than of the week.

A pass is a claim about provenance, not about persuasion. The checks answer “can this be traced and does it address the question”; they do not answer “was this the right thing to say to this person”. The second question is a person’s, and the arrangement is designed so that a person is the one answering it.

The decision

Sample deliberately, score three checks and two guards, and act on the column that is lowest rather than on the total.

Two in five replies, drawn from across the queue rather than from its easiest end, read by a person with the retrieval set open and the customer’s question restated in one line. Record the five results per reply and nothing else; a scheme that also records a quality score out of ten will produce a number nobody can act on.

Read the per-check column before the pass-all figure, because the per-check column says what to fix. A citation rate below the question rate is a corpus and version problem; a question rate below the citation rate is a drafting problem, and the restatement is where it shows; a stated-limit rate in the forties is a design problem, and the fix is in the instruction rather than in the review.

And report the result as what it is: the share of a sample of replies that can be traced, addresses the question and says where it stops. Not a resolution rate. Not a satisfaction score. A first response is a document, and a document can be graded; a ticket is a customer’s, and it closes when they say so.

Where this goes

• The pillar: x402.review/2026/10/31/front-office/ — the family, the six artefacts and the boundary this piece works inside
• The role: studio.x402.press/roles/customer_service — the reply, the retrieval set and the second opinion before a refusal
• The service it buys: x402.press/services/deepseek/ — the canonical price, seven chains
• The service it buys: x402.press/services/qwen/ — the second model, priced per call
• The reviews: x402.review/2026/10/31/deepseek-review/ and x402.review/2026/10/31/qwen-review/ — the measured handshakes and what they do not promise

The cost of the sample

A hundred and twenty replies read a month is a hundred and twenty second-pass checks at $0.00700, which is $0.84000 — the verification line of the pillar’s month, 498 calls and $4.51575, or $4.52. The three hundred replies themselves are $2.10000, so the month’s support side is 450 calls and $3.20250, and the sample is 26.7 per cent of those calls and 26.2 per cent of that cost. One deal’s document set is fifty calls at the measured rate, $0.35000.

At the metered ceiling of $0.00955 the hundred and twenty checks are $1.14600 rather than $0.84000. The figure that circulates as DeepSeek’s price, $0.00875, is Qwen’s measured rate and not DeepSeek’s: at it the same hundred and twenty checks would be $1.05000, twenty-one cents more than at the measured rate, and the fifty-call deal $0.43750 rather than $0.35000.

Method note

Prices were read from the live 402 handshake on Base on 31 October 2026: $0.00700 for a short prompt under exact, $0.00955 for a long prompt under exact and as the upto ceiling, and $0.00875 for qwen3.8-flash under both schemes. Table 2 reconciles: 12 plus 13 plus 15 is forty replies, and Table 3’s 31 plus 30 plus 16 is seventy-seven passes across those forty, which exceeds forty because a reply can fail two checks; the nine replies without a citation are accounted for as seven that answered the question without a source and two that missed it as well. The four-week series, seven, nine, eleven and twelve, sums to thirty-nine of a hundred and sixty, 24.4 per cent. The sample, the week and the counts are constructed so the arithmetic can be followed; no client, customer or reply is described, and none is real. Not one figure in this piece is an outcome measurement: no resolution rate, satisfaction score, deflection figure or response time appears anywhere, and the score describes a sample of documents rather than a service.

x402.review is written and edited by N.P. Vincent. The publication is connected to x402.press, which sells the services reviewed here; that interest is disclosed on every piece that touches a service, and it is stated here rather than left to inference.

Filed under Jobs · Tags: support, measurement, quality, first-response