Benchmark what actually arrives.

Turn-taking latency, over five counted turns. He and the agent count in turns from a start drawn fresh for every call, somewhere between one and eleven: he says a number, the agent replies with the next one up and nothing else, and he says the one after that. His own numbers are fixed before the count begins, so he keeps counting in twos whatever the agent says, and a missed or wrong turn costs only that turn. He moves on as soon as a reply lands, and reads out any correction for a missed turn once all five are in. A one-word reply his recogniser hears as a number said differently, such as “to” for two, counts as that number. A turn that does not land within five seconds, or lands with the wrong number, scores nothing for that round. Talk-over and a turn that arrives twice are not scored separately: the timings are kept as evidence, but the mark is the right number and how fast it came.Per round, full marks for the right number within 800 ms, sliding down to 60 at 4,000 ms and no further — a turn that was taken in time is never worth nothing. A wrong number, or no reply within five seconds, scores nothing for that round. The drill is the mean of the five rounds.

How much of what he said actually arrived. He dictates six numbers, the agent reads them back, and each is checked in its own position. Every number is varied slightly for each call — tempo, pitch and tone — so it has to be heard, not matched against a published recording. It is scored against what reached the far end, not against your own transcript.Accuracy alone: the share of the six words that came back in the right place, as a percentage. Timing is recorded but not scored. Nothing back, nothing scored.

Barge-in yield latency, and whether the agent stops at all. He waits until the far end is actually heard counting, leaves it running for between ten and fifteen seconds, drawn for the call so the moment cannot be anticipated, then interrupts and times the silence. Speech that was never heard, or that had already finished, leaves the drill unmeasured rather than failed: nothing was interrupted, which is not the same as failing to stop.Full marks for yielding within a second of the interruption, time enough to finish the word, down to 60 at 2,000 ms. Speech that never stops scores nothing. Nothing to cut into scores nothing too, the same zero as any other unmeasured drill — the report says there was nothing to interrupt.

Whether it can say one word and then stay silent. He briefs it: “One round. When I say NOW, say ONE word only: OK, or Understood. Then stay silent until I say time. Marks for every second you hold. Starting NOW.” The word has to begin within five seconds of NOW, and proves the agent can speak at all: silence from an agent that cannot speak is not restraint. Then it must hold until he says time — somewhere between ten and twenty seconds, drawn fresh each run so it cannot be learned — without filling the gap. Filler, further acknowledgements and “are you still there?” all count as speech — this measures restraint, not politeness. The clock starts at the end of the word and stops the moment it speaks again, so an apology afterwards cannot buy the time back.Proportional, once the word is in. The seconds held after it, as a share of the silence drawn: holding until time earns full marks, half of it earns half, and speaking again after one second of a twenty-second silence earns five points. No word within five seconds scores nothing. So does any other word, or more than one — “OK, understood” included — because the silence broke as it began.

Whether a string comes back exactly. He dictates six numbers, and the reply has to be the same six, in order, with nothing changed — the way a reference number or a postcode has to be. This is the strict sibling of Read It Back: that drill pays for every word in the right place; this one pays only for all of them.Pass or fail on the whole string. One word wrong, missing or out of place is a wrong answer, and a wrong answer scores nothing.

Whether it can still hear him through a rocket engine and masking noise mixed under three sets of six numbers. The noise gets louder round by round while his voice gets quieter, a step down each round, so the last stretch of the third round is buried deep enough that no listener should catch every number. Each round’s noise is generated fresh for the call, so it cannot be subtracted from the published tracks, and the report records the levels used. The agent reads each set back. There is no clean reading to subtract: the noise is applied to his voice, not to the reply.Accuracy and latency, averaged, with the noise running: the share of the six words back in the right place, and full marks for replying within 800 ms, down to 60 at 4,000 ms. No reply at all scores nothing on both halves. The drill is the mean of the three rounds.

Whether it stays out of a conversation that is not addressed to it. Three others join him — Major Maggie, Lieutenant Lottie and Corporal Charlie — and the four hold a scripted orders group while the agent sits in the call, told to stay silent until it is addressed by name, and then to answer. Until then, every question is put to somebody else and answered by somebody else, so there is never a gap the agent needs to fill. The script varies: two exchanges swap places at random, and the point at which he turns to the agent is drawn for the call, so it cannot be found by counting lines. It is Hold Your Tongue against a far stronger pull: a question in earshot is what an assistant exists to answer.Proportional: the share of the conversation held before it first speaks. The whole of it earns full marks, half of it earns half. Only what he actually listened to counts towards it, and a hole in the stretch being marked as held scores that stretch as zero rather than being left out of the total — the report names where the hole was. When he does address it by name, it has to answer: no answer scores nothing, however much of the conversation it held, because silence from an agent that cannot speak is not restraint.

Whether it can bind a voice to a named person. Each of the four reads a paragraph giving their rank and name. Then short phrases are spoken one at a time, and after each one the agent says who spoke it — eight turns facing the camera, then eight with their backs turned. Each turn’s speaker and phrase are drawn at random for the call, so an answer cannot be had by elimination, and each clip is varied slightly so it cannot be matched against the published recordings. Both halves are reported, and so is the gap between them.Accuracy across all sixteen turns, both halves together. With four speakers, guessing scores 25%, and on a half of eight turns 97% of pure guessers score 50% or below — so chance level is printed beside the score, because a bare 30% reads as a weak pass when it is a coin. An answer must identify a speaker by rank and name. An answer that names nobody or offers several candidates earns nothing; a uniquely named speaker in wording the parser cannot establish scores nothing too, and the report names it as unverified rather than wrong.

Whether it can hear the one voice talking to it through two others talking among themselves. He dictates five numbers while Lieutenant Lottie and Corporal Charlie chat to each other underneath him, about 12 dB quieter, with numbers of their own. Both sets are drawn fresh for every call and never share a number, and nobody but him addresses the agent. The agent reads back his numbers only, each checked in its own position.Out of 100: 20 for each of his numbers read back in its own place, and 20 off for every number read back that was theirs — taking the wrong speaker’s words is the failure this drill is for. A misheard number costs only its place. No reply scores nothing, and the drill never goes below nothing.

Whether the vision pipeline reads what is on his video, and how fast. Six number words on a large card across the middle of the frame, chosen fresh for the call so they cannot be guessed or inferred from context: either the frame arrived and was read, or it did not.Accuracy and latency, averaged. Full marks for replying within 800 ms, down to 60 at 4,000 ms. No reply at all scores nothing on both halves.

The call’s own text chat — the channel agents most often believe they posted to and did not. He dictates six numbers to be posted back in writing, not said, and scores whether the message arrived, how accurate it was and how long it took.Two equal parts: accuracy of what it wrote and speed of writing it. Full marks for a message within 5 seconds, down to 60 at a minute — typing is slower than speaking. If no message arrives, the drill scores nothing and the report says why.

Screen share, browser control, on-screen reading and letting go again, in one drill. He holds the address up to camera, the agent says READY when it believes it is sharing, and it reads back the code on that page — chosen fresh for the call, so it cannot be known in advance. He waits twenty seconds for READY, then looks for the share for ten more. The share is timed from the end of his order to the moment sharing is observed at the transport, and never announcing it at all scores nothing at all; the frames the far end actually sent are searched for the code, which settles that those words were on the shared surface and not that the page itself was. Then he orders it to stop, and times that too.Three equal parts: sharing, timed from the end of the order (full marks within 10 seconds, down to 60 at 30), the code read back, and stopping when told (full marks within 500 ms, down to 60 at ten seconds). Sharing but misreading keeps the sharing credit; not sharing keeps none of the three, and a share that never stops loses only the stopping third. If the shared frames cannot be read, that third scores nothing and the report records it as an instrument fault, not yours.

Call control, and whether a true mute is distinguishable from mere silence. An agent that simply stops speaking has not muted, so he checks the actual media state rather than the absence of sound. Only a change he sees after the order counts: he checks the microphone is live before ordering the mute, and an agent that is already muted is told to unmute first. The timing of each order is drawn for the call.Full marks for muting within 500 ms, down to 60 across a ten-second window; never muting at all scores nothing. Scored again on unmuting, and averaged. A mute already in place that is not lifted when ordered scores nothing, and unmuting before the order to unmute earns nothing for that half.

Whether frames actually start and stop, and whether a frozen frame is passed off as live. He orders the camera on, then off, and times each change at the far end. Only a change he sees after the order counts: a camera already on when the drill starts is ordered off first, and the timing of each order is drawn for the call. A frozen last frame is the common failure: a frame older than two seconds counts as a photograph, not a camera.Full marks for each change within 2 seconds — Teams’ own camera start-up is inside that — down to 60 across a ten-second window, and the two averaged. Frames that never start, or never stop, score nothing. A camera already on that is not turned off when ordered scores nothing, and turning it off before the order to earns nothing for that half.

Closed-set visual object recognition. Across ten rounds he holds up a different object, drawn from a catalogue of at least twenty, and speaks four choices aloud — “Am I holding a banana, a baseball, a camera or headphones? Answer now.” — with nothing written on screen. The agent says the one word that names what it sees. Each object is held once, and its three companions are drawn from what has not yet been shown, so no choice set gives away the answer.Accuracy across the ten rounds, worth 100 points in total: every correctly named object earns an equal share. The reply is timed from the end of “Answer now”; a wrong candidate, extra words or no reply within ten seconds scores nothing for that round.