Repeatability and question batching
Compare three questions together and separately, preserving every answer, time, and usage value. Errors remain in the report.
What the model decides. What code checks.
- Three repetitions alternate the order of one combined request and three individual requests. At most 12 real calls, with no parallel queue.
- The report retains answers and call indices for each block, Choice agreement, and the actual elapsed time of every call.
- The first error stops subsequent requests. Planned, attempted, and unstarted calls are visible; unknown usage is never replaced with zero.
Read. Run. Replay.
Unpack the archive and open its directory in a terminal. The first step needs neither a key nor a network connection.
python3 runner.py repeatability-batchingFor a real run, set SHVYDKO_API_KEY in your computer or server environment. Do not put the key in your browser or source code. Current account quotas and pricing apply.
python3 runner.py repeatability-batching --live --output repeatability-batching-run.jsonThe report contains requests, actual responses, elapsed times, and known usage. The file is created with private permissions and never overwrites an existing file. Network failures are not automatically retried: check Usage before another call.
python3 runner.py repeatability-batching --replay repeatability-batching-run.jsonReplay checks that the recipe, inputs, and requests match. A local file is editable, so it is not signed proof of where responses came from. If you replace the inputs with your own data, the report will contain those data too: keep it private.
An open contract, with every step visible.
The plan below was generated by the same runner from the bundled fixtures. It contains requests and author expectations, not model results.
{
"mode": "offline_plan",
"model_calls": 0,
"recipe": "repeatability-batching",
"author_expectation": {
"team": "engineering",
"urgent": true
},
"notice": "Requests only, no model answers. Hierarchy includes conditional branches; repetition requests are templates.",
"requests": [
{
"state": "A fictional support team receives a report that card checkout fails for every customer. Payment on delivery still works. No money has been charged. The team must review the incident urgently.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should investigate the root cause?",
"criteria": {
"engineering": "Checkout integration outage",
"finance": "Charged-payment dispute",
"logistics": "Delivery problem"
}
},
"urgent": {
"type": "noul",
"instructions": "The text explicitly requires urgent team review."
},
"impact": {
"type": "score",
"instructions": "How broadly does the incident block ordering?",
"criteria": [
"Does not block ordering",
"Blocks one payment method",
"Blocks every ordering method"
]
}
}
},
{
"state": "A fictional support team receives a report that card checkout fails for every customer. Payment on delivery still works. No money has been charged. The team must review the incident urgently.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should investigate the root cause?",
"criteria": {
"engineering": "Checkout integration outage",
"finance": "Charged-payment dispute",
"logistics": "Delivery problem"
}
}
}
},
{
"state": "A fictional support team receives a report that card checkout fails for every customer. Payment on delivery still works. No money has been charged. The team must review the incident urgently.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "The text explicitly requires urgent team review."
}
}
},
{
"state": "A fictional support team receives a report that card checkout fails for every customer. Payment on delivery still works. No money has been charged. The team must review the incident urgently.",
"questions": {
"impact": {
"type": "score",
"instructions": "How broadly does the incident block ordering?",
"criteria": [
"Does not block ordering",
"Blocks one payment method",
"Blocks every ordering method"
]
}
}
}
]
}