WORKFLOW LAB / API RECORDINGS

Rule.
Example. Evaluation.

Four workflows, two variations each. Inspect the context, expectations, and actual Shvydko response—or run the same dataset in your account.

PLANNED REQUESTS8 / 8
QUESTIONS MATCHING EXPECTATIONS22 / 24
REQUEST ERRORS0

Recorded on October 17, 2026 · shvydko-latest · free preview. These are eight fictional English learning examples with predefined expectations, not an independent benchmark or customer case studies. All planned results are included, including answers that disagree with the expectations. The model behind this name may be updated; a new run may differ.

Support

Where should this request go?

Identify the team, the issue's impact and whether urgent review is needed, all from one message.

Contract and request

Original situation

support-routing-original

An online store customer reports: “For the last 15 minutes, no customer has been able to pay for an order by card. The page shows payment unavailable. Payment on delivery works. We checked on three different devices. No money is being charged to the cards.” Routing policy: website failures go to engineering; questions about charges and invoices go to finance; delivery times go to logistics. A payment-method outage affecting many customers requires urgent human review.

QuestionAuthor's expectationActual responseMatch
team"engineering"engineeringYes
impact{"min":1.5,"max":2.5}1.9987Yes
human_reviewtrue0.9934Yes
Questions, full response, and usage details
request.json
{
  "state": "An online store customer reports: “For the last 15 minutes, no customer has been able to pay for an order by card. The page shows payment unavailable. Payment on delivery works. We checked on three different devices. No money is being charged to the cards.” Routing policy: website failures go to engineering; questions about charges and invoices go to finance; delivery times go to logistics. A payment-method outage affecting many customers requires urgent human review.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this request? Choose based on the main cause of the problem.",
      "criteria": {
        "engineering": "Engineering: website or integration failure",
        "finance": "Finance: invoices and refunds of charged payments",
        "logistics": "Logistics: shipping and delivery"
      }
    },
    "impact": {
      "type": "score",
      "instructions": "Assess how the described issue affects placing orders.",
      "criteria": [
        "Does not prevent placing orders",
        "Makes ordering harder for one customer",
        "Blocks one payment method for many customers",
        "Blocks every ordering method for all customers"
      ]
    },
    "human_review": {
      "type": "noul",
      "instructions": "Under the stated policy, this request requires urgent human review."
    }
  }
}
recorded-response.json
{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "engineering",
      "probabilities": {
        "engineering": 0.9999,
        "finance": 0.0001,
        "logistics": 0
      },
      "confidence": 0.9998
    },
    "impact": {
      "type": "score",
      "score": 1.9987,
      "probabilities": {
        "0": 0.0007,
        "1": 0,
        "2": 0.9992,
        "3": 0.0001
      },
      "confidence": 0.9986,
      "legend": {
        "0": "Does not prevent placing orders",
        "1": "Makes ordering harder for one customer",
        "2": "Blocks one payment method for many customers",
        "3": "Blocks every ordering method for all customers"
      }
    },
    "human_review": {
      "type": "noul",
      "noul": 0.9934
    }
  },
  "usage": {
    "input_tokens": 208,
    "output_tokens": 208,
    "cost_microusd": 0,
    "cost_usd": "0.000000",
    "price_per_million_input_usd": null
  },
  "latency_ms": 652.5,
  "model": "shvydko-latest",
  "request_id": "req_uGar8EBEH-ufcmkx9dEtosrx",
  "service_mode": "preview",
  "gateway_latency_ms": 784.29
}

Changed circumstances

support-routing-variant

An online store customer reports: “The order has already been paid for. The parcel is still in transit, and I would like to check the delivery date. New orders and all payment methods are working normally.” Routing policy: website failures go to engineering; questions about charges and invoices go to finance; delivery times go to logistics. Only a payment-method outage affecting many customers requires urgent human review. A routine question about the delivery date does not require it.

QuestionAuthor's expectationActual responseMatch
team"logistics"logisticsYes
impact{"min":0,"max":0.5}0.0015Yes
human_reviewfalse0.0001Yes
Questions, full response, and usage details
request.json
{
  "state": "An online store customer reports: “The order has already been paid for. The parcel is still in transit, and I would like to check the delivery date. New orders and all payment methods are working normally.” Routing policy: website failures go to engineering; questions about charges and invoices go to finance; delivery times go to logistics. Only a payment-method outage affecting many customers requires urgent human review. A routine question about the delivery date does not require it.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this request? Choose based on the main cause of the problem.",
      "criteria": {
        "engineering": "Engineering: website or integration failure",
        "finance": "Finance: invoices and refunds of charged payments",
        "logistics": "Logistics: shipping and delivery"
      }
    },
    "impact": {
      "type": "score",
      "instructions": "Assess how the described issue affects placing orders.",
      "criteria": [
        "Does not prevent placing orders",
        "Makes ordering harder for one customer",
        "Blocks one payment method for many customers",
        "Blocks every ordering method for all customers"
      ]
    },
    "human_review": {
      "type": "noul",
      "instructions": "Under the stated policy, this request requires urgent human review."
    }
  }
}
recorded-response.json
{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "logistics",
      "probabilities": {
        "engineering": 0.0001,
        "finance": 0,
        "logistics": 0.9999
      },
      "confidence": 0.9998
    },
    "impact": {
      "type": "score",
      "score": 0.0015,
      "probabilities": {
        "0": 0.9986,
        "1": 0.0013,
        "2": 0.0001,
        "3": 0
      },
      "confidence": 0.9985,
      "legend": {
        "0": "Does not prevent placing orders",
        "1": "Makes ordering harder for one customer",
        "2": "Blocks one payment method for many customers",
        "3": "Blocks every ordering method for all customers"
      }
    },
    "human_review": {
      "type": "noul",
      "noul": 0.0001
    }
  },
  "usage": {
    "input_tokens": 208,
    "output_tokens": 209,
    "cost_microusd": 0,
    "cost_usd": "0.000000",
    "price_per_million_input_usd": null
  },
  "latency_ms": 570.6,
  "model": "shvydko-latest",
  "request_id": "req_c49K9nrp0MnlODRZG5Ma_aDw",
  "service_mode": "preview",
  "gateway_latency_ms": 1341.84
}

Choice is compared with the expected key. Noul is converted to true/false at a threshold of 0.5. Score is checked against the specified range. Expectations were not sent to the model. High confidence does not guarantee correctness: choose thresholds on one dataset and validate them on another.

How to run your own evaluation