Guides / StrategyThread

Design an AI workflow with typed outputs and explicit failure states

A practical pattern for AI workflows that use typed outputs, explicit failure states, and tests to reject malformed or weakly supported results.

November 5, 2026 · 12 min read · StrategyThread

Consider a hypothetical research intake pipeline that fails without throwing an error. A teammate pastes a dense note, the model returns something that looks like JSON, your parser accepts it, and a later step treats the output as if it were verified. The problem may not be missing braces. It may be a neat-looking record with one invented category, one unsupported summary, or one extra field that your downstream code quietly ignores.

This guide proposes designing two things separately:

• a typed model output with a narrow schema
• an outer workflow that assigns explicit failure states when no trustworthy payload is available

Use those two records to distinguish format validation from source review. Structured output constraints help with shape, but they do not automatically prove that the content is well-supported. OpenAI’s Structured Outputs guide says Structured Outputs constrains responses to your supplied JSON Schema and describes a separate refusal indicator that application code can inspect OpenAI Structured Outputs guide. Those are useful guarantees for format and refusal handling, not for truth or business suitability.

This article gives a practical design you can adapt for extraction, research intake, triage, and review queues. The worked example uses hypothetical rules and synthetic data, labeled as such.

Separate payload outcomes from workflow failures

The first design move is conceptual: stop putting every possible condition into one status field.

A suggested heuristic is to define two layers of state.

Layer 1: payload outcomes

These are values the model may return inside the schema after a successful structured generation. Keep them tightly tied to the task.

For a research intake record, one compact set is:

• accepted
• needs_review
• rejected_input

Those values mean the model produced a valid payload for the task, even if the payload says the source note was not usable.

Layer 2: outer workflow failures

These are states your application assigns outside the schema when the run did not produce a usable typed payload.

Suggested workflow states:

• parsed
• refusal
• incomplete_response
• schema_error
• request_error

These names are editorial recommendations, not source-defined platform terms. In this proposed design, use the labels as follows:

• refusal means the model declined for safety reasons
• incomplete_response means generation stopped before a complete usable payload was obtained
• schema_error means the result did not validate against your contract
• request_error means the API call failed before you could evaluate content

The Structured Outputs guide includes examples of handling refusal via message.refusal, and it shows an example that checks finish_reason for a length-based incomplete result path OpenAI Structured Outputs guide. That supports treating refusal and incompletion as distinct conditions, even if your own workflow labels are application-defined.

For this proposed design, do not let the model return schema_error or request_error inside the payload. Those are orchestration concerns, not task outcomes.

Design the schema from downstream needs

A typed workflow starts with the consumer of the data, not with the prompt.

Ask these questions first:

• What exact fields does the next software step require?
• Which fields need restricted values, such as enums?
• What evidence should be stored separately from the conclusion?
• Which ambiguous cases should be stopped rather than passed through?

OpenAI’s Structured Outputs documentation shows strict JSON Schema usage, including required fields and additionalProperties: false, and it notes SDK support for typed schema definitions with Python pydantic.BaseModel and JavaScript z.object OpenAI Structured Outputs guide. That is the technical basis for treating the schema as a contract rather than as a formatting wish.

Here is a hypothetical schema for a research intake record:

{
  "type": "object",
  "properties": {
    "status": {
      "type": "string",
      "enum": ["accepted", "needs_review", "rejected_input"]
    },
    "topic": { "type": "string" },
    "claim_summary": { "type": "string" },
    "evidence_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "source_text": { "type": "string" },
          "supports_claim": { "type": "boolean" }
        },
        "required": ["source_text", "supports_claim"],
        "additionalProperties": false
      }
    },
    "reason": { "type": "string" }
  },
  "required": ["status", "topic", "claim_summary", "evidence_items", "reason"],
  "additionalProperties": false
}

Why this shape is useful:

• status keeps a small, enforceable set of business outcomes.
• claim_summary separates the conclusion from raw evidence.
• evidence_items creates an audit trail for later review.
• additionalProperties: false reduces silent drift in downstream handling.
• reason gives a human-readable explanation for logs and review queues.

A suggested heuristic is to avoid a free-form confidence field unless you already know how it will be used. In many workflows, explicit evidence checks are easier to review than an ungrounded score.

Build failure handling before the happy path

Define what the application does with each failure before adding the successful-response path.

Start with a decision procedure.

Then apply payload-level rules only after parsing succeeds.

Suggested payload interpretation table:

Use the two stages to distinguish infrastructure and format failures from a task-level rejection. If you blur refusal, truncation, schema failure, and business rejection together, your logs stop being diagnostic. If you separate them, you can answer basic operational questions: did the model decline, did the payload fail validation, or did the content simply not meet your own criteria?

A fully worked example: research intake with support checks

This section is a hypothetical example with synthetic data.

Step 1: define the acceptance rules

Suggested heuristics for this example:

• Count only distinct source snippets whose support has been independently checked, not model-generated supports_claim votes.
• With 0 verified supporting snippets, use rejected_input for this example's source-support rule.
• With 1, hold an otherwise eligible payload in needs_review.
• With 2 or more, an accepted payload is eligible to remain accepted; preserve an existing needs_review or rejected_input hold for another reason.
• Never promote an existing hold based on the count alone.

These thresholds are recommendations for illustration, not source-proven universal outcomes.

Step 2: synthetic source note

The team wants a workflow for screening incoming AI research notes.
The extracted result must be easy for software to parse.
Unsupported claims should be stopped before they reach later stages.

Step 3: synthetic structured payload

{
  "status": "accepted",
  "topic": "AI research intake workflow",
  "claim_summary": "The note describes a parsable research intake workflow that should block unsupported claims from moving forward.",
  "evidence_items": [
    {
      "source_text": "The extracted result must be easy for software to parse.",
      "supports_claim": true
    },
    {
      "source_text": "Unsupported claims should be stopped before they reach later stages.",
      "supports_claim": true
    }
  ],
  "reason": "Two source statements directly support the summary."
}

Step 4: verify the result

Check the result in two passes.

Pass A: schema checks

• Are all required top-level keys present?
• Is status one of the allowed enum values?
• Does each evidence item include only source_text and supports_claim?
• Are there any unexpected top-level keys?

Pass B: business checks

• Do the cited source_text snippets actually appear in the input?
• Does claim_summary stay within what the evidence supports?
• Does the evidence count match the chosen status rule?

For this synthetic example, independently read the two snippets against the summary. One states the parsing requirement and the other states the unsupported-claim requirement. They establish what the note requests, not that the proposed system is effective. Both snippets occur in the note, so the verified support count is 2 and accepted is consistent with the proposed rule. A model-generated supports_claim: true is not itself that independent check.

Step 5: apply outer workflow logic

Handle the actual SDK's request error, refusal, completion and schema-validation paths before calling the following application code. The documented message.refusal and finish_reason examples use a Chat Completions response; adapt those checks to the endpoint you actually call rather than inventing a shared response envelope.

This runnable helper implements only the proposed post-parse source-support rule:

def normalize_status(support_count, input_status):
    allowed = {"accepted", "needs_review", "rejected_input"}
    if type(support_count) is not int or support_count < 0:
        raise ValueError("invalid support count")
    if input_status not in allowed:
        raise ValueError("invalid payload status")
    if support_count == 0 or input_status == "rejected_input":
        return "rejected_input"
    if support_count == 1:
        return "needs_review"
    return input_status


def apply_reviewed_support(obj, note, reviewed_indices):
    # Call only after schema validation and an independent source review.
    # reviewed_indices comes from that review, never from the model's votes.
    verified = {
        item["source_text"]
        for index, item in enumerate(obj["evidence_items"])
        if index in reviewed_indices
        and item["source_text"].strip()
        and item["source_text"] in note
    }
    count = len(verified)
    return {
        **obj,
        "status": normalize_status(count, obj["status"]),
        "reason": f"Application check: {count} distinct reviewed snippets.",
    }

For the displayed payload, the completed source review identifies indices {0, 1}. Calling apply_reviewed_support(payload, note, {0, 1}) retains accepted and records two distinct reviewed snippets. With no verified snippets, even a model-labeled accepted payload becomes rejected_input. Duplicate text does not increase the count. The helper checks text presence; the independent review must still establish semantic support.

The design point is the order:

• detect refusal or incompletion
• validate structure
• apply your business rules
• only then pass the payload downstream

Test both your code and your model behavior

OpenAI’s evals guide describes a three-part loop: describe the task, run test inputs, then analyze results and iterate OpenAI evals guide. The guide recommends checking performance against your criteria when a model or other application setting changes OpenAI evals guide.

That suggests two distinct testing layers.

Layer 1: deterministic tests for your own logic

Your status normalization, schema handling, and rule checks should be tested without live model calls.

Python’s unittest documentation describes fixtures, test cases, suites, runners, and discovery as core parts of the standard library testing framework Python unittest guide. If you want concise test matrices, pytest.mark.parametrize runs the same test function with multiple argument sets pytest parametrization guide.

Example:

import unittest


class SupportRulesTests(unittest.TestCase):
    def test_status_matrix(self):
        cases = [
            (0, "accepted", "rejected_input"),
            (1, "accepted", "needs_review"),
            (2, "accepted", "accepted"),
            (0, "needs_review", "rejected_input"),
            (1, "needs_review", "needs_review"),
            (2, "needs_review", "needs_review"),
            (0, "rejected_input", "rejected_input"),
            (1, "rejected_input", "rejected_input"),
            (2, "rejected_input", "rejected_input"),
        ]
        for count, status, expected in cases:
            with self.subTest(count=count, status=status):
                self.assertEqual(normalize_status(count, status), expected)

    def test_duplicate_snippets_count_once(self):
        obj = {"status": "accepted", "topic": "Synthetic test",
               "claim_summary": "A sample claim", "reason": "Synthetic input",
               "evidence_items": [
            {"source_text": "Use typed records.", "supports_claim": True},
            {"source_text": "Use typed records.", "supports_claim": True},
        ]}
        result = apply_reviewed_support(obj, "Use typed records.", {0, 1})
        self.assertEqual(result["status"], "needs_review")

    def test_unreviewed_or_missing_snippets_do_not_pass(self):
        obj = {"status": "accepted", "topic": "Synthetic test",
               "claim_summary": "A sample claim", "reason": "Synthetic input",
               "evidence_items": [
            {"source_text": "Not present in the source.", "supports_claim": True},
        ]}
        result = apply_reviewed_support(obj, "Use typed records.", {0})
        self.assertEqual(result["status"], "rejected_input")
        unreviewed = apply_reviewed_support(obj, "Not present in the source.", set())
        self.assertEqual(unreviewed["status"], "rejected_input")

    def test_invalid_count(self):
        for value in [-1, True, "2"]:
            with self.assertRaises(ValueError):
                normalize_status(value, "accepted")


if __name__ == "__main__":
    unittest.main()

A suggested heuristic is to keep these tests fully deterministic. If a test is verifying your own rules, do not let model variation into the test.

Layer 2: eval-style cases for model behavior

Build a small corpus of representative task inputs. Record actual model responses and independently reviewed expected outcomes. The table below lists conditions the complete workflow must handle; it does not promise that every safety-related topic produces a refusal or that every short token budget produces truncation.

This does not prove universal reliability. It gives you a repeatable way to notice regressions when you change prompts, schemas, or model versions.

Reusable checklist, assumptions, and common mistakes

A StrategyThread-style editorial workflow is a good fit for this pattern because the site presents itself as AI automation and SaaS strategy research with articles arranged into a connected knowledge structure across multiple content buckets StrategyThread. For this proposed intake method, retain the evidence separately so a reviewer can compare it with the conclusion.

Use this suggested pre-ship checklist:

Assumptions behind this approach:

• you can define at least some acceptance criteria in advance
• typed records are more useful to downstream software than free text
• failing closed is preferable to silently passing uncertain output

Limits:

• A schema controls shape better than truth.
• Evals only measure the cases you chose.
• A valid payload can still be weakly supported.

Common mistakes:

• treating valid JSON as verified reasoning
• allowing the model to invent categories instead of using enums
• skipping evidence fields and making later review impossible
• mixing orchestration failures with task outcomes
• changing model or prompt behavior without rerunning evals

Before connecting the workflow to a later action, decide which states can proceed and test the paths that must stop. Keep the parsed payload, independent source check and resulting decision together.