Reasoning

Test-Time Compute

Test-time compute is extra inference work spent on a single problem: longer chains, multiple samples, search, or verifier passes. Accuracy often rises with that budget up to a point, then hits diminishing returns.

Card 194 of LLMs Visual Card

The card plots accuracy against inference compute for one hard task. The curve rises steeply at first, then flattens as more spending yields smaller gains. Annotations mark techniques that buy compute: chain-of-thought length, self-consistency sample count, beam-style search over reasoning steps, and verifier models that score candidate answers. The horizontal axis is not training FLOPs; it is per-query effort at deployment.

Training scales parameters and data; test-time compute scales deliberation per input. A smaller model with heavy sampling and verification can beat a larger model on some reasoning benchmarks when latency budgets allow. The trade is economic and operational: tokens cost money, parallel samples need orchestration, and users wait longer. The curve on the card is the planning tool: identify where your application sits on the slope.

Common patterns include generating several CoT traces and voting, iteratively refining an answer with critique prompts, tree search over partial solutions, and calling a stronger model only as a judge. Each adds a knob tied to quality. None removes the need for task-specific evaluation; some problems saturate after one extra pass, others keep gaining through dozens of samples.

Design with explicit budgets. Cap maximum tokens, wall clock, and sample count per user tier. Route easy queries to a fast single pass and hard queries to expanded compute only when a cheap classifier flags uncertainty. The card lists mechanisms; the engineering takeaway is allocation: treat inference like a resource you can spend where marginal accuracy still pays for itself.

Keep exploring

Each card is part of a larger map of LLM concepts. Move to the next card, follow a related concept, or return to the full curriculum view.

About the visual cards Browse the map