We tested 27 decision models. One was right every time.

Updated on 4 November 2026 with a second test, on new words.

Plainlamp turns a hard PDF, such as a paper, a policy or a standard, into a short narrated video lesson and a guide. First, our pipeline records what kind of document it is: a research paper, an organisation's policy, a published standard, or rules from an authority, such as a law. The kind goes into the lesson's own record of the document, so it has to be right.

The person gives the kind. If it is none of the four, the kind is "other", and the person describes the document in their own words. A small AI model then reads those words and sorts them into one of the four kinds, or into "none". If the model says "none", or is not sure enough, our pipeline stops. It does not guess.

That model is a decision model: a model made to answer a multiple-choice question about a piece of text. It gives one answer and says how sure it is, from 0 to 1. We take its answer at 0.6 or more.

To choose one, we tested every decision model we could reach, on our own question.

The question

The model gets the person's words and five options, each with a short definition:

• paper: a research article or study, such as a journal paper
• policy: an organisation's rules for its own people, such as a company policy or procedure
• standard: a published standard or framework, such as an ISO standard or a best-practice framework
• rules: a law, regulation or set of rules from an authority, such as a statute or a regulator's rulebook
• none: none of these

The test

• 38 made-up descriptions, written by us. We set the right answer for each before any model read it. All 38 are listed at the end of this post.
  • 22 plain ones: five papers, five policies, five standards, four sets of rules, and three that fit no kind (a cookbook, a washing machine manual, a novel). Some are short or misspelt, such as "iso 9001", "gdpr" and "the reserch paper on burnout in nurses".
  • 16 harder ones: a chatty sentence ("So my manager sent me this 40-page PDF about how we have to handle customer complaints…"), a vague one, standards and rules that are easy to mix up (IFRS 16, WCAG 2.2, the HIPAA privacy rule, the EU AI Act), a PhD thesis, a drug trial, a school's anti-bullying policy, three more that fit no kind (board minutes, a product datasheet and a trap), and four in Dutch, German, Spanish and French.
  • The trap: "My car insurance policy document." It has the word "policy", but an insurance policy is a contract, not an organisation's rules for its own people. The right answer is "none".
• 27 models: every decision model OpenRouter listed on 1 November 2026, after leaving out aliases and one free copy. All were called through OpenRouter's Decisions route, which OpenRouter marks as alpha: an early version that may still change. No fallback to another provider, and only endpoints that do not keep our words for training.
• Five reads of each description: a read is one call to the model with the words. Each description was read once, then three more times, then once more with zero data retention required (only endpoints that keep no data may answer). So most models answered 190 times.
• What counts: for each read, we ask what our pipeline would do with the answer. It takes the answer, which means it records that kind, when the model chose a kind at 0.6 or more. On "none", or under 0.6, it stops.
  • right: it takes the right kind, or it stops when no kind fits;
  • wrong kind taken: it takes a wrong kind. This is the harm: a lesson made for the wrong kind of document;
  • stopped, though a kind fit: the right kind exists, but the model said "none" or was under 0.6. The pipeline stops when it did not need to, but no wrong lesson is made.
• The order: fewest wrong kinds taken first, then fewest failed reads, then the most right, then speed.

All runs were on 1 November 2026, with the model versions offered that day. Our calls through OpenRouter cost about US$0.06 in all.

The results

Three more models never answered. Kev 4B: every call came back "no route available" that day. Span-01 and Span-01 Lite: in our first two runs, every call came back with an error (HTTP 400), and its text said that Respan accepts only another form of question. Span-02 answered the same question. In the last run, with zero data retention required, OpenRouter had no endpoint for Luna or for Respan's three models and refused those calls, so Luna and Span-02 read 152 times. Span-02's four wrong kinds are IFRS 16 read as "rules". We set "standard" as the answer before the test, but you could argue for "rules" (see the limits). Drex v2.0 failed one read: the first try failed, and the second got "model busy" back through OpenRouter. Fewest failed reads comes before most right in our order, so Drex v2.0 is listed below Luna. The cost is what OpenRouter reported for each call or, where it reported none, the input tokens at OpenRouter's listed price, for these short descriptions. Time is from one computer in the Netherlands; with an even number of reads, the median is the upper of the two middle times.

One was right every time

Perplexity's Decider V1.2 was right in all 190 reads: every one of the 38 descriptions, in all five reads. It never took a wrong kind, and it never stopped when a kind fit. It gave the same answer in all five reads of every description, and its confidence moved only once, by less than 0.0004. It is also cheap: about US$4.30 per million calls.

One other model, Mammoth 1 from Baseten, also chose the right kind in all 190 reads. But on the PhD thesis it was only 0.44 sure, so our pipeline would have stopped five times.

Of the 24 models that answered, 14 never took a wrong kind. The other 10 each took a wrong kind on one description only: nine on the trap below, and Span-02 on IFRS 16, where you could argue for its answer.

Is that lead chance? For 902 of the 912 pairs of model and description, every read gave the same result, so the reads are not separate tests: what counts is the 38 descriptions. Decider was right in every read of all 38, and ten other models in every read of 37. One description is a gap that could easily be chance. Seven of those ten only stopped, each on the one description it missed, which makes no wrong lesson; three took a wrong kind on the trap. So the first test cannot clearly put Decider ahead of any other model, once we allow for the many comparisons. There is more under "How sure we can be", below. After it comes a second test, on new words.

The trap split the field

"My car insurance policy document." split the 24 models that answered exactly in half:

• 12 said "none" in every read.
• 3 said "policy", but always under 0.6, so our pipeline would have stopped.
• 9 said "policy", and our pipeline would have taken it in at least one read. Elephant 1 was 0.89 sure, every time.

Every wrong kind in the table is this one description, except Span-02's four. Span-02 read IFRS 16 as "rules". IFRS 16 is an accounting standard, but you could argue for "rules" (see the limits below).

Even the winner was not sure. Perplexity's Decider said "none" at only 0.35. On "none" our pipeline stops whatever the confidence, so that is the right result. But it shows how close the trap is.

Right on the line

Six of the 24 models that answered chose the same kind for the same words both over and under our 0.6 line: sure enough in some reads, but not in others.

• On the trap, Jev 1.14 from TypeSafe said "policy" at 0.54 to 0.62. So our pipeline would have taken this wrong kind in four of five reads. The three reads of our second run alone gave 0.62, 0.61 and 0.54.
• On "A conference paper on machine learning for fraud detection.", d1 from LiquidAI said "paper" at 0.53 to 0.89, sure enough in four of five reads. d1.1 said "paper" in four reads and was sure enough in two.
• Solar Decide Flash from Upstage did this on three descriptions. On "the HIPAA privacy rule" it said "rules" at 0.52 to 0.71, sure enough in two of five reads.
• On "PCI DSS version 4, the security requirements for card payments.", Span-02 from Respan said "standard" at 0.49 to 0.77, sure enough in two of its four reads.

An answer near the line can go either way. Read once, a model can look fine, or bad, by chance.

The sixth model, Clef from Cloudflare, said "policy" to the trap at 0.604 in our first run, and at 0.595 in all four later reads. Clef always chose the same kind. But its confidence changed on 20 of the 38 descriptions between our first and second run, and on only one within the second run. So Clef's confidence changed mostly between runs, and hardly from one read to the next.

The thesis most models would not take

"A PhD thesis on supply chain resilience" should be a paper: a thesis is a study. Only 7 of the 24 models that answered were sure enough to take it in every read. Of the 14 that never took a wrong kind, 11 stopped on it: they said "none", or chose "paper" under 0.6. That stops the pipeline when it did not need to, but makes no wrong lesson.

Steady answers

Most models give the same answer every time. 20 of the 24 that answered chose the same answer for every description in all their reads. Four changed their answer on one or two descriptions between reads.

Zero data retention

The words people type are about their own documents. So in Plainlamp, the call goes only to an endpoint that keeps no data. Of the 27 models, 22 answered with that required. For GPT-6 Luna Decisions and Respan's three models, OpenRouter had no such endpoint and refused the calls before any answer. Kev 4B never answered at all. Our test used made-up words, so the first two runs did not require it of most models, and we could see how good each model that answered is before we chose.

How sure we can be

Every model read the same 38 descriptions. So to compare two models, we count the descriptions on which one of them had more of its reads right. If the two were equally good, those would split about evenly between them. We call a gap too big for chance when chance alone gives a split at least as uneven less than 1 time in 20 (an exact sign test). A 95% confidence interval is a range worked out so that, for tests like this, it holds the true share 19 times in 20.

• Descriptions, not reads. For 902 of the 912 pairs of model and description, every read gave the same result. So we compare descriptions: on each, the share of a model's reads that were right. "Right on 37" below means right in every read of 37 descriptions.
• The top: Decider was right in every read of all 38 descriptions, and ten other models in every read of 37. Each of the ten is one description behind Decider, which could be chance (one description alone cannot tell two models apart).
• How sure one count is: with 95% confidence, right on 38 of 38 means right on at least 90% of descriptions like ours, and 37 of 38 on at least 86% (Wilson intervals).
• Gaps too big for chance: against Decider, only two of the other 23. GPT-6 Luna Decisions was behind on 9 descriptions and ahead on none (about 1 time in 260). Solar Decide Flash was behind on 8 descriptions and ahead on none (about 1 time in 130). Most of their missed reads were stops: all 36 for GPT-6 Luna Decisions and 28 of 33 for Solar Decide Flash. How a description is counted matters for Solar Decide Flash: if one right read of a description were enough, it would be behind on 5, which could be chance (about 1 time in 16). Drex v2.0, with the next biggest gap, is 5 descriptions behind (one of them only because a read failed), which could be chance (about 1 time in 16).
• Many comparisons: we compared Decider with 23 models, and among that many, one gap can pass the 1-in-20 line by chance alone. Allowing for that (Holm's method), even the gap to Solar Decide Flash and the gap to GPT-6 Luna Decisions could be chance.
• A stop and a wrong kind count the same here. The table puts wrong kinds first, because a wrong kind makes a wrong lesson and a stop does not. This count ranks misses, not harm.
• Ours, not yours: we wrote the 38 descriptions; nobody drew them from what real people type. These ranges hold for descriptions like ours.

A second test, on new words

The 38 descriptions could not tell the top models apart, and we had chosen our model on them. So on 4 November 2026 we tested again, on 339 new descriptions written for this test, and decided how we would judge the result before any decision model saw them.

• 339 new descriptions: 70 papers, 69 policies, 65 standards, 69 sets of rules and 66 that fit no kind. 294 are in English and 45 in Dutch, German, French, Spanish, Italian, Portuguese or Romanian. There are short labels, plain sentences and chatty ones. The writers marked 147 as hard, such as seven insurance contracts that call themselves a "policy" (like the trap in the first test), and standards that a law requires.
• Who wrote them: three AI writers (Claude, from Anthropic) wrote 360 from our brief: the question, the five definitions, and our rules for hard cases, such as "an insurance policy document is a contract, so none", and that a published standard is a standard "even where a law points to it". We dropped 21 near-copies. Two AI readers, also Claude, then each set the answer for all 339 alone, without seeing the writers' answers or each other's. We kept a description only where the writer and both readers agreed, and they agreed on every one. That suggests our rules give one answer for each description, at least to these readers; it does not show that every person would agree. The models saw only the question and the five definitions, not our rules.
• Which models: Decider and the ten models one description behind it in the first test. With fewer comparisons, a smaller gap is enough to count as too big for chance.
• Fixed first: before any decision model read the new words, we wrote down in our own code (our repository is private) the models, the rule for each read (the same as above) and the test: each of the ten against Decider, on the descriptions only one of the two got right, then Holm's method over the ten. Only these verdicts were fixed first; the other counts below describe the result.
• The reads: one read of each description, with zero data retention required for every call, and a second read of 50 of them. The versions that answered were the ones of the first test (Clef and Clef Flash give no date, so for them we cannot tell). All our calls in the second test, Jev 1.14's included, cost about US$0.07.

The order is the first table's: fewest wrong kinds taken first, then fewest failed reads, then the most right. Microsoft-Decision-1 comes first because it never took a wrong kind. Decider took one (the privacy notice below), and one description alone cannot tell two models apart.

• Too big for chance, even allowing for ten comparisons (Holm's method): d1-ultra, Tev1 4B Experimental, Drex v1.5, Clef, Clef Flash and Classify-1 are behind Decider. Of these six, Clef's gap is the smallest: 13 descriptions to 2. On its own, chance would give a split this uneven only about 1 time in 140. But it is close: two of the 13 were the right kind at 0.5962 and 0.5955, just under our 0.6 line. Had one of them been at 0.6 or more, the gap would be 12 to 2, which after Holm's method could be chance (a check we made after the result).
• Could be chance: Mammoth 1, Mercury Decide, Microsoft-Decision-1 and Drex Max. The test cannot tell these four apart from Decider. Mammoth 1 and Microsoft-Decision-1 were each behind on 7 descriptions and ahead on 3 (about 1 time in 3).
• No model got more right than Decider, at our 0.6 line. At other lines it would differ (see the limits).
• Misses, not harm: as in the first test, these counts rank misses. For d1-ultra, Tev1 4B Experimental, Clef and Classify-1, most of the descriptions only Decider got right were stops, which make no wrong lesson (14 of 15, 18 of 29, 11 of 13 and 13 of 19).
• Decider's three misses: it took "policy" for "The privacy notice on a shopping website that tells customers what the shop does with their data.", and it chose the right kind but under our 0.6 line on "la méthode EBIOS Risk Manager pour l'analyse des risques cyber" (0.30) and "BS 7671, the British wiring regulations for electrical installations." (0.59).
• The hardest descriptions: when they finished, the readers named the descriptions they had found hardest to decide, eleven in all (we listed them before the runs). They are 3% of the set, but they hold 32 of the 150 times that Decider and one of the ten disagreed (one right, the other not). Two of Decider's three misses are among them.
• Wrong kinds, the harm: Microsoft-Decision-1 never took a wrong kind. Decider, Drex Max, Mammoth 1 and d1-ultra took one each; the others took 2 to 20 (see the table). The seven insurance contracts hold 20 of the ten models' 53 wrong kinds. And seven wrong kinds are standards read as "rules": three IFRS descriptions that say a law requires them, and "BS 7671, the British wiring regulations". Our rules count them as standards, but as with IFRS 16 in the first test, you could argue for "rules".
• How sure one count is: with 95% confidence, Decider's 336 of 339 means right on at least 97% of descriptions like these (a Wilson interval).
• Steady: on the 50 descriptions read twice, each model's result (right, wrong kind or stopped) was the same both times. Four models, Jev 1.14 among them, once chose a different answer the second time, but in both reads our pipeline would have taken no kind.
• Jev 1.14, added afterwards: two descriptions behind in the first test, it was not one of the ten. After we saw this result, we were asked about it. So we ran it on the same descriptions, and the same 50 again, and tested it on its own, apart from the ten: 330 of 339, behind Decider on 8 descriptions and ahead on 2, which could be chance (about 1 time in 9). It took a wrong kind 5 times, Decider once; on wrong kinds alone that is 4 descriptions to 0, which could be chance too. Three of Jev's five were insurance contracts read as "policy", like the trap in the first test. Decider was right on all seven.
• Ours, not yours: AI writers wrote these descriptions to our brief; nobody drew them from what real people type. Our brief asked for hard cases like the first test's trap, and the hard ones decide the result: on the 192 descriptions not marked hard, allowing for the ten comparisons, the test cannot tell any of the ten apart from Decider (a check we made after the result). On the hardest ones, the right answer follows our rules, which the models never saw.

What we chose

Perplexity Decider V1.2, through OpenRouter. Every call requires zero data retention, and there is no fallback to another provider. We pinned the dated version that answered every read in the first test. Our pipeline uses it. In that test it never took a wrong kind and never stopped when a kind fit, and it has one of the lowest prices. That was enough for us to choose it.

On the first test, no model did better, but once we allow for the many comparisons, that test could not clearly put it ahead of any other model. On 339 new descriptions (above), it was ahead of six of the ten closest models by a gap too big for chance, and no model got more right.

What this test does not tell you

• The 38 descriptions are made up by us. Real people's words will be messier.
• We chose our model on the same 38 descriptions, and we wrote the 16 harder ones after an earlier, smaller test with the 22 plain ones. The second test, on new words, came after we chose.
• Some right answers are our reading of our own definitions. You could argue that a thesis is not a paper, or that IFRS 16 counts as rules, because many countries make it law.
• It is one question. A model that is best here may not be best on yours.
• The 0.6 line is ours. With another line, other models would be right every time too: Mammoth 1 at any line up to 0.43, Joey 1 from 0.38 to 0.54, Clef from 0.61 to 0.62, Classify-1 from 0.72 to 0.74, and Clef Flash from 0.78 to 0.82. Decider V1.2 is right every time at any line up to 0.83.
• Five reads are few. A model whose answers sit near a line may do better or worse in the next five.
• The second test read each description once, and 50 of them twice. A read near our line can go either way.
• The 0.6 line decides the second test too. At 0.7, Mercury Decide would get more right than Decider; at 0.8, Mammoth 1, Mercury Decide, Microsoft-Decision-1, Drex Max, Clef Flash and Jev 1.14 would.
• OpenRouter's Decisions route is marked alpha. Its form and prices may change.
• Times are from one computer in the Netherlands, with six models read at once.
• Models and prices change. These are the ones of 1 November 2026. The second test was on 4 November 2026.

If you use a decision model with a confidence line, read each input more than once, and look hardest at the answers near your line.

How to repeat this test

• The call: POST https://openrouter.ai/api/alpha/decisions, with the description as the state and one question, under the key kind, of type choice. The model is named without a date (such as perplexity/pplx-decider-v1.2); the answer names the dated version that answered. Provider settings: no fallback (allow_fallbacks: false) and data_collection: "deny" in every run; zero data retention (zdr: true) for every model in the last run, and for Jev 1.14 in every run.
• The question's instruction, word for word: "The state is the words of a person who is about to study a document, describing what kind of document it is. Which kind of document do they describe? Choose "none" when the words fit none of the four kinds."
• The five options, word for word: paper: "A research article or study, such as a journal paper" · policy: "An organisation's rules for its own people, such as a company policy or procedure" · standard: "A published standard or framework, such as an ISO standard or a best-practice framework" · rules: "A law, regulation or set of rules from an authority, such as a statute or a regulator's rulebook" · none: "None of these".
• The line: an answer is taken at confidence 0.6 or more, on the confidence the route gives with each answer. "none" is never taken.
• The reads: each description once, then three times, then once with zero data retention required: five in all. A call that was busy, timed out (our limit: 60 seconds), hit a server error (HTTP 500, 502, 503, 524 or 529) or lost its connection was tried once more, 2 seconds later.
• The median time: with an even number of reads, the upper of the two middle times.
• The versions that answered: perplexity/pplx-decider-v1.2-20261016, liquid/d1-ultra-20261015, togethercomputer/tev1-4b-experimental-20260923, baseten/mammoth-1-20261021, nace-ai/drex-v1.5-20261009, inception/mercury-decide-20260930, microsoft/microsoft-decision-1-20261009, nace-ai/drex-max-20261016, liquid/d1-20260930, liquid/d1.1-20261027, baseten/joey-1-20261029, cloudflare/clef-omni-20261009, openai/gpt-6-luna-decisions-20261006, nace-ai/drex-v2.0-20261009, cloudflare/clef, typesafe/jev-1.14-20261014, upstage/solar-decide-20260928, respan/span-02-20261015, cloudflare/clef-flash, scaledown/classify-1-20260928, perplexity/pplx-decider-flash-v1.2-20261016, typesafe/jev-1.13-20260917, baseten/elephant-1-20261021, upstage/solar-decide-flash-20261008. Clef and Clef Flash answered without a date.
• The second test: the same call, question, options and line, with zero data retention (zdr: true) in every call and ignore: ["primeintellect"] in the provider settings (one of the hosts OpenRouter lists for Clef and Clef Flash, left out). One read of each of the 339 descriptions, and a second read of 50: the 50 whose id (such as "R001") has the lowest SHA-256. The first test's retry rule applied, but no call needed it: Classify-1's one failed read came back as HTTP 504, an error our code does not try again. The hosts that answered: Microsoft-Decision-1 on Azure, Decider on Perplexity, Drex Max on DeepInfra, Mammoth 1 on BaseTen, d1-ultra on Liquid, Mercury Decide on Inception, Clef on DeepInfra, Classify-1 on ScaleDown, Drex v1.5 on DeepInfra, Tev1 4B Experimental on Together, Clef Flash on Makora, Jev 1.14 on TypeSafe. The versions that answered: microsoft/microsoft-decision-1-20261009, perplexity/pplx-decider-v1.2-20261016, nace-ai/drex-max-20261016, baseten/mammoth-1-20261021, liquid/d1-ultra-20261015, inception/mercury-decide-20260930, cloudflare/clef, scaledown/classify-1-20260928, nace-ai/drex-v1.5-20261009, togethercomputer/tev1-4b-experimental-20260923, cloudflare/clef-flash, typesafe/jev-1.14-20261014. We have not published the 339 descriptions.
• The 38 descriptions, with the answer we set:
  • paper: "A research article from a psychology journal about how teams make decisions." · "My professor's study on change management in hospitals, published in 2019." · "A meta-analysis of studies on sleep and memory." · "the reserch paper on burnout in nurses" · "A conference paper on machine learning for fraud detection." · "a PhD thesis on supply chain resilience" · "un artículo científico sobre el cambio climático publicado en una revista" · "A randomised controlled trial of a new diabetes drug, from The Lancet."
  • policy: "Our company's rules for working from home." · "The bank's internal procedure that all employees follow to report suspicious transactions." · "The hospital's hand hygiene protocol that nurses must follow." · "my company's code of conduct" · "The employee handbook section on expenses and travel." · "So my manager sent me this 40-page PDF about how we have to handle customer complaints, and I need to understand it before Monday." · "The school's anti-bullying policy for pupils and staff." · "het privacybeleid dat mijn werkgever voor alle medewerkers heeft opgesteld"
  • standard: "ISO 27001, the information security management standard." · "The NIST Cybersecurity Framework." · "PCI DSS version 4, the security requirements for card payments." · "COBIT, the IT governance framework from ISACA." · "iso 9001" · "IFRS 16, the accounting standard for leases" · "The WCAG 2.2 web accessibility guidelines." · "la norme ISO 45001 sur la santé et la sécurité au travail"
  • rules: "The EU's General Data Protection Regulation." · "The Dutch tax authority's rules for VAT on services." · "A new law on rental contracts that our parliament passed." · "gdpr" · "the thing from the financial regulator about how banks must report IT outages" · "the HIPAA privacy rule" · "das deutsche Bürgerliche Gesetzbuch" · "The EU AI Act."
  • none: "A cookbook with Italian pasta recipes." · "The user manual for my washing machine." · "A fantasy novel I have to read for my book club." · "My car insurance policy document." · "Minutes of our last board meeting." · "A product datasheet for a network switch."