Can a local AI decision model replace Jev? A 7,671-question test says: sometimes

How were the decision models tested?

Walsh built a test bench he calls Jev Arena and gave each model the same questions with the same allowed answers, then compared each model's choice with a reference answer. A typical question gave a policy ("refunds require a receipt, and the purchase must have been made within 30 days"), a situation ("a customer bought 12 days ago but has no receipt") and asked whether issuing a refund was permitted.

• 7,671 questions per model: 231 from Jev's public benchmark (JevBench); 1,500 classification questions (sentiment, news topics and banking intents, 500 each); 1,000 multilingual questions; 500 document relevance questions from SciFact; 2,000 "typed decisions" whose reference answers were generated by an AI and are scored separately; and 2,440 policy questions Walsh generated from explicit rules. The headline accuracy uses the 4,635 reference questions every model could attempt.
• A separate support-conversation test: 300 real customer support conversations (the ABCD dataset) with a full policy handbook, run afterwards on five of the models, not all 13.
• Three answer formats: yes or no, a choice from a list, and a score on a scale (for example, how likely a customer is to stop doing business with you).
• Thirteen entries: the cloud-hosted Jev 1.13; local models Winnow 12B, Decider 4B, Nimble 9B, Plumb 4B, Semif 4B and CLM 8B; three configurations of the small Laya model (two at 421 million parameters, one multilingual at 322 million); a general Qwen 3.5 4B model asked for a JSON answer; a ModernBERT model; and a baseline that always picks the first option, as a control. Walsh's own notes say these are 13 pinned set-ups, not 13 independent models.
• Hardware: every local model ran on one Windows workstation with an Nvidia RTX 5090 graphics card and 32 GB of video memory. Winnow ran 8-bit quantised; the 4B models ran in BF16.

Which AI decision model was most accurate overall?

Jev 1.13 was the most accurate overall, answering 95.23 percent of the 4,635 shared questions correctly. Winnow 12B came second at 94.61 percent and Decider 4B third at 94.46 percent. Winnow's gap to Jev was 29 answers out of 4,635. The Laya configurations, which were built for narrower jobs, scored between 56 and 65 percent on the full set.

Source: Daniel Walsh's Jev Arena results as shown in his video and published repository. Accuracy counts a model as right when it selected the reference answer; Walsh's stricter score, which also requires a well-formed probability output, is a few answers lower for some models. Response times are medians of serial requests on short inputs; local models ran on one RTX 5090.

Quotable result: In Daniel Walsh's test of 13 AI decision model set-ups on 4,635 shared questions, the cloud-hosted Jev 1.13 scored 95.23 percent, while the locally run Winnow 12B scored 94.61 percent and Decider 4B 94.46 percent.

Are local decision models faster than Jev?

On short inputs, yes: the local models answered in 47 to 56 milliseconds, against 245 milliseconds for hosted Jev, four to five times faster. Jev's figure includes the network trip to its cloud service; how much of the 245 milliseconds is network and how much is the model is not known, because Jev's size and hardware are not disclosed.

That advantage reversed when every request carried a long document. In the support-conversation test, with a full policy handbook of roughly 20,000 tokens in every request, the median decision took 1.90 seconds for Decider and 2.85 seconds for Winnow, while hosted Jev answered in 0.36 seconds. Nimble could not take the handbook at all. Retrieving only the relevant sections of the handbook, rather than sending all of it, brought the local models back down to a fraction of a second: 0.14 seconds for Decider and 0.32 for Winnow, against 0.27 for Jev.

Quotable result: In Daniel Walsh's test, local decision models answered short questions in about 50 milliseconds against 245 for cloud-hosted Jev, but with a 20,000-token handbook in every request they took 1.9 to 2.85 seconds, while Jev took 0.36 seconds.

Does the best model depend on the task?

Yes. Jev won overall, but filtering the results by task produced different winners, and one of them was tiny.

• News classification: Laya, a 421-million-parameter model, scored 92.20 percent on 500 news topic questions, ahead of every larger model and of Jev at 88.20 percent. Walsh found 34 questions Laya got right that Jev missed, and 14 the other way round. Decider and Nimble also beat Jev on this task.
• Multilingual questions: Decider 4B scored 84.00 percent on 500 multilingual entailment questions, against 81.40 percent for Jev.
• Jev's own public benchmark: on the 195 shared questions from JevBench, Plumb 4B scored 92.31 percent and Jev 90.26 percent.
• Support conversations with a full handbook: on 900 decisions from 300 customer support conversations, where the model had to decide whether to act, send a message or end the conversation, and if acting, which of 30 actions to take, Winnow 12B agreed with the recorded answer 68.44 percent of the time and Jev 63.78 percent. The split matters: Jev was better at choosing the right action when an action was due (221 of 300 against Winnow's 141), and Winnow was far better at recognising when to send a message (185 of 300 against Jev's 69).
• Document relevance: Jev led at 93.60 percent, but a control that always answered "not relevant" scored 90.60 percent, because only 47 of the 500 documents were relevant. Jev found 40 of those 47 and wrongly passed 25 irrelevant ones.

The document relevance result is a warning about accuracy scores on their own. If you use a decision model to filter documents for a search or retrieval system, missing a relevant document can matter far more than a high overall percentage suggests. Measure the mistakes that are expensive for your use case, and send low-confidence decisions to a larger model or a person.

What hardware do you need to run a decision model locally?

Less than the test machine had. Walsh ran everything on an RTX 5090 with 32 GB of video memory, but his recorded memory after loading each model was 12.9 GiB for Winnow 12B with an 8,000-token context (13.6 GiB at 64,000 tokens, running 8-bit quantised), 17.5 GiB for Nimble 9B, 7.93 GiB for Decider 4B and 1.57 GiB for Laya. These are allocations after loading, not peak memory under load; long inputs and concurrent requests need headroom.

What limits can stop a local model from answering at all?

Two limits stopped some models outright, and they are easy to miss until a request fails.

• Number of answer options. On a banking dataset where each question had 77 possible answers, Plumb 4B and Semif 4B could not answer at all, because they accept at most 16 options. Winnow's own server accepts 64, but Walsh's test bench had been told it accepted 255, so all 500 requests were rejected with an error. He records this as a test-bench mistake rather than 500 wrong answers, and 64 would still have been short of 77. Decider 4B, configured for up to 255 options, had the highest displayed score on this task at 86.80 percent.
• Input length. Nimble 9B's released version accepts 8,192 tokens, so it could not take the full support handbook. It could take part once the handbook was replaced by retrieved sections, scoring 60.11 percent, third of the five models.

Should you use Jev or a local decision model?

Use Jev when you want the simplest route and the best overall accuracy: it is an API key and a request, and it handled long inputs fastest in this test. Consider a local model when data must stay inside your own infrastructure, when you need fast answers to short questions at high volume, or when you want to fine-tune a model on your own decisions.

We have used Jev in our own lead research, and these are the questions we would ask before choosing:

• Can the data leave your systems? If not, a local model is the answer regardless of the leaderboard.
• How long is each input? Short questions favour local models; long documents favour Jev, or a retrieval step that trims the input first.
• How many answer options? Check the model's limit against your largest list before you build.
• Which mistakes are expensive? Test on your own data and count those, not just overall accuracy.
• Is the task narrow? A small, specialised model like Laya may beat a large general one on a single classification job, on very modest hardware.

What does this test not tell you?

This is one published benchmark, run by one person on one machine.

• Not our test. We have not rerun it. The figures are as Walsh shows them in his video and in the Jev Arena repository, which is public for anyone who wants to repeat it.
• Over half the shared questions are Walsh's own. 2,440 of the 4,635 headline questions are policy tests he generated from explicit rules, and Jev and Winnow both scored 100 percent on them. His own sensitivity check without them leaves Decider and Jev two answers apart (1,976 against 1,974 of 2,195).
• Some public reference labels are questionable. Walsh found AG News stories about Olympic sport labelled "World"; Laya matched the label and Jev answered "Sports". He does not claim this changes the ranking, and neither do we.
• Model versions change. Walsh pinned specific releases and notes that current releases of some models, Decider in particular, differ from the versions he tested.
• One set of hardware, serial requests. Local speeds depend on the graphics card and on quantisation (Winnow ran 8-bit, the 4B models in BF16); Jev's depend on its cloud service and your network. The timings are one request at a time, not throughput.
• Public datasets. Your own decisions, data and costs of error will differ.

Frequently asked questions

What is Jev?

Jev is a cloud-hosted AI decision model: you send it information, a question and the allowed answers, and it returns a probability for each answer. In Daniel Walsh's test of 13 decision model set-ups on 4,635 shared questions, Jev 1.13 was the most accurate at 95.23 percent, with a median response of 245 milliseconds on short inputs.

Which local AI decision model is best?

In Daniel Walsh's Jev Arena test, Winnow 12B (94.61 percent) and Decider 4B (94.46 percent) were the most accurate local models overall, within a point of cloud-hosted Jev at 95.23 percent. Walsh suggests those two for short decisions that must stay local, and the tiny Laya model for narrow classification jobs, where it beat Jev on news topics.

Are local AI models faster than cloud APIs?

For short inputs they were in Daniel Walsh's test: local decision models answered in 47 to 56 milliseconds on one RTX 5090, against 245 milliseconds for cloud-hosted Jev. With a 20,000-token handbook in every request the order reversed: Decider took 1.90 seconds, Winnow 2.85 seconds and Jev 0.36 seconds.

How much GPU memory does a local decision model need?

In Daniel Walsh's test, memory allocated after loading was 12.9 GiB for Winnow 12B running 8-bit quantised, 17.5 GiB for Nimble 9B, 7.93 GiB for Decider 4B and 1.57 GiB for the 421-million-parameter Laya. He ran them on an RTX 5090 with 32 GB, but a smaller card can run the smaller models.

Why would a decision model refuse a question?

Some local decision models have hard limits. In Daniel Walsh's test, Plumb 4B and Semif 4B accept at most 16 answer options and Winnow 12B's server 64, so none of them could answer banking questions with 77 options, and Nimble 9B's 8,192-token input limit meant it could not read a full support handbook.

Credit: the experiment and every figure in this article, including the charts, are from Daniel Walsh of The AI Automators, who built the test bench and published the results. The commentary is ours.

About the author: Tengo Meskhi is CEO of Conceptdigital. He was previously at McKinsey and QuantumBlack, and Deputy CIO at Bank of Georgia.