Going directly for the logprobs is always icky when you use a chat model as base, because they are trained to write prose as output. So your "choice" tokens and thus their probabilities might get diluted in whatever else it wanted to say. If you have to do it in the same way as this post, at least add clear system instructions and a carefully worded beginning to the assistant output section of the prompt to lower the chances of it wandering off immediately.
I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).
The whole point is the quantified output. If you just ask an LLM to type out its confidence "manually", it'll make up some nonsense. The logprob numbers are more reliable.
I got this technique to work extremely reliably last year. However there were a bunch of caveats:
1) Firstly, you must institute a check that the multiple choice tokens dominate the output distribution. They should sum to 95% or more, ideally 99%, or the LLM is not following instructions properly. This is also the problem with constrained decoding - if the LLM really doesn't want to output a valid answer, the one you extract will not be high quality.
2) You need to ask it multiple times, permuting which option corresponds to which letter, and average the results. LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure.
3) For the same reason, performance improves if you frame the prompt as if it were the middle of a quiz. "Question 1" carries baggage that "Question 12" doesn't.
4) You must be exceedingly careful with tokenization.
But when all was said and done, I got a general purpose A/B classifier that gave high resolution quantitative output for the cost of a couple dozen tokens ingested and a couple inference passes.
>The whole point is the quantified output. If you just ask an LLM to type out its confidence "manually", it'll make up some nonsense. The logprob numbers are more reliable.
The whole point of my argument is that neither is good, but from a technical perspective logprobs is probably the worst unless you train a model on specific outputs. In which case you'd throw out the generality again, so when I think about it more, it's actually the worst overall. In my experiments, having the model simply assign "high" or "low" probability in a structured output generally performs best. You can try numbers, but you will never get anything close to what you could expect from traditional ML. And most certainly not from logprobs.
Yes, for vision classifiers we have in prod, I've seen a huge difference between A, B, C, 1, 2, 3 style answers and emitting a string. Even from just base model behavior pre-sft/rl. It was one of those obvious in retrospect moments.
> LLMs are surprisingly biased towards picking "A"
GP pointed at a causal explanation for this: almost every sentence in English that's a statement will start with "A" or "An", so "biased towards picking ''A''" will include most attempts at saying anything long-form for any reason.
I don't think that's the source of the bias I saw. I am confident that my prompting strategy eliminated attempts to generate long form content - specifically, I took care to wrap (A) and (B) in parentheses, so the completion looked like "Answer: (" - with this scheme an LLM is very unlikely to want to write "Answer: (A sentence goes here...". I know this, quantitatively, because I reliably got 99% distribution coverage with only A+B - that is, no inclination to write "The" or other common sentence starter. That's the beauty of the scheme - you can pretty directly and quantitatively validate how well the LLM understood the instructions. You expect it to only output A or B - so does it?
Meanwhile, the bias could be as much as 70% in favor of A in ambiguous cases - a signal completely drowning the <1% inclination to violate the format.
I would also note that models aren't people and don't think like people, so it's also possible that (at least for autoregressive ones) it could just be more likely to say "A" than "B" at that point, not necessarily because of "want" or "reason" but simply because that's what it was trained to do (such as in English writing).
What about switching to numbers or just some random Unicode character like smiley faces. Could be interesting if someone tested what LLMs like to say on a "cold start" lol.
Sure, it was high resolution (precise), how was accuracy compared to Jev (or existing open source implementations of the same concept, like laya)?
Also, Jev/laya do it in one forward pass, for multiple questions about the same state, rather than multiple passes for one question about that state. Well, for the usual multilingual configuration, two forward passes through different small models for laya, but that's because one is the router which chooses which model should do the real work, but still.
I never claimed that what I did was comparable to these modern options - I didn't validate it in more than an ad-hoc way anyway, and it used an off-the-shelf LLM rather than something specially trained. I didn't consider it worth releasing or making a big fuss of.
I contribute my experience here only because I've seen a lot of chatter lately about doing exactly this sort of thing, and I thought I'd share how I made it work for me. There are a lot of ways it can silently fail and give bad numbers if you aren't careful, and I wouldn't want people to think it doesn't work just because they used a vibe coded GitHub project from the last 48 hours that doesn't take these things into account.
In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].
I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.
+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper
If we're talking about running it locally, what about passing a partial response as part of the input?
Prompt part: "What is better, toast or bread?"
Incomplete answer part: "The answer to this question is "
and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.
Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.
To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.
Sure, you can fix that in a couple lines. Then a couple more lines for evaluating multiple questions on the same answer in parallel. Then a couple more lines for the confidence score (which is trivial to compute from all we have, but missing regardless). Then a harness to fine-tune an existing model to perform better on this specific task, and a collection of training data to use for that
I think we can all agree that Jev is not rocket science. It's a good idea executed well, with marketing that might have been a tad too bold
The confidence score is not trivial to compute. That is the whole point of the model. Even if you are using a proper scoring function such as NLL, it is not enough to ensure calibration in deep nets. So you have to do good post training to ensure it. These are all known techniques, but they are far from trivial, especially on large scale datasets.
Their docs at https://docs.typesafe.ai/confidence state "confidence is a statistic computed from the probability distribution the answer already gives you. TypeSafe computes it for you"
And further down "TypeSafe computes confidence from how the probability is spread across the options. All of it on one option gives 1.0; the more evenly it spreads, the lower the confidence. This demo uses (3 × largest probability − 1) / 2 to approximate confidence for three options."
So while we don't know the exact formula they use, it is just a function over the probabilities
I am open to the argument that this does not work well if you just plug in a qwen model instead of a model that is trained to output more statistically useful token distributions
we agree then, that is the entirety of my argument. Getting a deep net especially one that is anywhere near even SLM size to be calibrated is tough, especially across domains. They claim calibration across a variety of datasets which is interesting.
Best option would be reasoning + clear system instructions + constrained output. That is, if you have to use a chat model. Which works well enough to be sure, but hey I haven't tried raising millions of dollars when I did that 3 years ago. But perhaps I was the stupid one.
You’re still hoping that the model will respect your with to reply with a single letter. With Jev, the model doesn’t even have a concept of replying with something else.
A fundamental benefit of LLMs over Jev is that you can use test-time compute to improve the accuracy. Jev might eventually evolve to use test-time compute, but the formulation seems to more elusive to me than for LLMs.
Because of masked attention in LLMs, if you put the options before the body (the email to analyze), the transformer already knows what it needs to look for, and can use more tokens to create state to address that specific task (BERT has no mask in the attention, so tokens attend also to next tokens). You could also do a few examples in the system prompt to improve calibration.
Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."
Wow! TIL! I've been running a for loop around the two ordering variations to catch the winner of each turn and the difference is quite noticeable. In the options-after-body case in 47 of 100 attempts it classifies as phishing, whereas in the options-before-body case it classifies clearly as rickroll (94 out of 100 attempts)
Payroll sends you an email with a link to a Youtube video that plays a song.
Anyway, this for-looping stuff doing 100 calls to even a local VLLM API takes around 5 seconds in total, so this isn't anywhere close to sub-second Jev territory.
Beyond the missing latency and compute comparisons that Heaney commenter mentioned, also nothing about its error rate compared to Jev (nor if it even always outputs in a format the app can parse, not sure how solved that is).
But then at the end it says it’s parody. Maybe HN title should say it’s a joke.
Which massively slows down the output. Doing this with Qwen 9B already takes you into seconds per answer territory, and Jev is supposedly frontier level intelligence.
Non deterministic systems have furthered the "brain rot" in our industry.
Lots of people were happy to ignore the code in their "supply chain" before LLM's - but suddenly not reading the LLM's output is a problem. I get they are different but we're in the same realm.
The lack of real data on performance of what ever application that one is trying to pitch is getting appalling. It's a lot of "trust me bro" this works better hand waving. And it's getting gross.
And how do we even measure nondeterministic systems? Because if I told you that Anthropic was spending millions of dollars having 1000's of agents "pre solve" benchmarks to build into their next version of the system you would scream they were cheating. Every one is focused on the "hacking" in the hugging face incident and no one is looking why they were even playing with those benchmarks in the first place.
Yeah it says it's a parody, but then in the same sentence it refers to the other "OpenJev" implementations, which are basically the same thing with marginally more effort. And it doesn't imply that those things are parodies too (and I don't think they are parodies).
Somehow the HN crowd has a bunch of "professionals" who don't care about error rates and think that a Qwen model running on a potato is frontier intelligence.
You didn't specify time frames; 1) is doable for a very short time, with a lot of coulomb caching in between the computer and the potato :).
(For more realistic solution, surely someone must be working on optronics - these models just beg to have their weights cleverly etched into stacked sheets of plastic, so they can do inference for free on a beam of light.)
This assumes that you have a powerful computer that can run an LLM. Most people will have to call an LLM API every time, which is three orders of magnitude more expensive than Jev.
I am hearing about Jev for the first time here so no idea about the hype.
So their(Jev) is that the thing is faster at classification than a frontier model? Because the whole type safe aspect is already fully solvable with structured output.
But their example is classification but that would also be possible and faster with a classic BERT model.
So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?
Off the top of my head it's 3 things it advertises:
- By not being a optimised for chat, it can deliver confidence for answer and not for how an answer should be phrased
- Speed. It can take seconds for OpenAI to compile schemas, jev can respond before openAI has even begun thinking
- Token efficiency and price. I think its the output token they don't even charge for because they are negligible, and the tokens they do charge for are at a fraction of a comparable model.
If you are using structured output, I think those 3 together is a really big deal.
>But their example is classification but that would also be possible and faster with a classic BERT model.
I believe the things you can classify with ChatGPT without any tuning or training is way beyond what BERT can do.
You need to train data for a BERT-based classifier, and then there's a risk that it will pick up specific biases from the data instead of what you want.
As far as I understand, the idea of Jev is zero-shot or few-shot classifier: it learns a lot of stuff at pre-training, but unlike a classic LLM it doesn't need to learn how to chat, so it can be much smarter at a particular size
I am in no way trying to sell Jev here as some panacea of the modern world; I'm only responding to your questions:
> But their example is classification but that would also be possible and faster with a classic BERT model.
With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model. Jev is pitched as a zero- or 'few-shot' model. You define the schema in code, give it instructions, and it works without a traditional training pipeline.
> So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?
Yup; that about sums it up: it is more or less an optimized, task-specific small model with the flexible understanding of a traditional LLM.
> With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model.
BERT requires a huge corpus, but it isn't labeled. BERT is trained through self-supervised learning using mask tokens and next sentence prediction. Fine-tuning is useful for specific tasks, but isn't absolutely essential for the model to function.
I think you're possibly arguing a point I wasn't making? I'm not saying Jev invented zero-shot classification, or that there aren't already zero-shot classifiers on HF that can do classification without fine-tuning; I was responding to questions asked in a silo.
I think this discourse is still in the "figuring it out" phase. But here's where my thoughts are currently:
If you accept the premise that there are use cases where you might ask a frontier model a classification-shaped question and expect an ok enough answer, rather than creating a purpose specific classifier on some dataset that you have, then it follows that this is quite an inefficient thing to do, because you're doing extra work to turn the output tokens into a structured output and mostly throwing them away. So then if you could instead train a frontier level model that skips the output tokens and directly returns the structured classification information, that would be more efficient, and that's what jev seems to be.
But a lot rides on that initial premise of whether this is a use case that makes sense. But if you find yourself asking a model like Opus arbitrary yes/no questions and then maybe you switch to a faster and cheaper model because it's too slow and expensive, it seems like jev might be a great replacement for that.
This seems like an unusual definition of type safety. I certainly understand how every run deterministically giving the same schema (type) of data is a requirement to be "type-safe", but in my mind the content of the result is not relevant to the question of type safety. Am I not getting it?
There's still run-to-run variance because it's not fully deterministic. So runs with exact same inputs can return different outputs. Besides, though the output always conforms to the choices you specified, whether the probabilities attached to them are actually correct is a different issue.
Jev HAS hallucinations and it doesn't attempt to solve hallucination at all.
For three choices problem (A,B,C), what Jev guarantees is that it will give the choice in a defined schema (type-safe). It never guarantees that the choice is correct (hallucination).
I mean, the obvious analogy is to other llm hype cycles. When chatgpt came out, everyone wanted to figure out how to use it for everything. Turned out it really was good at a lot of things, while still being overhyped. Same thing when chain of thought models hit the scene. Same thing with coding harnesses. Same thing now.
My base case is that this will probably be pretty useful, and also not as useful as the current hype suggests.
there's some nugget of usefulness to it as an idea, but overall jev itself for some reason smells like a scam. scam in a sense of an engineered marketing push towards some bs that will turn into a saas. technical merit is next to none.
If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.
By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.
So, fast in the LLM space and comparable with Jev.
That's right. There's only so much optimization that you can make to a transformer-based model and any tricks that Jev is employing, any open-source LLM can also employ.
The number of - “I did/invented Jev last year”, or, “here’s a version of Jev I vibed up last night” is getting a bit ridiculous.
Especially ridiculous is how the hacker news crowd seems to be taking these at face value…
There was one the other day with a compelling demo. But when you looked closely at it, it was feeding in the options with the word “best” on the option to pick and a fine tuned model designed to recognise that word…
I thought about writing a blog post that used Jev as a jumping off point to talk about calibration metrics[1], when and why calibration is important[2], and various methods for doing post hoc recalibration[3] of poorly calibrated models. I thought it would be interesting and instructive to benchmark Jev's claims about calibration, find examples where it wasn't (relative to some data set, which shouldn't be hard to find), give an example of how its poor calibration could be exploited, and then show how it could be fixed with, say, isotonic regression. That would give practitioners a roadmap to using Jev successfully without blindly trusting it.
But I'm probably never going to write that article because, as your comment correctly points out, the tenor of the discourse isn't very healthy. Everyone is either "clowning on" Jev (as the kids say), or gulping down industrial quantities of Kool-Aid, or talking about the hype and branding instead of the math. I'll have to find a less contentious example if I want talk about calibration.
Edit: I just found something interesting: there already is a preprint case study[4] along the same lines as the one I outlined, posted just 2 days ago!
Unsurprisingly, they found that recalibration helps enormously, as you'd expect. I guess I could still talk about the Dutch book stuff if I gave a gambling/investing example, but that's pretty well-trodden territory. So now I have even less interest in writing that blog post.
Whilst I do like reading these things for technical know how, I can sympathise with the creator of jev who now presumably has to apply an order of magnitude effort to explain why the 100 smaller things done better than this add up to a much better product.
Replace 'explain' with 'sell'. Don't forget that it's a gold rush. There's no reason to sympathize with corporations in their rush for the slice of the pie.
You can be unsympathetic to the corporation’s bottom line while being sympathetic to the human beings that had their work trivialized by some cocky blog post.
You can also just acknowledge that we don't know what they did (and that is because they chose not to tell us). Apparently, until not so long ago Jev would spell out its identity as Q-w-e-n if asked, so we might as well assume that what they did is at least similar to what this blog poster did (who chose to tell us).
As a commercial artist seeing some pretty serious market disruption based on models that used my art and writing to create, without permission, credit, or compensation, I’d have a hard time not punching anyone in the AI business complaining about other people using their work uncredited. Then I’d probably have a real hard time not doing it a few more times. I’m confident I’d settle for just dressing them the fuck down, but it would take some real restraint.
My source: https://news.ycombinator.com/item?id=49783999#49785351 I don't have an account so couldn't check, a friend couldn't replicate with a one-letter attempt (first letter A 50%) - performed by his agent, though, so not sure what he did exactly.
From what I’ve learned about Jev I feel it’s just a very successful marketing campaign to developers not fully understanding data science (and deep learning). It’s nothing new, been around since 2022? Being local is an extreme advantage lol.
Fast and accurate general purpose classifiers DID NOT EXIST before Jev. You could either use an LLM to get a slow and accurate general purpose classifier, or you could use a smaller model to get a fast and inaccurate general purpose classifier, or you could fine tune your own model that would be both fast and accurate, but it wouldn't be general purpose.
Except in this case it's not technically correct. Jev's claim is that it's frontier intelligence and these guys are pretending that a 8-bit quantized 0.6B param Qwen model is that. There's no universe in which that claim is technically correct.
The one thing I can't wrap my head around with Jev is why they're trying to create that "System One" narrative.
In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two. So by definition what Jev does isn't System One thinking.
If anything, regular programming that automatically executes based on logic, without requiring "thinking" would be "System One".
> In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two.
I'd argue that most human classification is pre-conscious / System One. You see a table, you recognize it as a table without asking yourself "is this a table?"
I guess their marketing implies that it moves classification into system one response time.
>they're trying to create that "System One" narrative
I think you answered your own question. Executives are going to ask two questions, 1) how is this different/why does it matter and 2) how will i use it to make money?
Leya came to market more than a year before Jev, and failed because nobody understood how to use it, and he was unable to market it properly. Jev used this strategy and did not fail.
> In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two.
Huh? I guess that depends on the exact definition of "classification", but I think the bulk of basic classification tasks we make every day to make sense of our surroundings, such as object recognition is definitely done using system 1. So is higher-level "stereotyping" or anything you could described with "I know it when I see it".
Because those responses can be incorrect or even harmful, you would sometimes make use of system 2 to correct them - but that doesn't change that the initial response is from system 1.
Sure, object recognition is System 1, but Jev's own use cases list things like security incident triage, invoice approval, agent escalation, support actions, etc.
Those are generally the kind of tasks that require "System 2" in humans.
To be clear, I think the whole "System 1 vs System 2" framing is a pretty limiting way to think about AI (and thinking in general).
They were coined in the extremely influential foundational research the "pop science book" was based on, but that doesn't make it a marketing ploy; System 1 is very fast non-verbal subconscious heuristic operation, and a lot of our brain's classification takes place at that stage. Anyone familiar with the usage immediately had clear ideas about how it'd ideally be used (e.g. heuristics that reduce search space, manage context elements dynamically, etc) vs not (e.g. as a 1:1 replacement for something that could already be easily solved with a classifier).
I also built one, but mine uses embeddings. It classifies concepts defined by a collection of positive and negative examples. The classifier model is trained in <1 second using ridge regression. The model itself is exactly the same shape as the embedding, so it works as a concept embedding. Since I already have a dataset, I can use it to do conformal prediction in order to calibrate confidence scores. Jev, on the other hand, has a generic model, not trained on in-domain examples, so its confidence scores are uncalibrated for any non-generic task.
So you might ask: how do I obtain the training examples? Just collect samples and use a coding agent to classify them as match or no match. From time to time you can add more examples to the dataset to have your concept adapt to changes in input distribution. It's all automated, but it only uses LLMs to train concept vectors, after that it works like a regular embedding model with a calibrated classifier on top. It's also 20-30x faster than Jev, free, and runs on CPU.
this is such a trivial thing to do in DSPY, no one bothered to give it a name…
here's 7 lines
import os
import dspy
lm = dspy.LM("openrouter/z-ai/glm-5.3-flash", api_key=os.environ["OPENROUTER_API_KEY"])
jev = dspy.Predict('email:str -> choice:Literal["Legitimate", "Spam", "Phishing"]')
email = "Payroll asks for your password on a non-company sign-in page."
pred = jev(email=email, lm=lm)
print(pred.choice)
there are other options, obviously. you can choose to give it some tools, maybe some reasoning stage before picking a choice, and that's on top of the "reasoning" the llm model already does api side
I wonder if this could be a good stepping stone to write a local prompt router to optimise what model get what prompt. I.e. if the prompt is just a lookup, send it to haiku, if it's reasoning, send it to opus and if it's implementation send it to sonnet.
I tried it out, but with kev instead of this python script. The issue is that mid session swapping invalidates the cache, which drives cost quite a lot. Ended up loosing money when comparing prompts in most of my transcripts.
If you're not behind a walled garden like i am (vertex), you could probably experiment with routing on effort level instead. Anthropic supports it, but vertex has not added that feature yet.
Looking at the logprobs on tokens works for the local models, but not on the frontier ones. It's been more or less broken since GPT-4o for example. I wrote about it two years ago: https://medium.com/data-science/9-11-or-9-9-which-one-is-hig.... Also, I've done some work in estimating confidence and on rubric evals using the same method, and you actually get better correlation to "real confidence" by just getting the LLM to say it.
Interesting. Have you repeated these experiments with recent models? I'm thinking frontier models APIs have tools/MCPs for math stuff but curious about recent Qwen models, etc.
Haven't tried recent frontier ones, Astra for example doesn't support logprobs emission on the API, and Sol and Luna supposedly support it with reasoning disabled. Haven't tried local models like Qwen 3.8 27b (I'm actually exploring their thinking trace, it's a lot of fun)
What I'm missing here is also type guarantees. I don't think you can do it without token level logic which forces the model to output the tokens from a predefined pool of tokens. A logic like this given some JSON schema is not that difficult to implement. If the LLM must output JSON schema compatible value then you can also add that it doesn't "hallucinate". Which is funny too because just guaranteeing the type does not mean the model does not hallucinate but this is another story.
Any analogy works at certain abstraction level, and this works with a premise that it's a classifier that makes decisions.
Still good. In practice for Jev the devils in the details.
As you all know by now, it's easy to write PoC and understand with AIs (or even manually, which is now a prestious practice).
That demo will get you 80% there
Getting to that 100% or even 99% to JEV level will be hard with all the edge cases, infra, API, communications, etc.
I tried this in a custom ML array programming language with mixed results. I created a literate programming document about it. https://sw-ml-study.github.io/demo-decision-model/concise-he... -- it has some analysis of the python and some other approaches.
I should mention that I am trying to apply a Jev-inspired approch to a trained-from scratch, in-browser typed decision model that attempts to recreate the deterministic classic Eliza program.
Hey, I enjoyed the article! I think it’s helpful to break stuff down as much as possible to nail down what is happening. They did that. The important part is not importing a model or something, it’s what’s happening after. I learned something
Thoughtful demo. If possible, I'd want to see next is whether the probabilities it returns are calibrated and not just whether the top choice is right. I ran a small experiment on this, compared base and instruct models' calibration on option-letter logits, then fine-tuned with Brier loss vs cross-entropy. Have you looked at a reliability curve for this version?
Very nice as a conceptual thing, but there's a bit more to it. I've bolted Gemma 4 onto a custom pipeline for that (https://rcarmo.github.io/projects/go-system-one/) and it's OK-ish (a bit slow on my puny 3060, but I can use it to prototype a bunch of things locally until the Jev mania settles and we have better models).
Haha - I did enjoy this read! And there is a point to the whole marketing-dresses-up-stuff that is certainly true. I think it's worth pointing out the other HN story earlier https://news.ycombinator.com/item?id=49765348 about an open-weight model called Laya.
P.S. I am evaluating that model for a production use case where I would have used Jev
The argument the article makes is it's not that different. Jev's argument is that they have trained the model to better output probabilities (which is not necessarily a training object of LLMs but we don't actually know that)
Ultimately Jev claims to have a data advantage which is likely where the future lies. They'll have a unique edge in improving general purpose classification / decisioning.
If you believe the marketing, constraining the output this way can make the model much faster and much more type-safe (the model didn’t give you a fifth choice not present in the choices).
I missed the hypewave so can't say a lot about Jev, but the double standards are entertaining:
About Jev:
> We didn't train a model with Reinforcement Learning for Calibrated Decisions (RLCD) to calibrate the decisions and probabilities (even though they are not always correct).
Only 99% correctness! Borderline unusable!
About their model:
> It classifies: it gets a prompt with choices and outputs probabilities.
You want numbers, it gives you numbers! What more could you want?
a hile ago (when big providers still provided logprobs) i created a VS Code highlighter that visualizes unsure tokens.
Since most chat models want to answer with a human-readable message i think their logprobs are not as meaningful. It would be interesting to see if one choice is like "correct" and if the model wants to choose it more often, cause it might not answer the question but to prose to the user.
You can also go beyond Jev. Qwen 3.5 0.8B is fantastic at basic image classification/question answering (including OCR elements) also. Though rather than looking at logits, I get it to output a structured JSON object and it does simple object classification tasks on a Mac at under 500ms a pop (I forget how far, but I think it's like ~250ms) with good accuracy (depending on task).
I think it's like saying Jev has been trumpeting the invention of a spinning transportation system when we've known such things as a "wheel" for thousands of years. Classification models have been with us far longer than autoregressive LLMs.
not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.
p(y = next thinking+decision token | x = question) != p(y = next decision token | x = question)
The former is what LLMs are trained for, the latter is what Jev was likely trained on (likely used thinking alignment as an auxiliary loss, but not explicitly included in the probability calibration).
First release people are getting more balanced at least. Less 'it's gonna change everything' to 'might be viable use cases'. If I have to look at one more video high lighting google flights, I might loose it though.
i love that people are trying to make OS jevs but what is the point of doing all this work and not ask your coding agent to do a little benchmarking. selfishly want an open weight model to beat jev here
strong "You can build dropbox quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" vibes
You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)
> TypeSafe AI's Master Customer Agreement explicitly prohibits using the services or model outputs to develop a competing product, perform model distillation, or reverse engineer the service, which generally restricts competitive benchmarking aimed at replicating the model.
It doesn't explicitly prohibit benchmarking by name, but the previous terms (which seem aimed at preventing Jev being used to increase the value of competitive products) does seem to lean that direction.
That said, MsSQL had terms which prevented publishing benchmarks which compared it against other SQL DBs and that wasn't enough to prevent some companies from using it.
Why anyone would want to work for a company who thought so little of their own product that it couldn't stand up to customers using it for normal business processes is beyond me.
This, like the hype of Jev on Twitter, totally ignores accuracy and generality across domains.
In my experience even structured LLM output performs poorly on classifier tasks. LLMs are trained to talk and think longer. If you don't give LLM enough space to reason it would become very dumb.
I'm not saying that Jev is way better, but that people way overindexed cost and speed.
I'm surprised something like Jev came out "so late", but the hype has been ridiculous. Yes, it's a good idea. No, it only helps when fast and cheap are important and I guarantee existing labs will have this figured out in a matter of days.
Add visual understanding, add reasoning and bring down the size to run on my computer. That's when it will be interesting.
So many people that don't understand the tech jumped on the hype train because "it cannot hallucinate" and else. It's crazy.
Jev is so cheap I dont see anyone undercutting them on price. If people/products start using Jev underneath before the incumbents can release something then most likely everyone's jsut gonna stay with Jev
Companies like OpenAI or Anthropic could offer it basically for free as an extension to their existing models. You still need the generator, but having something fast (and maybe even local) to do browser nav would be awesome.
Should they be writing quicksort in assembly as a first step? I think its legitimate in this case given that Jev is likely using the same tools as the example. Showing how easily the core is created using those tools helps to dispel some of the mystery and hype.
Example why its legit:
I just invented a new "Regression Estimate Validator" aka Rev. It takes hundreds of input dimensions, then outputs an interpretable score. Its very fast and statistically robust. Response: Ok but you could just use `pytorch.nn.Linear(d_in, 1)`? True, it is equivalent, but that's concealing millions of lines of hand-tuned math libs, CUDA, python, and other stuff.
The fact that there are many lines of code underpinning the target functionality doesn't make it any harder to use, and doesn't increase the value of the sales pitch for the "new shiny thing" using those few lines of code.
However, I do sympathize with your frustration that people can just say "its 1 line of code" when that line is "invoke API" which is really millions of lines / databases, etc. as a way to dismiss legitimate work without understanding its implications.
I'm so sick of seeing these people who "made Jev in 25 lines of Python" or whatever the flavor of the day is. Do you people seriously think that Qwen3-0.6B-Q8_0.gguf is frontier intelligence? If you want to argue that Jev is NOT frontier intelligence, then go make that argument. Don't try to pretend that Qwen3-0.6B-Q8_0.gguf is frontier intelligence. That's retarded.
Don't cut the quote mid-sentence. Here's the full quote:
> note: this is a parody blog post, see these links for better/more complete open implementations of Jev: OpenJev, openjev-sglang, and OpenJev on DiffusionGemma.
Are OpenJev, openjev-sglang, and OpenJev on DiffusionGemma using Qwen3-0.6B-Q8_0.gguf, or did you just want to emphasize the part that was unrelated to your previous response?
Going directly for the logprobs is always icky when you use a chat model as base, because they are trained to write prose as output. So your "choice" tokens and thus their probabilities might get diluted in whatever else it wanted to say. If you have to do it in the same way as this post, at least add clear system instructions and a carefully worded beginning to the assistant output section of the prompt to lower the chances of it wandering off immediately.
I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).
The whole point is the quantified output. If you just ask an LLM to type out its confidence "manually", it'll make up some nonsense. The logprob numbers are more reliable.
I got this technique to work extremely reliably last year. However there were a bunch of caveats: 1) Firstly, you must institute a check that the multiple choice tokens dominate the output distribution. They should sum to 95% or more, ideally 99%, or the LLM is not following instructions properly. This is also the problem with constrained decoding - if the LLM really doesn't want to output a valid answer, the one you extract will not be high quality. 2) You need to ask it multiple times, permuting which option corresponds to which letter, and average the results. LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure. 3) For the same reason, performance improves if you frame the prompt as if it were the middle of a quiz. "Question 1" carries baggage that "Question 12" doesn't. 4) You must be exceedingly careful with tokenization.
But when all was said and done, I got a general purpose A/B classifier that gave high resolution quantitative output for the cost of a couple dozen tokens ingested and a couple inference passes.
>The whole point is the quantified output. If you just ask an LLM to type out its confidence "manually", it'll make up some nonsense. The logprob numbers are more reliable.
The whole point of my argument is that neither is good, but from a technical perspective logprobs is probably the worst unless you train a model on specific outputs. In which case you'd throw out the generality again, so when I think about it more, it's actually the worst overall. In my experiments, having the model simply assign "high" or "low" probability in a structured output generally performs best. You can try numbers, but you will never get anything close to what you could expect from traditional ML. And most certainly not from logprobs.
Yes, for vision classifiers we have in prod, I've seen a huge difference between A, B, C, 1, 2, 3 style answers and emitting a string. Even from just base model behavior pre-sft/rl. It was one of those obvious in retrospect moments.
> LLMs are surprisingly biased towards picking "A"
GP pointed at a causal explanation for this: almost every sentence in English that's a statement will start with "A" or "An", so "biased towards picking ''A''" will include most attempts at saying anything long-form for any reason.
I don't think that's the source of the bias I saw. I am confident that my prompting strategy eliminated attempts to generate long form content - specifically, I took care to wrap (A) and (B) in parentheses, so the completion looked like "Answer: (" - with this scheme an LLM is very unlikely to want to write "Answer: (A sentence goes here...". I know this, quantitatively, because I reliably got 99% distribution coverage with only A+B - that is, no inclination to write "The" or other common sentence starter. That's the beauty of the scheme - you can pretty directly and quantitatively validate how well the LLM understood the instructions. You expect it to only output A or B - so does it?
Meanwhile, the bias could be as much as 70% in favor of A in ambiguous cases - a signal completely drowning the <1% inclination to violate the format.
I would also note that models aren't people and don't think like people, so it's also possible that (at least for autoregressive ones) it could just be more likely to say "A" than "B" at that point, not necessarily because of "want" or "reason" but simply because that's what it was trained to do (such as in English writing).
What about switching to numbers or just some random Unicode character like smiley faces. Could be interesting if someone tested what LLMs like to say on a "cold start" lol.
> LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure.
Not nearly as sophisticated as myself who would mutter "When in doubt - Charlie out" before marking C.
Sure, it was high resolution (precise), how was accuracy compared to Jev (or existing open source implementations of the same concept, like laya)?
Also, Jev/laya do it in one forward pass, for multiple questions about the same state, rather than multiple passes for one question about that state. Well, for the usual multilingual configuration, two forward passes through different small models for laya, but that's because one is the router which chooses which model should do the real work, but still.
I never claimed that what I did was comparable to these modern options - I didn't validate it in more than an ad-hoc way anyway, and it used an off-the-shelf LLM rather than something specially trained. I didn't consider it worth releasing or making a big fuss of.
I contribute my experience here only because I've seen a lot of chatter lately about doing exactly this sort of thing, and I thought I'd share how I made it work for me. There are a lot of ways it can silently fail and give bad numbers if you aren't careful, and I wouldn't want people to think it doesn't work just because they used a vibe coded GitHub project from the last 48 hours that doesn't take these things into account.
A colleague of mine compared Jev with existing classification models, here the results: https://www.timetoact-group.at/en/techblog/techblog/jev-for-...
In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].
I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.
0: https://arxiv.org/pdf/1706.04599
[flagged]
+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper
https://til.simonwillison.net/llms/llama-cpp-python-grammars
Yes. But even then, the probabilities are not calibrated. In jev/laya, they are (well, relatively anyways).
If we're talking about running it locally, what about passing a partial response as part of the input?
Prompt part: "What is better, toast or bread?"
Incomplete answer part: "The answer to this question is "
and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.
Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.
To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.
Sure, you can fix that in a couple lines. Then a couple more lines for evaluating multiple questions on the same answer in parallel. Then a couple more lines for the confidence score (which is trivial to compute from all we have, but missing regardless). Then a harness to fine-tune an existing model to perform better on this specific task, and a collection of training data to use for that
I think we can all agree that Jev is not rocket science. It's a good idea executed well, with marketing that might have been a tad too bold
The confidence score is not trivial to compute. That is the whole point of the model. Even if you are using a proper scoring function such as NLL, it is not enough to ensure calibration in deep nets. So you have to do good post training to ensure it. These are all known techniques, but they are far from trivial, especially on large scale datasets.
Their docs at https://docs.typesafe.ai/confidence state "confidence is a statistic computed from the probability distribution the answer already gives you. TypeSafe computes it for you"
And further down "TypeSafe computes confidence from how the probability is spread across the options. All of it on one option gives 1.0; the more evenly it spreads, the lower the confidence. This demo uses (3 × largest probability − 1) / 2 to approximate confidence for three options."
So while we don't know the exact formula they use, it is just a function over the probabilities
I am open to the argument that this does not work well if you just plug in a qwen model instead of a model that is trained to output more statistically useful token distributions
> I am open to the argument
we agree then, that is the entirety of my argument. Getting a deep net especially one that is anywhere near even SLM size to be calibrated is tough, especially across domains. They claim calibration across a variety of datasets which is interesting.
For N options, it's (N x Max Probability - 1) / (N - 1). It's verified in this article: https://bernoulli.app/articles/is-jev-confident
It means confidence is just a converted max probability and not an independent signal.
That's the approach that daseinlabs/open-jev takes, in contrast to the above, which is what TheoLeeCJ/openjev and ekzhang/openjev-sglang do
https://sgnt.ai/p/jev/
Seems like all normal english words could risk the same, so would using short but random strings be even better?
Actually to me it sounds it could be benchmarked if this kind of effect exists in the first place.
Best option would be reasoning + clear system instructions + constrained output. That is, if you have to use a chat model. Which works well enough to be sure, but hey I haven't tried raising millions of dollars when I did that 3 years ago. But perhaps I was the stupid one.
Isn't that effectively the same as the blog post? You are just pushing the token filter to the sampling step.
"Reply with just the letter A, B, or C."
There, I fixed your problem.
You’re still hoping that the model will respect your with to reply with a single letter. With Jev, the model doesn’t even have a concept of replying with something else.
A fundamental benefit of LLMs over Jev is that you can use test-time compute to improve the accuracy. Jev might eventually evolve to use test-time compute, but the formulation seems to more elusive to me than for LLMs.
Because of masked attention in LLMs, if you put the options before the body (the email to analyze), the transformer already knows what it needs to look for, and can use more tokens to create state to address that specific task (BERT has no mask in the attention, so tokens attend also to next tokens). You could also do a few examples in the system prompt to improve calibration.
Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."
Wow! TIL! I've been running a for loop around the two ordering variations to catch the winner of each turn and the difference is quite noticeable. In the options-after-body case in 47 of 100 attempts it classifies as phishing, whereas in the options-before-body case it classifies clearly as rickroll (94 out of 100 attempts)
Payroll sends you an email with a link to a Youtube video that plays a song.
Options after body:
Options before body: This was Gemma4-26B-A4B-NVFP4 by the way.EDIT
Gemma4-12B-it-NVFP4 seems way less sensitive to option/body ordering:
Options after body:
Options before body: Anyway, this for-looping stuff doing 100 calls to even a local VLLM API takes around 5 seconds in total, so this isn't anywhere close to sub-second Jev territory.Yah, that's what I use: https://rcarmo.github.io/projects/go-system-one uses Gemma, and that's partly why. Seems less prone to getting distracted with ordering.
Does this imply that bigger models aren’t affected by this as much and therefore won’t see much improvement?
What a time to be alive, repeating questions to a model twice to increase accuracy.
Repitation always helped make your point stronger. Repitation always helped make your point stronger.
I guess repeating a mistake helps make it more obvious too.
With enough repitation you might get repitition.
Use repetition to avoid trepidation if you have low reputation.
Or even repetition, repetition.
Mr Smith agrees.
You can say that again!
Repetition Legitimizes
I use ROT13 twice for extra security
I had an issue with accuracy a bit ago. So I repeated a couple of things without understanding why and it solved the problem.
I am glad there is an actual reason.
fuck this timeline
I feel you
Bro this timeline makes no sense. Repeating instruction to a data center of geniuses.
Djinniuses
Prompt Repetition Improves Non-Reasoning LLMs: https://arxiv.org/abs/2512.14982
This works very very well :).
https://github.com/Mushroom-Systems/lichen
Did I read this right?
This repo is really outperforming the OG Jev in the public benchmarks?
There was no time to benchmaxx. How is this possible?
Beyond the missing latency and compute comparisons that Heaney commenter mentioned, also nothing about its error rate compared to Jev (nor if it even always outputs in a format the app can parse, not sure how solved that is).
But then at the end it says it’s parody. Maybe HN title should say it’s a joke.
The parody note at the end is bizarre. Everything above it reads pretty seriously:
> Everyone on Twitter is all over Jev, how it's the next frontier of large language models and the AI paradigm. We don’t really think so.
latency and compute comparisons highly depends on your local setup.
you can swith to a better model for lower error rate.
Which massively slows down the output. Doing this with Qwen 9B already takes you into seconds per answer territory, and Jev is supposedly frontier level intelligence.
> nothing about its...
Non deterministic systems have furthered the "brain rot" in our industry.
Lots of people were happy to ignore the code in their "supply chain" before LLM's - but suddenly not reading the LLM's output is a problem. I get they are different but we're in the same realm.
The lack of real data on performance of what ever application that one is trying to pitch is getting appalling. It's a lot of "trust me bro" this works better hand waving. And it's getting gross.
And how do we even measure nondeterministic systems? Because if I told you that Anthropic was spending millions of dollars having 1000's of agents "pre solve" benchmarks to build into their next version of the system you would scream they were cheating. Every one is focused on the "hacking" in the hugging face incident and no one is looking why they were even playing with those benchmarks in the first place.
"Trust me Bro"...
Yeah it says it's a parody, but then in the same sentence it refers to the other "OpenJev" implementations, which are basically the same thing with marginally more effort. And it doesn't imply that those things are parodies too (and I don't think they are parodies).
Somehow the HN crowd has a bunch of "professionals" who don't care about error rates and think that a Qwen model running on a potato is frontier intelligence.
1) get local model to run on the electrical output of a potato 2) accept Nobel price
You didn't specify time frames; 1) is doable for a very short time, with a lot of coulomb caching in between the computer and the potato :).
(For more realistic solution, surely someone must be working on optronics - these models just beg to have their weights cleverly etched into stacked sheets of plastic, so they can do inference for free on a beam of light.)
> "25 lines of python" > "import Solution" ok
>we didn't call an api
>calls an api
ok
not really fair to call downloading a model once to run locally the same thing as calling an api which happens for every question in jev
This assumes that you have a powerful computer that can run an LLM. Most people will have to call an LLM API every time, which is three orders of magnitude more expensive than Jev.
most people will, this tongue-in-cheek demo does not though. downloading assets to run a program is not what we refer to when we say "API calls"
The 25 lines is the only thing that makes Jev different compared to Solution apparently.
I am hearing about Jev for the first time here so no idea about the hype. So their(Jev) is that the thing is faster at classification than a frontier model? Because the whole type safe aspect is already fully solvable with structured output. But their example is classification but that would also be possible and faster with a classic BERT model. So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?
Off the top of my head it's 3 things it advertises:
- By not being a optimised for chat, it can deliver confidence for answer and not for how an answer should be phrased
- Speed. It can take seconds for OpenAI to compile schemas, jev can respond before openAI has even begun thinking
- Token efficiency and price. I think its the output token they don't even charge for because they are negligible, and the tokens they do charge for are at a fraction of a comparable model.
If you are using structured output, I think those 3 together is a really big deal.
>But their example is classification but that would also be possible and faster with a classic BERT model.
I believe the things you can classify with ChatGPT without any tuning or training is way beyond what BERT can do.
You need to train data for a BERT-based classifier, and then there's a risk that it will pick up specific biases from the data instead of what you want.
As far as I understand, the idea of Jev is zero-shot or few-shot classifier: it learns a lot of stuff at pre-training, but unlike a classic LLM it doesn't need to learn how to chat, so it can be much smarter at a particular size
I am in no way trying to sell Jev here as some panacea of the modern world; I'm only responding to your questions:
> But their example is classification but that would also be possible and faster with a classic BERT model.
With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model. Jev is pitched as a zero- or 'few-shot' model. You define the schema in code, give it instructions, and it works without a traditional training pipeline.
> So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?
Yup; that about sums it up: it is more or less an optimized, task-specific small model with the flexible understanding of a traditional LLM.
> With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model.
BERT requires a huge corpus, but it isn't labeled. BERT is trained through self-supervised learning using mask tokens and next sentence prediction. Fine-tuning is useful for specific tasks, but isn't absolutely essential for the model to function.
If something is task-specific (well understood) wouldn't this be a good candidate for a computer program?
Not necessarily. It may be well-understood but still require judgment calls.
couldn't be more wrong - there are so many zero shot classifiers available on HF which do the same thing.
I think you're possibly arguing a point I wasn't making? I'm not saying Jev invented zero-shot classification, or that there aren't already zero-shot classifiers on HF that can do classification without fine-tuning; I was responding to questions asked in a silo.
>With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model.
I think they were responding to this. You can use BERT to provide zero shot classification predictions.
I guess I assumed they meant BERT, not some specific BERT-base model. Vanilla BERT does not support zero shot.
I think this discourse is still in the "figuring it out" phase. But here's where my thoughts are currently:
If you accept the premise that there are use cases where you might ask a frontier model a classification-shaped question and expect an ok enough answer, rather than creating a purpose specific classifier on some dataset that you have, then it follows that this is quite an inefficient thing to do, because you're doing extra work to turn the output tokens into a structured output and mostly throwing them away. So then if you could instead train a frontier level model that skips the output tokens and directly returns the structured classification information, that would be more efficient, and that's what jev seems to be.
But a lot rides on that initial premise of whether this is a use case that makes sense. But if you find yourself asking a model like Opus arbitrary yes/no questions and then maybe you switch to a faster and cheaper model because it's too slow and expensive, it seems like jev might be a great replacement for that.
> is already fully solvable with structured output.
Not particularly. There is still the problem of hallucinations and varying results across runs.
That's more of what type-safety means for their team. Every run gives the same results. It's type-safe
This seems like an unusual definition of type safety. I certainly understand how every run deterministically giving the same schema (type) of data is a requirement to be "type-safe", but in my mind the content of the result is not relevant to the question of type safety. Am I not getting it?
There's still run-to-run variance because it's not fully deterministic. So runs with exact same inputs can return different outputs. Besides, though the output always conforms to the choices you specified, whether the probabilities attached to them are actually correct is a different issue.
Jev HAS hallucinations and it doesn't attempt to solve hallucination at all.
For three choices problem (A,B,C), what Jev guarantees is that it will give the choice in a defined schema (type-safe). It never guarantees that the choice is correct (hallucination).
One thing I would like to know is how fast it is when it's being presented with a 8000 ctx prompt? 16k? 32k?
no one knows but everyone pretends so go along with it.
I'm getting flashbacks to when everyone was doing map:reduce for things
I mean, the obvious analogy is to other llm hype cycles. When chatgpt came out, everyone wanted to figure out how to use it for everything. Turned out it really was good at a lot of things, while still being overhyped. Same thing when chain of thought models hit the scene. Same thing with coding harnesses. Same thing now.
My base case is that this will probably be pretty useful, and also not as useful as the current hype suggests.
there's some nugget of usefulness to it as an idea, but overall jev itself for some reason smells like a scam. scam in a sense of an engineered marketing push towards some bs that will turn into a saas. technical merit is next to none.
It's fast.
If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.
By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.
So, fast in the LLM space and comparable with Jev.
That's right. There's only so much optimization that you can make to a transformer-based model and any tricks that Jev is employing, any open-source LLM can also employ.
The number of - “I did/invented Jev last year”, or, “here’s a version of Jev I vibed up last night” is getting a bit ridiculous.
Especially ridiculous is how the hacker news crowd seems to be taking these at face value…
There was one the other day with a compelling demo. But when you looked closely at it, it was feeding in the options with the word “best” on the option to pick and a fine tuned model designed to recognise that word…
I thought about writing a blog post that used Jev as a jumping off point to talk about calibration metrics[1], when and why calibration is important[2], and various methods for doing post hoc recalibration[3] of poorly calibrated models. I thought it would be interesting and instructive to benchmark Jev's claims about calibration, find examples where it wasn't (relative to some data set, which shouldn't be hard to find), give an example of how its poor calibration could be exploited, and then show how it could be fixed with, say, isotonic regression. That would give practitioners a roadmap to using Jev successfully without blindly trusting it.
But I'm probably never going to write that article because, as your comment correctly points out, the tenor of the discourse isn't very healthy. Everyone is either "clowning on" Jev (as the kids say), or gulping down industrial quantities of Kool-Aid, or talking about the hype and branding instead of the math. I'll have to find a less contentious example if I want talk about calibration.
Edit: I just found something interesting: there already is a preprint case study[4] along the same lines as the one I outlined, posted just 2 days ago!
Unsurprisingly, they found that recalibration helps enormously, as you'd expect. I guess I could still talk about the Dutch book stuff if I gave a gambling/investing example, but that's pretty well-trodden territory. So now I have even less interest in writing that blog post.
[1]: https://en.wikipedia.org/wiki/Brier_score#Decompositions
[2]: https://en.wikipedia.org/wiki/Dutch_book_arguments
[3]: https://scikit-learn.org/stable/modules/calibration.html
[4]: https://arxiv.org/abs/2609.24052
No I'm s̶p̶a̶r̶t̶a̶c̶u̶s̶ Jev!
Highly suspect of content marketing.
Ends with referring to a product, and saying "this is a parody post", after pretending to make a serious point.
It's not suspect of content marketing. It IS content marketing. Their product is the punchline. It's an ad.
How to write Jev in 25 lines of Python:
Whilst I do like reading these things for technical know how, I can sympathise with the creator of jev who now presumably has to apply an order of magnitude effort to explain why the 100 smaller things done better than this add up to a much better product.
Replace 'explain' with 'sell'. Don't forget that it's a gold rush. There's no reason to sympathize with corporations in their rush for the slice of the pie.
You can be unsympathetic to the corporation’s bottom line while being sympathetic to the human beings that had their work trivialized by some cocky blog post.
You can also just acknowledge that we don't know what they did (and that is because they chose not to tell us). Apparently, until not so long ago Jev would spell out its identity as Q-w-e-n if asked, so we might as well assume that what they did is at least similar to what this blog poster did (who chose to tell us).
As a commercial artist seeing some pretty serious market disruption based on models that used my art and writing to create, without permission, credit, or compensation, I’d have a hard time not punching anyone in the AI business complaining about other people using their work uncredited. Then I’d probably have a real hard time not doing it a few more times. I’m confident I’d settle for just dressing them the fuck down, but it would take some real restraint.
I didn't know that - I revoke my OP if true!
My source: https://news.ycombinator.com/item?id=49783999#49785351 I don't have an account so couldn't check, a friend couldn't replicate with a one-letter attempt (first letter A 50%) - performed by his agent, though, so not sure what he did exactly.
IT's the infamous "OneDrive in 10 lines of code (SFTP)"
While technically correct, it's not the same thing
It's not the same thing but it have advantages Jev don't have like ... being local.
From what I’ve learned about Jev I feel it’s just a very successful marketing campaign to developers not fully understanding data science (and deep learning). It’s nothing new, been around since 2022? Being local is an extreme advantage lol.
Fast and accurate general purpose classifiers DID NOT EXIST before Jev. You could either use an LLM to get a slow and accurate general purpose classifier, or you could use a smaller model to get a fast and inaccurate general purpose classifier, or you could fine tune your own model that would be both fast and accurate, but it wouldn't be general purpose.
But we still don't have a fast and accurate classifier.
All we have is a company that claims to have created one, with no proof.
I've seen enough proof to convince myself, but unfortunately I don't have enough to convince you. Maybe others will publish proper evals.
Except in this case it's not technically correct. Jev's claim is that it's frontier intelligence and these guys are pretending that a 8-bit quantized 0.6B param Qwen model is that. There's no universe in which that claim is technically correct.
Isn’t Jev built on top of Qwen?
Is Jev built on top of 8-bit quantized 0.6B param Qwen model? No.
The one thing I can't wrap my head around with Jev is why they're trying to create that "System One" narrative.
In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two. So by definition what Jev does isn't System One thinking.
If anything, regular programming that automatically executes based on logic, without requiring "thinking" would be "System One".
> In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two.
I'd argue that most human classification is pre-conscious / System One. You see a table, you recognize it as a table without asking yourself "is this a table?"
I guess their marketing implies that it moves classification into system one response time.
I assumed they call it System One just because it's fast and there's no chain of thought/reasoning.
Either way it's an analogy that's bound to be loose as Kahneman's modes are about humans.
>they're trying to create that "System One" narrative
I think you answered your own question. Executives are going to ask two questions, 1) how is this different/why does it matter and 2) how will i use it to make money?
Leya came to market more than a year before Jev, and failed because nobody understood how to use it, and he was unable to market it properly. Jev used this strategy and did not fail.
> In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two.
Huh? I guess that depends on the exact definition of "classification", but I think the bulk of basic classification tasks we make every day to make sense of our surroundings, such as object recognition is definitely done using system 1. So is higher-level "stereotyping" or anything you could described with "I know it when I see it".
Because those responses can be incorrect or even harmful, you would sometimes make use of system 2 to correct them - but that doesn't change that the initial response is from system 1.
Sure, object recognition is System 1, but Jev's own use cases list things like security incident triage, invoice approval, agent escalation, support actions, etc.
Those are generally the kind of tasks that require "System 2" in humans.
To be clear, I think the whole "System 1 vs System 2" framing is a pretty limiting way to think about AI (and thinking in general).
They are "System 2" in humans, because we haven't evolved the necessary wetware circuitry for it to be "System 1".
But with models, we can train them to answer such questions without verbal reasoning.
It's just marketing. More specifically, it's an answer for why their model can't answer questions that require reasoning.
Oh, I thought it was some weird branding thing. So then I looked it up.
"System One" and "System Two" were coined in some pop science book...so back to its usage being a marketing ploy.
They were coined in the extremely influential foundational research the "pop science book" was based on, but that doesn't make it a marketing ploy; System 1 is very fast non-verbal subconscious heuristic operation, and a lot of our brain's classification takes place at that stage. Anyone familiar with the usage immediately had clear ideas about how it'd ideally be used (e.g. heuristics that reduce search space, manage context elements dynamically, etc) vs not (e.g. as a 1:1 replacement for something that could already be easily solved with a classifier).
Uninformed hype for their startup. And they did a great job.
I also built one, but mine uses embeddings. It classifies concepts defined by a collection of positive and negative examples. The classifier model is trained in <1 second using ridge regression. The model itself is exactly the same shape as the embedding, so it works as a concept embedding. Since I already have a dataset, I can use it to do conformal prediction in order to calibrate confidence scores. Jev, on the other hand, has a generic model, not trained on in-domain examples, so its confidence scores are uncalibrated for any non-generic task.
So you might ask: how do I obtain the training examples? Just collect samples and use a coding agent to classify them as match or no match. From time to time you can add more examples to the dataset to have your concept adapt to changes in input distribution. It's all automated, but it only uses LLMs to train concept vectors, after that it works like a regular embedding model with a calibrated classifier on top. It's also 20-30x faster than Jev, free, and runs on CPU.
An illustration of how it defines a concept as opposed to simple cosine similarity: https://github.com/horiacristescu/semlabel/raw/main/images/c...
Can you elaborate a bit on the semlabel? And/or share a repo?
Yes, it is at https://github.com/horiacristescu/semlabel
Thank you. Super impressive.
this is such a trivial thing to do in DSPY, no one bothered to give it a name…
here's 7 lines
there are other options, obviously. you can choose to give it some tools, maybe some reasoning stage before picking a choice, and that's on top of the "reasoning" the llm model already does api sideafter looking closer at the typesafe's jev, i want to point out that i misunderstood the significance of this jev model
it is the latency that makes it significant
What about the probabilities?
so you can pick the choice with the highest probability?
the example uses an external api, and i don't think they return probabilities from those anyway.
I wonder if this could be a good stepping stone to write a local prompt router to optimise what model get what prompt. I.e. if the prompt is just a lookup, send it to haiku, if it's reasoning, send it to opus and if it's implementation send it to sonnet.
I was thinking the same. Haven't tried it out.
I tried it out, but with kev instead of this python script. The issue is that mid session swapping invalidates the cache, which drives cost quite a lot. Ended up loosing money when comparing prompts in most of my transcripts.
If you're not behind a walled garden like i am (vertex), you could probably experiment with routing on effort level instead. Anthropic supports it, but vertex has not added that feature yet.
Looking at the logprobs on tokens works for the local models, but not on the frontier ones. It's been more or less broken since GPT-4o for example. I wrote about it two years ago: https://medium.com/data-science/9-11-or-9-9-which-one-is-hig.... Also, I've done some work in estimating confidence and on rubric evals using the same method, and you actually get better correlation to "real confidence" by just getting the LLM to say it.
Interesting. Have you repeated these experiments with recent models? I'm thinking frontier models APIs have tools/MCPs for math stuff but curious about recent Qwen models, etc.
Haven't tried recent frontier ones, Astra for example doesn't support logprobs emission on the API, and Sol and Luna supposedly support it with reasoning disabled. Haven't tried local models like Qwen 3.8 27b (I'm actually exploring their thinking trace, it's a lot of fun)
What I'm missing here is also type guarantees. I don't think you can do it without token level logic which forces the model to output the tokens from a predefined pool of tokens. A logic like this given some JSON schema is not that difficult to implement. If the LLM must output JSON schema compatible value then you can also add that it doesn't "hallucinate". Which is funny too because just guaranteeing the type does not mean the model does not hallucinate but this is another story.
The type is guaranteed because you only read the probability from your set of choice tokens
You don't actually use the "next token" that the model chooses
That's what the author is doing in this part
This works because the model is always producing probabilities for all tokensAny analogy works at certain abstraction level, and this works with a premise that it's a classifier that makes decisions.
Still good. In practice for Jev the devils in the details. As you all know by now, it's easy to write PoC and understand with AIs (or even manually, which is now a prestious practice).
That demo will get you 80% there
Getting to that 100% or even 99% to JEV level will be hard with all the edge cases, infra, API, communications, etc.
Still a good article.
The website is currently returning "Site not available"
Here is an archive: https://web.archive.org/web/20260923122959/https://www.nobod...
I tried this in a custom ML array programming language with mixed results. I created a literate programming document about it. https://sw-ml-study.github.io/demo-decision-model/concise-he... -- it has some analysis of the python and some other approaches.
I should mention that I am trying to apply a Jev-inspired approch to a trained-from scratch, in-browser typed decision model that attempts to recreate the deterministic classic Eliza program.
Hey, I enjoyed the article! I think it’s helpful to break stuff down as much as possible to nail down what is happening. They did that. The important part is not importing a model or something, it’s what’s happening after. I learned something
Thoughtful demo. If possible, I'd want to see next is whether the probabilities it returns are calibrated and not just whether the top choice is right. I ran a small experiment on this, compared base and instruct models' calibration on option-letter logits, then fine-tuned with Brier loss vs cross-entropy. Have you looked at a reliability curve for this version?
Very nice as a conceptual thing, but there's a bit more to it. I've bolted Gemma 4 onto a custom pipeline for that (https://rcarmo.github.io/projects/go-system-one/) and it's OK-ish (a bit slow on my puny 3060, but I can use it to prototype a bunch of things locally until the Jev mania settles and we have better models).
We have Jev at home
Haha - I did enjoy this read! And there is a point to the whole marketing-dresses-up-stuff that is certainly true. I think it's worth pointing out the other HN story earlier https://news.ycombinator.com/item?id=49765348 about an open-weight model called Laya.
P.S. I am evaluating that model for a production use case where I would have used Jev
Could someone explain how Jev is different from using any old model and constraining the output to "My choice is a/b/c..."?
The argument the article makes is it's not that different. Jev's argument is that they have trained the model to better output probabilities (which is not necessarily a training object of LLMs but we don't actually know that)
Ultimately Jev claims to have a data advantage which is likely where the future lies. They'll have a unique edge in improving general purpose classification / decisioning.
The model can spend more “mental energy” on the decision because it doesn’t have to spend any on phrasing the output.
LLMs are constant time per token, if you only let it choose a/b/c/d then there's nothing else it can "spend".
If you believe the marketing, constraining the output this way can make the model much faster and much more type-safe (the model didn’t give you a fifth choice not present in the choices).
How is that different from just not letting it output option 5 when you sample from the probabilities?
This is a system two model, and not a system one. To get the performance of Jev, and you want to run locally, use Laya. It is up on Hugging Face.
I missed the hypewave so can't say a lot about Jev, but the double standards are entertaining:
About Jev:
> We didn't train a model with Reinforcement Learning for Calibrated Decisions (RLCD) to calibrate the decisions and probabilities (even though they are not always correct).
Only 99% correctness! Borderline unusable!
About their model:
> It classifies: it gets a prompt with choices and outputs probabilities.
You want numbers, it gives you numbers! What more could you want?
a hile ago (when big providers still provided logprobs) i created a VS Code highlighter that visualizes unsure tokens.
Since most chat models want to answer with a human-readable message i think their logprobs are not as meaningful. It would be interesting to see if one choice is like "correct" and if the model wants to choose it more often, cause it might not answer the question but to prose to the user.
You can also go beyond Jev. Qwen 3.5 0.8B is fantastic at basic image classification/question answering (including OCR elements) also. Though rather than looking at logits, I get it to output a structured JSON object and it does simple object classification tasks on a Mac at under 500ms a pop (I forget how far, but I think it's like ~250ms) with good accuracy (depending on task).
[dead]
This is like saying that cars are useless because you can achieve the same thing by removing the cannon from a tank.
I think it's like saying Jev has been trumpeting the invention of a spinning transportation system when we've known such things as a "wheel" for thousands of years. Classification models have been with us far longer than autoregressive LLMs.
Classification models, sure. But classification models that don’t require training data?
What I don’t understand is, why would you not want “reasoning” in a classifier?
Speed and cost are obvious reasons, but isn’t this a tradeoff?
not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.
[dead]
I've built something similar, i am hosting it.
You can test Jev like model at 26B parameter count here (built few weeks ago): https://gambler-relay-us-west1.leo-fish.ts.net/demo (might not stay up for long)
Typesafe compatible API
This is just running on old hardware.
Am I missing something here:
p(y = next thinking+decision token | x = question) != p(y = next decision token | x = question)
The former is what LLMs are trained for, the latter is what Jev was likely trained on (likely used thinking alignment as an auxiliary loss, but not explicitly included in the probability calibration).
First release people are getting more balanced at least. Less 'it's gonna change everything' to 'might be viable use cases'. If I have to look at one more video high lighting google flights, I might loose it though.
Maybe someone can explain why RL is even needed for post training with Jev? We have supervised labels.
I guess it's due to the calibrated decision part (and that's what LLMs tell me).
But I figure some supervised classification post training would still improve the model.
Pretty interesting how a simple example like this makes the idea so easy to understand.
> We didn't train a model with Reinforcement Learning for Calibrated Decisions (RLCD) to calibrate the decisions and probabilities
But aren’t calibrated predictions one of the defining features?
i love that people are trying to make OS jevs but what is the point of doing all this work and not ask your coding agent to do a little benchmarking. selfishly want an open weight model to beat jev here
just found this one https://huggingface.co/spaces/multimodalart/jev-decision-ind...
> "25 lines of python" > "import Solution"
ok
strong "You can build dropbox quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" vibes
You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)
You beat me to it!
Reminds me of my own tiny Lisp interpreter attempts; that moment when it first evaluates a simple expression is pure magic.
This makes me wonder about using Jevko for configuration instead of YAML, especially with such a compact parser.
to be fair a Jev architecture would be better optimized for this particular workflow than an LLM.
but letting an LLM think would trade latency and performance for significant reliability above that of Jev.Latency and compute comparison needed.
Is benchmarking Jev still a ToS violation?
Was it? That would make it unusable in any corporate setting.
I heard it was and just repeated what I heard.
A Google AI prompt says
> TypeSafe AI's Master Customer Agreement explicitly prohibits using the services or model outputs to develop a competing product, perform model distillation, or reverse engineer the service, which generally restricts competitive benchmarking aimed at replicating the model.
It doesn't explicitly prohibit benchmarking by name, but the previous terms (which seem aimed at preventing Jev being used to increase the value of competitive products) does seem to lean that direction.
That said, MsSQL had terms which prevented publishing benchmarks which compared it against other SQL DBs and that wasn't enough to prevent some companies from using it.
Why anyone would want to work for a company who thought so little of their own product that it couldn't stand up to customers using it for normal business processes is beyond me.
Oracle also famously forbids posting public benchmarks of the DB.
Latency-calibration charts or it didn't happen. (LLMs are not optimized for calibration.)
This guy just seems a bit salty
> Email: {email}\n\n{options}<|im_end|>
Why do I have to feed my e-mail into the model?
You don’t that’s a python f string
name = “qurren” print(f”hello {name}”)
kinda feel like the "this is a parody blog post, see these links for better/more complete open implementations of Jev..."
is a legal cya a la "Nathan For You" 's Dumb Starbucks
Startup coming out of 2 years of stealth to be reproduced this easily
It wasn't
This, like the hype of Jev on Twitter, totally ignores accuracy and generality across domains.
In my experience even structured LLM output performs poorly on classifier tasks. LLMs are trained to talk and think longer. If you don't give LLM enough space to reason it would become very dumb.
I'm not saying that Jev is way better, but that people way overindexed cost and speed.
THis is not Jev.
I'm surprised something like Jev came out "so late", but the hype has been ridiculous. Yes, it's a good idea. No, it only helps when fast and cheap are important and I guarantee existing labs will have this figured out in a matter of days.
Add visual understanding, add reasoning and bring down the size to run on my computer. That's when it will be interesting.
So many people that don't understand the tech jumped on the hype train because "it cannot hallucinate" and else. It's crazy.
> I guarantee existing labs will have this figured out in a matter of days.
Exactly.
Jev has no moat, and the incumbents will devour their lunch if Jev actually starts gaining traction.
Jev is so cheap I dont see anyone undercutting them on price. If people/products start using Jev underneath before the incumbents can release something then most likely everyone's jsut gonna stay with Jev
Companies like OpenAI or Anthropic could offer it basically for free as an extension to their existing models. You still need the generator, but having something fast (and maybe even local) to do browser nav would be awesome.
This is basically Temu Jev
Now, can you do it in <200ms for 45 questions at once, have 0% malformed output, and any kind of meaningful benchmark? We’ll wait!
127ms latency on 26B model, here you go: https://gambler-relay-us-west1.leo-fish.ts.net/demo
Running on old home hardware, Jev is probably running on a very powerful cluster.
How it's done: https://news.ycombinator.com/item?id=49813610
Nothing has malformed output if you coerce it's output into a statically defined set of options
> <200ms for 45 questions at once
Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.
So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.
Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.
> have 0% malformed output
Pretty trivial; only the allowed output is selectable :)
So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?
Yeah but a lot of developers who didn't even know that this was a possibility now do, and will probably find use cases for it.
You can probably even share context between questions by cleverly manipulating the attention mask.
Nice idea! Didn't think about that; a single linear memory allocation could do the trick
The principle is this.
I get dishonest vibes from this post? Jev claims to be cheaper/more efficient, and the post claims just to achieve the same functionality.
> note: this is a parody blog post
Jev is .. Hype Slop ... ain't life grand.
Nothing I hate more than bullshit articles claiming X in Y lines of code, only to use libraries abstracting hundreds of thousands of lines of code.
Should they be writing quicksort in assembly as a first step? I think its legitimate in this case given that Jev is likely using the same tools as the example. Showing how easily the core is created using those tools helps to dispel some of the mystery and hype.
Example why its legit:
I just invented a new "Regression Estimate Validator" aka Rev. It takes hundreds of input dimensions, then outputs an interpretable score. Its very fast and statistically robust. Response: Ok but you could just use `pytorch.nn.Linear(d_in, 1)`? True, it is equivalent, but that's concealing millions of lines of hand-tuned math libs, CUDA, python, and other stuff.
The fact that there are many lines of code underpinning the target functionality doesn't make it any harder to use, and doesn't increase the value of the sales pitch for the "new shiny thing" using those few lines of code.
However, I do sympathize with your frustration that people can just say "its 1 line of code" when that line is "invoke API" which is really millions of lines / databases, etc. as a way to dismiss legitimate work without understanding its implications.
> Example why its legit:
Nowhere in your example do you claim that it's written in X lines of code, so that's perfectly fine.
Don't tell me something takes 25 lines of code if it obviously takes much more.
Can you replicate Jev from A to Z in 25 lines? No. Then don't claim to be doing so.
[flagged]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
I'm so sick of seeing these people who "made Jev in 25 lines of Python" or whatever the flavor of the day is. Do you people seriously think that Qwen3-0.6B-Q8_0.gguf is frontier intelligence? If you want to argue that Jev is NOT frontier intelligence, then go make that argument. Don't try to pretend that Qwen3-0.6B-Q8_0.gguf is frontier intelligence. That's retarded.
I wonder if Qwen 3.0 0.6B q8 would have noticed
> note: this is a parody blog post
Don't cut the quote mid-sentence. Here's the full quote:
> note: this is a parody blog post, see these links for better/more complete open implementations of Jev: OpenJev, openjev-sglang, and OpenJev on DiffusionGemma.
Are OpenJev, openjev-sglang, and OpenJev on DiffusionGemma using Qwen3-0.6B-Q8_0.gguf, or did you just want to emphasize the part that was unrelated to your previous response?
Yes, all of those are using small models (not this particular model, but small models nonetheless). Small models are not frontier intelligence.