It uses this model, MoritzLaurer/ModernBERT-large-zeroshot-v2.0, which can be run on pretty quaint hardware. You likely do not need a GPU, probably 2gb ram minimum on top of whatever you are already hosting
What about data residency and retention? They quickly mention ZDR for enterprise accounts only on their support but that’s it. At least to us, that’s a hard showstopper.
Too bad they seem too busy to answer these questions, IMO that feels like a missed early leverage opportunity for them. We benchmarked a bit self hosted alternative, and nothing reach quite the cost/performance of Jev, yet.
I second that. If they would offer zero data retention and optional even EU residency, we would immediately use it. In particular with the new image feature.
It's a big issue for US startups that don't immediately build for GDPR compliance - however there's also a very simple solution that is to deploy on AWS and let AWS handle the heavy workload. Not every startup has the time to digest real compliance efforts that require experts - and implement the 2000 task list that's required, especially for enterprise who will look at that in big details - if you're into big headaches -> enjoy article 45 and next of the GDPR -> https://www.legiscope.com/blog/cross-border-data-transfers.h...
I don't get this whole thing. This sounds like a small model, that's very possible to run on the edge, and it seems like most of the things it does actually are useful there.
The extra funny thing about this is that the target audience is people who use agentic AI systems (system one thinking as a compliment to system two thinking). Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpphttps://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
The main benefit of Jev seems to be broader knowledge? But local models (especially the ones actually trained specifically for decisions, rather than ones that "support" it like embeddinggemma) are probably good enough for what most people use it for. They also seem to be quite easy to fine tune (I haven't tried, but seems to be what people say).
it still needs the full semantic "understanding" of natural language. So I don't see how a smaller model one-shotting a single token could be any better than other bigger models
> Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpphttps://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
I mean, yes, that works at dev time; but going from zero to nonzero local inference components makes an agent implementation suddenly much more complex to ship to production / drastically narrows the set of hosts such an implementation can be deployed onto.
Not sure why you're being downvoted. This is very reasonable to expect people to run themselves if they have an expectation of privacy. There is basically no reason to trust someone at their word about data retention when you can run it yourself as easily as this.
I haven been requesting access to their trust center for weeks, even emailed them about it multiple times only for them to direct me to their trust center.
It would be very useful for some use-case I have around deciding if an input has potentially sensitive data, to avoid sending it to AI providers. Having to call an external API makes this a catch 22.
I love everything about this launch post. They feel like people i would get along with.
And i already sub to jev and i think this launch is going to make me a much more loyal user now lol
> We’ve heard you loud and clear on context lengths. Jev actually supports a much longer context length than we’ve enabled in production. We chose to limit it intentionally because Jev, like all models, is less reliable on longer contexts, and this constraint encourages more effective usage patterns in prod.
> We aren’t your parents though, so while we strongly recommend limiting context length, you no longer have to:
They’re upfront about their decisions and intentions. All of which make perfect sense. What gives it shady vibes for you?
Eg: Longer context has long been known to reduce response accuracy, so it makes perfect sense why they’d limit it especially at launch. Personal evals are in fact more useful than public benchmarks so it makes sense (and is refreshing) that they’re cautioning against them…
Cool to see another model with support for bounding boxes on images. Really wish to see segmentation in a future release; Gemini 3.5 Flash-Lite is mandatory in my use case as it's the best model for semantic image segmentation in my experience.
Using it as part of an agentic CAD workflow, where the model needs to be able to selectively segment from parts of a user's image as it works on a project. Using OpenSCAD so being a general purpose model is very useful to me.
If openscad is the thing Google models has consistently outpermormed the comparable competition in my experiences. Few other models has better physical spatial reasoning skills.
I've found OpenAI models the best (although I haven't tried Argon) for OpenSCAD.
I have found a couple of problems Opus could fix that Sol couldn't (smoothing a three way join between curved edges and a bezier curve face is one I can remember). But it's much slower than Sol and generally doesn't produce as nice results.
There's no reason that an agentic workflow has to use the primary agent model for every intuitive computing task; in fact, that would've sounded absurd before the singularity. Maybe an old flash model is the best in class for this, but A) I doubt it and B) it's still gonna be wildly more expensive.
Assuming I'm not just misunderstanding something critical (happens often!) and you haven't explored this already (very possible!), pointing your agent at this link and asking for some prototypes might be illuminating: https://huggingface.co/models?pipeline_tag=image-segmentatio...
I've had pretty good luck having Gemini or Claude use SAM as a tool. Gemini is really good at identifying things in images, but it doesn't do a great job doing segmentation per se, but SAM on its own can't really decide what you need to segment.
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)
I like to test the decision models with coin flips. This has a well known answer and I haven't seen one give it back. It undermines my trust in every other decision.
This is common knowledge though, and not some deep math problem, small children understand this concept.
If they cannot get something so simple right, while at the same time offered as a decision with likelihoods machine, why should I trust their other answers with "probability" they produce?
I think a better way to view these numbers is that they are "confidence" levels, not probabilities in the sense you are interpreting.
You should never trust the machine. ;) But you can have a level of confidence with it, and you can know where it's strong and weak.
This is a language association machine, not a math and probability machine. It is specifically weak with any real math.
As simple as a coin flip is -- as simple as a child may be able to understand it -- the machine is specifically not designed to produce an accurate mathematical probability.
That's just not how LLMs work. It's not how they're designed. Nor these decision models which use identical principals.
All that aside... agreed regardless. Irreversible choices should never be finalized by such a volatile technology, regardless of how well we benchmark it. If there's even a chance it could be wrong, that is unacceptable.
> not probabilities in the sense you are interpreting
I am not interpreting them that way, that's how they are being sold by the companies behind them. I do not think "confidence" is accurate either. As far as we can tell, and in the open source/model versions, it's derived logit values.
It's essentially, "predict the outcome of a coin flip", "what is the result?", and "heads" and "tails" as the answer options. With the Jev website, they do not show the "this is not a real probability" disclaimer until after you get the answer (misleading imo, should be upfront and more prominent, eg. by the input form like the gemini app)
When I give this to an LLM, it will generally respond with something like 50/50, 100% confidence.
100% agree, have found through tests a big discrepancy between models optimised for SWE benchmarks and usage in prod - there is a somewhat subjective element to it as well in terms of output that can't be quantified effectively (less so with decision models naturally)
I think the point is we need better and open benchmark players which can bring and monetise on domain specific benchmarks for end users and not just selling them to the big labs.
They are? In all my evals at work, Jev seems to win, despite me wanting the open weight versions to.
I imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
Same here, and I have a project 100% hosted in Cloudflare ecosystem (AI Gateway, Workers, Queues, R2, D1, DO, etc) I was so happy when I saw clef. Tested, and it's bad. Slower and wronger than Jev.
Still waiting on Decisions API using Luna, as this what I was using before.
For anyone interested in a GUI around Jev I built Decision Studio for experimenting with it in our self-hosted OSS llms.py [1]
You can define tasks like sentiment analysis, support triage, or claim support, tweak the questions and criteria, and inspect the answers and probability distributions. You can also import existing recipes, save examples, and share recipes with recorded results.
If you're looking for a decent local model, JevK5-4B is excellent. On my own benchmark for the things I work with (seeing if the answer to a question was correct given the evidence that was provided), it was within a point of jev.
I used the model and it was good enough for email classification.
I was wondering if I'm supposed to be doing multiple runs for topic classification with grouping related topics i.e. if its sport,economy or tech related news and then another run for soccer,basketball or something else
or batching them like the News article topic classification example in a single run.
It's also open source, https://github.com/nicobrenner/jeffy it's in the roadmap to build something for the browser. Do you want to take a stab at it and submit a PR?
Going from "I'll need a research team and 5 years" to Jev-based bird classifiers in just 12 years is really nice. I'm happy that some problems get easier, and I'm happy that people make things and share their work so we can all benefit.
Let's test it in a real-world trading scenario. Jev 1.13 had quite good results, only beaten by Microsoft-Decision-1: https://pilabs.space/trading-arena
I use it as a drop in replacement for LLMs I had previously doing filtering and classification in a pipeline. It's loads cheaper and faster with roughly equivalent performance on my own internal tests. The confidence scores (which are also reasonably well calibrated on my data) are a super nice bonus.
My favorite use case, can't take credit for it, saw it on a twitter post early on, is inlining it as a sql function so you can do fuzzy classifier based joins
it's actually more usefull than it looks. it is not much useful on its own but it can play a big role in a complex workflow or custom harness.
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
You can turn a lot of process and workflow systems into deep decision trees.
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
I found an example online, I cannot find the github again, but it was interesting: an instant dynamic generation of React component, you type what you want to build using natural language, and it directly add one to the page.
If I remember well, in the background, it does the multi-choice question: "What is the component requested ? --> Calendar, Slider, ..."
I've used it in a bunch of places, but my favorite use. I have an app (https://plonkit.app/) i'm the only user... because i built it for myself. In the app I have a shopping list feature. When you add an item it auto assigns it to the right shopping list. It's a very simple feature, but with Jev, it's quick, and it stayed simple.
Effort router. Dynamically changes the reasoning level for OpenAI and Anthropic models based on Jev's perceived difficulty of the task to be done. Only these two providers can support this because they both support changing effort level without busting the cache.
Anything that can't be solved with a regex and LLM feels like an overkill. There are surprisingly tons of things that falls in this category. E.g Filter, classification, preprocessing micro decisions, etc.
Well, they do need to try to compete with SubQ’s longer context and image processing that has been coming out ahead in benchmarks, plus the thousand self-hosted options that now exist.
Main thing I need from Jev is high rate limits. I have about 500-600 I need to fire off concurrently. I use Clef and now OpenAI decisions to help make up the difference.
- add to your README links to download the relevant .gguf files – I assume from HuggingFace?
- or even implement your own equivalent of `ollama pull`/`list`/`rm`/etc
- add some CI using Github Actions (or an alternative if you prefer)
- as well as `go install github.com/computerex/dlgo/cmd/dlgo@latest`, also document `go install github.com/computerex/dlgo/cmd/dlgo-vision@latest`, `go install github.com/computerex/dlgo/cmd/dlgo-image@latest`, etc (obvious to anyone experienced with Go, not to many inexperienced with it)
actually, I try running `go install github.com/computerex/dlgo/cmd/...@latest` on my Mac and I get a lot of errors; some of them are because `cmd/gpu-watchdog` is trying to use Windows-specific APIs (e.g. `syscall.NewLazyDLL`) even on macOS
A lot of the week 1 Jev demos involved getting image data into it somehow, via description I guess? Should be interesting to see what we get with 100ms latency decisions on visual inputs. To me the latency / response time is just a huge game changer -- one to two orders of magnitude more responsiveness opens up a large design space.
Interesting way to release this, I'm not sure how to say this, but this has a real "enterprise"/old school taste to it verses how all the other models/AI companies are releasing things.
The font rendering on this site is something else. Is that a stroke being applied to every character? I thought this was a Linux glitch but it exists on iPhone too.
I have to admit, despite my initial scepticism of Jev, it is actually pretty fucking good at making decisions. I’m trying to break it with weird situations and I just can’t. And it’s so cheap!
They don't. Generally we already have trillions tokens worth of classifier datasets so there's no reason to create synthetic data that will be worse than the data it was trained on.
Ok. Obviously these are cool people trying to do something useful, and my real name is on this site so I definitely don't want to come off as hostile. But I have three problems with their whole... thing, here:
1. Adding bounding boxes only compounds the biggest issue with Jev, which is that it's using a massive model to do what we've been using purpose-built[1] models to do for years(/decades), only a little worse and a lot more expensive. This is fine --great, even-- if presented as a tradeoff.[2] And a product that enables "smart if statements" that are "sci-fi" does not sound like a tradeoff at all. In that light, ending the post with facetiously-labeled graphs that just show it beating a bunch of random names (including models with twenty-seven BILLION parameters) on unknown metrics feels borderline dishonest.
2. "We love the community's desire to help make models better!" feels like Manufacturing Consent but with way lower stakes. I turn on various experimental features and pick nightly builds and all that like many of us do, but this feels... well, at best it's kinda offputting rhetoric for a practice I otherwise condone. IMHO.
3. Their job postings go beyond groan-inducing to vaguely insulting. Take this "Data Demon" posting[3], which is presumably for a data scientist (data engineer...?) but is branded around mocking the Turing-award-winning scientists trying to warn the world about superintelligence. Yes, I had the same reaction: what? Why?? It's giving a corporate culture where A) executives feel entitled to shoot from the hip and inject their personal crusades on a whim, and where B) scientists/science isn't appreciated (or, worse: understood). Criticizing the concept of measurement isn't helping, nor is their choice to label output percentages as confidence scores.
I guess I hope I'm misunderstanding, cause they're gonna make millions regardless, I'm sure. I just know that I can't think of when I'd reach for Jev 1.*[4] for a prod usecase, and this is an awfully big company for a prototyping shortcut.
[4]: Which, btw, what happened to semver? Did we stop doing semver cause python did? Did I miss a memo/blog? Not a rhetorical question, I swear. Deprecating minor versions just seems unholy, much less minor version thirteen!
So they held back a bunch of features because they knew they had basically no moat, and now they're trickling out a bit more functionality to try to convince people they are better than others and move fast? And using Apple advert slogans to show how cool they are?
The way that site charges a fee[1] for benchmarking models, makes any benchmarks on it very sus, IMO. TBF, the benchmarking site is probably more profitable than TypeSafe, just like the joke about how WinRAR is more profitable than OpenAI[2]. :)
If you want a local alternative; I added image support to OpenDecision three days ago - https://github.com/deepanwadhwa/OpenDecision
Videos will be next.
I see it runs locally, yes? What are hardware requirements?
It uses this model, MoritzLaurer/ModernBERT-large-zeroshot-v2.0, which can be run on pretty quaint hardware. You likely do not need a GPU, probably 2gb ram minimum on top of whatever you are already hosting
Just added this at the top of the repo. For text only, 1.1GB ram, for images 1.4GB. context windows for text (2 model choices) - 512-8192 tokens.
What about data residency and retention? They quickly mention ZDR for enterprise accounts only on their support but that’s it. At least to us, that’s a hard showstopper.
Too bad they seem too busy to answer these questions, IMO that feels like a missed early leverage opportunity for them. We benchmarked a bit self hosted alternative, and nothing reach quite the cost/performance of Jev, yet.
I second that. If they would offer zero data retention and optional even EU residency, we would immediately use it. In particular with the new image feature.
Maybe it's possible via their Digital Ocean offering https://ideas.digitalocean.com/changelog/now-available-jev-f...
It's a big issue for US startups that don't immediately build for GDPR compliance - however there's also a very simple solution that is to deploy on AWS and let AWS handle the heavy workload. Not every startup has the time to digest real compliance efforts that require experts - and implement the 2000 task list that's required, especially for enterprise who will look at that in big details - if you're into big headaches -> enjoy article 45 and next of the GDPR -> https://www.legiscope.com/blog/cross-border-data-transfers.h...
I don't get this whole thing. This sounds like a small model, that's very possible to run on the edge, and it seems like most of the things it does actually are useful there.
Why does this thing have to be another Web API?
The extra funny thing about this is that the target audience is people who use agentic AI systems (system one thinking as a compliment to system two thinking). Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpp https://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
The main benefit of Jev seems to be broader knowledge? But local models (especially the ones actually trained specifically for decisions, rather than ones that "support" it like embeddinggemma) are probably good enough for what most people use it for. They also seem to be quite easy to fine tune (I haven't tried, but seems to be what people say).
it still needs the full semantic "understanding" of natural language. So I don't see how a smaller model one-shotting a single token could be any better than other bigger models
> Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpp https://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
I mean, yes, that works at dev time; but going from zero to nonzero local inference components makes an agent implementation suddenly much more complex to ship to production / drastically narrows the set of hosts such an implementation can be deployed onto.
> What about data residency and retention?
If you are sending data over the internet to someone else's computer, who you can't or won't audit, you should keep you expectations realistic.
You can be sure that comapnies will only do ZDR if your data is more expensive to keep than its utility to them, even if they claim something else.
That's why we sign DPA in Europe, it's a matter of liability transfer. I agree on your point, but that is not the issue here.
Not sure why you're being downvoted. This is very reasonable to expect people to run themselves if they have an expectation of privacy. There is basically no reason to trust someone at their word about data retention when you can run it yourself as easily as this.
I sent an email, they were radio silent for 2 weeks , and then replied said ZDR is on. So give it a try.
I haven been requesting access to their trust center for weeks, even emailed them about it multiple times only for them to direct me to their trust center.
Damn, I must have had a bad roll then, been a couple weeks here too
It would be very useful for some use-case I have around deciding if an input has potentially sensitive data, to avoid sending it to AI providers. Having to call an external API makes this a catch 22.
[dead]
I love everything about this launch post. They feel like people i would get along with.
And i already sub to jev and i think this launch is going to make me a much more loyal user now lol
> We’ve heard you loud and clear on context lengths. Jev actually supports a much longer context length than we’ve enabled in production. We chose to limit it intentionally because Jev, like all models, is less reliable on longer contexts, and this constraint encourages more effective usage patterns in prod.
> We aren’t your parents though, so while we strongly recommend limiting context length, you no longer have to:
> TypeSafe-Danger-Zone-Long-Context: true
For me it's the reverse, it gives very shady vibes.
Remember the relatioship incentive is skewed, you are giving in your precious data inexchange for random logprobs, that they offer to "cheap".
They’re upfront about their decisions and intentions. All of which make perfect sense. What gives it shady vibes for you?
Eg: Longer context has long been known to reduce response accuracy, so it makes perfect sense why they’d limit it especially at launch. Personal evals are in fact more useful than public benchmarks so it makes sense (and is refreshing) that they’re cautioning against them…
Listen to Diogo's episode on Latent Space. Dude's an interesting guy, if not a tad eccentric.
Ooo i musta missed this in the last email
Will look for it thanks! Good call out :)
I had the opposite feeling tbh
"...there's no way that our team has a full understanding of what is needed to spark delight in the world and build sick sci-fi."
"Size does matter, apparently "
Cool to see another model with support for bounding boxes on images. Really wish to see segmentation in a future release; Gemini 3.5 Flash-Lite is mandatory in my use case as it's the best model for semantic image segmentation in my experience.
Curious why you use Gemini 3.5 Flash Lite instead of SAM/Moondream? Any specific workflows where it outperforms?
Using it as part of an agentic CAD workflow, where the model needs to be able to selectively segment from parts of a user's image as it works on a project. Using OpenSCAD so being a general purpose model is very useful to me.
If openscad is the thing Google models has consistently outpermormed the comparable competition in my experiences. Few other models has better physical spatial reasoning skills.
I've found OpenAI models the best (although I haven't tried Argon) for OpenSCAD.
I have found a couple of problems Opus could fix that Sol couldn't (smoothing a three way join between curved edges and a bezier curve face is one I can remember). But it's much slower than Sol and generally doesn't produce as nice results.
There's no reason that an agentic workflow has to use the primary agent model for every intuitive computing task; in fact, that would've sounded absurd before the singularity. Maybe an old flash model is the best in class for this, but A) I doubt it and B) it's still gonna be wildly more expensive.
Assuming I'm not just misunderstanding something critical (happens often!) and you haven't explored this already (very possible!), pointing your agent at this link and asking for some prototypes might be illuminating: https://huggingface.co/models?pipeline_tag=image-segmentatio...
Semantic segmentation is a lot different, he needs to segment one off things in CAD it sounds like.
I've had pretty good luck having Gemini or Claude use SAM as a tool. Gemini is really good at identifying things in images, but it doesn't do a great job doing segmentation per se, but SAM on its own can't really decide what you need to segment.
> Warning: AI benchmarks. Please don't trust them
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
That’s true, for example here, this benchmark chooses tasks and datasets that Jeffy is really good at, better than Jev or Laya: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)
I generally trust benchmarks but my personal experience in the last 6-9 months is they have little relation to real LLM performance, unfortunately.
I've had better luck just running a few of my own simple tests.
It's almost turning into something as good as standardized testing for humans...helpful indicator, but can be gamed.
I like to test the decision models with coin flips. This has a well known answer and I haven't seen one give it back. It undermines my trust in every other decision.
It's a fun question but the models aren't designed for questions like that.
iirc, Jev has a "jaggedness" page that lists known problem areas with the model, and plain mathematics (which I'd argue this is) is one of them.
This is common knowledge though, and not some deep math problem, small children understand this concept.
If they cannot get something so simple right, while at the same time offered as a decision with likelihoods machine, why should I trust their other answers with "probability" they produce?
Here's another post that shows how fragile or inconsistent, even dangerous, these decision models can be: https://zachlandes.com/trolley-problem/
Don't use decision models with important decisions, especially those that involve human lives or irreversible choices
I think a better way to view these numbers is that they are "confidence" levels, not probabilities in the sense you are interpreting.
You should never trust the machine. ;) But you can have a level of confidence with it, and you can know where it's strong and weak.
This is a language association machine, not a math and probability machine. It is specifically weak with any real math.
As simple as a coin flip is -- as simple as a child may be able to understand it -- the machine is specifically not designed to produce an accurate mathematical probability.
That's just not how LLMs work. It's not how they're designed. Nor these decision models which use identical principals.
All that aside... agreed regardless. Irreversible choices should never be finalized by such a volatile technology, regardless of how well we benchmark it. If there's even a chance it could be wrong, that is unacceptable.
> not probabilities in the sense you are interpreting
I am not interpreting them that way, that's how they are being sold by the companies behind them. I do not think "confidence" is accurate either. As far as we can tell, and in the open source/model versions, it's derived logit values.
> You should never trust the machine
Rule #2 https://susam.net/inverse-laws-of-robotics.html
> coin flip...
LLM's get this one exactly right in my limited testing "50/50, 100% confident"
How do you test that? Are you looking for 50% confidence in an answer? I imagine it’s more than that, I’m curious.
A very simple setup, you can try here for Jev: https://jevplayground.com/
It's essentially, "predict the outcome of a coin flip", "what is the result?", and "heads" and "tails" as the answer options. With the Jev website, they do not show the "this is not a real probability" disclaimer until after you get the answer (misleading imo, should be upfront and more prominent, eg. by the input form like the gemini app)
When I give this to an LLM, it will generally respond with something like 50/50, 100% confidence.
They also link to another post they made about AI benchmarks that I really like: https://typesafe.ai/blog/antibenchmaxxing
100% agree, have found through tests a big discrepancy between models optimised for SWE benchmarks and usage in prod - there is a somewhat subjective element to it as well in terms of output that can't be quantified effectively (less so with decision models naturally)
I think the point is we need better and open benchmark players which can bring and monetise on domain specific benchmarks for end users and not just selling them to the big labs.
Or, you know, it's an easy excuse because they're already behind on many of them.
They are? In all my evals at work, Jev seems to win, despite me wanting the open weight versions to.
I imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
Same here, and I have a project 100% hosted in Cloudflare ecosystem (AI Gateway, Workers, Queues, R2, D1, DO, etc) I was so happy when I saw clef. Tested, and it's bad. Slower and wronger than Jev.
Still waiting on Decisions API using Luna, as this what I was using before.
Run your own evals, folks.
We evaluated the Decisions API at work and it was worse in all dimensions (price, speed, and quality of the result).
Or they use it as marketing wording to get liked by devs.., which is most likely given we know that’s typesafes usp
we've been public about this before the category was even created
[dead]
For anyone interested in a GUI around Jev I built Decision Studio for experimenting with it in our self-hosted OSS llms.py [1]
You can define tasks like sentiment analysis, support triage, or claim support, tweak the questions and criteria, and inspect the answers and probability distributions. You can also import existing recipes, save examples, and share recipes with recorded results.
An OpenRouter API key is required to use Jev.
[1] https://llmspy.org/docs/features/decision-studio
If you're looking for a decent local model, JevK5-4B is excellent. On my own benchmark for the things I work with (seeing if the answer to a question was correct given the evidence that was provided), it was within a point of jev.
In case you need something a bit more lightweight, or want privacy. You can get amazing (better in a lot of cases) accuracy and performance from local models that run and train on CPU: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
I used the model and it was good enough for email classification. I was wondering if I'm supposed to be doing multiple runs for topic classification with grouping related topics i.e. if its sport,economy or tech related news and then another run for soccer,basketball or something else or batching them like the News article topic classification example in a single run.
[flagged]
Anything like this that can just run entirely in browser without needing an external application?
Yes, you can adapt it to run on the browser: http://jeffyclassify.com/blog/mind-reader/
It's also open source, https://github.com/nicobrenner/jeffy it's in the roadmap to build something for the browser. Do you want to take a stab at it and submit a PR?
I just got it to work using python in wasm from pyodide.
Love it! Do you have a gist or a PR?
Sorry only a one shot prompt so probably no value to add
Going from "I'll need a research team and 5 years" to Jev-based bird classifiers in just 12 years is really nice. I'm happy that some problems get easier, and I'm happy that people make things and share their work so we can all benefit.
[0]: https://xkcd.com/1425/
Not only that, but scalable, pay as you go, easy to get solutions.
You could also pay for a ML product, but not so easily and cheap. This enables PoCs and small products.
Slightly unrelated, but I really like the website styling. Looks like an improved version of Linux From Scratch dos.
Swiss Style/Typographic Style.
too bad it's just LARPing, underneath the covers it's the typical bloated Framer site.
Let's test it in a real-world trading scenario. Jev 1.13 had quite good results, only beaten by Microsoft-Decision-1: https://pilabs.space/trading-arena
[dead]
I'm curious what people are building with these models
I use it as a drop in replacement for LLMs I had previously doing filtering and classification in a pipeline. It's loads cheaper and faster with roughly equivalent performance on my own internal tests. The confidence scores (which are also reasonably well calibrated on my data) are a super nice bonus.
For me it's using 1 Jev call to ask 5 questions, instead of what I had before.
What I had before was basically:
- 1 classifier question
- if ok, do some heavy computation
- after heavy computation, another classifier
So now what I do is:
- Jev with the 1st and 3rd classifier
- if ok, heavy computation
I can classify many things at once, and it's cheaper and faster than 1 for sure and 1 maybe classification that I had before.
My favorite use case, can't take credit for it, saw it on a twitter post early on, is inlining it as a sql function so you can do fuzzy classifier based joins
Can you share any more details? That sounds really interesting.
Very weird and very cool :)
^ Same here. so much faster and cheaper than using an LLM to return structured data
it's actually more usefull than it looks. it is not much useful on its own but it can play a big role in a complex workflow or custom harness.
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
Automatic tagging of images is what I'm doing with it. I'm using a modified Gemma model, which ingests images (and audio, for that matter).
You can turn a lot of process and workflow systems into deep decision trees.
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
[1] - https://github.com/prasanthj/duckdb-dual-cognition
[2] - https://demos.isotopes.ai/demo/apps/chain-command
I found an example online, I cannot find the github again, but it was interesting: an instant dynamic generation of React component, you type what you want to build using natural language, and it directly add one to the page.
If I remember well, in the background, it does the multi-choice question: "What is the component requested ? --> Calendar, Slider, ..."
https://shapeshiftui.vercel.app/ sounds similar.
I've used it in a bunch of places, but my favorite use. I have an app (https://plonkit.app/) i'm the only user... because i built it for myself. In the app I have a shopping list feature. When you add an item it auto assigns it to the right shopping list. It's a very simple feature, but with Jev, it's quick, and it stayed simple.
Effort router. Dynamically changes the reasoning level for OpenAI and Anthropic models based on Jev's perceived difficulty of the task to be done. Only these two providers can support this because they both support changing effort level without busting the cache.
Anything that can't be solved with a regex and LLM feels like an overkill. There are surprisingly tons of things that falls in this category. E.g Filter, classification, preprocessing micro decisions, etc.
It’s good for automatically categorising or reviewing stuff which is free text.
Judging from the tilte, I guess dreams.
Silly toy experiments: https://letsplay2048.lol/
scoped classification
Sounds like an incremental change with context size change, new "box" type, and composable images.
As it's minor change did not expect much, but having a higher context helps for sure. I will own the errors for passing bigger context.
Well, they do need to try to compete with SubQ’s longer context and image processing that has been coming out ahead in benchmarks, plus the thousand self-hosted options that now exist.
Main thing I need from Jev is high rate limits. I have about 500-600 I need to fire off concurrently. I use Clef and now OpenAI decisions to help make up the difference.
Shameless plug to my own inference engine: https://github.com/computerex/dlgo
- Inference for a bunch of open weights architectures
- Diffusion image generation fir Z ai turbo
- Vision input into LLM for inference
- Vulkan/fast CPU, optimized kernels
- Playground for testing
- Last but not least, system one fast inference mode with same API semantics as Jev
Decision models work by using a backbone model like qwen 3.5 9b, along with a special linear head that provides the readout for the forward pass.
Suggestions:
- add to your README links to download the relevant .gguf files – I assume from HuggingFace?
- or even implement your own equivalent of `ollama pull`/`list`/`rm`/etc
- add some CI using Github Actions (or an alternative if you prefer)
- as well as `go install github.com/computerex/dlgo/cmd/dlgo@latest`, also document `go install github.com/computerex/dlgo/cmd/dlgo-vision@latest`, `go install github.com/computerex/dlgo/cmd/dlgo-image@latest`, etc (obvious to anyone experienced with Go, not to many inexperienced with it)
actually, I try running `go install github.com/computerex/dlgo/cmd/...@latest` on my Mac and I get a lot of errors; some of them are because `cmd/gpu-watchdog` is trying to use Windows-specific APIs (e.g. `syscall.NewLazyDLL`) even on macOS
Wonderful feedback. I implemented all of above.
I'm aware this is an unproductive comment, but why are all their releases so cringe? It's like they're written by a Reddit mod or something...
> For the builders AND dreamers
> Enter the danger zone at your own risk.
> Size does matter, apparently
> We’re trying something new out here: we’re going to treat you guys like adults
> Lots of people love to ask what kind of sci-fi we’re releasing next
That said, their docs are great and don't seem to include all the Reddit cringe.
I walked away with _mostly_ the opposite impression. They have a specific tone of voice! I think that's good.
I've yet to see any company match the copy writing skills of Valve though.
It's too bad the tone is of a guy I do not want to be around lol
Fascinating.. one could say that goes both ways? Came off refreshing (cut the ai marketing bs).
Toobad the fonts are ridiculous bad though. Relax.
it’s so authentic
A lot of the week 1 Jev demos involved getting image data into it somehow, via description I guess? Should be interesting to see what we get with 100ms latency decisions on visual inputs. To me the latency / response time is just a huge game changer -- one to two orders of magnitude more responsiveness opens up a large design space.
A shameless plug for my engine which supports different backends, and lets an LLM create rules on the fly for it
https://github.com/jlt-commons/lev
Interesting way to release this, I'm not sure how to say this, but this has a real "enterprise"/old school taste to it verses how all the other models/AI companies are releasing things.
Very refreshing.
of all comments, "enterprise" taste is among those I'd least expect
I haven’t used jev yet. I’m curious if it can be used for entity labeling or extraction in text somehow?
I see box is now a thing but no span?
So render text to image then use box right?
[flagged]
The font rendering on this site is something else. Is that a stroke being applied to every character? I thought this was a Linux glitch but it exists on iPhone too.
I have to admit, despite my initial scepticism of Jev, it is actually pretty fucking good at making decisions. I’m trying to break it with weird situations and I just can’t. And it’s so cheap!
I Love this > "Warning: AI benchmarks. Please don't trust them"
Nice release post
How does JEV prevent distillation?
They don't. Generally we already have trillions tokens worth of classifier datasets so there's no reason to create synthetic data that will be worse than the data it was trained on.
I don't think they can. But they probably monitor usage and test for patterns to then block/ban accounts
[flagged]
the discord jam was awesome! kudos to these guys!!
"Our philosophy: add a dark outline on all text on our website so it becomes basically unreadable."
Ok. Obviously these are cool people trying to do something useful, and my real name is on this site so I definitely don't want to come off as hostile. But I have three problems with their whole... thing, here:
1. Adding bounding boxes only compounds the biggest issue with Jev, which is that it's using a massive model to do what we've been using purpose-built[1] models to do for years(/decades), only a little worse and a lot more expensive. This is fine --great, even-- if presented as a tradeoff.[2] And a product that enables "smart if statements" that are "sci-fi" does not sound like a tradeoff at all. In that light, ending the post with facetiously-labeled graphs that just show it beating a bunch of random names (including models with twenty-seven BILLION parameters) on unknown metrics feels borderline dishonest.
2. "We love the community's desire to help make models better!" feels like Manufacturing Consent but with way lower stakes. I turn on various experimental features and pick nightly builds and all that like many of us do, but this feels... well, at best it's kinda offputting rhetoric for a practice I otherwise condone. IMHO.
3. Their job postings go beyond groan-inducing to vaguely insulting. Take this "Data Demon" posting[3], which is presumably for a data scientist (data engineer...?) but is branded around mocking the Turing-award-winning scientists trying to warn the world about superintelligence. Yes, I had the same reaction: what? Why?? It's giving a corporate culture where A) executives feel entitled to shoot from the hip and inject their personal crusades on a whim, and where B) scientists/science isn't appreciated (or, worse: understood). Criticizing the concept of measurement isn't helping, nor is their choice to label output percentages as confidence scores.
I guess I hope I'm misunderstanding, cause they're gonna make millions regardless, I'm sure. I just know that I can't think of when I'd reach for Jev 1.*[4] for a prod usecase, and this is an awfully big company for a prototyping shortcut.
[1]: https://huggingface.co/models
[2]: of dev speed for cost and accuracy, which is lowkey another way to say efficiency and efficacy. Two pretty massive things to trade!
[3]: https://jobs.ashbyhq.com/typesafe-ai/d315d436-e517-4f17-82bb...
[4]: Which, btw, what happened to semver? Did we stop doing semver cause python did? Did I miss a memo/blog? Not a rhetorical question, I swear. Deprecating minor versions just seems unholy, much less minor version thirteen!
[dead]
[dead]
I am starting to like Typesafe more and more lately. It's plainly simple but the next time I see the same problem, Jev just fits right.
> Surprise: the Jev you all know and love (jev-1.13.0) was multimodal all along!
It really feels like they're just admitting that they had it disabled because they needed investors to subsidize turning it on.
So they held back a bunch of features because they knew they had basically no moat, and now they're trickling out a bit more functionality to try to convince people they are better than others and move fast? And using Apple advert slogans to show how cool they are?
With the speed they're adding features shows you're correct.
They need to patent it.
Which others?
There are several public benchmarks for decision models where they don't top every single one. https://benchmarkheaven.com/jev-models/all
I guess their cheeky "Warning: AI Benchmarks, don't trust them" is their way out of that one.
The way that site charges a fee[1] for benchmarking models, makes any benchmarks on it very sus, IMO. TBF, the benchmarking site is probably more profitable than TypeSafe, just like the joke about how WinRAR is more profitable than OpenAI[2]. :)
[1]: https://benchmarkheaven.com/submit
[2]: https://news.ycombinator.com/item?id=49797811
https://huggingface.co/spaces/multimodalart/jev-decision-ind... is also ranking just for the love of the game.
Given that benchmarks are useful, why not give them economic value?
There's nothing wrong with them wanting to capture some economic value, but the way they're monetizing it is a slippery slope of incentives.
OTOH, I trust the benchmarks from Apolinario's HF space that you've linked to.