A decision model takes your data and typed questions, and returns a probability in a fraction of a second, with no text to parse. Which decision model? compares all of them. This is the family I'd try first on a client system, for three reasons that have little to do with the benchmark.
The first is the exit. The category is weeks old, and I don't want a client depending on a closed model from a company that may look different in a year. Baseten's weights are MIT, so the worst case is a GPU of your own. The second is the paperwork: zero data retention by default, SOC 2 Type II and HIPAA, which shortens procurement. The third is that the three models share one API, so you can start high and move down by changing a name.
Then there's the score. On JevBench, mammoth-1 leads the decision models I compared at 85.2, with the best calibration at 94.3. That benchmark is one person's project, so treat it as a lead, not a verdict, and test on the client's own labelled cases.
This is how I'd wire it for a client: mammoth-1 behind a thin adapter, called from a Lambda with the key in Secrets Manager, and a threshold per question. Above it, the code acts. Below it, an LLM or a person looks. If the volume makes the bill matter, test elephant-1 on the same cases and move down only if it holds.
I'd stop short of calling it my default. I haven't run it in production, nobody has tested these models against injection, and they've only been out for weeks. That's what would have to change.
Use it when
- A wrong answer costs money, and the data may leave Australia: mammoth-1 has the best text score and calibration in the comparison.
- The same job at high volume: elephant-1 has the same API and terms, at about a third of the price and a lower score.
- A client needs zero data retention, SOC 2 Type II or HIPAA without a long negotiation: Baseten's zero retention is on by default for Model APIs.
- You want a way out: the weights are MIT, so the same model can run on your own GPU in Sydney if the client's rules change.
Don't use it when
- Data that must stay in Australia on the hosted API: no Sydney region is documented. Self-host the weights in ap-southeast-2, or use Claude on Amazon Bedrock's AU profile with a schema.
- Text written by people trying to fool you, until it's been tested: none of these models was in DecisionRobust, the one independent injection test. d1-ultra did best there.
- joey-1 as the only gate on anything: it scores 35.91 on the Decision Index's private tests, against 63.3 on its public one.
- Long documents: the hosted limit is 16,384 tokens, and the state counts in every question. Perplexity's Decider takes 256K.
- You need the reason as well as the answer: decision models don't write text. Use an LLM.
Pricing
Per million input tokens, output free: mammoth-1 $0.10, elephant-1 $0.035, joey-1 $0.01. At 500 tokens a decision, a million decisions cost $50, $17.50 and $5. Dedicated GPUs are billed by the minute.
Gotchas, with fixes
Gotcha: Hosted rate limits are low: 120 requests a minute on Pro, about 2 a second. A request can also get a 429 when there's no capacity.
Fix: Ask for Enterprise limits or a dedicated deployment before launch, put several questions in one request, and retry with backoff.
Gotcha: Slugs end in -latest, so the weights can change under you, and no deprecation policy is published.
Fix: Keep a labelled test set and re-run it on any change. If you need frozen behaviour, self-host a pinned revision of the weights.
Gotcha: A request over the 16,384 token limit is refused whole.
Fix: Count tokens first, keep the state short, and use max_len to fail fast. Refusal beats Clef's silent truncation, but your code has to handle it.
Gotcha: Gateways don't carry everything: OpenRouter doesn't list these models, and Vercel lists them as text-only, though Baseten documents images.
Fix: Call Baseten directly for now, behind a thin adapter so you can switch.
Gotcha: Image limits and the token cost of an image aren't published, and none of these models is on an image benchmark.
Fix: Measure usage.input_tokens on your own images, and keep a person behind any image decision.
Provisioning
Pulumi via a Terraform provider · Community Terraform provider
basetenlabs/baseten v0.1.0, community tier, for dedicated deployments. The shared Model APIs need only an API key: keep it in AWS Secrets Manager.
Instead, consider
- Cloudflare Workers AI Your code already runs in a Worker: Clef Flash is one binding away, at $0.038 per million, with a lower score.
- Jev To see what started this category, in an experiment. Early access and closed, so there's no way out.
- Anthropic API (for client systems) You need more accuracy or a written reason, and can afford seconds: an LLM still wins on accuracy.
- Amazon Bedrock The data must stay in Australia and you won't run a GPU: Claude on the AU profile with a schema, with no probabilities.
Decision guides
The concept behind it
Sources
- Baseten decision models docs
- Baseten pricing and limits
- mammoth-1 in Baseten's library
- JevBench, open-weights board
- Decision Index
Last reviewed by Nicolas Dao