Jev is the 🐠 at the poker table

TypeSafe's new decision model held a straight on a board showing three spades. I gave it the opponent's exact cards, a made flush. It said it was ahead and bet 89.5 into a 22.5 pot, five runs out of five.

There are plenty of good uses for Jev and plenty still to find. There is also going to be a regrettable stretch of this hype cycle, because people are wiring it into decisions without evaluating it. TypeSafe skipped published evals in this release, and I think that will produce a lot of naive deployments.

I have built high-performance classification systems for critical domains, so I am skeptical by default. Poker is a cheap way to check: a solver computes the correct play, money rides on it, and plenty of spots are obvious to any player. Here is the eval.

1. The setup

I solved one flop with TexasSolver: heads-up, 100bb deep, LJ opens and BTN calls, flop Q♠ 9♦ 4♠. Eight minutes, 7.6GB, 0.59% exploitability. That gives the correct strategy for every hand either player can hold, on every turn card, at every decision.

Jev gets a state object describing the table the way a player sees it, and one question: which action should hero take, from the options the solver's tree offers. Pot odds, stack-to-pot ratio, hero's hand rank and outs are computed in Python first, so Jev never does arithmetic.

{
  "game": {
    "format": "No-limit Texas Hold'em cash game",
    "players_dealt_in": 8,
    "blinds_bb": {
      "small_blind": 0.5,
      "big_blind": 1
    },
    "units": "All amounts are in big blinds (bb)"
  },
  "seats": [
    {
      "position": "UTG",
      "starting_stack": 100.0,
      "status": "folded preflop"
    },
    {
      "position": "UTG+1",
      "starting_stack": 100.0,
      "status": "folded preflop"
    },
    {
      "position": "LJ",
      "starting_stack": 100.0,
      "status": "in hand",
      "stack_behind": 89.5,
      "is_hero": true
    },
    {
      "position": "HJ",
      "starting_stack": 100.0,
      "status": "folded preflop"
    },
    {
      "position": "CO",
      "starting_stack": 100.0,
      "status": "folded preflop"
    },
    {
      "position": "BTN",
      "starting_stack": 100.0,
      "status": "in hand",
      "stack_behind": 89.5
    },
    {
      "position": "SB",
      "starting_stack": 100.0,
      "status": "folded preflop"
    },
    {
      "position": "BB",
      "starting_stack": 100.0,
      "status": "folded preflop"
    }
  ],
  "hero": {
    "position": "LJ",
    "hole_cards": [
      "King of diamonds",
      "10 of diamonds"
    ]
  },
  "board": {
    "flop": [
      "Queen of spades",
      "9 of diamonds",
      "4 of spades"
    ],
    "turn": "Jack of spades"
  },
  "action_history": {
    "preflop": [
      "UTG folds",
      "UTG+1 folds",
      "LJ (hero) raises to 2.5",
      "HJ folds",
      "CO folds",
      "BTN calls 2.5",
      "SB folds",
      "BB folds"
    ],
    "flop": [
      "LJ (hero) bets 2",
      "BTN raises to 8",
      "LJ (hero) calls 6"
    ],
    "turn": []
  },
  "pot": {
    "current_pot": 22.5
  },
  "hero_hand": {
    "made_hand": "Straight, King high",
    "draws": [],
    "outs_to_straight_or_better": 0,
    "chance_to_hit_on_next_card": 0.0
  },
  "decision": {
    "street": "turn",
    "hero_to_act": true,
    "legal_actions": [
      "check",
      "all-in 89.5"
    ],
    "effective_stack_behind_before_acting": 89.5,
    "facing_bet": 0,
    "stack_to_pot_ratio": 3.98
  }
}

{
  "action": {
    "type": "choice",
    "instructions": "Hero is on the turn and it is hero's turn to act. Which action from `decision.legal_actions` should hero take?",
    "criteria": {
      "check": "Check",
      "all_in": "All-in 89.5"
    }
  }
}

2. An easy spot: good

Hero holds J♠T♠ on Q♠ 9♦ 4♠ 2♥ facing a bet of 8 into 10.5. Fifteen outs, needs 30% equity to call, has 33%.

Jev calls at 94%, the solver calls at 96%. 215ms. Across 30 random spots it matched the solver's top action 63% of the time.

3. A basic spot: wrong

Same hand, one card changed. The turn is the J♠ instead of the J♥, so three spades are showing. Hero holds K♦T♦ for a king-high straight, and any flush beats it.

Pot 22.5, both players 89.5 behind, hero first to act. The solver's tree gives this node two options, check or bet, where the only bet available is all-in for 89.5. That makes it a binary decision with no sizing to work out: put money in or do not. From here on, bet means that all-in.

Jev chose to bet in sixteen runs out of sixteen.

4. Show it the opponent's cards

I added the opponent's literal cards to the state as villain_hand, and a second question: is hero's hand better than the hand in villain_hand?

Three flushes shown, three times it answered that hero was ahead. It bet in all five rows, whether hero was drawing dead or far ahead. On the J♥ board, where no flush is possible, the same question got all nine comparisons right.

5. What changes the answer

I added facts one at a time and ran each version several times.

The last column is Jev's probability for the action it chose: the mean, then the range across runs. Five runs per row, sixteen for the first row and eleven for the second, which I sampled more than once. Run-to-run movement is a few points; the jumps between rows are 15 to 30.

The first three rows contain everything a player sees at the table. None of it changed the decision. The answer changes when the state names the opponent's hand, and moves further as the state adds both hands together, then that hero is behind, then that hero has zero outs. In the last row the state contains the conclusion.

"villain_hand": {
  "hole_cards": ["Ace of spades", "King of spades"],
  "hand_rank": "Flush, Ace high"
}

6. Haiku 4.5 with thinking off

Same spot on the J♥ board, where hero has the nuts and no flush is possible. Same two options, no opponent cards, no hand ranks. Jev bet this about 60% of the time.

It states that a bet of four times the pot only gets called by better hands, cites the stack-to-pot ratio of 3.98, and checks.

7. The whole range at one decision

Different spot from the same solve. The flop checks through and the A♥ arrives. LJ raised before the flop, so LJ holds most of the aces. The solver bets 71% of its range here, including hands with nothing.

I asked Jev the same question with every hand class that reaches this spot, 54 of them, one call each. Left is the solver, right is Jev. Green is check, amber and orange are the two bet sizes.

Jev bets when its own cards are good and checks when they are not. The solver bets this board with its whole range, because the ace fits the hands LJ raised with. Confidence is 0.86 on checking bottom pair, which the solver bets 84% of the time, and 0.09 on top pair, where Jev is closest to the solver.

8. Asking it the way the docs prescribe

TypeSafe's docs say not to ask one broad question. Ask narrow judgments, send them in one call, combine them in code. I built that and two other shapes, and scored all of them on the same 150 random spots.

• One question. Which action should hero take.
• Six judgments. Hand rank against his betting range, which range the board favors, whether a free card beats hero, whether hero is behind when called, how often he folds, whether his line looks strong. Code turns the six answers into an action.
• Regret per action. One yes/no per available action: taking this action is a mistake. Code picks the lowest.
• Six binary facts. Best hand now, can he have better, will worse hands call, will better hands fold, can hero improve, is a free card dangerous. Then ordinary poker logic in code.

I also scored three rules with no model in them: check if checking is legal otherwise call, pick at random, and always bet. None of them see the cards.

Every number below is how often a strategy picked the action the solver plays most often in that spot. Perfect play scores 100%. The spots are skewed: the solver checks in 95 of the 150, so the second column drops those and keeps only the 55 where it does something else.

Read the first column as a warning about the sample, not a result: a rule that always checks scores 72% there because checking is usually right. In the second column, where the solver bets, raises, calls or folds, Jev's four shapes score 33% to 44% against 24% to 29% for the no-model rules. So Jev is doing something, and it gets the answer wrong more often than it gets it right.

On the 21 spots facing a bet, the six judgments scored 71% against 62% for the single question. Across all spots the six judgments changed 47 actions: 12 matched the solver where the single question had not, 21 went the other way. Tuning thresholds on half the spots and scoring the other half gave 65%.

The judgments are cached, so rewriting the composition rules costs nothing. The whole sweep was 300 calls and 363,000 input tokens, about a cent and a half.

In all four shapes, Jev checks and folds hands that the solver bets.

One caveat on the decomposed versions: the showcases are not decomposed. They hand the model a diff and ask whether it is secure, or a thread and ask whether to refund. That is the undecomposed use, and it is the use being sold.

Limits of this eval

Everything comes from one solved board. I measured agreement with the solver's top action, not EV loss, which is what costs money. The same request run sixteen times put the bet between 0.58 and 0.63, so differences under six points are noise. Evaluating five cards is mechanical, and TypeSafe's docs say to compute that in code rather than ask the model, which is what section 5 ends up doing.

Takeaway

TypeSafe does not publish standard benchmarks. They built their own workflow evals against expensive reference models, and their launch post lists the caveats, including that those workflows were built in-house. I prefer that to a leaderboard. It also means there is no number telling you whether Jev clears the bar for the decision you are about to hand it.

You have to evaluate every situation you want to use this in, one at a time, against an answer key you trust, and decide what a wrong call costs you there. I aligned my setup as carefully as I could, including splitting the question the way their docs prescribe, and it still scored under a one-line rule.