Cuarenta AI

September 24, 2026

• Cuarenta
• Reinforcement Learning
• Incomplete Information Games
• Monte Carlo Tree Search

About Cuarenta and why building this?

Cuarenta is a traditional Ecuadorian card game. It’s specially played (and celebrated) on Fiestas de Quito (Quito’s foundation holidays).

I was searching for an excuse to use reinforcement learning (RL) to solve something uncommon. That led me to apply it to cuarenta: I ended up deciding to build a cuarenta AI engine.

Here’s a summary of the game’s rules:

• 4 players (it can be played by 2 as well, but that’s completely uncommon), with 2 teams. Players are positioned in circle alternating according to team, so you always sit in front of your teammate.
• It’s played with a 40-card classic deck: 4 suits - brillos (diamonds), corazones rojos (hearts), corazones negros (spades) and tréboles (clubs) - and 10 ranks per suit: 1 through 7, plus the J, Q and K.
• Each player is dealt 5 cards. The rest of the deck stays face down and is dealt again when the hands run out.
• On your turn you must play one card from your hand; you either leave it on the table or use it to capture:
  • a table card of the same rank;
  • a pair of table cards whose values add up to your card’s value (the J, Q, K cannot do this one);
  • a run of consecutive ranks (the run bridges the gap between the 7 and the 10, since there is no 8 or 9 in this deck);
  • if your capture takes the card that is sitting on top of the table, the one that was just played, that is a caída and pays 2 points;
  • if your capture takes everything that is on the table, that is a limpia, also 2 points (it only pays while your team is below 38 points).
• Everything a team captures goes to its cartón (collection of their captured cards). When the deal ends, the cartón is counted: At card 20 you start to get 1 point for each extra card, starting from 6 points, rounded up to an even number. For instance, if you count 19 cards, you get no cartón points. If you have more than 19, then you count them like this: 20, 6 points; 21, 7 points; 22, 8 points; 23, 9 points. If the points are odd, then you approximate to the next even number (7 -> 8 points, 9 -> 10 points)
• The first team to reach 40 points wins.

So: cooperative (you and the player in front of you), turn-based, imperfect information (you cannot see anybody else’s hand) and small enough that a single machine can play millions of games.

TL;DR

I built an AI engine for the cuarenta game. The engine consists of a neural network trained by self-play reinforcement learning that plays with search: an Information Set Monte Carlo Tree Search (ISMCTS) samples hidden-card worlds consistent with what the acting player knows, searches them, and aggregates the outcome from multiple simulations. The same network guides the search (the policy) and evaluates its leaves (the value) in an AlphaGo/AlphaZero-inspired style. Against a handwritten heuristic-based player (that plays like a human that knows the rules and takes advantage of rewarding actions, but does not have very good memory of past turns), the raw policy on its own is a coin flip (49.0% ±3.5% over 200 games), while the same weights played with the search reach 77.5% ±3.5% over 200 games (72.0% at 128 search simulations). The engine is also shipped as a playable game: a Go service that runs the exported model (ONNX), and a SvelteKit UI where you can play against it. The two experiments that did not work so well are as informative as the one that did: a ReBeL-style solver, and a policy with no search at all trained with PPO.

All the three models played against the heuristic agent and against each other head to head, 100 games each:

The search-based model beats all. The analysis section has the per-game metrics, the paired deals, the cost and others.

Here you can see the training charts, as produced by the evaluation against the heuristic agent. _for suffix corresponds to the currently-training agent and the _against suffix corresponds to the heuristic opponent:

A super quick and domain-specific introduction to AI if you don’t have idea on this topic

Let’s define artificial intelligence as the capacity to solve problems and accomplish goals autonomously. From a basic perspective, a problem is solved by reasonably searching for the best (according to the goal) solution for the problem and implementing it.

For instance, a player has to pick (i.e. search) the most convenient card to play between all its playing options according to what it sees, remembers and hypothesises (about what it cannot see) considering what has happened in the game up to that point with the goal of winning the game.

As we’ll see, search has a special importance.

Three important concepts:

• State: everything that defines the position of the game (whose turn it is, the table, the hands, the scores).
• Policy: the rule that says which action to take in a state. Here, it is a neural network: you give it the state and it gives you a score per candidate action.
• Value: the rule that says how good a state is. A network too: you give it a state, and it returns how much that position is expected to pay off.

The value tells you which positions are good, the policy tells you which of your actions lead to them, and the search is the part that actually tries several of them (a few moves deep) before committing to one.

The other concept I need is the difference between knowing and believing. In a fully observable problem (chess, Go, most Atari games) the state is the same for everybody, and people speak of an MDP, a Markov Decision Process. In cuarenta you cannot see the other players’ hands: the actual state is hidden, and your best information is your own hand, the table, the captured piles and the history of what everybody has played. That is a POMDP, a Partially Observable MDP. The set of all the states that are consistent with what you know is called an information set, and the art of playing these hypothetical games is mostly to reason over that set instead of over one state filled of unknowns.

Development

I had already used DQN and PPO for simple problems here and there, but all of them were single player games (or problems) with a completely visible MDP (Markov Decision Process). Cuarenta is different with respect that multiplayer, cooperative, incomplete information game (you cannot see the other players’ hands) being it represented by a Partially Observable Markov Decision Processes (POMDP).

For complete information games like chess, go, Some atari games, etc., there’s been famous innovations in recent years, so there’s a convenient blueprint to follow. Systems like AlphaGo and AlphaZero put together neural networks and search trees (e.g. Monte Carlo Tree Search a.k.a. MCTS) to solve them.

Using the MCTS approach for search made sense to me as a good starting point because it’s more efficient than minimax in a heavily-probability-based problem like cuarenta (e.g. when trying to determine what cards are in opponents’ hands) due to its “let’s not strictly try all the possibilities, but try the most promising”. Still, there was something important to consider: MCTS is supposed to work on complete information search spaces and, as said before, we have incomplete information in cuarenta. So, looking for possible alternatives, I found out about the Information Set Monte Carlo Tree Search (ISMCTS) algorithm.

ISMCTS differs from the classic MCTS in that each simulation is run on a determinization — one concrete world drawn from the information set, i.e. one guess of the opponents’ hands that is consistent with everything you have seen — and the results of the simulations are aggregated at the information set level. You do not pick the action that is best in one guessed world; you pick the action that is best on average over the worlds you sampled.

So the engine I built is AlphaZero’s shape - network plus search, trained by self-play - with the search replaced by ISMCTS. Concretely, the loop is:

1. Observe. The public information is: my own hand, the cards on the table, both teams’ captured piles (the cartón), the scores, and the history of plays (plies). Everything else is hidden.
2. Hypothesise. Fill in the hidden part: deal the unseen cards into the other hands in a way that is consistent with the history (how many cards each player has played, what has been captured, what is left in the deck). ISMCTS does this by sampling; the network learns to skip the explicit reasoning and predict from the observation alone.
3. Search. For each candidate action, run simulations: play the action, then keep playing both sides with the policy, using the value network to score the leaves. Repeat over many sampled worlds and average, so the action survives being wrong about the hidden cards.
4. Act and learn. Play the most promising action, and train the network on the search’s own improved action distribution (the policy target) plus the actual outcome of the game (the value target), then use the improved network in the next self-play iteration.

What I tried, and what actually worked

Plain PPO no search.

As a baseline I trained the same network with PPO self-play and no search at all. It is a perfectly fine learner and it is much cheaper to run (one forward pass per decision), but it plateaued at 49.0% ±3.5% against the heuristic agent over 200 games: a coin flip.

ReBeL

ReBeL is an interesting idea for exactly incomplete info games: it replaces the game tree over hidden states with a public tree plus a belief over states, and it trains a value network whose job is to be the value of a public state, refined by solving subgames. I implemented an adapted version for cuarenta and it did not bring good results. My reading of why: ReBeL is designed to work on two-player zero-sum games with a well-defined game value per public state, and its training loop depends on re-solving subgames to convergence. That inner solve is the method, and it needs a value network that is already decent, plus a lot of compute per iteration. In my setting (four players, two teams, a big hidden-card space, one machine) I never got the value network to the point where the subgame solving was worth it, so the whole thing was an expensive way to be worse than ISMCTS.

ISMCTS + actor/critic net.

This is the one that works. Playing the trained network against the heuristic opponent, 100 games, got these results:

simulations per turn	win rate vs the heuristic
0 (raw policy)	49.0% ±3.5% (200 games)
32	57.0%
64	59.0%
128	72.0%
400	77.0% ±4.2% (100 games)

The cost: about 10 ms per decision at 128 simulations and ~31 ms at 400, single-threaded. That was cheap enough to serve in the web game.

How the environment observations are structured and handled by the policy net

One implementation detail shaped everything else: the legal action set changes every turn. Sometimes you can only dump a card on the table; sometimes you can pick between two captures of different sizes; the same card can capture different sets. A classification head over a fixed list of actions would be a big mess of mostly-impossible slots.

So both pipelines share one representation: a 208-dimensional observation plus 84 features per candidate action, padded to a maximum of 64 slots with a mask saying which slots are legal.

The observation is five 40-bit vectors (one bit per card in the deck) for my hand, the table, my team’s cartón, the opponent’s cartón and the last card placed — that last one is the caída reference — followed by my team’s score, the opponent’s score, whose turn it is, how many cards are left in the deck, and how many plays each player has made this deal (for determinization calculations). The 84 action features are, in order: the 40 bits of the card I would play, the 40 bits of the cards it would take (all zeros when the action only places a card), a flag for “this is a capture”, a flag for “this is a caída” and a flag for “this is a limpia”. The last two are read from the scoring function itself rather than re-derived, so the features and the rules cannot drift apart. The policy is then a scoring function: it scores each action from its features and softmaxes over the legal ones. That same block of features is what the ISMCTS uses as its prior.

Reward shaping, and the value head

The reward is dense, from the acting team’s point of view: +1 per captured card, +2 for a caída, +2 for a limpia, the cartón points when the deal ends, and a win bonus of +20. I measured the mixture over three games with the trained network: the two teams’ combined return was ~137, of which captures were 46%, caídas/limpias 39% and the win bonus only 15%. Which is a good reminder that “the agent plays to win” is not free: at 20, the win bonus is the smallest term in the reward; if you want winning to dominate, raise it.

Two implementation details about the value head:

• What the value predicts matters. With a win target the head estimates a win probability in [-1, 1] and the reward terms are ignored; with a return target it estimates the shaped return (in game points, divided by a scale constant). The scale started at 100, which made the targets’ variance tiny; 30 gives roughly unit variance and behaves much better. That is also a classic way to fool yourself: my first “working” value head had an in-sample R² of 0.77 while the held-out R² was −0.27, with a mean absolute prediction about 70× too large. It had memorised the training games and was confidently wrong everywhere else - and since the search uses the value to score its leaves, a confidently wrong head makes the search worse than no search at all.
• Bootstrapping helps only if the value is good. I added TD(λ) returns (λ ≈ 0.9) instead of pure Monte-Carlo outcomes. With a silent (useless) value head it took the error from ~34.7 to 9.5 points; with an overfit head from 24.6 to 33.5. The trick is not that λ-returns are better; it is that they make the target depend on the value network, so they amplify whatever that network is.

How I measured agent strength

The opponent for every number in this post is a hand-written rule-based player: a heuristic agent with a four-line priority list — take a caída if one is available, then a limpia, then the capture that takes the most cards, and otherwise place a card at random. — Everything is measured head-to-head, in complete games, with a fixed seed per game.

The heuristic agent is a baseline, not a strong player: it does not do the things a good cuarenta player does - it does not avoid handing the opponent a caída, does not do cartón arithmetic to choose between two captures, and does not keep a safe card in hand for later. So “77% against this bot” means “comfortably better than a decent simple opponent”, and it is not evidence of beating strong humans.

Analysis

Against the heuristic

side	win rate	score	return	caídas	limpias	plies	s/game
ISMCTS + network (400 sims)	77.0% ±4.2%	36.2–27.9	82.7–62.1	9.04–7.88	3.36–2.49	124.7	1.13
ReBeL (32 CFR iters, depth 3, 16 particles)	54.0% ±5.0%	32.2–30.8	71.3–69.3	8.18–7.99	3.56–2.73	122.0	1.0
searchless PPO (one forward pass)	48.0% ±5.0% (CI 38.2–57.8)	31.3–33.1	67.9–70.8	8.02–8.74	3.47–3.50	116.7	0.02

Three things I read out of it:

• The raw policy is not a brainless lucky coin flip. It’s just a slightly worse player. It loses the underlying numbers too: 31.3 vs 33.1 points, 8.02 vs 8.74 caídas, a slightly negative return (67.9 vs 70.8).
• ReBeL’s 54% comes without a collecting edge. It is behind on caídas (7.99 vs 8.18) and its return is level (71.3 vs 69.3), and it takes the most limpias of the three sides (3.56 vs the bot’s 2.73). Limpias on their own are not a strength signal: the search engine also takes more than the bot (3.36 vs 2.49) and wins 77%. What separates the two models is the other two columns — ISMCTS beats the bot on caídas (9.04 vs 7.88) and on return (82.7 vs 62.1, a +20.7 margin), ReBeL on neither. So whatever produces those 54%, it is a margin/timing edge inside the noise, not extra scoring volume.
• At the top the two measures agree. ISMCTS has both the highest win rate (77%) and the largest margin (36.2 vs 27.9, +8.3 points; return 82.7 vs 62.1), and game length stays in the same 117–125 band, so nothing is being won by dragging games long.

Head to head

match	ISMCTS + network	searchless PPO	ReBeL	95% CI	verdict
ISMCTS vs PPO	72.0%	28.0%	—	PPO 19.2–36.8%	ISMCTS ahead
ISMCTS vs ReBeL	63.0%	—	37.0%	ISMCTS 53.5–72.5%	ISMCTS ahead
PPO vs ReBeL	—	54.0%	46.0%	PPO 44.2–63.8%	no clear winner

The numbers under the win rates

Some extra game details per side, per game:

match	side	score	return	caídas	limpias
ISMCTS vs PPO	ISMCTS + network	35.08	79.99	8.95	3.49
	searchless PPO	28.36	61.75	7.70	2.74
ISMCTS vs ReBeL	ISMCTS + network	34.50	79.17	8.86	2.65
	ReBeL	28.98	64.08	7.64	3.18
PPO vs ReBeL	searchless PPO	33.08	75.06	8.42	3.12
	ReBeL	29.98	66.76	7.72	3.56

Some time related details:

match	plies	seconds/game
ISMCTS vs PPO	122.0	1.52
ISMCTS vs ReBeL	125.4	5.45
PPO vs ReBeL	121.2	3.78

Three things I interpret out of these two tables:

• Score, return and caídas track the win rate. Limpias do not. The winner takes more caídas in all three matches, but in both ReBeL matches the losing side takes more limpias (3.18 vs 2.65, and 3.56 vs 3.12). A limpia only pays while your team is below 38 points, so it is a situational bonus, not a strength proxy.
• ReBeL is the most expensive thing here and the least rewarding. The PPO side of a match costs essentially nothing (one forward pass per decision), so a match’s wall-clock is dominated by the other side: ~1.5 s per game around ISMCTS at 200 simulations, ~3.8 s per game around ReBeL, ~5.5 s per game when it is ReBeL against ISMCTS. ReBeL costs roughly 2.5× what the search engine costs per game and loses both matches.
• Game length. Mean plies land in a 121–125 band across all three matches, so the win-rate differences apparently are not correlated to one side dragging games long or short.

Conclusions

• Search is real strength The same weights went from a coin flip to 77% by putting ISMCTS in front of them, and 128 simulations per turn (~10 ms) already capture most of that.
• Head to head, the search engine is the winner. ISMCTS beat both other checkpoints (72% against searchless PPO, 63% against ReBeL), while searchless PPO and ReBeL came out a statistical tie.
• The value head is hard to tune with cuarenta’s sparse rewards. A value that is confidently wrong makes search harmful, and in-sample metrics will not tell you — held-out ones will. Training a relatively good value function needed a tuned scale, an honest held-out metric and bootstrapping only once the head was actually useful. Still, I think there’s room for improvement if explore more creative approaches.
• Determinization is a good, cheap approximation. ISMCTS never reasons about the belief exactly (it samples worlds), which is why it cannot be the theoretical ceiling.