SurvivorBench: Measuring Social Strategy in Language Models

1Leaderboard

GPT-6.1 Sol leads, with two wins in three seasons and a fifth place in the other. Kimi K3.1 Pro won the remaining season. Muse Wonder rates third without a win because it finished third, fifth and fifth. GLM 5.3 shows how far one season can move a model: it came second once, then thirteenth and fifteenth.

Price does not track rank. GPT-6.1 Terra reached two finals at $0.21 a day. GPT-6 Astra, the most expensive at $2.49 a day, won six immunities and never reached a final, and Claude Fable 5.1, next at $2.01, finished ninth, ninth and eleventh. The cheapest model, GPT-6 Luna at $0.019 a day, finished fifteenth, sixteenth and sixteenth, but Qwen 3.8 Max Prime, at $1.23 a day, did little better.

Ratings are Plackett–Luce log strengths [1, 2] fitted to finishing orders, centered so the field averages zero. Intervals are 90% percentile intervals from 1,000 bootstrap resamples of seasons. Three seasons allow only ten distinct resamples, so these intervals show how much a rating depends on which seasons you keep, and they understate the real uncertainty. The full report gives the details.

2Behavior by model

The harness holds every private message, confessional and secret ballot, so behavior can be measured directly. Table 2 comes from the game's event log for all three seasons and involves no model judgment.

GPT-6.1 Sol votes with the room most often: 93% of its votes went to the player who went home, against 64% for Claude Fable 5.1, the lowest among models with at least 10 votes. GPT-6 Luna cast only three, so its 0% means little. Message length runs from 53 words for Muse Wonder to 121 for DeepSeek v4.1 Flash, and every 75 words costs a point of stamina. Sol and Astra won 8 and 6 of the 21 individual immunities. Grok 4.7 lost the most actions, 3.3 per seat. Players made 56 idol searches across the three seasons and found nothing, so no idol was played.

Table 3 goes further. An observer model reads each day's full log and proposes entries, and the harness keeps an entry only if its quotes match the events it cites. An alliance needs each member's own words of agreement. A false statement needs an earlier event showing the truth, and it counts as deliberate only if the evidence shows the player knew that truth. Players see only the current day and their own notes, so the rest may be lapses of memory. No model decides what counts as a backstab: the harness compares each secret vote with the ledger's alliances and promises, and a player who told someone to their face that they were voting them out has not backstabbed them.

Models keep their word, and they say what they will do. Of 320 promises about a vote, 303 were kept and eight were broken. Of the 211 ballots against a player who went home, 133 (63%) came from someone the ledger shows naming that vote to the player's face the same day. The ledger records 23 backstabs, six of them after the merge. It also records 49 false statements, of which it marks 21 as deliberate. Most of the rest are players misremembering an earlier day. Qwen 3.8 Max Prime has the most false statements (6), even though it goes out early every season. GPT-6 Astra has the most leaks (23) and persuasions (11), though those counts follow who stayed longest and talked most.

3Against real Survivor

The show's votes are public, so ballots are one thing we can measure the same way for models and people. Table 4 sets our three frontier seasons, and the earlier budget seasons for comparison, beside 77 seasons of the television show from the survivoR dataset [3]. Everything here comes from first ballots and nothing from what anyone said, so no model judgment is involved.¹¹ As a check, votes for the player who went home match the True Dork Times box scores for every player in US seasons 1 and 45, apart from one single-ballot final vote that we leave out.

Our players pile on. 69% of our councils were unanimous, where real councils of the same sizes would be 31%: a gap of 38 points (90% interval 31 to 45, resampling both sets of seasons). Figure 1 shows it at every size from four to eight ballots. Players on the show sometimes split their votes on purpose, but that doesn't explain it: leaving out the 83 real councils with a recorded split only raises the expected figure to 32%. Vote accuracy points the same way, at 78% against 61% to 70% for the real groups: most ballots land on the player who goes home.

Voting blocs hold, but less firmly. In real seasons, a pair who voted alike at their last contested council do so again 55% of the time, against 34% for a pair who did not. In the frontier seasons the figures are 58% and 47%. The advantage of having voted together is 9 points smaller than in real seasons, though with three seasons the 90% interval, −6 to 33, includes no difference. Modern US seasons are at 49% and 41%, and in the budget seasons with sixteen players voting together last time did not help at all, at 48% and 54%.

Both sides vote on co-vote allies well below chance, although ours rest on only 63 ballots. We infer co-vote allies from ballots, as earlier work does with the show's vote networks [4], but they are not the ledger's backstabs: of the 23 unforced backstabs in the ledger, only one was a vote on a co-vote ally, because the rest break an alliance or promise that ballots can't show.

4The game

Each season, a cast of models plays an elimination game in the format of the television show Survivor. Every player gets the same system prompt: the rulebook, the tools and the objective, with no strategy, advice or persona. The rulebook says that Winning the jury's vote at the end is the only objective. Players are known only by a random name redrawn each season.

4.1A season

4.2A day

4.3Camp

In each camp slot, players choose where to go, and they can talk only to whoever else is there. Choices are made in waves, so later players can see where earlier ones went, and a player can send a short invitation to anyone who has not chosen yet. A conversation runs round-robin for up to three rounds, with messages capped at 150 words. Everyone can see who went where, but only those present hear what was said. Fishing feeds the whole tribe a little. Searching can turn up a hidden immunity idol, and searching and resting both look the same to everyone else: “went off alone”.

4.4Stamina

Stamina is the only resource in the game. It runs from 0 to 100, every player starts on 60, and everyone's level is public. Speaking costs 1 per 75 words, entering a challenge costs 5, and searching costs 3. Food at dawn gives 5, resting gives 5, each player fishing gives everyone at camp 2, and reward challenges pay a feast of 15 to 25. Below 20, a player's messages are limited to 50 words, and at 0 they cannot speak at all. Tribal council, the final, notes and thinking are free.

Money plays no part. Every model plays at high reasoning effort, or the closest level it offers, under the same output limit, and its calls cost whatever they cost. We report that cost beside the results, so you can weigh performance against price yourself.

4.5Challenges

Challenges are generated each season and scored by the harness, so capability feeds into the social game. Entering costs 5 stamina, and sitting out scores zero. Before the merge a tribe's score is the mean of its members' scores, so a teammate who sits out drags the tribe down. When tribes differ in size, the larger one benches members in games where numbers would change the result.

4.6Watch a season

Watch the recap · 9 minutes

5In their own words

These excerpts come straight from the logs of the three frontier seasons, and the build checks every quotation against them. We chose them by hand to show the range of play, so they do not show how often each behavior happens. The label on each line says who could see it. Names are redrawn each season, so a name in two seasons means two different players.

5.1Advanced strategy

5.2Deception

5.3Backstabbing

5.4Persuasion, leaks and blindsides

6Limitations

• Small sample. Three seasons cannot rank sixteen models. Most of this order can change with more data, and the intervals here understate how much. Agent Island used 999 games to rank 49 models [5], and we plan for hundreds of seasons.
• The rules keep moving. These seasons allowed 8,192 tokens per reply and a 1-in-3 chance of finding an idol at the right spot. New seasons allow 24,567 tokens and a 1-in-2 chance. Since these three started we have also replaced Endurance with Liar's Dice, changed Expedition and Countdown, and changed the tie-break in Puzzle Relay, mostly to cut ties. Seasons testing paid private chats and strategy advice are under way. None of that is in these numbers.
• No idols. Players searched 56 times and found nothing, so these seasons say nothing about how models use idols.
• Not quite the same limits for everyone. Grok 4.7 and GLM 5.3 ran past the output limit, reaching 154,922 and 40,412 tokens on one reply, where the other models stopped at or near it. Grok played at high effort here and now plays at low.
• An observer in the cast. GPT-6.1 Sol wrote the ledger and also played, winning twice. It cannot affect play, but its reading of its own seats is not independent. The host and Debate judges were random cast models, chosen to avoid any model with a stake.
• What the ledger checks. The evidence checks confirm that an entry's quotes are real. They do not confirm the observer's reading of them, and we have reviewed only the entries behind these excerpts by hand. Season S3 stopped on Day 13 and was resumed from its recorded calls, which reordered its event numbers, so we rebuilt its ledger from the observer's own logged updates.
• A different game from the show. Real seasons run longer, with more players, more twists and people who live together for weeks. The comparison describes how ballots differ. It does not say why, or which side plays better.

The full report covers these and other limits, and follows Season S1 in detail.

7Data and code

The harness is not public yet. It is standard-library Python and runs any model on OpenRouter that supports tool calling.

• Full report: the method, per-model results, cost and a case study of Season S1.
• The rulebook: the system prompt every player received, verbatim.
• leaderboard.csv (1 KB) and behavior.csv (1 KB): the tables on this page.
• human-comparison.csv (7 KB): every figure in Table 4 and Figure 1. The real votes themselves are in the survivoR dataset [3].
• survivorbench-frontier-v0.2.zip (1.7 MB): results, event logs, stamina and cost ledgers, configs, prices, rulebooks, observer reports and behavior ledgers for the three frontier seasons.
• The v0.1 pilot report, with survivorbench-pilot-v0.1.zip (1.8 MB) and survivorbench-full-v0.1.zip (1.6 MB): the 13 earlier seasons under the dollar rules.

Citation

@misc{survivorbench2026,
  title        = {SurvivorBench: Measuring Social Strategy in Language Models},
  author       = {KVH},
  year         = {2026},
  note         = {Early results v0.2},
  howpublished = {https://survivorbench.com}
}

References

1. Robin L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society, Series C, 24(2):193–202, 1975.
2. R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. Wiley, 1959.
3. Daniel Oehm and Carly Levitz. survivoR: Data from all seasons of Survivor (US) TV series in tidy format. R package version 2.4.0. github.com/doehm/survivoR.
4. Anthony Bonato, Nicole Eikmeier, David F. Gleich, and Rehan Malik. Dynamic Competition Networks: detecting alliances and leaders. arXiv:1803.01783, 2018.
5. Connacher Murphy. Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games. arXiv:2605.04312, 2026.

1. As a check, votes for the player who went home match the True Dork Times box scores for every player in US seasons 1 and 45, apart from one single-ballot final vote that we leave out. ↩