An agent is given the manual for a small programming language, forty example programs with the exact output each one produced, and a button. The button runs any program the agent writes and shows what came out. It can be pressed two hundred times (eighty, on half the tasks).
The button does not run the language in the manual. It runs a dialect: the same language, changed
in four to sixteen places. Lists might start at 1. print might be spelled say. A function might
remember its variables as they were when it was written, not as they are when it runs. Nobody tells
the agent which.
The agent must write an interpreter that matches the button byte for byte on three hundred programs it never sees.
This is Dialect Recon, an environment we built to train and measure one skill: working out how a system really behaves when its documentation is wrong. We ran two frontier coding agents on it. For each hidden change in each task they attempted, we checked whether their final interpreter got it right.
- hidden changes the examples show, found
- 106 / 106
- hidden changes only an experiment reveals, found
- 21 / 48
- hidden changes the examples show, found
- 204 / 204
- hidden changes only an experiment reveals, found
- 25 / 90
Source: saved regrade records and transcript review. A change that appears in a task attempted three times counts three times. Some changes have no isolated test programs, so their classification also depends on the transcript. One example-visible change was only partly implemented.
Both agents found every change shown in the examples, though one implementation was incomplete. They missed many of the changes that required a new experiment. We built Dialect Recon to measure that gap. The runs also exposed problems in the environment that we had to fix before trusting it.
IReading versus asking
Coding agents can do well in unfamiliar languages when they have the rules and can test their code. Given the README of MNM Lang, a language invented in March 2026, Claude Opus 4.6 running through Claude Code solved all 26 of its challenges, with the author pointing out wrong outputs during debugging (Acher, 2026).
On esoteric languages such as Brainfuck, frontier models writing code directly score 0 to 11% (Sharma & Chopra, 2026). Coding agents that can run their code solve 64 (Claude Opus 4.6) and 79 (GPT-5.4) of the same 80 Brainfuck problems (Sharma, Thorat & Chopra, 2026, Table 1).
Real engineering often gives you a different job. An old interpreter has corner cases nobody wrote down. A vendor's SQL is almost standard SQL. The only specification of a network protocol that matters is the server on the other end. In those cases, an engineer has to find where the running system differs from the documentation, test the difference, and account for it in their code.
Related studies have tested how models handle changed or hidden rules:
| Study | Setting | Result |
|---|---|---|
| Wu et al., 2023, Table 20 | Python with 1-based lists. The model is told about the change. | GPT-4 predicting program output, with chain of thought: 73.5% → 24.8% |
| Thimmaiah et al., 2025 | Familiar operators with new meanings. The new rules are supplied. | Models that reach 90% on normal semantics drop by up to 40 to 60 points |
| Wei et al., 2025 (CodeARC) | Recover one hidden function by querying it | Best of 18 models: 52.7% on anonymized functions |
| Yin et al., 2025 (ORACLE) | Work out black-box environments by interacting | o3 exceeded 70% on most easy environments, but fell below 40% on some hard tasks |
These studies cover supplied rule changes, hidden functions and black-box environments. Two recent benchmarks ask agents to rebuild whole programs from their behaviour (MirrorCode, ProgramBench), but those programs are real and public, so a model may partly remember them.
In Dialect Recon, an agent has to recover the rules of a whole language from examples and a limited number of experiments. Grading uses programs the agent has never seen.
The idea itself is older than language models. In programming-language courses at Brown, students are given "mystery languages": variants of a language that differ in hidden ways, which they must tell apart by writing experiments (Pombrio, Krishnamurthi & Fisler, SNAPL 2017). Dialect Recon asks the same thing of an agent, at a scale of four hundred languages and with an exact grader.
IIFour hundred generated languages
Every dialect starts from one small base language: integers, strings, booleans, nil, lists,
functions with closures, if, while, for, and print. A program ends normally (exit code 0),
prints <<error>> and exits with code 1 on a runtime error, or exits with code 2 if it does not
parse. The manual describes all of it.
A dialect is made by turning dials. There are 48 of them in eight families, and each changes one rule of the language:
| Family | Example dial | Base language | Dialect |
|---|---|---|---|
| Syntax | keyword spelling | print 5; |
say 5; |
| Arithmetic | division rounding | -7 / 2 is -3 |
-7 / 2 is -4 |
| Indexing | first index | [10, 20][1] is 20 |
[10, 20][1] is 10 |
| Scope | when closures read variables | reads the current value | reads the value at definition |
| Evaluation order | argument order | left to right | right to left |
| Types | mixed comparison | 1 < "a" is an error |
1 < "a" is false |
| Printing | boolean form | true |
True |
| Errors | undefined variables | an error | nil |
The scope dials come from work by Lu and Krishnamurthi. They collected the misconceptions students
hold about variables, mutation, scope and closures, and turned each one into a misinterpreter: an
interpreter that runs code the way a student with that misconception expects
(OOPSLA 2024, Distinguished Paper). Several of our dials
are those misinterpreters written as settings: variables passed by reference, one global scope for
everything, let that reassigns instead of declaring, closures that copy their environment. These
let us test whether an agent checks familiar behaviour such as scope and argument passing.
A dialect turns 4 to 16 dials (median 10, our measurement over all 400), taken from at least four families. The environment has 400 dialects. We generated their rule combinations instead of choosing existing languages. That alone does not establish what a model has seen in training. Each dialect has an exact answer key: it is whatever our reference interpreter does with those dials set.
Most programs cannot tell a difference
Take the division dial. These three programs look like fair tests of it:
print 6 / 3; # 2 in both languages
print 7 / 2; # 3 in both languages
print -7 / 2; # -3 in the base language, -4 in the dialect
Only the last one can tell the two languages apart. Rounding only matters when the result is negative and inexact. The same is true of almost every dial: it changes behaviour only in a narrow corner.
That is why our first attempt, generating programs at random, failed. 88 to 94% of random programs printed exactly the same output with a dial on or off (our prototype measurement). They almost never divided a negative number, read past the end of a list, or captured a loop variable in a closure.
For each dial we wrote small program templates that test its corner case, and mixed them into random,
well-typed code. Then we test each program: run it on the dialect,
then again with one dial switched back to the base language. If the output changes, the program is
sensitive to that dial, like the -7 / 2 line above. Every dial in every dialect controls at least
20 of the 300 graded programs (our measurement), so we can measure which programs test each rule.
The dials also interfere. With "lists start at 1" switched on, our closure templates produced no sensitive program in 30,830 tries (prototype measurement). They indexed lists from 0, crashed on the first index, and never reached the closure they were written to test. The fix was to render every template in the dialect's own rules. We also found 13 pairs of dials that hide each other's effects, and those pairs are never drawn together.
IIIHiding the evidence
For each dialect we generate 60 example programs and show the agent 40. The 40 are chosen so that two to four of the dialect's changes never appear in any of them. Those changes cannot be read off the examples. The agent has to suspect them, write a program that would reveal them, and spend a button press on it.
Here is a change the examples do show. One dialect gives the agent an example much like this:
let q = [14];
print q[1];
print 2 * 3 + 4 * 5;
The manual predicts an error on line 2, since a one-element list has nothing at index 1. The dialect
prints 14 and then 70. Two changes are visible: lists start at 1, and + binds tighter than *,
so the last line means 2 * (3 + 4) * 5.
Here is a change the examples in that same dialect never show:
let r = 1;
fn k() {
return r;
}
r = 5;
print k();
The manual says this prints 5. The dialect prints 1, because its functions snapshot their
variables when they are defined. None of the 40 examples does anything like this. The only way to
find out is to write a program that tests when the function reads those variables.
Hidden changes are always about meaning, never spelling. A renamed keyword shows up in almost every example, and a blind guess at one gets back only "parse error", which teaches nothing.
Where the points are
For every dialect we measure how far each kind of knowledge gets you on its 300 graded programs. Across all 400 dialects, the manual alone gives a median score of 0. Applying everything the 40 examples show gives a median of 143. Recovering all the rules gives 300. We call the score from applying the examples perfectly the examples-only score. Going beyond it requires discovering changes the examples do not show.
Hidden programs reproduced (of 300)
Show the data
| Dialect | Manual alone | With the examples | Experiment only |
|---|---|---|---|
| 1 | 5 | 46 | 254 |
| 2 | 0 | 67 | 233 |
| 3 | 4 | 71 | 229 |
| 4 | 1 | 77 | 223 |
| 5 | 0 | 80 | 220 |
| 6 | 0 | 81 | 219 |
| 7 | 2 | 82 | 218 |
| 8 | 0 | 83 | 217 |
| 9 | 3 | 83 | 217 |
| 10 | 0 | 84 | 216 |
| 11 | 0 | 84 | 216 |
| 12 | 0 | 84 | 216 |
| 13 | 0 | 85 | 215 |
| 14 | 4 | 85 | 215 |
| 15 | 5 | 85 | 215 |
| 16 | 7 | 86 | 214 |
| 17 | 0 | 87 | 213 |
| 18 | 0 | 88 | 212 |
| 19 | 0 | 88 | 212 |
| 20 | 2 | 90 | 210 |
| 21 | 3 | 90 | 210 |
| 22 | 3 | 90 | 210 |
| 23 | 1 | 91 | 209 |
| 24 | 7 | 91 | 209 |
| 25 | 0 | 92 | 208 |
| 26 | 2 | 92 | 208 |
| 27 | 0 | 93 | 207 |
| 28 | 0 | 94 | 206 |
| 29 | 13 | 94 | 206 |
| 30 | 0 | 96 | 204 |
| 31 | 0 | 96 | 204 |
| 32 | 11 | 96 | 204 |
| 33 | 0 | 97 | 203 |
| 34 | 0 | 97 | 203 |
| 35 | 0 | 100 | 200 |
| 36 | 0 | 100 | 200 |
| 37 | 1 | 100 | 200 |
| 38 | 15 | 100 | 200 |
| 39 | 5 | 101 | 199 |
| 40 | 0 | 102 | 198 |
| 41 | 0 | 102 | 198 |
| 42 | 0 | 102 | 198 |
| 43 | 10 | 102 | 198 |
| 44 | 9 | 105 | 195 |
| 45 | 15 | 105 | 195 |
| 46 | 0 | 107 | 193 |
| 47 | 0 | 108 | 192 |
| 48 | 0 | 109 | 191 |
| 49 | 15 | 110 | 190 |
| 50 | 15 | 111 | 189 |
| 51 | 7 | 112 | 188 |
| 52 | 0 | 113 | 187 |
| 53 | 9 | 113 | 187 |
| 54 | 0 | 114 | 186 |
| 55 | 1 | 114 | 186 |
| 56 | 2 | 114 | 186 |
| 57 | 0 | 116 | 184 |
| 58 | 0 | 116 | 184 |
| 59 | 4 | 116 | 184 |
| 60 | 8 | 116 | 184 |
| 61 | 0 | 117 | 183 |
| 62 | 0 | 118 | 182 |
| 63 | 0 | 118 | 182 |
| 64 | 1 | 118 | 182 |
| 65 | 1 | 118 | 182 |
| 66 | 8 | 118 | 182 |
| 67 | 13 | 118 | 182 |
| 68 | 25 | 120 | 180 |
| 69 | 0 | 121 | 179 |
| 70 | 0 | 121 | 179 |
| 71 | 15 | 122 | 178 |
| 72 | 2 | 123 | 177 |
| 73 | 3 | 124 | 176 |
| 74 | 0 | 125 | 175 |
| 75 | 0 | 125 | 175 |
| 76 | 3 | 125 | 175 |
| 77 | 5 | 125 | 175 |
| 78 | 16 | 125 | 175 |
| 79 | 1 | 126 | 174 |
| 80 | 15 | 127 | 173 |
| 81 | 0 | 128 | 172 |
| 82 | 0 | 128 | 172 |
| 83 | 5 | 128 | 172 |
| 84 | 10 | 128 | 172 |
| 85 | 0 | 129 | 171 |
| 86 | 2 | 129 | 171 |
| 87 | 6 | 129 | 171 |
| 88 | 13 | 129 | 171 |
| 89 | 0 | 130 | 170 |
| 90 | 0 | 130 | 170 |
| 91 | 0 | 130 | 170 |
| 92 | 0 | 130 | 170 |
| 93 | 0 | 130 | 170 |
| 94 | 0 | 130 | 170 |
| 95 | 3 | 130 | 170 |
| 96 | 9 | 130 | 170 |
| 97 | 14 | 130 | 170 |
| 98 | 0 | 131 | 169 |
| 99 | 10 | 131 | 169 |
| 100 | 29 | 131 | 169 |
| 101 | 0 | 132 | 168 |
| 102 | 0 | 132 | 168 |
| 103 | 9 | 132 | 168 |
| 104 | 0 | 133 | 167 |
| 105 | 1 | 133 | 167 |
| 106 | 6 | 133 | 167 |
| 107 | 0 | 134 | 166 |
| 108 | 0 | 134 | 166 |
| 109 | 0 | 135 | 165 |
| 110 | 0 | 135 | 165 |
| 111 | 0 | 135 | 165 |
| 112 | 3 | 135 | 165 |
| 113 | 15 | 135 | 165 |
| 114 | 0 | 136 | 164 |
| 115 | 0 | 136 | 164 |
| 116 | 4 | 136 | 164 |
| 117 | 8 | 136 | 164 |
| 118 | 0 | 137 | 163 |
| 119 | 0 | 137 | 163 |
| 120 | 0 | 137 | 163 |
| 121 | 0 | 137 | 163 |
| 122 | 0 | 137 | 163 |
| 123 | 0 | 137 | 163 |
| 124 | 0 | 137 | 163 |
| 125 | 0 | 137 | 163 |
| 126 | 0 | 138 | 162 |
| 127 | 0 | 138 | 162 |
| 128 | 0 | 138 | 162 |
| 129 | 0 | 138 | 162 |
| 130 | 0 | 138 | 162 |
| 131 | 0 | 138 | 162 |
| 132 | 0 | 138 | 162 |
| 133 | 0 | 138 | 162 |
| 134 | 0 | 138 | 162 |
| 135 | 0 | 138 | 162 |
| 136 | 1 | 138 | 162 |
| 137 | 16 | 138 | 162 |
| 138 | 0 | 139 | 161 |
| 139 | 0 | 139 | 161 |
| 140 | 0 | 139 | 161 |
| 141 | 1 | 139 | 161 |
| 142 | 1 | 139 | 161 |
| 143 | 3 | 139 | 161 |
| 144 | 6 | 139 | 161 |
| 145 | 9 | 139 | 161 |
| 146 | 0 | 140 | 160 |
| 147 | 0 | 140 | 160 |
| 148 | 0 | 140 | 160 |
| 149 | 0 | 140 | 160 |
| 150 | 0 | 140 | 160 |
| 151 | 2 | 140 | 160 |
| 152 | 3 | 140 | 160 |
| 153 | 3 | 140 | 160 |
| 154 | 13 | 140 | 160 |
| 155 | 15 | 140 | 160 |
| 156 | 0 | 141 | 159 |
| 157 | 0 | 141 | 159 |
| 158 | 0 | 141 | 159 |
| 159 | 0 | 141 | 159 |
| 160 | 0 | 141 | 159 |
| 161 | 0 | 141 | 159 |
| 162 | 0 | 141 | 159 |
| 163 | 0 | 141 | 159 |
| 164 | 0 | 141 | 159 |
| 165 | 7 | 141 | 159 |
| 166 | 12 | 141 | 159 |
| 167 | 14 | 141 | 159 |
| 168 | 15 | 141 | 159 |
| 169 | 0 | 142 | 158 |
| 170 | 0 | 142 | 158 |
| 171 | 0 | 142 | 158 |
| 172 | 0 | 142 | 158 |
| 173 | 0 | 142 | 158 |
| 174 | 0 | 142 | 158 |
| 175 | 0 | 142 | 158 |
| 176 | 0 | 142 | 158 |
| 177 | 1 | 142 | 158 |
| 178 | 3 | 142 | 158 |
| 179 | 5 | 142 | 158 |
| 180 | 5 | 142 | 158 |
| 181 | 6 | 142 | 158 |
| 182 | 11 | 142 | 158 |
| 183 | 13 | 142 | 158 |
| 184 | 14 | 142 | 158 |
| 185 | 16 | 142 | 158 |
| 186 | 0 | 143 | 157 |
| 187 | 0 | 143 | 157 |
| 188 | 0 | 143 | 157 |
| 189 | 0 | 143 | 157 |
| 190 | 0 | 143 | 157 |
| 191 | 0 | 143 | 157 |
| 192 | 0 | 143 | 157 |
| 193 | 0 | 143 | 157 |
| 194 | 0 | 143 | 157 |
| 195 | 0 | 143 | 157 |
| 196 | 1 | 143 | 157 |
| 197 | 1 | 143 | 157 |
| 198 | 2 | 143 | 157 |
| 199 | 4 | 143 | 157 |
| 200 | 7 | 143 | 157 |
| 201 | 9 | 143 | 157 |
| 202 | 0 | 144 | 156 |
| 203 | 0 | 144 | 156 |
| 204 | 0 | 144 | 156 |
| 205 | 0 | 144 | 156 |
| 206 | 0 | 144 | 156 |
| 207 | 0 | 144 | 156 |
| 208 | 0 | 144 | 156 |
| 209 | 0 | 144 | 156 |
| 210 | 0 | 144 | 156 |
| 211 | 0 | 144 | 156 |
| 212 | 0 | 144 | 156 |
| 213 | 1 | 144 | 156 |
| 214 | 4 | 144 | 156 |
| 215 | 4 | 144 | 156 |
| 216 | 5 | 144 | 156 |
| 217 | 8 | 144 | 156 |
| 218 | 14 | 144 | 156 |
| 219 | 0 | 145 | 155 |
| 220 | 0 | 145 | 155 |
| 221 | 0 | 145 | 155 |
| 222 | 0 | 145 | 155 |
| 223 | 0 | 145 | 155 |
| 224 | 0 | 145 | 155 |
| 225 | 0 | 145 | 155 |
| 226 | 0 | 145 | 155 |
| 227 | 0 | 145 | 155 |
| 228 | 0 | 145 | 155 |
| 229 | 0 | 145 | 155 |
| 230 | 3 | 145 | 155 |
| 231 | 3 | 145 | 155 |
| 232 | 3 | 145 | 155 |
| 233 | 3 | 145 | 155 |
| 234 | 3 | 145 | 155 |
| 235 | 5 | 145 | 155 |
| 236 | 6 | 145 | 155 |
| 237 | 8 | 145 | 155 |
| 238 | 9 | 145 | 155 |
| 239 | 15 | 145 | 155 |
| 240 | 15 | 145 | 155 |
| 241 | 16 | 145 | 155 |
| 242 | 0 | 146 | 154 |
| 243 | 0 | 146 | 154 |
| 244 | 0 | 146 | 154 |
| 245 | 0 | 146 | 154 |
| 246 | 0 | 146 | 154 |
| 247 | 0 | 146 | 154 |
| 248 | 0 | 146 | 154 |
| 249 | 0 | 146 | 154 |
| 250 | 0 | 146 | 154 |
| 251 | 0 | 146 | 154 |
| 252 | 0 | 146 | 154 |
| 253 | 0 | 146 | 154 |
| 254 | 0 | 146 | 154 |
| 255 | 0 | 146 | 154 |
| 256 | 0 | 146 | 154 |
| 257 | 0 | 146 | 154 |
| 258 | 0 | 146 | 154 |
| 259 | 0 | 146 | 154 |
| 260 | 0 | 146 | 154 |
| 261 | 0 | 146 | 154 |
| 262 | 0 | 146 | 154 |
| 263 | 1 | 146 | 154 |
| 264 | 2 | 146 | 154 |
| 265 | 2 | 146 | 154 |
| 266 | 3 | 146 | 154 |
| 267 | 6 | 146 | 154 |
| 268 | 9 | 146 | 154 |
| 269 | 0 | 147 | 153 |
| 270 | 0 | 147 | 153 |
| 271 | 0 | 147 | 153 |
| 272 | 0 | 147 | 153 |
| 273 | 0 | 147 | 153 |
| 274 | 0 | 147 | 153 |
| 275 | 0 | 147 | 153 |
| 276 | 0 | 147 | 153 |
| 277 | 0 | 147 | 153 |
| 278 | 0 | 147 | 153 |
| 279 | 0 | 147 | 153 |
| 280 | 0 | 147 | 153 |
| 281 | 0 | 147 | 153 |
| 282 | 0 | 147 | 153 |
| 283 | 0 | 147 | 153 |
| 284 | 0 | 147 | 153 |
| 285 | 0 | 147 | 153 |
| 286 | 0 | 147 | 153 |
| 287 | 0 | 147 | 153 |
| 288 | 0 | 147 | 153 |
| 289 | 0 | 147 | 153 |
| 290 | 1 | 147 | 153 |
| 291 | 1 | 147 | 153 |
| 292 | 1 | 147 | 153 |
| 293 | 4 | 147 | 153 |
| 294 | 4 | 147 | 153 |
| 295 | 4 | 147 | 153 |
| 296 | 4 | 147 | 153 |
| 297 | 5 | 147 | 153 |
| 298 | 5 | 147 | 153 |
| 299 | 7 | 147 | 153 |
| 300 | 8 | 147 | 153 |
| 301 | 0 | 148 | 152 |
| 302 | 0 | 148 | 152 |
| 303 | 0 | 148 | 152 |
| 304 | 0 | 148 | 152 |
| 305 | 0 | 148 | 152 |
| 306 | 0 | 148 | 152 |
| 307 | 0 | 148 | 152 |
| 308 | 0 | 148 | 152 |
| 309 | 0 | 148 | 152 |
| 310 | 0 | 148 | 152 |
| 311 | 0 | 148 | 152 |
| 312 | 0 | 148 | 152 |
| 313 | 0 | 148 | 152 |
| 314 | 0 | 148 | 152 |
| 315 | 0 | 148 | 152 |
| 316 | 0 | 148 | 152 |
| 317 | 1 | 148 | 152 |
| 318 | 1 | 148 | 152 |
| 319 | 2 | 148 | 152 |
| 320 | 4 | 148 | 152 |
| 321 | 4 | 148 | 152 |
| 322 | 4 | 148 | 152 |
| 323 | 5 | 148 | 152 |
| 324 | 5 | 148 | 152 |
| 325 | 5 | 148 | 152 |
| 326 | 6 | 148 | 152 |
| 327 | 11 | 148 | 152 |
| 328 | 12 | 148 | 152 |
| 329 | 0 | 149 | 151 |
| 330 | 0 | 149 | 151 |
| 331 | 0 | 149 | 151 |
| 332 | 0 | 149 | 151 |
| 333 | 0 | 149 | 151 |
| 334 | 0 | 149 | 151 |
| 335 | 0 | 149 | 151 |
| 336 | 0 | 149 | 151 |
| 337 | 0 | 149 | 151 |
| 338 | 0 | 149 | 151 |
| 339 | 0 | 149 | 151 |
| 340 | 0 | 149 | 151 |
| 341 | 1 | 149 | 151 |
| 342 | 1 | 149 | 151 |
| 343 | 2 | 149 | 151 |
| 344 | 2 | 149 | 151 |
| 345 | 2 | 149 | 151 |
| 346 | 3 | 149 | 151 |
| 347 | 4 | 149 | 151 |
| 348 | 6 | 149 | 151 |
| 349 | 8 | 149 | 151 |
| 350 | 11 | 149 | 151 |
| 351 | 0 | 150 | 150 |
| 352 | 0 | 150 | 150 |
| 353 | 0 | 150 | 150 |
| 354 | 0 | 150 | 150 |
| 355 | 0 | 150 | 150 |
| 356 | 0 | 150 | 150 |
| 357 | 0 | 150 | 150 |
| 358 | 0 | 150 | 150 |
| 359 | 0 | 150 | 150 |
| 360 | 0 | 150 | 150 |
| 361 | 0 | 150 | 150 |
| 362 | 0 | 150 | 150 |
| 363 | 0 | 150 | 150 |
| 364 | 0 | 150 | 150 |
| 365 | 0 | 150 | 150 |
| 366 | 0 | 150 | 150 |
| 367 | 0 | 150 | 150 |
| 368 | 0 | 150 | 150 |
| 369 | 0 | 150 | 150 |
| 370 | 0 | 150 | 150 |
| 371 | 0 | 150 | 150 |
| 372 | 0 | 150 | 150 |
| 373 | 0 | 150 | 150 |
| 374 | 0 | 150 | 150 |
| 375 | 0 | 150 | 150 |
| 376 | 0 | 150 | 150 |
| 377 | 0 | 150 | 150 |
| 378 | 1 | 150 | 150 |
| 379 | 2 | 150 | 150 |
| 380 | 3 | 150 | 150 |
| 381 | 4 | 150 | 150 |
| 382 | 4 | 150 | 150 |
| 383 | 5 | 150 | 150 |
| 384 | 5 | 150 | 150 |
| 385 | 5 | 150 | 150 |
| 386 | 6 | 150 | 150 |
| 387 | 6 | 150 | 150 |
| 388 | 6 | 150 | 150 |
| 389 | 6 | 150 | 150 |
| 390 | 7 | 150 | 150 |
| 391 | 8 | 150 | 150 |
| 392 | 8 | 150 | 150 |
| 393 | 13 | 150 | 150 |
| 394 | 14 | 150 | 150 |
| 395 | 17 | 150 | 150 |
| 396 | 17 | 151 | 149 |
| 397 | 0 | 159 | 141 |
| 398 | 11 | 166 | 134 |
| 399 | 0 | 172 | 128 |
| 400 | 1 | 172 | 128 |
Figure 1. Source: our measurement over all 400 dialects. Lines span the lowest to highest score, circles mark the medians, and the square marks the full score. The examples-only score includes what the manual earns. The remaining points require experiments. The table contains every dialect's scores.
Why a list of dials is not enough
With a fixed menu of 48 dials, an agent could try to learn the menu and test each dial once. In our prototype, a script that knew every dial did exactly that: one test per dial, and a perfect score on every dialect.
Then we gave it a dialect with 16-bit integers, a width it had never seen. Its single width test was
designed to tell 32-bit, 64-bit and unlimited integers apart, by adding numbers near two billion. In a
16-bit language, numbers wrap around much earlier: print 30000 + 30000; prints -5536, not
60000. The answer matched none of the options the script knew, so it picked the nearest one, 32-bit,
and every program that touched a number above 32,767 came out wrong. Its score fell to about 120 of
300 (prototype measurement).
So 15 dial settings are reserved for the dialects we evaluate on, including keywords such as puts
and val, a nil that prints as <nil>, and division by zero that returns its left operand. An agent
that has memorised the menu should fail on them. An agent that knows how to experiment should not.
IVAn exact grader
The reward is one number: how many of the 300 hidden programs the agent's interpreter reproduces exactly, output bytes and exit code both, from 0 to 300. There is no fuzzy matching and no judge model. A program matches or it does not.
We record which programs respond to each dial, so we can inspect a submission's errors alongside its transcript to judge which changes it found. Some programs depend on several dials, so their contributions cannot simply be added. The tables use that combined evidence, with isolated tests where available.
The button reports what happened, never why. It returns the program's output and exit code. A
runtime error comes back as <<error>> and code 1, with no message. If it answered "index 3 is out of
range", the agent could learn the indexing rule from the error text, and we would be measuring reading
again. With only the output and exit code, the agent has to test what caused the error.
The dialect runs outside the agent's machine, and the agent has no network. On every task we check the extremes before trusting any score: the reference interpreter scores 300, an empty submission scores 0, and an interpreter that follows the manual scores close to 0.
VWhat testing found
Testing turned up problems in timing, the probe service and the way tasks were packaged.
The score depended on the machine. Our first design limited each program's wall-clock time. On a busy grading machine a correct interpreter could run out of time, so the same submission could score differently on two runs. Worse, a submission that hit the limit on every program could hold the grader for 340 programs × 60 seconds, almost six hours, longer than the agent's whole episode. We changed the scoring limit to 2 seconds of CPU time per program, so waiting for a busy host no longer consumes that allowance.
The button could stop working. The language allows up to 200 nested function calls. The service behind the button used a lower Python recursion limit than our reference interpreter, so a program recursing 199 levels deep, well within the rules, crashed it for the rest of the episode. Agents test documented limits, so this was common: the button died in 13 of 22 graded Claude Sonnet 5 runs and 5 of 34 GPT-5.6-terra runs (our measurement). We gave the service the same limit as the interpreter and made it survive any request, then replayed all 136,000 example and hidden programs in the dataset through it. Every one returned its recorded answer. The agent results below were measured before this fix. We have not rerun the agents to measure how much the failures affected their scores.
What we tested was not what users download. Our tests ran each task from our own copy. The public registry delivers files differently, and from there the grader failed before it started. Every check had passed and nothing worked. We now run every check on the downloaded task, exactly as users get it.
VIAttacking our own grader
We ask agents to try to earn points without doing the task. A separate model reviews their submissions, the grader and the recorded results for evidence of unearned reward. Where a control is available, we also score the submission with the suspected attack removed to see what it contributed.
In Dialect Recon's earlier checks, printing the same thing every time failed admission: the 40 examples have at least 35 different outputs. Memorising those answers passed admission but scored almost nothing, because none of the examples appears among the 300 graded programs.
One attempt did reach the maximum score without interpreting any program. An earlier round had missed it because the submissions never reached the grader. We fixed that test setup and isolated untrusted code from the hidden answers, giving it only the program being graded. After the fix, the shortcut scored 0, the reference scored 300, and all 34 agent submissions we re-graded kept their earlier scores. None of the agent runs in this post used the shortcut.
Source: our Dialect Recon validation record and subsequent red-team implementation history.
VIIResults
We ran both agents on the same ten tasks, drawn with a fixed seed, with up to four attempts per task. Claude Sonnet 5 (high) ran through Claude Code, GPT-5.6-terra ran through the Codex CLI at its highest reasoning setting. Every number below is our measurement from the raw runs. We re-graded all 53 finished submissions ourselves and every one reproduced its recorded score exactly.
| Claude Sonnet 5 (high) | GPT-5.6-terra | |
|---|---|---|
| Finished runs | 19 | 34 |
| Mean score (of 300) | 199 | 171 |
| Median score | 182 | 169 |
| Perfect runs (300) | 4 | 2 |
| Runs exactly on the examples-only score | 7 | 13 |
| Runs below it | 0 | 1 |
| Median working time (limit 120 min) | 19 min | 16 min |
For comparison, on these ten tasks the manual alone scores 4.6 on average and the examples alone score 129.3. Both agents clear the examples-only line on average, and both reach a perfect score at least twice, so the task is solvable from scratch in two hours. Claude Sonnet 5 scores higher on average and higher on 8 of the 10 tasks.
Hidden programs reproduced (of 300)
Show the data
| Task | Examples-only | Manual-only | Claude Sonnet 5 (high) runs | GPT-5.6-terra runs |
|---|---|---|---|---|
| Task 1 | 108 | 0 | 108, 108 | 108, 108, 108, 108 |
| Task 2 | 110 | 15 | 214 | 110, 110, 214 |
| Task 3 | 114 | 1 | 114, 204, 300 | 102, 114, 204, 300 |
| Task 4 | 128 | 0 | 231 | 128, 128, 168 |
| Task 5 | 129 | 6 | 245, 300 | 245, 245, 300 |
| Task 6 | 130 | 0 | 300 | 169, 169, 169 |
| Task 7 | 133 | 1 | 175, 234 | 175, 175, 175, 175 |
| Task 8 | 145 | 16 | 145, 145, 145 | 145, 211, 211 |
| Task 9 | 147 | 4 | 182, 182 | 147, 182, 182 |
| Task 10 | 149 | 3 | 149, 300 | 149, 149, 217, 217 |
Figure 2. Source: saved regrade scores and task baseline receipts. Each row is one task. Short tick: manual-only score. Tall tick: examples-only score. Circles and squares: individual runs, with ×n where several runs scored the same.
Side by side, task by task
| Task | Hidden changes (not in examples) | Examples-only score | Claude Sonnet 5 runs | GPT-5.6-terra runs |
|---|---|---|---|---|
| 1 | 6 (2) | 108 | 108, 108 | 108, 108, 108, 108 |
| 2 | 9 (3) | 110 | 214 | 110, 110, 214 |
| 3 | 7 (2) | 114 | 114, 204, 300 | 102, 114, 204, 300 |
| 4 | 16 (4) | 128 | 231 | 128, 128, 168 |
| 5 | 10 (3) | 129 | 245, 300 | 245, 245, 300 |
| 6 | 9 (3) | 130 | 300 | 169, 169, 169 |
| 7 | 9 (3) | 133 | 175, 234 | 175, 175, 175, 175 |
| 8 | 5 (2) | 145 | 145, 145, 145 | 145, 211, 211 |
| 9 | 9 (3) | 147 | 182, 182 | 147, 182, 182 |
| 10 | 8 (2) | 149 | 149, 300 | 149, 149, 217, 217 |
Source: our measurement. Tasks are ordered by their examples-only score.
Nobody beat Task 1. Both of its hidden changes (ranges that include their end value, and arguments passed by reference) raise no error and are never hinted at. All six runs, from both models, scored exactly 108.
GPT-5.6-terra often repeated the same result. On Task 7 it scored 175 four times, on Task 1 108 four times, on Task 6 169 three times, and in each case the replay shows the same changes found and missed. Claude Sonnet 5 varies more between attempts on one task (Task 3: 114, 204, 300), which gives it more chances at a perfect run.
On Task 2 the only GPT-5.6-terra run that divided by zero scored
214, and the two that never tried scored 110. On Task 10 one extra probe of a top-level return
separates 217 from 149. In these runs, the score depended on whether the agent tested that behaviour.
Which changes get found
Both agents found every change that the examples show, in every finished run (one GPT-5.6-terra run implemented one of them only partly, described below). Every change they missed completely was one the examples hide. Some of those changes were found far more often than others:
Share of dialects with this hidden change where the run found it
Show the data
| Hidden change | Claude Sonnet 5 (high) found | GPT-5.6-terra found |
|---|---|---|
| list items evaluated right to left | 1/1 | 3/3 |
| variables scoped to the function | 2/2 | 3/3 |
| and/or return an operand | 2/2 | 4/4 |
| negative indexes wrap or clamp | 2/2 | 2/3 |
| an error erases earlier output | 2/3 | 2/4 |
| return at top level ends the program | 1/2 | 2/4 |
| ranges include their end | 2/4 | 3/7 |
| division by zero gives a value | 2/2 | 1/6 |
| one global scope for everything | 2/4 | 2/7 |
| let on an existing name reassigns it | 0/3 | 2/3 |
| number vs string compares false | 2/4 | 1/7 |
| reading past the end gives nil | 2/6 | 0/10 |
| division rounds down | 1/3 | 0/6 |
| arguments passed by reference | 0/3 | 0/7 |
| lists copied into functions | 0/2 | 0/3 |
| and/or evaluate both sides | 0/1 | 0/3 |
| arguments evaluated right to left | 0/1 | 0/3 |
| extra arguments silently dropped | 0/1 | 0/3 |
| string index gives a character code | 0/2 | 0/4 |
Figure 3. Source: saved per-change regrades and transcript review. For each hidden change, the share of runs that implemented it correctly. Labels show found of total.
How agents find hidden changes explains the split. The manual lists its runtime errors in one table,
and the most productive habit in both models' transcripts was to run each listed error through the
button and compare. A dialect change that turns one of those errors into a normal result (division by
zero returning a value, a top-level return that ends the program quietly, comparing a number with a
string giving false) gets found this way.
The changes found least often raise no error at all. Arguments passed by reference (0 of 3 for Claude Sonnet 5, 0 of 7 for GPT-5.6-terra), lists copied when passed to a function, a string index returning a character code, arguments evaluated right to left. Nothing in the manual or the examples points at them. To find one, an agent has to imagine it first.
VIIIHow each model works
We read every transcript: 22 graded Claude Sonnet 5 runs and 36 GPT-5.6-terra runs. Both models read the manual, compared outputs, built an interpreter and tested it, but differed in when and how they used their experiments.
| Behaviour | Claude Sonnet 5 (high) | GPT-5.6-terra |
|---|---|---|
| First move after reading | Probes first. In 21 of 22 graded runs the first probe comes before any interpreter code, at about 2 minutes. | Builds first. Writes a complete interpreter, makes all 40 examples pass, then probes. |
| Probes used (median) | 33.5, about a third of the budget | 17.5, about a sixth of the budget |
| Most probes in one run | 152 of 200 | 83 of 200 |
| Probe design | Tracer functions, powers-of-two scans, binary searches over the socket, side-by-side checks against its own interpreter | Batches of one-line questions, and an error-table sweep in its best runs |
| Same task, repeated | Scores vary between attempts | Often the identical score and the identical changes found |
| Final message | A detailed summary, sometimes claiming checks it did not run | Always begins "Implemented submission/interp.py… matches all 40 visible examples". Never mentions what was left untested. |
Source: our reading of every transcript. Probe counts are exact where the agent printed the server's remaining budget, and counted from commands elsewhere.
Minutes the agent worked before declaring itself done
Show the data
| Model | Runs | Fastest (min) | Median (min) | Slowest (min) |
|---|---|---|---|---|
| Claude Sonnet 5 (high) | 19 | 10 | 19 | 51 |
| GPT-5.6-terra | 34 | 9 | 16 | 28 |
Figure 4. Source: saved run timestamps. Minutes each finished run worked before declaring itself done. The limit was 120.
Both models stopped with time and experiments left. No run used more than 51 minutes of 120, and no run used its whole probe budget. The median Claude Sonnet 5 run used a third of it, the median GPT-5.6-terra run a sixth. GPT-5.6-terra runs never discussed how many hidden changes might remain, although the instructions say there are between four and sixteen. Several Claude Sonnet 5 runs misread what is graded: the instructions say the withheld programs are checked, but not that the score counts exact matches on them, and some runs concluded that only crash-avoidance was scored and stopped as soon as the 40 examples passed.
In every perfect run the agent sent questions about meaning (evaluation order, scope, comparisons) to the button and to its own interpreter, side by side, and treated every disagreement as a lead. The runs stuck on the examples-only score either never asked the question that mattered, or asked it only of their own interpreter.
Moments from the transcripts
Each block below is copied from the recorded transcript. Times are seconds from the start of the run.
Claude Sonnet 5, Task 10: one comparison turns 149 into 300
While checking comparisons, the agent ran the same program through the button and through its own interpreter (848 s):
cat > /tmp/e14.src << 'EOF'
print 1 < 2 < 3;
EOF
python3 /tmp/probe2.py /tmp/e14.src
echo "=== mine ==="
python3 submission/interp.py /tmp/e14.src; echo status=$?
--- result ---
false
exit_status=0 remaining=177
=== mine ===
<<error>>
status=1
Its next message (894 s): "…the dialect may silently change other error behaviors too. Let me systematically test each documented error condition against the real probe." That sweep found the task's second hidden change. Another Claude Sonnet 5 run on the same task never sent these probes and scored 149, exactly the examples-only score.
Claude Sonnet 5, Task 6: tracer functions reveal evaluation order
A single 349-byte probe (594 s) uses a function that prints its argument, to show the order in which things are evaluated:
fn P(tag) { echo tag; return tag; }
fn two(a, b) { return a; }
echo two(P(1), P(2));
echo [P(3), P(4), P(5)];
...
--- result ---
1
2
1
5
4
3
[3, 4, 5]
Arguments run left to right, but list items run right to left (5 4 3). No example shows this, and no
error table would lead to it. This run scored 300. Another Claude Sonnet 5 run on the same task, which ended
early and is not among the finished runs, never tested evaluation order this way and scored 224.
Claude Sonnet 5, Task 3: "no scoping at all"
The agent compared the button's answer with its own interpreter on a closure-in-a-loop program (637 s):
let fs = [];
for (i in 0 .. 3) { fn g() { return i; } push(fs, g); }
print fs[0]();
print fs[2]();
--- result (probe, then own interpreter) ---
{"remaining": 150, "status": 0, "stdout": "2\n2\n"}
0
2
After two more probes it concluded (757 s): "This is a fundamental discovery … the dialect has no lexical/block scoping at all", just one flat global namespace. It rewrote the interpreter and scored 300. It is the only Claude Sonnet 5 run on this task to find that change, which controls 96 hidden programs.
Claude Sonnet 5, Task 4: one output line reveals two hidden changes
The agent first found that integers wrap at 8 bits (print 128; prints -128). Later (493 s):
print 200 / 3;
print 200 % 3;
--- result ---
STATUS=0 REMAINING=48
-19
-2
200 wraps to −56, and −56 / 3 rounds down to −19 instead of toward zero (−18). One line exposed both the 8-bit width and a hidden rounding change.
Claude Sonnet 5, Task 8: the right test, written and never sent
The agent wrote eight small programs, one per documented error, and ran all eight on its own
interpreter. One of them, e7, read past the end of a three-element list, exactly the dialect's
hidden change. Then (691 s):
"Now let's verify these against the real dialect via probe (combining calls isn't possible since errors halt execution, so I'll check a representative subset to confirm error format/status)"
for f in e1 e4 e5 e8; do echo "=== $f ==="; python3 t/myprobe.py t/$f.src; done
e7 was not in the subset. The run stopped at 746 s with 55 of 80 probes unused, and scored 145, the
examples-only score. The missed change controls 89 hidden programs.
Claude Sonnet 5, Task 5: asking its own interpreter instead of the button
The agent tested a mixed comparison, but only locally (787 s):
print 1 < "a";
=== cmpmix ===
<<error>>
status=1
"All correct, matching the spec's documented examples exactly."
On the same task another Claude Sonnet 5 run sent print 1 < true; to the button and got false. That run
scored 300. This one scored 245. Comparing against the button caught the change that the local test missed.
GPT-5.6-terra, Task 3: one hidden change hides another, and the run falls below the floor
The agent probed negative string indexes (about 467 s):
print "abc"[-1];
print "abc"[-4];
--- output
<<error>>
The first line really printed c. The second index was out of range and raised an error, and this
dialect has a second hidden change: an error erases everything printed before it. So the c vanished.
The agent concluded that negative indexes do not work on strings and restricted them to lists:
sed -i '180c\ if i<0 and type(a)is list:i+=len(a)' submission/interp.py
It lost 12 programs that an examples-only interpreter gets right and scored 102, the only run of either
model below the examples-only score. Another GPT-5.6-terra run tested "abc"[-1] and "abc"[-3], both in range,
saw no error, and scored 300.
GPT-5.6-terra, Task 2: one division by zero, +104 programs
print 1 / 0;
---
REF
1
OURS status=1
<<error>>
Its next probe divided and took remainders of 7, -7 and 0 by zero, and all returned the left
operand. The agent: "One additional dialect rule surfaced: division and remainder by zero do not error",
and both return the left operand. This run scored 214. The other two GPT-5.6-terra runs on the task never divided
by zero and scored exactly 110.
GPT-5.6-terra, Task 8: the same question, asked two ways
Two runs tested whether let inside a block creates a new variable. One printed the outer variable
afterwards:
let a = 1;
if (true) { let a = 2; }
print a;
---
2
It concluded that let rebinds an existing name, and scored 211. The other read the variable only
through a closure inside the block, which prints the same value under either rule, reported that
scope "matches the base language", and scored 145, the examples-only score.
GPT-5.6-terra, Task 9: evidence against its own hypothesis, ignored
The agent decided negative indexes wrap Python-style. A later probe (about 640 s) said otherwise:
let a = [1, 2];
a[1] = 9;
print a;
a[-1] = 8;
print a;
{'remaining': 71, 'status': 1, 'stdout': '[1, 9]\n[8, 9]\n<<error>>\n'}
Python-style wrapping would give [1, 8]. The dialect gave [8, 9]: it clamps -1 to the first
element. The agent kept its original rule, scored 0 of 56 on that change, and finished on the
examples-only score of 147. Both other GPT-5.6-terra runs on the task found the clamp and scored 182.
GPT-5.6-terra, Task 10: one probe worth 68 programs
fn f() { print 7; return 1; }
print 1;
return f();
--- probe vs ours
1
7
[ours]
1
<<error>>
status=1
The agent: "a top-level return is valid, evaluates its expression, and ends the program successfully.
I’m incorporating that behavior now." The two GPT-5.6-terra runs that sent this probe scored 217. The two that
did not scored 149. The whole gap is this one question.
IXDid either model try to cheat?
We found no evidence of it. We searched every command, every file read and every file written in all 62 transcripts (26 Claude Sonnet 5, 36 GPT-5.6-terra) for any attempt to reach the hidden programs, the dialect's settings, the reference interpreter, the grader or the network, or to influence the score. We found none in either model.
The few commands that looked outside the task folder were diagnostic: locating the task directory,
checking why ./probe would not run, and checking whether the button's service was still alive after it
stopped answering. Some runs wrote their own client for the button's socket to see the exit code and
remaining budget. That uses the documented interface, and the service still enforced the budget. Every
submission we examined is a genuine interpreter that reads only the program it is given.
Conclusion
What stood out to us was how often the agents stopped without testing the hidden changes. They found every change shown by the 40 examples, with one partial implementation, but often left behaviour such as argument passing and evaluation order unchecked. No run used even half of its two hours. The median run used a third (Claude Sonnet 5) or a sixth (GPT-5.6-terra) of its experiments.
The transcripts helped explain the scores. On Task 2, a division-by-zero probe was worth 104 programs.
On Task 10, a top-level return was worth 68. Comparing the button's answer with the agent's own
interpreter exposed differences that testing the interpreter against the manual could not. We want
to train agents to make those comparisons more often, including when nothing has gone visibly wrong.
The manual-only and examples-only scores let us see what the agents had actually discovered. We also had to fix the grader and check that our red-team submissions had reached it. The results here still describe the earlier agent runs, including the probe-service failures.
References
- Acher, M. (2026). Can Coding Agents Program in M&Ms Language? blog.mathieuacher.com
- Lu, K.-C., & Krishnamurthi, S. (2024). Identifying and Correcting Programming Language Behavior Misconceptions. OOPSLA 2024. Author publication page, DOI 10.1145/3649823
- Pombrio, J., Krishnamurthi, S., & Fisler, K. (2017). Teaching Programming Languages by Experimental and Adversarial Thinking. SNAPL 2017. doi:10.4230/LIPIcs.SNAPL.2017.13
- Sharma, A., & Chopra, P. (2026). EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages. arXiv:2603.09678
- Sharma, A., Thorat, S., & Chopra, P. (2026). Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages. arXiv:2606.10933
- Thimmaiah, A., Zhang, J., Srinivasa, J., Li, J. J., & Gligoric, M. (2025). LLMs Lean on Priors, Not Programming Language Semantics (PLSemanticsBench). arXiv:2510.03415
- Wei, A., et al. (2025). CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis. arXiv:2503.23145
- Wu, Z., et al. (2023). Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. NAACL 2024. arXiv:2307.02477
- Yin, C., et al. (2025). Investigating Advanced Reasoning of LLMs via Black-Box Environment Interaction (ORACLE). arXiv:2508.19035
- Adamczewski, T., et al. (2026). MirrorCode: AI can rebuild entire programs from behavior alone. arXiv:2606.30182
- Yang, J., et al. (2026). ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv:2605.03546