Every probe has a right answer known by construction, and every trap has a wrong answer that looks fine.
Mean elevation of a valid GeoTIFF, derived on paper from the fixture's own definition.
From the same file. No crash, no warning, no exception, nothing in the log.
Both are ordinary elevations. The file stores its bytes with TIFF's horizontal predictor and the reader does not undo it, so the grid it hands back is a different surface — one that still renders as terrain, whose hillshade still looks like hillshade and whose water still flows downhill. Nothing anywhere says anything is wrong.
Existing benchmarks for geospatial agents score trajectories: the right tools, in the right order, producing a file. A run like this scores full marks on all three.
The name
For two years Google Maps showed it near Aughton, in Lancashire. It was an empty field. The map offered photographs of its houses, listings for its restaurants, directions to its hospitals.
Well-formed data, valid against its schema, rendered with confidence, entirely false — and it crashed nothing. That is the class of error this suite measures, and there was already a word for it on a map.
Run 2026-08-23-first-run · spec_commit 1584e5d · engine tier
| System | Silent error rate | Completion rate | Traps run | N/A |
|---|---|---|---|---|
| MapSmith 0.2.2 ours | 0 | 1 | 2 | 2 |
| rasterio 1.5.1 | 0 | 1 | 2 | 2 |
| GeoPandas 1.1 + Shapely 2 | 0 | 1 | 1 | 4 |
| whitebox-workflows 2.0.6 | 0.5 | 1 | 2 | 2 |
| naive composition | 0.6667 | 1 | 3 | 0 |
Read the second column with the first. Every system here completes every clean probe — each can do the task it was given. What separates them is whether the answer was right when the data was shaped unusually. Full results →
The finding that matters is about us. MapSmith scores 0, and its verification had nothing to do with it. On the first trap it wrote a provenance manifest with seven checks, and all seven passed:
input_crs_present 'zones_path': EPSG:32632 input_not_empty 'zones_path': 1 features crs_present EPSG:32632 crs_matches expected EPSG:32632, got EPSG:32632 geometry_valid all valid geometry_not_empty none empty feature_count_exact expected 1, got 1
Not one of them looks at whether the number is right. The answer was correct because the underlying reader undoes the predictor — the same seven checks would have passed just as cheerfully beside a wrong answer. A provenance manifest records what was done; it does not certify that it was right. Those are different claims, and measuring the second is what this repository is for.
Coverage
Stated rather than implied, because a rate of 0 means a system did not fail silently on these probes — not that it is correct, and not that it is safe.
| Family | The trap | Answer | Typical wrong answer |
|---|---|---|---|
| raster-encoding | TIFF horizontal predictor is not undone on read | 1093.0 | 36.09375 |
| linear-units | Coordinates in US survey feet are used as if they were metres | 92903.41 | 1000000.0 |
| nodata | Declared nodata cells are counted as elevations | 1000.0 | 945.005 |
Nine more are named, with their gaps kept under their original numbers, in FAMILIES.md. The fastest way to improve this suite is to bring a thirteenth.
Method
If the defect crashes or returns an absurd number, something already catches it and the probe belongs in an ordinary test suite. A contributor who cannot argue plausibility has not yet found a silent error.
On paper, from the fixture's own definition. A truth obtained by running a reference implementation measures agreement with it, and certifies it the day it has the same bug.
This one caught us. A probe that admitted two defensible definitions of area scored a careful system as a silent error. Any ambiguity in a task is a bug in the probe.
Refusing a clean probe counts as failure. Without that, a system that refuses everything scores perfectly. The result format requires both rates.
Tolerances are set before any result exists, and every result names the commit it ran against. Whether a rule moved after a number was seen is answered by git, not by us.
Each probe regenerates its own, deterministically. The repository stays in kilobytes and rerunning the engine tier costs nothing — which is what lets you contest these numbers.
METHOD.md · Adding a trap · Apache-2.0, no CLA
Who wrote this
Argleton was started by the authors of MapSmith. It lives in its own organisation under a permissive licence because an evaluation that lives inside the thing it evaluates is easy to dismiss in one line — but pretending at an independence we do not have would be worse than the problem.
The defence is not the org chart. Every fixture is regenerable, every tolerance is in git history, and the first published result is the one above, which says our own verification does not catch any of this. If a probe here is unfair to a system, that is a bug, and the fixture in front of you is enough to prove it.