Work · Verifying AI systems
Quantitative method
Every system that grades its own exam passes it. This is a method that makes a negative result possible — built on crypto markets, applied to the AI systems we ship.
Ten coin flips the same way is not rare, and in enough data you will always find a rule that looks true. That alone is why this ledger exists: every line is a way we could have fooled ourselves, and a rule written in advance that closes it.
Press, and watch the two numbers move in opposite directions.
Standing rules, written into the project's policy before any run — not a tally of results.
Three independent out-of-sample checks, on every line.
- Test until something passesThe hypothesis is registered in writing before it runs: population, horizon, feature, predicted direction — and the condition that would falsify it.Closed
- Interpret the result after seeing itThe statement stands verbatim as written, and the falsification is reported rather than edited away.Closed
- A single arm that looks goodHypotheses are registered as opposed pairs — a single arm that looks good cannot be falsified at all.Closed
- Not knowing what chance looks likeA null control runs through the same pipeline: one that finds an edge in shuffled data is broken, and that has to be found before it costs money.Closed
- Look again every time the data growsNo look before the registered date or the required count, whichever is later — and any earlier look is written down first.Closed
- Test on what you developed onA holdout of instruments never used in development, not analysed and with no look spent on it.Closed
- Run whatever looks goodA written gate governs what runs live, with a kill switch, risk caps, and a forward ledger that records every signal from the moment it fires.Closed
There is deliberately no counter here. The number of negative results changes every week; what makes a result worth believing is these rules, and they do not.
The problem
Ten coin flips landing the same way is not rare enough to mean anything, and with enough data and enough attempts you will always find a rule that looks true about the past. Randomness manufactures convincing patterns; that is a property of it, not a mistake. So most failure modes are ways of fooling yourself: fitting noise, testing until something passes, or measuring against the wrong benchmark. Crypto markets never close and move in minutes, which multiplies the number of attempts and the coincidences that come with them. And the problem is not particular to markets: any system — a machine-learning model, a quality gate, a language model — measured against a benchmark drawn from itself will always pass its own exam. The engineering problem is a process that can return a negative answer and be believed.
Approach
- Each hypothesis is registered in writing before it is tested — population, horizon, feature, predicted direction, and the condition that would falsify it — so a result cannot be reinterpreted once it appears. They are registered as opposed pairs, because a single arm that looks good cannot be falsified at all. When one falls, its statement stands verbatim and the falsification is published rather than the statement edited.
- From the record, in its own words: one hypothesis predicted the probability of backtest overfitting would exceed 0.5 on the exit and stop grids, on the grounds that "near-identical variants make the in-sample winner a coin flip" — falsified, and the opposite happened. A second predicted a classifier AUC above 0.55 — also falsified: it came in at 0.5357, inside the null distribution itself.
- Before any positive result is trusted, the shape of a false positive is established: an empirical null control is run through the same pipeline to see what it produces when there is nothing there to find. A check whose failure mode is unknown is not a check.
- Validation runs out of sample in three independent ways, one of them a holdout of instruments never used during development.
- What holds for a trading rule holds literally for a machine-learning model. Classifiers inside the platform are measured with purged cross-validation and a time embargo that stops the future leaking into training, preceded by a shuffled-label control run first to establish what the pipeline produces when there is nothing there to find. That is the same audit we put any AI system through before we hand it over: the shape of a false result is fixed before a real one is believed.
- The method catches its own errors first. A number that three documents in the project described as "unfiltered" turned out to be a doubly-filtered arm on a much smaller sample — caught by an independent replay that two separate code paths reproduced to the last digit. Another — that 1% of trades carry 55% of the profit — is true in percent and false in the ledger's own unit; the fat tail was an artefact of the measure, not a property of the result.
- A written policy governs what may run live, the executor carries a kill switch and risk caps, and a forward ledger records every signal from the moment it fires onward — so what gets measured later cannot be selected out of the past.
- The tooling is built in the studio: algorithms, software, bots, and an automated platform that runs in real time on a cloud server without interruption. It scans crypto markets on intervals of minutes, computes features and machine-learning labels, tests strategies, and sends alerts to a messaging bot.
Measured
Built with
- Python
- pandas
- NumPy
- SQLite
- systemd
Stated plainly
Presented as methodology only; the work under way today is on crypto markets. No performance figures are shown and none should be inferred — a research process transfers between domains, a track record does not.