The research continues. None of the current deliverables can be shared yet: they are in progress, and one of them is headed for anonymous review, which rules out describing it here. What can be described is the shape the work is taking, because that shape has changed.
One stage at a time
Until September the programme lived in a single working paper. By v7 it carried a taxonomy, an argument about epistemic self-classification, a procedure for causal claims, a corpus, a vertical validation, a multi-model control and a pre-registered experiment. Each part needed the others to be read, and the whole was hard to follow and harder to place with any journal. Splitting it into an ontological and an empirical line was the first step. The second is to give the work a fixed order, in three stages, each with its own document:
- a theoretical paper that sets out the model against the literature;
- a check of that model on a corpus;
- an experimental validation run with other people.
Three documents look more complicated than one long draft. In practice each stage can be read, judged and, if it fails, corrected without dragging the others along. A reader who wants the argument does not have to go through the statistics first, and a referee who doubts the statistics does not have to accept the argument in order to test them.
α1 — the theory
A purely theoretical paper, built on the literature: an argument, with no data and no experiment. While it is being prepared for double-blind review I can say almost nothing more about it. It will have a note of its own when the process allows.
α2 — the check
The first verification of the model, and the point where the material changes. Until now the taxonomy of claims has been tested on text written by language models. The theoretical work made one gap plain: so far the taxonomy has only been applied to the kind of text it was built from. Before any experimental phase, it has to meet text it was not built from.
α2 applies it to the propositions of War and Peace, in Louise and Aylmer Maude’s English translation. The choice is partly a provocation. In one closed text, Tolstoy’s novel holds nearly every kind of statement the taxonomy has to tell apart: narrated fact, historical claim, a character’s belief reported by the narrator, and long essayistic passages on causation and the laws of history, up to the second epilogue. If the categories hold on a canonical human text of that size, model output becomes one case among many. If they fail, the failures will show where.
2027 — the empirical line
Once α1 and α2 are complete, the empirical line resumes as the third stage. It keeps what was announced in September (vertical validation, blind coding, the uplift experiment) and is scheduled for 2027. One person cannot run it: blind and double-blind coding and inter-annotator agreement need annotators who did not design the categories. If you work in annotation, corpus linguistics or philosophy of science and would consider taking part, write to claudio@claudiocammarano.com.
The next note will be about α1 alone.
The formal apparatus of this note was built in dialogue with a language model and verified line by line. The verification is the part that counts: no step was accepted on the strength of the output’s apparent authority.