Claims › Concept · How a verdict is made
Severity
When does passing a test count as evidence?
A result counts as evidence only if the procedure that produced it could have caught the error: agreement a method was bound to find shows nothing.
Linked concepts: Verdict · Convergent paths · Falsification
Where it sits
How its claims lean on each other
The claims
-
“One does not have evidence for a claim if nothing has been done to rule out ways the claim may be false. If data x agree with a claim C but the method used is practically guaranteed to find such agreement, and had little or no capability of finding flaws with C even if they exist, then we have bad evidence, no test (BENT).”
Agreement between data and a claim is not evidence when the method could not have found the claim false. This is the negative, minimal criterion the programme uses to tell a validation that tested something from one that only confirmed what it presupposed.
Mayo 2018 · p. 5, Excursion 1, Tour I, §1.1 · assumed · weight 1 · open C012
-
“We have evidence for a claim C just to the extent it survives a stringent scrutiny. If C passes a test that was highly capable of finding flaws or discrepancies from C, and yet none or few are found, then the passing result, x, is evidence for C.”
The positive counterpart of the weak requirement: evidence is graded by how capable the test was of finding flaws. It matters because it turns severity from a veto into a measure, which is what a validation protocol needs to rank results rather than just reject them.
Mayo 2018 · p. 14, Excursion 1, Tour I, §1.2 · assumed · weight 1 · extends C012 · open C013
-
“An overall inference can be more reliable and precise than its premises individually.”
Convergent, independent checks can warrant a conclusion more strongly than any single one of them. This is the epistemic ground for aggregating multiple fallible runs or judges instead of demanding a single infallible one.
Mayo 2018 · p. 15, Excursion 1, Tour I, §1.2 · used · weight 1 · open C014
-
“There is evidence an error is absent to the extent that a procedure with a very high capability of signaling the error, if and only if it is present, nevertheless detects no error.”
Absence of a detected error counts only if the procedure would have signalled that error had it been there. It is the operational form of severity, and the test any judge-based validation has to pass before its silence can be read as a result.
Mayo 2018 · p. 16, Excursion 1, Tour I, §1.2 · assumed · weight 1 · refines C013 · open C015
-
“Statistical methods should not be seen as tools for what philosophers call “rational reconstruction” of a piece of reasoning. Rather, they are forward-looking tools to find something out faster and more efficiently, and to discriminate how good or poor a job others have done.”
Mayo treats inferential methods as instruments for finding things out and for grading others' work, not as after-the-fact justifications. The programme reads validation protocols the same way: as probes that can fail, not as rationalisations of verdicts already reached.
Mayo 2018 · p. 8, Excursion 1, Tour I · used · weight 1 · open C016
-
“A severity scrutiny is going to be a key method of our meta-methodology. It is fairly obvious that we want to scrutinize how capable a statistical method is at detecting and avoiding erroneous interpretations of data.”
Severity is applied one level up, to methods rather than to claims: a method is judged by its capacity to catch erroneous interpretations. This is the move the programme makes when it treats severity as a meta-level property of validation procedures.
Mayo 2018 · p. 9, Excursion 1, Tour I, Methodology and Meta-methodology · assumed · weight 1 · presupposes C013 · open C017
-
“What happened to the goal of scrutinizing BENT science by the severity criterion? Neither “probabilism” nor “performance” directly captures that demand.”
Neither of the two dominant philosophies of statistics, assigning degrees of belief or securing long-run error rates, asks whether a test could have caught the error. That gap is why the programme cannot borrow its validation criterion from either school and has to state severity on its own.
Mayo 2018 · p. 13, Excursion 1, Tour I · assumed · weight 1 · presupposes C012 · open C030
-
“The term “severity” is Popper’s, though he never adequately defined it. It’s not part of any statistical methodology as of yet.”
Severity comes from Popper but arrived without a working definition and without a place in any statistical method. Mayo’s contribution, and the programme’s starting point, is that operational definition: the word is old, the criterion is not.
Mayo 2018 · p. 9, Excursion 1, Tour I, Methodology and Meta-methodology · used · weight 1 · refines C017 · open C031
-
“P-values can be readily invalidated due to how the data (or hypotheses!) are generated or selected for testing.”
A significance level is valid only relative to how data and hypotheses were selected; selective reporting makes a hard result look easy to explain away as chance. For the programme this is the warning attached to any choice of which outputs or cases enter a validation set.
Mayo 2018 · p. 17, Excursion 1, Tour I · used · weight 1 · open C032
-
“After learning enough about certain types of mistakes, we may construct (often from other tests) a testing procedure with an overwhelmingly good chance of revealing the presence of a specific error, if it exists-but not otherwise. Such a testing procedure may be called a severe (or reliable) test, or a severe error probe.”
A severe test is built from prior knowledge of specific mistakes: a probe that would reveal a given error if present, and stay silent otherwise. This is the original formulation of the idea, and it ties severity to an error repertoire rather than to probability in a hypothesis.
Mayo 1996 · p. 7, ch. 1, §8 · assumed · weight 1 · confirms C015 · open C033
-
“The idea of error statistical foundations for Bayesian tools is not as preposterous as it may seem. The concept of severe testing is sufficiently general to apply to any of the methods now in use.”
Severity is not tied to one statistical school: any inference can be judged by whether it withstood severe testing. That generality is what licenses the programme to apply it to LLM validation, a domain Mayo did not consider.
Mayo 2018 · p. 27, Excursion 1, Tour I · assumed · weight 1 · extends C017 · open C036
-
“methods for inductive inference in science are error correcting; the justification for inductive methods of experimental testing in science is that they are self-correcting.”
Peirce justified induction not by the truth of its conclusions but by the method’s capacity to correct its own errors over time. It is the older root of error statistics, and the ground for treating a validation protocol as justified by what it would catch, not by what it has found so far.
Mayo 2005 · p. 299 · used · weight 1 · presupposes C038 · open C037
-
“Induction is the experimental testing of a theory. The justification of it is that, although the conclusion at any stage of the investigation may be more or less erroneous, yet the further application of the same method must correct the error.”
Induction is justified by the method, not by any single conclusion: persisting in it corrects whatever error a stage contains. It is the original of the self-correcting thesis that Mayo turns into the basis of error statistics.
Peirce 1931–1958 · CP 5.145 · quoted in Mayo 2005, p. 299 · used · weight 1 · open C038
-
“The theory here proposed does not assign any probability to the inductive or hypothetic conclusion, in the sense of undertaking to say how frequently that conclusion would be found true. […] The theory here presented only says how frequently, in this universe, the special form of induction or hypothesis would lead us right.”
Probability attaches to the method’s track record, not to the hypothesis. The programme inherits the split: a validation reports how often a procedure gets it right, never how probable a given verdict is.
Peirce 1931–1958 · CP 2.748 · quoted in Mayo 2005, p. 300 · assumed · weight 1 · open C039
-
“The process of testing it will consist, not in examining the facts, in order to see how well they accord with the hypothesis, but on the contrary in examining such of the probable consequences of the hypothesis . . . which would be very unlikely or surprising in case the hypothesis were not true.”
A test does not collect agreement; it looks for consequences that would be surprising if the hypothesis were false. This is severity before the word, a century before Mayo, and the reason fit with the data is never enough on its own.
Peirce 1931–1958 · CP 7.231 · quoted in Mayo 2005, p. 304 · assumed · weight 1 · confirms C013 · open C040
-
“It is true that the observed conformity of the facts to the requirements of the hypothesis may have been fortuitous. But if so, we have only to persist in this same method of research and we shall gradually be brought around to the truth.”
A lucky agreement is exposed by persisting with the same method: replication is the correction. For validation, a single passing run is not a result until the procedure has been run again.
Peirce 1931–1958 · CP 7.115 · quoted in Mayo 2005, p. 309 · used · weight 1 · refines C038 · open C041
-
“Nor must we lose sight of the constant tendency of the inductive process to correct itself. This is of its essence. This is the marvel of it. ...even though doubts may be entertained whether one selection of instances is a random one, yet a different selection, made by a different method, will be likely to vary from the normal in a different way, and if the ratios derived from such different selections are nearly equal, they may be presumed to be near the truth.”
Two selections made by different methods err in different ways, so their agreement is evidence about the truth. It is the convergence argument the programme uses when independent runs or coders reach the same verdict.
Peirce 1931–1958 · CP 2.729 · quoted in Mayo 2005, p. 315 · assumed · weight 1 · confirms C014 · open C042
-
“...the negative fact supposed by me [no mysterious force interferes with the action of chance] is merely the denial of any major premise from which the falsity of the inductive conclusion could be deduced. Actually so long as the influence of this mysterious source not be overwhelming, the wonderful self-correcting nature of the ampliative inference would enable us, even so, to detect and make allowance for them.”
Induction needs no uniformity of nature, only the absence of an overwhelming hidden bias, and lesser biases are themselves detectable by the method. A validation does not have to assume a clean world, only one where its errors are not engineered against it.
Peirce 1931–1958 · CP 2.749 · quoted in Mayo 2005, p. 316 · used · weight 1 · open C043
Reading cards on this concept