Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, 2018. §1.1, §1.2, §1.3.
- Publisher
- Cambridge University Press
- ISBN
- 9781107286184
- DOI
- 10.1017/9781107286184
- Shelf
- Severity and error
- Concepts
- Severity, Convergent paths
Why it is hereThe book the programme takes its working definition of evidence from: a claim is warranted to the extent it has survived a test that could have caught it out. Excursion 1, Tour I sets the definition and the stakes.
The underlinings across the pages
pp. 4–28 · 49 marks
structure (red in Zotero) definition (green) key passage (yellow) book pages · pages not read are folded
1.1 Severity Requirement: Bad Evidence, No Test (BENT)
pp. 5–12
If a method was practically bound to find agreement with a claim, the agreement is no evidence for it. That minimal requirement is what the statistics wars keep losing sight of.
- The weak requirement Data that agree with a claim are bad evidence when the method could not have found the claim’s flaws had there been any.C012 →Conceptual tool
- Only if you want to learn A genuine test means risking that the conjecture is found wanting. Perverse publishing incentives push the other way, so the wise invite methods that expose their errors quickly.
- Forward-looking tools Big Data and machine learning bring results without statistical thinking. Statistical methods are not for reconstructing reasoning after the fact: they are for finding things out, and for judging how well others have done it.C016 →Working method
- A meta-methodology Severity is applied one level up, to methods: how capable is a method of detecting and avoiding wrong readings of data? The word is Popper’s, and it has no definition and no place in any statistical method yet.
p. 9“term “severity” is Popper’s, though he never adequately defined it”
C017 →Working methodC031 →Technical reference
1.2 Probabilism, Performance, and Probativeness
pp. 13–22
The two reigning philosophies of statistics ask either how probable a claim is or how often a method errs in the long run. Neither asks what severity asks: how well this claim, here, was probed.
- Neither probabilism nor performance What is at stake is a critical standpoint, and understanding inference is about reasoning with integrity, not only about mathematics.C030 →Conceptual tool
- The strong requirement A claim is supported to the extent it passed a test highly capable of finding its flaws and few or none were found.C013 →Conceptual tool
- Lift-off and drag-down An overall inference can be more reliable than any of its premises, or only as reliable as its weakest; convergent independent checks give lift-off.C014 →Conceptual tool
- Arguing from error An error is shown absent when a procedure very capable of signalling it, if and only if it is present, signals nothing.C015 →Working method
- Selection invalidates P-values and every error probability can be ruined by how data or hypotheses were selected: apophenia, the fund that mails out only its luckiest portfolio, the Texas sharpshooter.C032 →Corrective evidence
1.3 The Current State of Play in Statistical Foundations
pp. 23–29
Severity is what we already demand as consumers of claims, and it is not the property of one school: the Bayesian–frequentist dispute is a dispute between probabilism and performance, and severe testing can be applied to the tools of either.
C036 →Conceptual tool
Claims drawn from this card
Each claimA sentence from a source, kept verbatim, that the programme stands on, uses or has to answer to. shows its ENTIAThe notation this site uses on itself: what each piece is for, read through five layers. functionWhat a claim does for the programme: a conceptual tool, a working method, corrective evidence, a technical reference.; open + for its stanceHow the programme holds the claim: assumed, used, or contested., claim typeWhat kind of statement it is, so that the right test can be applied. Proposed until checked by hand. and relationsHow one claim bears on another, in the CiTO vocabulary: refines, extends, presupposes, confirms, contests..
+
assumed D1.8 ethical-normative+
used D1.8 ethical-normative+
used D1.3 logical-formal+
used D1.4 mathematical-analyticParaphrase CC BY-NC 4.0, Claudio Cammarano. Quoted passages belong to their author and are kept short; the text itself is not distributed. Notes from the Zotero library, updated weekly.