Claims

A scientific claim is not a headline: it withholds itself until you have the context, and no two readers arrive with the same context. So this is not a list to scroll — it is 15 questions, and the claims come through them.

These are not claims in general. Every one of them is on file because this programme stands on it, borrows a method from it, or has to answer to it. Nothing is here for completeness, and the archive is not trying to cover a field: it is showing what its own work rests on, including the parts that push back.

The claims, cored

Every claim is a core drilled through the five layers of ENTIA, from the least to the most exposed to error. Read down a column for one claim, across a band for one layer; the arcs underneath are the relations between claims.

1 · Free tag2 · Topic3 · Theme4 · Concept5 · Geometryweight, 0.5 → 2The claimsCaCammaranoCaCammaranoCaCammaranoMiMitchell · KrakauerBeBender · KollerAnAndreasArArtstein · PoesioSrSrivastavaLiLiangMayoMayoBaBaiVaVaswaniClaim typesClaim typesIntros…IntrospectionMeaningMeaningAnnotationAnnotationEvalua…EvaluationSeveritySeverityFalsificationFalsificationTru…TruthfulnessArchit…ArchitectureWhat a machine cannot be said to …What a machine cannot be said to doHow a verdict is madeHow a verdict is madeWhat scale d…What scale does not buyClaim typeClaim typeEpiste…Epistemic self-classificationFor…Form–meaning gapAgent modelVer…VerdictCod…Coder agreementVerdictMeasurement artefactDeclared incompletenessSeveritySeverityFalsificationFalsificationInverse scalingImitative falsehoodMachine judgeConvergent pathsSequence without recurrenceAlso: Measurement artefactAlso: Agent modelAlso: Coder agreementAlso: Declared incompletenessAlso: Measurement artefactAlso: Convergent pathsAlso: Machine judgeAlso: Inverse scalingAlso: Convergent pathsAlso: Convergent pathsAlso: Imitative falsehood×1.5×1.5×1.5×1.5C001C001 · Before asking whether a statement is true, you have to settle what kind of statement it is: the test that fits a measurement does not fit a value judgement.C046C046 · Conceptual frameworks are made, not given, and can be rebuilt critically.C002C002 · Two claims of different kinds share no standard of truth, so a procedure that works on one returns a meaningless verdict on the other.C003C003 · One dimension of the taxonomy decides what a claim is; the other six describe how it behaves once that is settled.C004C004 · A measurement, an experiment and an argument are accountable to different things, so they cannot be asked to share one document and one standard of proof.C005C005 · Two independent findings pointing the same way is not evidence that they are connected.C009C009 · The programme used to assert flatly that the producing system performs no metacognition over its own outputs.C010C010 · The premise the whole framework rests on, in its narrow form: a model may represent that a passage is in Spanish, or that it is being evaluated, without representing that the assertion it is about to make is causal rather than correlational.C011C011 · What a model reports about its own workings is not a log of them.C007C007 · The dispute over machine understanding is not one question but two pictures of what understanding is: tracking statistical regularities, or holding a causal model of how things work.C008C008 · A system trained only on the shape of language has nothing in its input that could tell it what the language is about.C006C006 · Predicting the next word can require the model to build a partial picture of what the writer believed and wanted.C019C019 · A machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent.C022C022 · A number nobody else can recompute from the deposited materials is testimony, not evidence.C018C018 · When measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are.C020C020 · Thirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways.C021C021 · Asked to rank answers, a strong model agreed with human raters about as often as human raters agreed with each other, above eighty per cent.C024C024 · Abilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric.C027C027 · An evaluation should publish the distance between what it set out to measure and what it actually measured.C012C012 · Agreement between data and a claim is not evidence when the method could not have found the claim false.C013C013 · The positive counterpart of the weak requirement: evidence is graded by how capable the test was of finding flaws.C014C014 · Convergent, independent checks can warrant a conclusion more strongly than any single one of them.C015C015 · Absence of a detected error counts only if the procedure would have signalled that error had it been there.C016C016 · Mayo treats inferential methods as instruments for finding things out and for grading others' work, not as after-the-fact justifications.C017C017 · Severity is applied one level up, to methods rather than to claims: a method is judged by its capacity to catch erroneous interpretations.C030C030 · Neither of the two dominant philosophies of statistics, assigning degrees of belief or securing long-run error rates, asks whether a test could have caught the error.C031C031 · Severity comes from Popper but arrived without a working definition and without a place in any statistical method.C032C032 · A significance level is valid only relative to how data and hypotheses were selected; selective reporting makes a hard result look easy to explain away as chance.C033C033 · A severe test is built from prior knowledge of specific mistakes: a probe that would reveal a given error if present, and stay silent otherwise.C036C036 · Severity is not tied to one statistical school: any inference can be judged by whether it withstood severe testing.C037C037 · Peirce justified induction not by the truth of its conclusions but by the method’s capacity to correct its own errors over time.C038C038 · Induction is justified by the method, not by any single conclusion: persisting in it corrects whatever error a stage contains.C039C039 · Probability attaches to the method’s track record, not to the hypothesis.C040C040 · A test does not collect agreement; it looks for consequences that would be surprising if the hypothesis were false.C041C041 · A lucky agreement is exposed by persisting with the same method: replication is the correction.C042C042 · Two selections made by different methods err in different ways, so their agreement is evidence about the truth.C043C043 · Induction needs no uniformity of nature, only the absence of an overwhelming hidden bias, and lesser biases are themselves detectable by the method.C045C045 · Dogmatic falsificationism treats the experimenter as Nature’s neutral voice, delivering verdicts the theory cannot touch.C051C051 · Once ceteris paribus clauses are allowed, a theory that survives every challenge looks more scientific, not less: the reductio of dogmatic falsificationism.C034C034 · Falsificationism explains why failing a test counts against a hypothesis, but not why passing one counts for it.C035C035 · Beating rivals is not passing a severe test: the test has to probe the ways this particular hypothesis could be wrong.C044C044 · If every universal theory has zero probability on any evidence, probability cannot be what separates good theories from bad ones.C047C047 · Every test bottoms out in basic statements that someone decides to accept; logic alone never forces the stop.C048C048 · What is decided by agreement is the singular observation, not the theory, and no experience justifies it: it is accepted.C049C049 · A theory forbids an event only under a ceteris paribus clause, so a failed prediction never refutes it alone.C050C050 · Neither proof, nor probability, nor refutation can be read off the facts: all three fail together.C052C052 · An observation counts as basic because a trained technique makes it decidable, not because it is given.C053C053 · Every test decides what is under test and what is taken for granted, and that line is a decision, not a fact.C054C054 · Methodological falsificationism keeps hard criticism while admitting every statement is fallible, by making the decisions explicit.C026C026 · On questions written to echo common misconceptions, the larger models answered falsely more often than the smaller ones, reversing the usual pattern.C028C028 · Across the field, hallucination means generated content that is nonsensical or unfaithful to the source it was given: a definition about fidelity to the input, not about truth in the world.C023C023 · Harmlessness was trained by having the model criticise and revise its own answers against a written list of principles, instead of having people label every harmful output.C025C025 · When several different chains of reasoning arrive at the same answer, that agreement is itself evidence for the answer.C029C029 · The architecture behind current language models dropped step-by-step processing of a sequence in favour of computing every relation between positions at once.C001 refines C002C003 extends C001C004 extends C001C005 refines C020C010 refines C009C010 refines C006C011 confirms C010C007 presupposes C008C006 contests C008C022 refines C019C018 refines C019C020 uses-method-in C019C021 contests C019C013 extends C012C015 refines C013C017 presupposes C013C030 presupposes C012C031 refines C017C033 confirms C015C036 extends C017C037 presupposes C038C040 confirms C013C041 refines C038C042 confirms C014C051 presupposes C049C034 refines C031C035 extends C034C044 confirms C039C048 refines C047C050 extends C044C052 refines C048C054 contests C045

Read it

Point at a column, or tap it, to read one claim through its five layers. Each mark says what kind of warrant the claim lives on: • observed, ∴ derived, ∃ what there is, ! what should be done. Green comes from the literature, bordeaux from the research log; a ring means the programme contests it.

What a machine cannot be said to do

Meaning

  1. C007Form–meaning gapAgent modelThe dispute over machine understanding is not one question but two pictures of what understanding is: tracking statistical regularities, or holding a causal model of how things work.Mitchell & Krakauer 2023 · Mitchell · Krakauer
  2. C008Form–meaning gapA system trained only on the shape of language has nothing in its input that could tell it what the language is about.Bender & Koller 2020 · Bender · Koller
  3. C006Agent modelPredicting the next word can require the model to build a partial picture of what the writer believed and wanted.Andreas 2022 · Andreas

Claim types

  1. C001Claim typeBefore asking whether a statement is true, you have to settle what kind of statement it is: the test that fits a measurement does not fit a value judgement.Building the Taxonomy ¶4 · Cammarano
  2. C046Claim typeConceptual frameworks are made, not given, and can be rebuilt critically.Lakatos 1978
  3. C002Claim typeTwo claims of different kinds share no standard of truth, so a procedure that works on one returns a meaningless verdict on the other.Building the Taxonomy ¶4
  4. C003Claim typeOne dimension of the taxonomy decides what a claim is; the other six describe how it behaves once that is settled.Building the Taxonomy ¶3 · Cammarano
  5. C004Claim typeA measurement, an experiment and an argument are accountable to different things, so they cannot be asked to share one document and one standard of proof.What Counts as One Paper ¶10
  6. C005Claim typeMeasurement artefactTwo independent findings pointing the same way is not evidence that they are connected.What Counts as Introspection ¶11

Introspection

  1. C009Epistemic self-classificationThe programme used to assert flatly that the producing system performs no metacognition over its own outputs.What Counts as Introspection ¶6
  2. C010Epistemic self-classificationThe premise the whole framework rests on, in its narrow form: a model may represent that a passage is in Spanish, or that it is being evaluated, without representing that the assertion it is about to make is causal rather than correlational.What Counts as Introspection ¶8 · Cammarano
  3. C011Epistemic self-classificationWhat a model reports about its own workings is not a log of them.What Counts as Introspection ¶10

How a verdict is made

Annotation

  1. C019VerdictCoder agreementA machine-produced verdict counts here only after a person has recoded a sample by hand, without seeing what the machine decided, and the two agree above ninety per cent.What Counts as a Verdict ¶2
  2. C022VerdictDeclared incompletenessA number nobody else can recompute from the deposited materials is testimony, not evidence.Coding Manual v2.1 ¶2
  3. C018Coder agreementWhen measuring whether two people label things the same way, correcting for each coder’s personal bias matters less than how many coders there are.Artstein & Poesio 2008 · Artstein · Poesio
  4. C020Coder agreementMeasurement artefactConvergent pathsThirty differently worded answers to the same prompt are not thirty disagreements: they can be one conclusion reached thirty ways.Vertical Validation ¶11

Evaluation

  1. C021VerdictMachine judgeAsked to rank answers, a strong model agreed with human raters about as often as human raters agreed with each other, above eighty per cent.Zheng et al. 2023
  2. C024Measurement artefactInverse scalingAbilities that look as though they switch on all at once with scale often improve smoothly underneath: the jump belongs to an all-or-nothing metric.Srivastava et al. 2023 · Srivastava
  3. C027Declared incompletenessAn evaluation should publish the distance between what it set out to measure and what it actually measured.Liang et al. 2022 · Liang

Severity

  1. C012SeverityAgreement between data and a claim is not evidence when the method could not have found the claim false.Mayo 2018 · Mayo
  2. C013SeverityThe positive counterpart of the weak requirement: evidence is graded by how capable the test was of finding flaws.Mayo 2018 · Mayo
  3. C014SeverityConvergent pathsConvergent, independent checks can warrant a conclusion more strongly than any single one of them.Mayo 2018 · Mayo
  4. C015SeverityAbsence of a detected error counts only if the procedure would have signalled that error had it been there.Mayo 2018 · Mayo
  5. C016SeverityMayo treats inferential methods as instruments for finding things out and for grading others' work, not as after-the-fact justifications.Mayo 2018 · Mayo
  6. C017SeveritySeverity is applied one level up, to methods rather than to claims: a method is judged by its capacity to catch erroneous interpretations.Mayo 2018 · Mayo
  7. C030SeverityNeither of the two dominant philosophies of statistics, assigning degrees of belief or securing long-run error rates, asks whether a test could have caught the error.Mayo 2018
  8. C031SeveritySeverity comes from Popper but arrived without a working definition and without a place in any statistical method.Mayo 2018
  9. C032SeverityA significance level is valid only relative to how data and hypotheses were selected; selective reporting makes a hard result look easy to explain away as chance.Mayo 2018
  10. C033SeverityA severe test is built from prior knowledge of specific mistakes: a probe that would reveal a given error if present, and stay silent otherwise.Mayo 1996
  11. C036SeveritySeverity is not tied to one statistical school: any inference can be judged by whether it withstood severe testing.Mayo 2018
  12. C037SeverityPeirce justified induction not by the truth of its conclusions but by the method’s capacity to correct its own errors over time.Mayo 2005
  13. C038SeverityInduction is justified by the method, not by any single conclusion: persisting in it corrects whatever error a stage contains.Peirce 1931–1958
  14. C039SeverityProbability attaches to the method’s track record, not to the hypothesis.Peirce 1931–1958
  15. C040SeverityA test does not collect agreement; it looks for consequences that would be surprising if the hypothesis were false.Peirce 1931–1958
  16. C041SeverityA lucky agreement is exposed by persisting with the same method: replication is the correction.Peirce 1931–1958
  17. C042SeverityConvergent pathsTwo selections made by different methods err in different ways, so their agreement is evidence about the truth.Peirce 1931–1958
  18. C043SeverityInduction needs no uniformity of nature, only the absence of an overwhelming hidden bias, and lesser biases are themselves detectable by the method.Peirce 1931–1958

Falsification

  1. C045FalsificationDogmatic falsificationism treats the experimenter as Nature’s neutral voice, delivering verdicts the theory cannot touch.Lakatos 1978
  2. C051FalsificationOnce ceteris paribus clauses are allowed, a theory that survives every challenge looks more scientific, not less: the reductio of dogmatic falsificationism.Lakatos 1978
  3. C034FalsificationFalsificationism explains why failing a test counts against a hypothesis, but not why passing one counts for it.Mayo 1996
  4. C035FalsificationBeating rivals is not passing a severe test: the test has to probe the ways this particular hypothesis could be wrong.Mayo 1996
  5. C044FalsificationIf every universal theory has zero probability on any evidence, probability cannot be what separates good theories from bad ones.Lakatos 1978
  6. C047FalsificationEvery test bottoms out in basic statements that someone decides to accept; logic alone never forces the stop.Popper 1959
  7. C048FalsificationWhat is decided by agreement is the singular observation, not the theory, and no experience justifies it: it is accepted.Popper 1959
  8. C049FalsificationA theory forbids an event only under a ceteris paribus clause, so a failed prediction never refutes it alone.Lakatos 1978
  9. C050FalsificationNeither proof, nor probability, nor refutation can be read off the facts: all three fail together.Lakatos 1978
  10. C052FalsificationAn observation counts as basic because a trained technique makes it decidable, not because it is given.Lakatos 1978
  11. C053FalsificationEvery test decides what is under test and what is taken for granted, and that line is a decision, not a fact.Lakatos 1978
  12. C054FalsificationMethodological falsificationism keeps hard criticism while admitting every statement is fallible, by making the decisions explicit.Lakatos 1978

What scale does not buy

Truthfulness

  1. C026Inverse scalingImitative falsehoodOn questions written to echo common misconceptions, the larger models answered falsely more often than the smaller ones, reversing the usual pattern.Lin et al. 2022
  2. C028Imitative falsehoodAcross the field, hallucination means generated content that is nonsensical or unfaithful to the source it was given: a definition about fidelity to the input, not about truth in the world.Ji et al. 2022

Architecture

  1. C023Machine judgeHarmlessness was trained by having the model criticise and revise its own answers against a written list of principles, instead of having people label every harmful output.Bai et al. 2022 · Bai
  2. C025Convergent pathsWhen several different chains of reasoning arrive at the same answer, that agreement is itself evidence for the answer.Wang et al. 2022
  3. C029Sequence without recurrenceThe architecture behind current language models dropped step-by-step processing of a sequence in favour of computing every relation between positions at once.Vaswani et al. 2017 · Vaswani

The 15 concepts

Layer 4 is where ideas are asserted and linked. Each concept has a page of its own: the question it answers, where it sits in the other layers, and its claims.

All the claims, one by one →How they are classified →