Replication crisis

• data-based claims turned out to be less reliable than previously believed
  • statistical claims could not be replicated with new data
• large-scale replications brought attention to the issue
  • e.g., Nieuwland et al. (2018); Open Science Collaboration (2015)
• this has also led to distrust in findings in academia and the public

The problem with p-values

• issues relating to reported findings:
  • misuse of misinterpretation of p-values in Null Hypothesis Significant Testing (NHST; e.g., Ioannidis, 2005)
  • study design
  • improper use of statistical methods
    • stemming from inadequate teaching
  • selective reporting
• in other words, HARKing and p-hacking (whether consciously done or not)

Solving the problem

• these could be mitigated with Open Science practices
  • transparency in writing, analyses, planning/hypothesising stages
  • reproducibility of analyses
  • greater value given to replication studies
  • embracing and addressing uncertainty (Vasishth & Gelman, 2021)
• in sum: “conscientious practice” (Sönning & Werner, 2021, p. 1182)

The garden of forking paths

• or ‘researcher degrees of freedom’ (Simmons et al., 2011)
• the problem: there are many plausible ways to analyse any given data set
• there are many choices researchers make in:
  • experimental design
  • data collection
  • data preprocessing
  • data analyses
  • reporting
• the path we happen to go down can seem pre-determined (Gelman & Loken, 2014)
  • but can amount to HARKing, p-hacking, fishing
• the fastest solution: share everything and write transparently

The current state of quantitative linguistics

• there is a trend towards empirical methods throughout linguistics
  • we should pay attention to methodological discussions in related fields
• we also find ourselves in a state of methdological crisis

Kuhn’s structure of scientific revoluations

• Thomas Kuhn’s The Theory of Scientifc Revoluations (1962)
  • based on socio-historical observation
  • the evolution of scientificy theory is cycical
  • crisis leads to revolution
• also applies to research methodology

Previous cycles of statistical analyses

• proprietary, point-and-click software (e.g., SPSS)
  • move to open source programming languages (e.g., R, Python, Julia)
• ANOVAs
  • move to linear regression
  • then linear mixed models
    • random-intercepts only models
    • maximal models
    • parsimonious models
• now a trend towards Bayesian regression

What do statisticians say?

• Wasserstein et al. (2019): list of Do’s and Don’t from statisticians
  • Don’t base conclusions on p-values
  • Do think about ATOM: Accept uncertainty, be Thoughtful, Open, and Modest
• Wasserstein & Lazar (2016): the American Statistical Association’s statement on p-values
  • p-values are often misused and misinterpreted
  • good statistical practice is part of good scientific practice
    • as such, relies on good study design and conduct
    • interpretation in context
    • complete and transparent reporting

Revolution

Simple fixes

• planning and design
  • large sample sizes
  • establish pre-processing/analysis steps a priori
• methodologically
  • select variables based on theory and research questions
  • model non-independence of data points (mixed models)
  • move towards estimation and away from arbitrary significance thresholds
• writing
  • be transparent about choices made

Words of comfort

Less experienced scholars must not fear methodological attacks on their analyses, which are instead seen as informing interim interpretations that may require future modification.

— Sönning & Werner (2021), p. 1199

• in some sub-fields linear mixed models are still not considered the standard
  • so you’re well situated despite the doom around p-values
• moving from frequentist (NHST) framework to the Bayesian framework is relatively painless
  • in this class we will run a LMM with lme4 (Frequentist) and with brms (Bayesian)

Replication value

• replication value (RV): “the expected utility of a finding before replication” (Isager, 2020, p. 6)
  • (scientific) value of the research claim
    • importance to the field, to policy, health etc.
  • the uncertainty of our knowledge about the claim
    • validity of study design, statistical power, bias, etc.
• replication aims to reduce uncertainty
  • which also increases utility of the claim

Quantifying RV

• how to quantify value and uncertainty?

• Isager et al. (2021) suggest using…

  • average yearly citation count to estimate value
    • the more citations, the higher the impact the original study had
  • and sample size to estimate uncertainty (\(\frac{1}{\sqrt{n}}\))
    • the higher the sample size, the more precise the estimate
    • the lower the sample size, the greater the uncertainty
    • i.e., \(n\) is inversely correlated with uncertainty

\(RV_{Cn}\)

Isager et al. (2021):

\[ RV_{Cn} = value \ x \ uncertainty \]