Single case vs. control sample
Modified t-test (Crawford & Howell 1998). Compares one individual to a small control sample. The control mean and SD are treated as sample estimates rather than population values.
Reverse: threshold score for a given p
Uses the control mean, SD, N and hypothesis entered above.
Notes
t = (case − mean) / (SD·√((N+1)/N)) with df = N−1. The “estimated % of controls below the case” is the point estimate of how abnormal the score is (Crawford & Garthwaite 2002). zCC is the effect size.
For measures where higher is worse (RT or errors) choose “Case higher than controls”.
References
Crawford, J. R., & Howell, D. C. (1998). Comparing an individual’s test score against norms derived from small samples. The Clinical Neuropsychologist, 12(4), 482–486. https://doi.org/10.1076/clin.12.4.482.7241
Crawford, J. R., & Garthwaite, P. H. (2002). Investigation of the single case in neuropsychology: Confidence limits on the abnormality of test scores and test score differences. Neuropsychologia, 40(8), 1196–1208. https://doi.org/10.1016/S0028-3932(01)00224-X
Dissociation between two tasks
Crawford & Garthwaite (2005). Tests whether the difference between the case's scores on two tasks is larger than the differences seen in controls.
Notes
RSDT (Revised Standardized Difference Test) compares z-scores and is the general choice. UDT (Unstandardized Difference Test) compares raw differences and is only valid when both tasks use the same scale. SDs are assumed to use N−1.
A classical dissociation also requires a deficit on one task and no deficit on the other. The single-task Crawford–Howell tests are shown for that purpose.
References
Crawford, J. R., & Garthwaite, P. H. (2005). Testing for suspected impairments and dissociations in single-case studies in neuropsychology: Evaluation of alternatives using Monte Carlo simulations and revised tests for dissociations. Neuropsychology, 19(3), 318–331. https://doi.org/10.1037/0894-4105.19.3.318
Crawford, J. R., & Howell, D. C. (1998). Comparing an individual’s test score against norms derived from small samples. The Clinical Neuropsychologist, 12(4), 482–486. https://doi.org/10.1076/clin.12.4.482.7241
χ² test, 2×2
Compares the success rate in two conditions (or two people or two tasks).
Enter any two of the three. Typing a total updates Errors; typing errors updates Total. If Correct is empty, Errors and Total give Correct. The calculated cell is shaded.
χ² test, N×2: distribution across item types
Do two conditions differ in how their errors (or correct responses) are distributed across item types? The test ignores differences in the overall number of errors.
Notes
This is the standard χ² test of independence on the 2×N table. The same test works for counts of correct responses.
χ² goodness of fit: observed vs. predicted counts
Tests whether the observed count of each item type differs from a specific a-priori prediction.
Notes
If the predictions sum to the observed total the test concerns only the distribution across types (df = k−1). Otherwise it also covers the overall number of errors (df = k). The df is chosen automatically. Each predicted count should be at least 5.
Fisher's exact test
Exact test for comparing two proportions. Use it instead of χ² when expected counts are small.
Enter any two of the three. Typing a total updates Errors; typing errors updates Total. If Correct is empty, Errors and Total give Correct. The calculated cell is shaded.
Many comparisons at once
McNemar's test (exact)
Binary outcomes (correct/incorrect) for the same items or participants in two conditions. This is an exact binomial test on the discordant pairs.
Binomial probability
What is the chance of getting this many successes by guessing?
Paired-samples t-test
Two scores per participant (or item).
Independent-samples t-test
Two separate groups. The columns can have different lengths.
Wilcoxon signed-rank test
Non-parametric alternative to the paired t-test. The p-value is exact for any N and handles ties.
Notes
Zero differences are dropped. Tied absolute differences get average ranks. The exact p-value comes from the full permutation distribution of the observed ranks so ties are handled correctly. A normal approximation (z with tie correction) and r = |z|/√N are also shown.
Pearson correlation
Linear correlation between two continuous variables.
Point-biserial correlation
Correlation between a continuous item property (e.g. word length) and success/failure.
In the 0/1 format each row is one item. Add 0/1 columns to get one correlation per participant. In the frequency format each row is one value of X.
Outliers: Grubbs' test
Tests whether the most extreme value is an outlier. If so the value is removed and the test is repeated on the remaining values.
Notes
Critical values are computed exactly from the t distribution: two-sided when testing both directions and one-sided when testing high or low values only. The thresholds are mean ± Gcrit × SD of the values that remain after the last removal. The test assumes the remaining data are roughly normal. Repeating Grubbs' test can miss outliers that mask each other.
Outliers: interquartile range
A value is an outlier if it lies below the 25th percentile minus X% of the interquartile range (IQR) or above the 75th percentile plus X% of the IQR.
Notes
Percentiles are computed as in Excel's PERCENTILE.INC / QUARTILE.INC. Tukey's classic rule uses X = 150 and his “far out” rule uses X = 300.
Normal distribution
z-score and p-value of a score given a known population mean and SD.
Reverse: threshold score for a given p
Uses the mean and SD entered above.
Notes
Use this only when the mean and SD are population values (e.g. test norms). To compare one person to a small sample use the Crawford–Howell test.
d′: signal detection sensitivity
For yes/no or same/different tasks. d′ separates sensitivity from a general bias to answer “yes”.