What our library's numbers show, and what they don't
We looked for patterns across the studies in our research library: how the score varies by topic and year, how it relates to how often other researchers cite a study, and what the links between studies show. Several of the patterns turned out to come from how the library was built, and we say so for each one.
- 15,036scored studies analysed
- 154,409citation links between library studies
- 6,838studies with an outside citation measure (iCite RCR)
- 73%with no study design we could read
1. Most of the spread in our score depends on whether we could read a study design from the abstract
In the version analysed here, our score was the sum of 6 parts. Two of them, study design and strength of evidence, account for 79% of the variation in scores across the library. Journal tier accounts for 9% and recency for 7%. For 10,918 of 15,036 studies, our automatic reader found no study design in the abstract, so they got 0 of the 25 design points.
That explains most of the gap between topics. Topics differ in score (rank-based effect size ε² = 0.16 across 31 topics), and a topic's average score closely tracks how many of its studies have no readable design (Pearson r = -0.90 across topics). In each of the three topics with the lowest median score among those with 100 or more studies, Inflammation and the immune system (88.2% without a design), Down syndrome and Alzheimer's (88.5% without a design), Risk genes beyond APOE (95.9% without a design), the reader found no design for most of the studies. A low topic average here mostly reflects that missing input. It is not a judgement that the science in that topic is weaker.
A known gap in our own data: of the 637 studies that PubMed tags as randomised controlled trials, 460 have no design recorded in our data and so get 0 design points. The same is true of 115 of 577 tagged meta-analyses and systematic reviews. We have not changed any scores on this page. This is reported so it can be fixed in the rubric's input data, not quietly patched here.
How computed: share = cov(component points, score) / var(score) over the 14,920 uncapped studies (the six shares sum to 1). ε² = Kruskal–Wallis H / (n − 1).
2. Newer studies score higher, and the recency points account for almost all of that
Across the library, later publication year goes with a higher score (Spearman ρ = +0.38, 95% CI +0.36 to +0.39, n = 15,032). Take away the points the rubric gives for being recent and almost nothing is left: ρ = +0.036. That rank measure is dominated by studies from 2000 on, which score about the same without their recency points. Studies from before 2000 still average several points lower (see the table), and our reader found no design for more of them.
| Years | Studies | Average score | Average without recency points | No design recorded |
|---|---|---|---|---|
| 1980-1989 | 143 | 34.2 | 33.2 | 98.6% |
| 1990-1999 | 823 | 37.0 | 36.0 | 89.1% |
| 2000-2009 | 2,488 | 41.2 | 39.7 | 79.5% |
| 2010-2014 | 2,062 | 43.1 | 41.1 | 75.3% |
| 2015-2019 | 2,471 | 45.9 | 42.3 | 71.3% |
| 2020-2023 | 2,108 | 50.0 | 43.6 | 66.5% |
| 2024-2026 | 4,921 | 50.3 | 40.5 | 67.7% |
The 2024–2026 rows mostly come from a different collection round than the older ones, so the periods differ in how they were collected as well as in when the studies were published.
3. Our score barely tracks how often other researchers cite a study, and journal tier accounts for the link
For 6,838 studies we hold the NIH's Relative Citation Ratio (RCR), a citation measure adjusted for field and year. Score and RCR are only weakly linked (ρ = +0.10, 95% CI +0.08 to +0.13). The journal-tier part of the score carries that link (ρ = +0.23). The design part (+0.05) and evidence part (-0.03) carry almost none, and once journal tier is held fixed the link is gone (partial ρ = +0.003).
| RCR fifth | RCR range | Studies | Average score |
|---|---|---|---|
| 1 | 0.0–2.67 | 1,367 | 44.2 |
| 2 | 2.67–4.93 | 1,368 | 45.9 |
| 3 | 4.93–8.21 | 1,367 | 47.2 |
| 4 | 8.21–14.09 | 1,368 | 46.9 |
| 5 | 14.09–819.88 | 1,368 | 48.0 |
This does not show that either measure is wrong. Citations measure attention and our score measures design and venue, which are different things. RCR exists mainly for the older part of the library (average publication year 2012, against 2020 for studies without it), so this result may not hold for the newest studies.
4. The links between our study pages agree with outside citation counts
Each study page lists the other library studies it cites, taken from OpenAlex reference lists. How often a study is cited within our library matches its outside RCR well (ρ = +0.58, 95% CI +0.56 to +0.60, n = 6,838). This suggests the citation links on our pages are a reasonable guide to which studies the rest of the library builds on.
One trap: across the whole library, more in-library citations go with a slightly lower score (ρ = -0.09). That is a year effect. Older studies have had more time to be cited (ρ with year = -0.61), and they also get fewer recency points. With year held fixed the sign flips to positive (partial ρ = +0.188).
5. Patterns that turned out to be artefacts, or nothing
Open access and citations: no link once year is held fixed
Across all studies with RCR, open-access studies are cited no more than others (ρ = -0.004, n = 6,673). Within topics, open access seemed to go with higher RCR (ρ = +0.11, n = 6,667), but that disappears once publication year and journal tier are held fixed (ρ = +0.009). Year is the confounder. Only 2 of 1,314 studies from before 2005 are open access, and in all 14 of 14 topics newer studies have higher RCR (ρ +0.07 to +0.46). Our likely explanation, which we have not tested: the library was filled by raw citation count, so a recent paper had to be unusually well cited for its age to get in.
Citation impact by topic says more about how we collected than about the field
Median RCR varies hugely by topic (ε² = 0.52, 14 topics, n = 6,792). Each topic was filled by taking the most-cited results of its own search, so how broad that search was likely shapes the topic's numbers (we did not measure search breadth). We don't read this as one research area mattering more than another.
Computed 2026-10-07 from a snapshot of the library's data files taken 2026-10-07, when it held 15,036 records. Studies added since then are not included. Every figure above is a rank (Spearman) correlation unless marked otherwise. Confidence intervals come from 500 bootstrap resamples. Scores and records were not changed. The per-study scores can be downloaded from the data page. That download holds the current version of the score, so it reproduces these figures only while the version matches the one named above.