A new study published conducted exploratory analyses to examine the quality of data and reporting in two large, publicly available datasets on stroke and diabetes, widely used in clinical prediction models.
Study
Two popular, publicly available health datasets with likely poor data provenance were selected for their high download counts and relevance to clinical prediction model research. One dataset focused on stroke and the other on diabetes, with both accessed from Kaggle on August 27, 2025. The current study aimed to highlight data provenance issues in clinical prediction models using these datasets.
Each dataset was evaluated using nine TRIPOD+AI data provenance items, and exploratory analyses were conducted to assess authenticity, including checks for simulated data, unexpected correlations among variables, abnormal distributions, and duplicate rows. Public Kaggle discussions about data provenance were also reviewed, and concerns were raised with Kaggle.
Google Scholar was searched to identify peer-reviewed articles that used these datasets for model development or validation, and full texts were screened for inclusion. Exclusions included non-peer-reviewed works and non-English articles. The authors noted that this search strategy likely underestimated the use of these datasets because studies that did not include direct Kaggle links would not have been captured.
Inconsistencies in reporting were documented, along with checks for dataset origin disclosures and reviews of statements about ethical approval and potential clinical use. Policy uptake was examined via Altmetric and Overton. Author affiliations by country were analyzed, and research volume over time was plotted using OpenAlex.
Results
An assessment of two widely used Kaggle health datasets for clinical prediction model research uncovered major concerns about data provenance and authenticity. Out of 653 research outputs identified, 125 published articles developed or validated clinical prediction models using these datasets.
Evaluation using nine TRIPOD+AI items revealed serious deficiencies in both datasets: neither provided information on when, where, why, or how the data were collected, nor could authenticity be independently verified. Both datasets failed all nine TRIPOD+AI data provenance assessment items.
The stroke dataset included 5,110 cases, which contained irregular patient IDs, improbable blood glucose and age distributions, and unrealistically little missing data. Similarly, the diabetes dataset comprised 100,000 cases that contained repetitive and unnatural values, artificial correlations, and many duplicate entries. Together, these findings indicate that both datasets are likely synthetic, fabricated, or otherwise unreliable and therefore unsuitable for research or clinical application.
The 125 included articles originated from 32 countries. However, reporting on ethical approval was rare, and most articles lacked sufficient information about data provenance. Only a small number described their data sources, and the majority did not meet basic transparency standards.
Nevertheless, these datasets were widely cited and frequently used to make recommendations for clinical care. Of the 125 studies, three models showed evidence of potential use in practice, one was cited in a medical device patent, and 86 review articles referenced these models.
Some articles described actual or potential use in clinical settings, and 11 studies developed web- or app-based prediction tools with graphical user interfaces, two of which were publicly accessible. None of the studies were referenced in policy documents.
The number of publications using these datasets has continued to rise, despite ongoing concerns about the quality and authenticity of the underlying data.
Conclusion
The current study highlights the urgent need to address the use of unreliable data in clinical prediction model research. Reliable data and transparent methods are essential to ensure trustworthy clinical decisions and safeguard patient care. The authors recommend action by journals, publishers, data repositories, researchers, and clinicians to improve standards and promote responsible research practices.
The authors also emphasized that this study examined only two publicly available Kaggle datasets and that it remains unclear how widespread similar data provenance issues are across other datasets and repositories.
Source:
https://www.news-medical.net/news/20260714/Unreliable-datasets-are-shaping-clinical-prediction-models.aspx