Inter hospital external validation of interpretable machine learning based triage score for the emergency department … – Nature.com
Study design and setting
This retrospective and validation study was executed across from 3 ED in Korea (A, B and C). A, B and C are tertiary hospitals located in a metropolitan city in Korea. Respectively, the hospital has approximately 2000, 1000, and 1000 inpatient beds. Approximately more than 80,000, 90,000 and 50,000 patients visit the ED annually. There are 16, 20 and 7 specialists working at each institution, respectively. All data were mapped to the Observational Medical Outcome Partnership Common Data Model (OMOP-CDM) for the multicenter study. This study was approved by the Samsung Medical Center Institutional Review Board (2023-02-036), and a waiver of informed consent was granted for EHR data collection and analysis because of the retrospective and de-identified nature of the data. All methods were performed in accordance with the relevant guidelines and regulations.
Initially, ED patients from 2016 to 2017 were included for each hospital. Patient older than 18 with disease patients were included. We also excluded patient with left without being seen or death on arrival/cardiopulmonary resuscitation patients. We split into two cohort: development (70%) cohort for training the interpretable ML model and test (30%) for evaluation from each hospital.
We extracted data from each hospitals electronic medical records system which all patient information was deidentified. Candidate input variables were considered with available features at the stage of ED triage including demographic characteristics such as age, gender, administrative variables including time of ED visit and clinical variables such as severity index, consciousness, and initial vital sign. Comorbidities were also obtained from hospital diagnosis records in the preceding 5years before patients emergency visit and compared for each hospital. They were extracted from International Statistical Classification of Diseases and Related Health Problems, Tenth Revision(ICD-10). The list and description of candidate predictors and comorbidities are given in the supplementary Tables6 and 7.
Emergency patients with semi-acute conditions typically undergo surgical procedure or are admitted to Intensive care unit (ICU) following emergency room treatment and given the imperative for patients to survive. Our primary outcome was 2-day mortality which was the target feature for analysis to build the interpretable ML model for each hospital.
For the multicenter study, we adopted OMOP CDM from the research network Observational Health Data Sciences and Informatics (OHDSI)28 for standardized structure and vocabularies to map emergency department data based on Systematized Nomenclature of MedicineClinical Terms (SNOMED-CT) and Logical Observation Identifiers Names and Codes (LOINC) as example shown Supplementary Fig.1. Extract, Transformation and Load (ETL) process was performed with structured query language. Each ED care and diagnosis related information was mapped into proper CDM tables as shown in Fig.2. For example, patient demographics and vital sign are mapped to Person and Measurement table, respectively. After transformation was completed into CDM format, all hospital can get the same structure and vocabularies, for executing same research query. All details of transformation and code are accessible on Gitgub29.
Table mapping for converting clinical to common data model tables. CDM: common data model; ED: Emergency department.
AutoScore Framework is a machine learning-based clinical score generator, consisting of six modules developed from Singapore12. Module 1 uses a random forest for ranking variables according to their importance. Module 2 transforms variables by categorizing continuous variables to improve interpretation with quantile information. Module 3 makes scores for each variable based on a logistic regression coefficient. Module 4 selects which variables could be included in the scoring model. In Module 5, clinical domain knowledge is incorporated to the score and cutoff points can be defined when categorizing continuous variables. Module 6 evaluates the performance of the score in a separate test dataset. The AutoScore framework provides a systematic and automated approach to develop score automatically, combining of advantage of machine learning for discriminating and the strength of logistic regression in its interpretability. For the overall score generation, We considered weighted average scores across all institutions. For each institutionsi, a weight({w}_{i})was formulated as ({w}_{i})=(left(sqrt{{(AUC}_{i})} times {N}_{i}^{3}right))/({sum }_{i=1}^{M}sqrt{{(AUC}_{i})} times {N}_{i}^{3}))100%where({N}_{i})was the sample size,({AUC}_{i}) was the AUC value obtained based on the validation set, andMwas the total number of institutions. Overall score was calculated with weighted score based on ({w}_{i}).
We defined our new novel framework CDM Autoscore for ED, combination of CDM based standardized format and autoscore based interpretable framework shown in Fig.3. The analysis and preparation code using CDM format was also shared on GitHub29.
Overall process of CDM Autoscore for ED. Each Institutions conducted Extract, Transformation and Load process for converting local data into CDM format. Algorithms from each of institution were derived using interpretable machine learning framework and validated inter-and intra- institutionally. EMR: Electronic medical records; ETL: Extract, transformation and Load; OMOP CDM: Observational Medical Outcome Partnership Common Data Model.
Categorical features were expressed as frequency and percentages and continuous features were expressed as means and standard deviations. Comparison tests for each hospital were performed with analysis of variance and chi-square tests at 5% significance levels. Standardized mean difference (SMD) was also calculated for comparing each hospital. Two types of validations for this study were conducted. First, we executed internal-institutional validation for each hospitals score. We also performed intra-institutional validation pair-wisely for the external validation. Area under the curve in the receiver operating characteristic (AUROC) and 95% confidence interval (CI) with 1000 times of bootstrap was reported. Other metrics including accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were also reported. SMOTE was conducted for handling the imbalance problem. Twice of minority was oversampled and same number of majorities according to the number of minority was sampled with fixed seed number.
- Optimization of wear parameters for ECAP-processed ZK30 alloy using response surface and machine learning ... - Nature.com - April 22nd, 2024 [April 22nd, 2024]
- Machine learning approach predicts heart failure outcome risk - HealthITAnalytics.com - April 22nd, 2024 [April 22nd, 2024]
- Practical approaches in evaluating validation and biases of machine learning applied to mobile health studies ... - Nature.com - April 22nd, 2024 [April 22nd, 2024]
- Application of power-law committee machine to combine five machine learning algorithms for enhanced oil recovery ... - Nature.com - April 22nd, 2024 [April 22nd, 2024]
- Free tool uses machine learning to pick better molecules for testing new reactions - Chemical & Engineering News - April 22nd, 2024 [April 22nd, 2024]
- Automated Analysis of Nuclear Parameters in Oral Exfoliative Cytology Using Machine Learning - Cureus - April 22nd, 2024 [April 22nd, 2024]
- An AI Ethics Researcher's Take On The Future Of Machine Learning In The Art World - SlashGear - April 22nd, 2024 [April 22nd, 2024]
- Enhancing Emotion Recognition in Users with Cochlear Implant Through Machine Learning and EEG Analysis - Physician's Weekly - April 22nd, 2024 [April 22nd, 2024]
- Imageomics Applies AI and Vision Advancements to Biological Questions - Photonics.com - April 22nd, 2024 [April 22nd, 2024]
- Machine learning reveals the control mechanics of an insect wing hinge - Nature.com - April 22nd, 2024 [April 22nd, 2024]
- The Future of ML Development Services: Trends and Predictions - FinSMEs - April 22nd, 2024 [April 22nd, 2024]
- CSRWire - Island Conservation Harnesses Machine Learning Solutions From Lenovo and NVIDIA To Restore Island ... - CSRwire.com - April 22nd, 2024 [April 22nd, 2024]
- Investigation of the effectiveness of a classification method based on improved DAE feature extraction for hepatitis C ... - Nature.com - April 22nd, 2024 [April 22nd, 2024]
- Machine Learning Uncovers New Ways to Kill Bacteria With Non-Antibiotic Drugs - ScienceAlert - April 22nd, 2024 [April 22nd, 2024]
- Formal Interaction Model (FIM): A Mathematics-based Machine Learning Model that Formalizes How AI and Users Shape One Another - MarkTechPost - April 22nd, 2024 [April 22nd, 2024]
- A secure approach to generative AI with AWS | Amazon Web Services - AWS Blog - April 22nd, 2024 [April 22nd, 2024]
- Imbalanced Learn: the Python library for rebuilding ML datasets - DataScientest - April 22nd, 2024 [April 22nd, 2024]
- AI has a lot of terms. We've got a glossary for what you need to know - Quartz - April 22nd, 2024 [April 22nd, 2024]
- Texxa AI, Where ideas take flight: Revolutionizing AI Solutions for Businesses and Individuals - GlobeNewswire - April 22nd, 2024 [April 22nd, 2024]
- Using machine learning to identify patients with cancer that would benefit from immunotherapy - Medical Xpress - April 22nd, 2024 [April 22nd, 2024]
- Researchers at Stanford University Explore Direct Preference Optimization (DPO): A New Frontier in Machine Learning and Human Feedback - MarkTechPost - April 22nd, 2024 [April 22nd, 2024]
- Machine Learning Helps Scientists Locate the Neurological Origin of Psychosis - ExtremeTech - April 22nd, 2024 [April 22nd, 2024]
- Slack delivers native and secure generative AI powered by Amazon SageMaker JumpStart | Amazon Web Services - AWS Blog - April 22nd, 2024 [April 22nd, 2024]
- Accurate and rapid antibiotic susceptibility testing using a machine learning-assisted nanomotion technology platform - Nature.com - March 20th, 2024 [March 20th, 2024]
- AI reveals the complexity of a simple birdsong - The Washington Post - March 20th, 2024 [March 20th, 2024]
- Researchers from MIT and Harvard Developed UNITS: A Unified Machine Learning Model for Time Series Analysis that Supports a Universal Task... - March 20th, 2024 [March 20th, 2024]
- Undergraduate Researchers Help Unlock Lessons of Machine Learning and AI - College of Natural Sciences - March 20th, 2024 [March 20th, 2024]
- Machine Learning Accelerates the Simulation of Dynamical Fields - Eos - March 20th, 2024 [March 20th, 2024]
- HEAL: A framework for health equity assessment of machine learning performance - Google Research - March 20th, 2024 [March 20th, 2024]
- Expert on how machine learning could lead to improved outcomes in urology - Urology Times - March 20th, 2024 [March 20th, 2024]
- Unlock the potential of generative AI in industrial operations | Amazon Web Services - AWS Blog - March 20th, 2024 [March 20th, 2024]
- Optimize price-performance of LLM inference on NVIDIA GPUs using the Amazon SageMaker integration with NVIDIA ... - AWS Blog - March 20th, 2024 [March 20th, 2024]
- Orange isn't building its own AI foundation model here's why - Light Reading - March 20th, 2024 [March 20th, 2024]
- Wall Street's Favorite Machine Learning Stocks? 3 Names That Could Make You Filthy Rich - InvestorPlace - March 20th, 2024 [March 20th, 2024]
- Edge Impulse machine learning platform adds support for NVIDIA TAO Toolkit and Omniverse - CNX Software - March 20th, 2024 [March 20th, 2024]
- MIT Researchers Developed an Image Dataset that Allows Them to Simulate Peripheral Vision in Machine Learning Models - MarkTechPost - March 20th, 2024 [March 20th, 2024]
- 18 Cutting-Edge Artificial Intelligence Applications in 2024 - Simplilearn - March 20th, 2024 [March 20th, 2024]
- Machine-learning-based global optimization of microwave passives with variable-fidelity EM models and response ... - Nature.com - March 20th, 2024 [March 20th, 2024]
- Benchmarking machine learning and parametric methods for genomic prediction of feed efficiency-related traits in ... - Nature.com - March 20th, 2024 [March 20th, 2024]
- PyCaret: Everything you need to know about this Python library - DataScientest - March 20th, 2024 [March 20th, 2024]
- Crypto Entities That Neglect AI and Machine Learning Investment Will Lag Behind, Warns Binance CTO Bitcoin News - Bitcoin.com News - March 20th, 2024 [March 20th, 2024]
- VictoriaMetrics Machine Learning takes monitoring to the next level - The Bakersfield Californian - March 20th, 2024 [March 20th, 2024]
- How Marketers Can Elevate Creative Performance with AI-Driven Format Optimisation - ExchangeWire - March 20th, 2024 [March 20th, 2024]
- Revolutionizing carbon neutrality: Machine learning paves the way for advanced CO reduction catalysts - EurekAlert - March 20th, 2024 [March 20th, 2024]
- BurstAttention: A Groundbreaking Machine Learning Framework that Transforms Efficiency in Large Language Models with Advanced Distributed Attention... - March 20th, 2024 [March 20th, 2024]
- Construction of environmental vibration prediction model for subway transportation based on machine learning ... - Nature.com - March 20th, 2024 [March 20th, 2024]
- Introducing 'Get started with generative AI on AWS: A guide for public sector organizations' | Amazon Web Services - AWS Blog - March 20th, 2024 [March 20th, 2024]
- Generative deep learning for the development of a type 1 diabetes simulator | Communications Medicine - Nature.com - March 20th, 2024 [March 20th, 2024]
- Integrating core physics and machine learning for improved parameter prediction in boiling water reactor operations ... - Nature.com - March 11th, 2024 [March 11th, 2024]
- Top AI Certification Courses to Enroll in 2024 - Analytics Insight - March 11th, 2024 [March 11th, 2024]
- Machine learning techniques applied to construction: A hybrid bibliometric analysis of advances and future directions - ScienceDirect.com - March 11th, 2024 [March 11th, 2024]
- Artificial Intelligence Market towards a USD 2,745 bn by 2032 - Market.us Scoop - Market News - March 11th, 2024 [March 11th, 2024]
- Data Maturation Represents the Essential Reason for Deploying Machine Learning Today | By Adam Mogelonsky - Hospitality Net - March 11th, 2024 [March 11th, 2024]
- The Top 3 Machine Learning Stocks to Buy in March 2024 - InvestorPlace - March 11th, 2024 [March 11th, 2024]
- How to Learn the Math Needed for Data Science - Towards Data Science - March 11th, 2024 [March 11th, 2024]
- This AI Paper from Huawei Introduces DenseSSM: A Novel Machine Learning Approach to Enhance the Flow of Hidden Information between Layers in State... - March 11th, 2024 [March 11th, 2024]
- Machine learning and the prediction of suicide in psychiatric populations: a systematic review | Translational Psychiatry - Nature.com - March 11th, 2024 [March 11th, 2024]
- Machine learning algorithms show applications in OAB, antibiotic resistance - Urology Times - March 11th, 2024 [March 11th, 2024]
- Scientists develop new machine learning method for modeling chemical reactions - Phys.org - March 11th, 2024 [March 11th, 2024]
- Machine learning developed a CD8+ exhausted T cells signature for predicting prognosis, immune infiltration and drug ... - Nature.com - March 11th, 2024 [March 11th, 2024]
- Single Transit Detection In Kepler With Machine Learning And Onboard Spacecraft Diagnostics - Astrobiology - Astrobiology News - March 11th, 2024 [March 11th, 2024]
- Meta AI Proposes Wukong: A New Machine Learning Architecture that Exhibits Effective Dense Scaling Properties Towards a Scaling Law for Large-Scale... - March 11th, 2024 [March 11th, 2024]
- Putting the AI in NIA: New opportunities in artificial intelligence - National Institute on Aging - March 11th, 2024 [March 11th, 2024]
- Revolutionizing LLM Training with GaLore: A New Machine Learning Approach to Enhance Memory Efficiency without Compromising Performance - MarkTechPost - March 11th, 2024 [March 11th, 2024]
- Uncertainty-aware deep learning for trustworthy prediction of long-term outcome after endovascular thrombectomy ... - Nature.com - March 11th, 2024 [March 11th, 2024]
- AI Engineer Salary: The Lucrative World of AI Engineering - Simplilearn - March 11th, 2024 [March 11th, 2024]
- Multimodal artificial intelligence-based pathogenomics improves survival prediction in oral squamous cell carcinoma ... - Nature.com - March 11th, 2024 [March 11th, 2024]
- Northrop Grumman Partners to Advance Deep Sensing for the US Army | Northrop Grumman - Northrop Grumman Newsroom - March 11th, 2024 [March 11th, 2024]
- Global cellular IoT connections to grow 90% to 6.5 bn by 2028: Juniper Research - ETTelecom - March 11th, 2024 [March 11th, 2024]
- Enhancing statistical reliability of weather forecasts with machine learning - Phys.org - March 11th, 2024 [March 11th, 2024]
- Inside AI: Talking to the Data - Inside Unmanned Systems - March 11th, 2024 [March 11th, 2024]
- Anemond's Factoid 2 is an experimental sampler plugin that uses machine learning to "decompose", remix and ... - MusicRadar - March 11th, 2024 [March 11th, 2024]
- Advancing Chemistry with AI: New Model for Simulating Diverse Organic Reactions - Lab Manager Magazine - March 11th, 2024 [March 11th, 2024]
- Generative AI: Understand the challenges to realize the opportunities | Amazon Web Services - AWS Blog - March 11th, 2024 [March 11th, 2024]
- How To Specialize in Artificial Intelligence - Troy Today - Troy University - March 11th, 2024 [March 11th, 2024]
- Google DeepMind Introduces Two Unique Machine Learning Models, Hawk And Griffin, Combining Gated Linear Recurrences With Local Attention For Efficient... - March 11th, 2024 [March 11th, 2024]
- Unlocking Innovation: AWS and Anthropic push the boundaries of generative AI together | Amazon Web Services - AWS Blog - March 11th, 2024 [March 11th, 2024]
- Introducing Microsoft's AI Red Team And PyRIT - AiThority - March 11th, 2024 [March 11th, 2024]
- Unveiling the World of Artificial Intelligence: A Beginner's Guide - Medium - January 3rd, 2024 [January 3rd, 2024]
- How machine learning might unlock earthquake prediction - MIT Technology Review - January 3rd, 2024 [January 3rd, 2024]
Tags: