Optimising retraining frequency for a paediatric emergency department admission prediction model: development and Temporal validation using real-world data
Publication Details
Williams, E.,
Sinha, T.,
Lyttle, M.,
&
Kanagasingam, Y.
(2026).
Optimising retraining frequency for a paediatric emergency department admission prediction model: development and Temporal validation using real-world data.
Emergency Medicine Australasia, 38 (3).
Abstract
Objective: To analyse temporal performance drift and optimal retraining frequency for an ensemble machine learning model to predict inpatient admission from paediatric emergency department (ED) triage data.
Methods: This study utilised 409,307 ED presentations from 1 July 2018 to 30 June 2024 at Perth Children's Hospital. An ensemble stacking model (XGBoost, TabNet, multi-layer perceptron and logistic regression base learners with a logistic regression meta-learner) incorporated structured triage features and tuned BioClinicalBERT-derived embeddings from free-text notes. The model ran prospectively through a 5-year rolling-window simulation, testing nine retraining cadences from weekly to triennia land a static model. Training, retraining and validation datasets were temporally separate and prior to the test set. Primary out-comes were discrimination via the area under the receiver operator characteristic (AUROC) and calibration as absolute mean daily bed error (AMDBE).
Results: Weekly retraining achieved a mean AUROC of 0.843 (SD 0.016) and AMDBE of 2.57 (SD 1.79) over the 5-year simulation. Fortnightly and monthly cadences were non-inferior (AMDBE 2.61 and 2.73), whereas longer intervals showed progressive calibration degradation (p < 0.001) and stable AUROC. Concept drift was most pronounced in the static model, with a mean AMDBE of 10.6 in 2024 compared to 1.79 for the weekly model. Notably, monthly retraining required only 25% of the weekly computational burden with non-inferior performance.
Conclusion: Monthly, or more frequent, model retraining sustains discrimination and calibration for paediatric ED admission prediction. This effectively mitigated concept drift and enabled accurate simulated daily bed-demand forecasting, providingevidence to support the clinical testing of such modelling.
Keywords
admission prediction, artificial intelligence, concept drift, machine learning, paediatric emergency medicine
Link to Publisher Version (URL)
