Journals | Policy | Permission

Journal of Endocrinology & Metabolism

Original Article

Volume 9, Number 3, June 2019, pages 43-50

Development and Validation of a Primary Care Tool to Identify Patients With Type 2 Diabetes Mellitus at High Risk of Hypoglycemia-Related Inpatient Admissions

Hypoglycemia is a condition caused by a very low level of blood sugar, and is a common adverse consequence of drug treatment for diabetes which has been identified as a significant barrier to glycemic control [1-3]. According to a Scottish population-based diabetes register, the incidence of hypoglycemia stands at 35 episodes per 100 patient-years among insulin-treated type 2 diabetes mellitus (T2DM) patients [4]. A meta-analysis conducted by Edridge et al reported the prevalence to be 45% for mild/moderate and 6% for severe episodes [5]. This figure is even higher amongst those on insulin, at 50% for mild/moderate episodes and 21% for severe episodes [5]. Severe hypoglycemia is defined as an event requiring assistance of another person to actively administer carbohydrate, glucagon, or other resuscitative actions. Although most episodes are mild, severe cases of hypoglycemia may lead to falls [6, 7], automobile accidents [7], dementia [8], and can even be potentially life-threatening [1].

Despite being potentially preventable, hypoglycemia is associated with a considerable economic and resource burden to the health sector [1, 2] and may lead to poorer quality of life [9-11] and increased patient distress [12].

In current endocrinology literature, many measures have been recognized as risk factors for hypoglycemia [4, 13-26]. They include advanced age [13-16], type of drugs prescribed [17-19], low glomerular filtration rate (GFR) [20, 21], long duration of insulin treatment [4, 22], low body mass index (BMI) [16] and stringent glycemic treatment goals [2, 4, 13, 18, 19, 23, 24]. Other measures such as hemoglobin A1c (HbA1c) and gender have also been identified as likely risk factors with almost significant statistical evidence [4, 19].

A risk-scoring tool based on the above risk factors will be useful for clinicians to identify high-risk patients for implementation of timely changes to treatment plan to avert the otherwise impending episodes of hypoglycemia. Such a tool had recently been developed and validated by Karter et al for a Western population in the United States [25]. However, a scoring index to predict inpatient admission due to hypoglycemia has yet to be developed for use in the Asian population. A recent review highlighted that there are several reasons to believe Southeast Asians might be of a different risk profile for hypoglycemia compared with Western populations [26]. In view of this gap in current research, we seek to develop and validate a hypoglycemia risk scoring index on a multi-ethnic Asian population using data from the National Healthcare Group (NHG) Chronic Disease Management System (CDMS).

The study has received ethical approval from the NHG Domain Specific Review Board (DSRB Ref: 2014/00030). We used a retrospective cohort study design to develop a risk tool to categorize the 6-month risk of hypoglycemia-related inpatient admission. We identified potential risk factors based on literature review, expert opinion and preliminary analysis for hypoglycemia admission rates by selected risk factors. Using a split-sample design, we constructed logistic regression and neural network models to predict hypoglycemia inpatient admission within 6 months. A risk index was then developed to score the inpatient admission risk. The C statistic was used to measure discriminative ability, and calibration was assessed using the Hosmer-Lemeshow goodness-of-fit test.

Patients with T2DM were identified from the CDMS database maintained by NHG based on the International Classification of Diseases Ninth Revision (ICD-9-CM) with diagnostic codes (250.x0, 250.x2, 357.2 or 362.xx) [27]. We identified 63,806 T2DM patients based on their reference visit in 2012 at NHG primary care institutions: Choa Chu Kang Polyclinic, Bukit Batok Polyclinic, Jurong Polyclinic, Hougang Polyclinic, Toa Payoh Polyclinic, Clementi Polyclinic and Ang Mo Kio Polyclinic. Of these, 16,402 patients were excluded due to incomplete data on at least one of the candidate risk factors. The remaining 47,404 patients were randomly split into a 67% (31,603 patients) tool development sample and a 33% (15,801 patients) internal validation test sample.

The general guide for predicting a dichotomous outcome variable is as follows: for every predictor variable, there should be at least 10 positive cases. Based on JurongHealth statistics, the rate of inpatient admission due to hypoglycemia is 0.4%. Assuming a total of six predictor variables, and with the sample divided into two groups: development (67%) and validation (33%), we would require a sample size of at least 15,000.

Predictors found to be significant in literature [4, 13-26] and which are available in our database were included: patient demographics (age, race, gender), last HbA1c record, largest reduction in HbA1c within 1 year, last BMI, last eGFR and usage of sulphonylurea and/or insulin and hospital inpatient admission in the past 1 year. The outcome monitored was hypoglycemia hospital inpatient admissions to any of the following acute hospitals: National University Hospital (NUH), Alexandra Hospital (AH), Tan Tock Seng Hospital (TTSH). We identified hypoglycemia-related inpatient admissions using the ICD-9 codes as described by Ginde et al [28].

The main source of data was the NHG CDMS database. This databank links administrative and clinical data of patients with diabetes mellitus across the NHG healthcare cluster of primary care clinics and acute hospitals, allowing for presentation of clinical outcomes for evidence-based population management [27].

We compared the distribution of the predictor variables in our model development and validation samples (Table 1).

Click to view

Table 1. Characteristic of the Development and Validation of Type 2 Diabetes Mellitus Sets

The proportion of patients who were admitted for hypoglycemia during the follow-up in the development set was 0.4%.

Coefficients of the rare-event bias-corrected multivariate logistic regression model based on the development set are displayed in Table 2. The bias-correction factor (τ) was fixed at 0.4% to correspond with the estimated prevalence of hypoglycemia admissions in the study population [29]. The candidate risk factors: age, largest reduction in HbA1c within 1 year, last BMI, last eGFR and usage of sulphonylurea and/or insulin, were found to be statistically significant predictors of hypoglycemia admission by model selection using the Likelihood Ratio Statistic and were thus retained in the simplified model. The Hosmer and Lemeshow’s C statistic for the simplified model is 9.87 (df = 8; P = 0.27) indicating no evidence of poor fit.

Click to view

Table 2. Rare-Event Bias Corrected Logistic Regression Analysis of Hypoglycemia Admission During 6-Month Follow-Up

The Hypoglycemia Risk Score (Table 3) was constructed by scaling the coefficients; the category of each predictor with the lowest risk was set as baseline and allocated a score of 0. The score range was from 0 to 100. Scores for the remaining categories in each risk factor were calibrated based on the odds ratio corresponding to the midpoint of each category. The overall Hypoglycemia Risk Score is equivalent to the sum across scores awarded for each predictor.

Click to view

Table 3. Hypoglycemia Admission Risk Scoring System

The proportion of patients who were admitted for hypoglycemia during the follow-up in the testing set was 0.4%. The demographic and candidate risk factor profiles of patients in the testing set were similar to those in the development set. The Hypoglycemia Risk Score was calculated for each patient in the testing set. The receiver-operating characteristics (ROC) curve is provided in Figure 1. The area under the ROC curve for the testing set is 0.805 (95% CI: 0.763, 0.855), which suggests that the risk score is a good predictor of hypoglycemia admission. The sensitivity and specificity of the risk score at selected thresholds are listed in Table 4. The appropriate threshold may be determined based on the required limits to false positive and false negative cases. For example, a threshold of 32 may be chosen when a sensitivity of at least 80% is required. When applied to the testing set, this leads to 10 false negative and 6,324 false positive cases in the test sample. In the context of the current study, a threshold of 38 was preferred as it minimizes the difference between sensitivity (73.8%) and specificity (73.1%) leading to 17 false negative and 4,216 false positive cases in the test dataset.

Click for large image

Figure 1. Receiver operating characteristic (ROC) curves for performance of risk score in development and testing set.

Click to view

Table 4. Sensitivity and Specificity of the Hypoglycemia Risk Score at selected thresholds

Logistic regression may not provide the best fit for the data due to non-linear relationships that may be present. In exploring machine learning methods, we constructed multilayer perceptron and radial basis function ANN models. For comparison purposes, we used the same training and validation dataset that were used for the development of our risk-scoring tool. The ANN model trained on the seven predictors achieved an AUC of 0.844, a marginal improvement over the previous 0.805 achieved by our risk score tool.

Although the frequency of inpatient admissions for hypoglycemia was low among T2DM patients [4], it should be noted that hypoglycemia may lead to dangerous treatment complications such as convulsions and death [30]. In addition, the detrimental consequences induce fear among patients and affect their willingness to comply with an intensive glycemic control plan [31, 32]. By implementing timely changes to patients’ treatment plan, their risk of inpatient admission due to hypoglycemia may be reduced. This in turn can reduce the burden on local healthcare resources and the mortality rate due to treatment-related complications. As such, there is impetus to identify high risk patients and administer appropriate preventive interventions to avoid hospitalization.

Due to the unpredictability of the medical condition and frequent changes in clinical measurements, the task of assessing patients’ risk of hypoglycemia inpatient admissions may be challenging for clinicians. Early interventions can only be applied when we can accurately identify at-risk patients before the deterioration of their health status. Healthcare systems currently lack a standard evidence-based method to systematically identify patients at risk of hypoglycemia-related hospital admissions. Thus, the risk scoring tool was created to implement an efficient, cost-effective approach to manage hypoglycemia risk amongst T2DM patients, allowing for the segregation of patients into different risk strata so that the appropriate educational, technological, and/or pharmacological interventions may be applied for each group. The relevant intervention bundles would then appropriately be targeted at patients in accordance with their predicted risk strata through a stepped care approach [33].

The risk-scoring tool was developed and validated on a multi-ethnic Asian sample population and would be much more useful when applied to Asian populations [26]. This tool may be implemented into an EMR to allow for automated risk stratification based on the EMR’s data, thus allowing clinicians to easily assess patients’ risk of inpatient admission due to hypoglycemia.

In the development of our risk-scoring tool, we had also explored machine learning methods such as ANN. These models could offer new insights into the predictive quality of risk factors [34] and may provide a significant improvement in predictive performance [35]. In our study, the ANN model we trained provided higher predictive accuracy and better discrimination compared to our risk scoring tool (AUC 0.844 vs. 0.805). Although the ANN model would provide improvements for predictive accuracy, discrimination and dynamic improvement of the model, there remains significant practical barriers in implementing machine learning models within a healthcare system’s EMR [36]. This is in contrast with the risk scoring tool, which is simple to calculate and easy to use, and can even be calculated manually. Thus, the risk scoring tool still remains the preferred option when considering adoptability and ease of use.

Our study had several limitations. Firstly, in Singapore, patients are able to transit freely between the private and public health care sectors. Thus, baseline variables such as health services utilization in the last one-year may be biased if they were receiving care outside of the NHG system because it will not be captured in our diabetes registry.

Secondly, government polyclinics account for 20% of all primary care attendances. This may limit the generalizability of the study. However, the representativeness of our sample is supported given that the government polyclinics were found to manage a disproportionately higher share of chronic conditions (41%) [37].

Thirdly, other possible risk factors for hypoglycemia such as the duration of diabetes, cognitive impairment, etc. were excluded from our study as the data was not available in the NHG CDMS.

Lastly, the index cannot be used reliably in other patient populations that were not involved in its derivation. Further work is required to externally validate the risk index before using it in clinical settings.

A risk-scoring tool using commonly available data can help to identify those at risk of hypoglycemia admission with a satisfactory level of accuracy. Implementation of this tool could increase primary care clinicians’ awareness of patients’ hypoglycemia risk. This can aid in early and efficient identification of patients with high risk for timely intervention treatment care plans to reduce hypoglycemia inpatient admissions. The model can be used to assess the likelihood of hypoglycemia based on the constructed Hypoglycemia Risk Scoring Index. The index can identify patients at risk for hypoglycemia inpatient admission and reduce incidences of hypoglycemia and its associated morbidity and mortality. Our future plan is to embed this risk-scoring predictive tool into the Electronic Medical Records for the automatic generation of the risk scores, to allow clinicians and case managers to easily assess patients’ risk of inpatient admission due to hypoglycemia.

Acknowledgments

The authors would like to acknowledge Mr. Yu Weichang for his help and contribution on statistical analysis and methodology.

Financial Disclosure

The authors received no specific funding for this work.

Conflict of Interest

The authors declare that they have no competing interest.

Informed Consent

Informed consent was waived based on ethical approval from the NHG Domain Specific Review Board (DSRB Ref: 2014/00030).

Author Contributions

KC participated in the conception and design of the tool as the clinician lead and the principal investigator of the project. KPT contributed to the statistical analysis, model development and the revision of the manuscript. FWLP wrote the manuscript, and participated in the statistical analysis and model development. Toh MPHS and Seah DEJ contributed to data collection and the revision of the manuscript. CXW contributed to the study concept and design, statistical analysis, interpretation of data and revision of the manuscript. All the authors reviewed and approved the final manuscript.

1. Davis S, Alonso MD. Hypoglycemia as a barrier to glycemic control. J Diabetes Complications. 2004;18(1):60-68.
   doi
2. Cryer PE. The barrier of hypoglycemia in diabetes. Diabetes. 2008;57(12):3169-3176.
   doi pubmed
3. Tan GH. Hypoglycemia: a preventable and avoidable barrier to glycemic control. Annals Diabetes Res. 2017;1(1):1001.
4. Donnelly LA, Morris AD, Frier BM, Ellis JD, Donnan PT, Durrant R, Band MM, et al. Frequency and predictors of hypoglycaemia in Type 1 and insulin-treated Type 2 diabetes: a population-based study. Diabet Med. 2005;22(6):749-755.
   doi pubmed
5. Edridge CL, Dunkley AJ, Bodicoat DH, Rose TC, Gray LJ, Davies MJ, Khunti K. Prevalence and incidence of hypoglycaemia in 532,542 people with type 2 diabetes on oral therapies and insulin: a systematic review and meta-analysis of population based studies. PLoS One. 2015;10(6):e0126427.
   doi pubmed
6. Johnston SS, Conner C, Aagren M, Ruiz K, Bouchard J. Association between hypoglycaemic events and fall-related fractures in Medicare-covered patients with type 2 diabetes. Diabetes Obes Metab. 2012;14(7):634-643.
   doi pubmed
7. Signorovitch JE, Macaulay D, Diener M, Yan Y, Wu EQ, Gruenberger JB, Frier BM. Hypoglycaemia and accident risk in people with type 2 diabetes mellitus treated with non-insulin antidiabetes drugs. Diabetes Obes Metab. 2013;15(4):335-341.
   doi pubmed
8. Whitmer RA, Karter AJ, Yaffe K, Quesenberry CP, Jr., Selby JV. Hypoglycemic episodes and risk of dementia in older patients with type 2 diabetes mellitus. JAMA. 2009;301(15):1565-1572.
   doi pubmed
9. Barendse S, Singh H, Frier BM, Speight J. The impact of hypoglycaemia on quality of life and related patient-reported outcomes in Type 2 diabetes: a narrative review. Diabet Med. 2012;29(3):293-302.
   doi pubmed
10. Rombopoulos G, Hatzikou M, Latsou D, Yfantopoulos J. The prevalence of hypoglycemia and its impact on the quality of life (QoL) of type 2 diabetes mellitus patients (The HYPO Study). Hormones (Athens). 2013;12(4):550-558.
    doi pubmed
11. Laiteerapong N, Karter AJ, Liu JY, Moffet HH, Sudore R, Schillinger D, John PM, et al. Correlates of quality of life in older adults with diabetes: the diabetes & aging study. Diabetes Care. 2011;34(8):1749-1753.
    doi pubmed
12. Nicolucci A, Pintaudi B, Rossi MC, Messina R, Dotta F, Frontoni S, Caputo S, et al. The social burden of hypoglycemia in the elderly. Acta Diabetol. 2015;52(4):677-685.
    doi pubmed
13. Walter RM, Jr. Hypoglycemia: still a risk in the elderly. Geriatrics. 1990;45(3):69-71, 74-65.
14. Huang ES, Laiteerapong N, Liu JY, John PM, Moffet HH, Karter AJ. Rates of complications and mortality in older patients with diabetes mellitus: the diabetes and aging study. JAMA Intern Med. 2014;174(2):251-258.
    doi pubmed
15. Budnitz DS, Lovegrove MC, Shehab N, Richards CL. Emergency hospitalizations for adverse drug events in older Americans. N Engl J Med. 2011;365(21):2002-2012.
    doi pubmed
16. Kong AP, Yang X, Luk A, Ma RC, So WY, Ozaki R, Ting R, et al. Severe hypoglycemia identifies vulnerable patients with type 2 diabetes at risk for premature death and all-site cancer: the Hong Kong diabetes registry. Diabetes Care. 2014;37(4):1024-1031.
    doi pubmed
17. Shorr RI, Ray WA, Daugherty JR, Griffin MR. Incidence and risk factors for serious hypoglycemia in older persons using insulin or sulfonylureas. Arch Intern Med. 1997;157(15):1681-1686.
    doi pubmed
18. Kalra S, Deepak MC, Narang P, Singh V, Uvaraj MG, Agrawal N. Usage pattern, glycemic improvement, hypoglycemia, and body mass index changes with sulfonylureas in real-life clinical practice: results from OBSTACLE Hypoglycemia Study. Diabetes Technol Ther. 2013;15(2):129-135.
    doi pubmed
19. Stahn A, Pistrosch F, Ganz X, Teige M, Koehler C, Bornstein S, Hanefeld M. Relationship between hypoglycemic episodes and ventricular arrhythmias in patients with type 2 diabetes and cardiovascular diseases: silent hypoglycemias and silent arrhythmias. Diabetes Care. 2014;37(2):516-520.
    doi pubmed
20. Moen MF, Zhan M, Hsu VD, Walker LD, Einhorn LM, Seliger SL, Fink JC. Frequency of hypoglycemia and its significance in chronic kidney disease. Clin J Am Soc Nephrol. 2009;4(6):1121-1127.
    doi pubmed
21. Hodge M, McArthur E, Garg AX, Tangri N, Clemens KK. Hypoglycemia incidence in older adults by estimated GFR. Am J Kidney Dis. 2017;70(1):59-68.
    doi pubmed
22. Group UKHS. Risk of hypoglycaemia in types 1 and 2 diabetes: effects of treatment modalities and their duration. Diabetologia. 2007;50(6):1140-1147.
    doi pubmed
23. Miller ME, Bonds DE, Gerstein HC, Seaquist ER, Bergenstal RM, Calles-Escandon J, Childress RD, et al. The effects of baseline characteristics, glycaemia treatment approach, and glycated haemoglobin concentration on the risk of severe hypoglycaemia: post hoc epidemiological analysis of the ACCORD study. BMJ. 2010;340:b5444.
    doi pubmed
24. Cryer PE. Hypoglycaemia: the limiting factor in the glycaemic management of Type I and Type II diabetes. Diabetologia. 2002;45(7):937-948.
    doi pubmed
25. Karter AJ, Warton EM, Lipska KJ, Ralston JD, Moffet HH, Jackson GG, Huang ES, et al. Development and validation of a tool to identify patients with type 2 diabetes at high risk of hypoglycemia-related emergency department or hospital use. JAMA Intern Med. 2017;177(10):1461-1470.
    doi pubmed
26. Goh SY, Hussein Z, Rudijanto A. Review of insulin-associated hypoglycemia and its impact on the management of diabetes in Southeast Asian countries. J Diabetes Investig. 2017;8(5):635-645.
    doi pubmed
27. Toh MP, Leong HS, Lim BK. Development of a diabetes registry to improve quality of care in the National Healthcare Group in Singapore. Ann Acad Med Singapore. 2009;38(6):546-546.
28. Ginde AA, Blanc PG, Lieberman RM, Camargo CA, Jr. Validation of ICD-9-CM coding algorithm for improved identification of hypoglycemia visits. BMC Endocr Disord. 2008;8:4.
    doi pubmed
29. King G. ZL: Logistic regression in rare events data. Political Analysis 2001;9:137-163.
    doi
30. Mayo Clinic. Diabetic hypoglycemia. (2015, February 20). Retrieved from https://www.mayoclinic.org/diseases-conditions/diabetic-hypoglycemia/symptoms-causes/syc-20371525.
31. Polonsky WH, Henry RR. Poor medication adherence in type 2 diabetes: recognizing the scope of the problem and its key contributors. Patient Prefer Adherence. 2016;10:1299-1307.
    doi pubmed
32. Larkin ME, Capasso VA, Chen CL, Mahoney EK, Hazard B, Cagliero E, Nathan DM. Measuring psychological insulin resistance: barriers to insulin use. Diabetes Educ. 2008;34(3):511-517.
    doi pubmed
33. Yeoh E, Choudhary P, Nwokolo M, Ayis S, Amiel SA. Interventions that restore awareness of hypoglycemia in adults with type 1 diabetes: a systematic review and meta-analysis. Diabetes Care. 2015;38(8):1592-1609.
    doi pubmed
34. De Beule M, Maes E, De Winter O, Vanlaere W, Van Impe R. Artificial neural networks and risk stratification: A promising combination. Mathematical and Computer Modelling. 2007;46(1-2):88-94.
    doi
35. Yoo TK, Kim DW, Choi SB, Oh E, Park JS. Simple scoring system and artificial neural network for knee osteoarthritis risk prediction: a cross-sectional study. PLoS One. 2016;11(2):e0148724.
    doi pubmed
36. Brandon Ballinger. Three challenges for artificial intelligence in medicine. Cardiogram. 2016, September 19. Retrieved from https://blog.cardiogr.am/three-challenges-for-artificial-intelligence-in-medicine-dfb9993ae750.
37. Integrated Health Service Division, Ministry of Health. Primary Care Survey 2014. Singapore: Ministry of Health.

This article is distributed under the terms of the Creative Commons Attribution Non-Commercial 4.0 International License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Journal of Endocrinology and Metabolism is published by Elmer Press Inc.