Reliability and validation of an electronic penicillin allergy risk-assessment tool in a pregnant population

Abstract

Background

Penicillin allergy adversely impacts patient care, yet most cases do not have true allergies. Clinicians require efficient, reliable clinical tools to identify low risk patients who can be safely de-labeled. Our center implemented the FIRSTLINE electronic point-of-care decision support tool to help non-allergist practitioners risk stratify patients with penicillin allergy. We sought to explore the reliability and validity of this tool in relation to allergist assessment and actual patient outcomes. We additionally compared it with two other published stratification tools, JAMA and PENFAST, to assess ability to accurately identify low risk patients appropriate for direct oral challenge.

Methods

In this single-center, retrospective, observational study, 181 pregnant females with self-reported penicillin allergy between July 2019 to June 2021 at BC Women’s Hospital, Vancouver, Canada were used to assess the reliability and validity of all three tools. Physician-guided history of penicillin use and symptoms were used for scoring. Results and recommendations were compared to actual patient outcomes after clinician decision for direct oral challenge or intradermal tests. We compared the performance of JAMA, PENFAST and FIRSTLINE.

Results

181 patients were assessed. 176/181 (97.2%) patients were deemed not allergic. Each risk stratification tool labelled majority of patients as low risk with 88.4% of patients PENFAST 0–2, 60.2% of patients JAMA low risk, 86.7% of patients FIRSTLINE very low risk.

Conclusion

We demonstrate that our point-of-care electronic algorithm is reliable in identifying low risk pregnant patients, as compared to an allergist assessment. To our knowledge, this is the first study to provide direct comparison between multiple decision support tools using the same population, minimizing participant bias. Providing clinical algorithms to risk stratify patients, can enable healthcare professionals to safely identify individuals who may be candidates for direct penicillin oral challenges versus needing referral to specialists. This increases the generalizability and efficiency of penicillin allergy de-labeling.

Background

Unconfirmed beta-lactam antibiotic allergies adversely impact patient care as using alternative antibiotics leads to increased risk of multi-drug resistant bacterial infections, prolonged hospitalization, and greater healthcare utilization [1]. Reported international penicillin allergy prevalence is approximately 10%, with most cases not having clinically significant hypersensitivity [2]. A systematic review revealed 96.5% of patients could be de-labeled without serious adverse reactions [3]. This is further supported by recent evidence whereby low risk penicillin allergy patients can safely receive direct oral challenge [4]. Our group previously described safety of DOC in pregnant people with low risk penicillin allergy histories [5]. Investigating penicillin allergy can be time and resource intensive, including an allergist referral, intradermal skin testing (IDT), and subsequent direct oral challenge (DOC).

Due to the high prevalence of reported penicillin allergy and limited allergist availability, non-allergist clinicians require an efficient, reliable approach to identify low risk patients who can be safely de-labeled. We have previously published a study showing that various healthcare professionals were able to safely and accurately risk stratify penicillin allergies with an electronic clinical algorithm in pediatric patients [6]. The electronic algorithm has been adapted for clinical use through an electronic decision support tool hosted by an antimicrobial stewardship mobile platform: FIRSTLINE (https://app.firstline.org/en/clients/39-bc-womens-hospital/steps/6158). Other risk stratifying algorithms exist, including: JAMA toolkit, a manual stratification and management tool, and PENFAST, a four-question tool providing the risk of true penicillin allergy (Supplementary material) [6,7,8].

For the pregnant population, many are hesitant to proceed with penicillin allergy de-labeling despite previous published literature on the safety of DOC during pregnancy [9]. To our knowledge there has been no specificity nor sensitivity comparison between the various de-labeling tools to inform practitioners on their ability to effectively identify low risk individuals who would qualify for a direct oral challenge test.

In this study, we sought to explore the reliability and validity of our electronic point-of-care decision support tool compared to an allergist assessment of “low risk”. We additionally compared the ability of the decision support tool to two other published stratification tools to identify patients at low risk of true penicillin allergy: JAMA, and PENFAST, to assess their ability to accurately identify low risk patients appropriate for direct oral challenge. We hypothesized that our electronic tool would be reliable and have high specificity for identifying low risk patients when compared with other decision support tools.

Methods

This is a single-center, retrospective, observational study with 181 pregnant females between July 2019 to June 2021. Patients with reported penicillin allergy were referred to the BC Women’s Hospital Penicillin Allergy clinic in Vancouver, Canada and were included regardless of medical, surgical, or medication history. Self-reported allergy histories required for each tool were routinely collected prior to patients’ first clinic visit. Information was stored on REDCap. This cohort was included in previous studies investigating the safety of DOC in pregnant populations [9].

The primary outcome of this study was to assess reliability of the electronic decision support tool compared with allergist assessment. Allergist assessment and to either pursue IDT and gold standard of DOC were previously recorded. Allergy histories were applied to the electronic decision support tool and the risk categories decided by the clinical assessment of the electronic decision tool were compared with allergist assessment. Risk categories included: allergic, very low risk of allergy, possible allergy, not allergic. The secondary outcome was to compare our electronic decision support tool with other published assessment tools to identify low risk patients appropriate for DOC. Patients were risk stratified by each tool into high, intermediate, and low risk categories.

With our interest specifically to identify patients at lowest risk for penicillin allergy, we opted to categorize all patients in the low risk stratification tier of each tool as screen negative this included those scoring PENFAST = 0–2 (very low and low risk), JAMA = low risk, FIRSTLINE = very low risk. All remaining patients were considered test screen positive. All patients deemed very low risk by clinician judgment received DOC. All remaining patients received IDT with benzylpenicilloyl polylysine and penicillin G. Positive IDT is defined as 3 × 3 mm greater than the negative control or a wheal size > 5 mm with underlying erythema, interpreted at 15 minutes [10,11,12]. If IDT was negative, patients then proceeded to a DOC. All equivocal IDT and delayed skin rash were considered test positive. All equivocal DOC without specifiers of exact symptoms experienced by patients were considered test negative. Examples of delayed DOC were those with delayed rash or delayed emesis (Type 4 hypersensitivity). None of the equivocal reactions had documented symptoms.

The assessor was blinded to patient information during data analysis. Collection of our data as a prospective cohort had received institution ethics approval. The screening tool results were then compared to clinician judgement to IDT and DOC. Additional ethics and consent waiver from the University of British Columbia was obtained for utilizing the data for algorithm validation.

Results

The study included 181 pregnant females between 22 and 48 years-old, referred by obstetricians, general practitioners, registered midwives, or pharmacists (Supplementary material). Patient reported reaction histories were collected (Table 1). Most patients had reactions over ten years ago (80.1%) and could not recall which penicillin drug was used (30.3%), nor the details of their reactions. The most reported adverse reactions were maculopapular rashes (49.2%), followed by urticaria (32.6%). Reaction histories were applied to the electronic de-labeling tool in order to obtain a risk category. Based on the tool recommendations: 7 patients screened as very low risk from the tool proceeded with a IDT based on clinician judgment while the remaining 150 patients with a DOC, while all 11 patients labeled as possible risk proceeded with DOC. Overall, 20 patients received IDT prior to DOC (Table 2). Of this, one patient (0.6%) had a positive result from an equivocal IDT. 161 (89.0%%) patients who had low risk assessments or moderate risk assessments and subsequent negative skin testing received DOC. Of this, four patients had delayed rashes and were deemed allergic. All of these four patients experienced skin rashes with two documented as morbilliform rash, none with documented time from oral challenge to adverse reaction (Table 3). In total, five patients were deemed allergic, however, none experienced anaphylaxis. Four patients with mild symptoms thought to be unrelated to the DOC were considered not allergic. One patient with equivocal DOC with repeated delayed emesis was thought secondary to pregnancy was also considered not allergic. Overall, 176 (97.2%) patients were deemed not allergic by an allergist. Patients were scored on each de-labeling tool (Table 4).

Table 1 Patient phenotype based on self-reported penicillin history. *Patients presented with a combination of symptoms.+Patients include those with delayed OC and equivocal ST. ^aReactions already prior: no details documented by patients. ^bOther symptoms including: face swelling, teeth discoloration, cold sensation, blurred vision, subjective weakness, feet swelling, canker sore, throat discomfort, lost voice, headache, HSP. ^cOther treatments included: Epinephrine: 0, antihistamines: 22, salbutabmol: 0, IV fluids: 1, steroids: 2, other: 12 (given another steroid, topical steroids, calamine lotion, unclear)

Table 2 Penicillin allergy test results. Result + are those who are deemed penicillin allergic, result- are those who are deemed penicillin nonallergic

Table 3 Demographic information on patients who experienced delayed oral challenge reaction

Table 4 Clinician decision to pursue intradermal skin test versus oral challenge compared to risk stratification via penicillin de-labeling tool

Based on the above results, our electronic assessment tool was able to label 86.7% of patients as very low risk, compared to 60.2% for JAMA low risk, and 88.4% for PENFAST score 0–2. Of these stratified low risk patients 80.7%, 58.6%, and 82.9% proceeded with a DOC based on clinician judgement for PENFAST, JANA, and FIRSTLINE respectfully.

Discussion

Congruent with previous data, we found low rates of true penicillin allergy in the pregnant population [13]. We acknowledge that our cohort has a low test positivity rate, as the purpose of our penicillin clinic is for “de-labeling”, thus is likely a referral bias towards lower risk patients. Many patients who were stratified as relatively higher risk by the allergist required skin testing but were still mostly deemed not allergic. The low prevalence of allergy in our cohort limits the ability for us to assess comparative efficacy at “ruling in an allergy”. It is difficult to assess the risk stratification capacity in this study as it includes mainly low-risk patients. Future validation should occur in larger cohorts across more varied risk levels.

Despite these limitations, clinical decision tools were designed to identify patients at low risk to facilitate safe de-labeling efforts. All three tools are aimed at helping the clinician conduct a penicillin de-labeling assessment. Our electronic tool provides specific question phrases which clinicians can ask while PENFAST and JAMA tools require clinicians to word their own questions. Our study shows that our electronic assessment tool is a reliable electronic tool to safely identify low risk patients similar to a board certified allergist assessment. The detailed and extensive algorithm likely contributes to its increased rates to identify low-risk patients, however, the time required may be more difficult to integrate into a busy practice. The high percentages of low-risk patients which later proceeded with oral challenge based on our electronic tool confirms our initial hypothesis and supports its use to DOC patients. We acknowledge that electronic tools do not necessarily replace physician judgment, as indicated by a proportion of patients stratified as low risk by the tool, who went on to receive IDT. There are multiple factors that can influence this decision, including economic and patient -related factors, such as anxiety.

Another limitation of this study is there may be recall bias while patients recollected past information for the questionnaire. The decision to label all equivocal IDT results was made to maximize safety in pregnant women. However, if a gold standard DOC had been completed, the patient may have tolerated it and been de-labeled. Additionally, we completed this validation study at a single centre with non-random recruitment, therefore extrapolating data to other populations requires caution.

To our knowledge, this is the first study to provide comparison between various available clinical decision support tools for penicillin allergy risk assessment. Our electronic tool has the benefit of both being freely accessible online and providing direct management recommendations. Use of this tool can empower non-allergist healthcare providers to safely identify and manage patients at low risk of true penicillin allergy. This allows for more patients to be de-labeled in community and hospital-settings more quickly and reduces waiting times for allergists to assess and manage patients who are at higher risk of true allergy.

Conclusion

Highly specific de-labeling tools can identify low risk patients while preserving safety. This tool can empower non-allergist healthcare providers to manage low risk patients in a resource limited healthcare system. We herein provide an assessment of reliability and show validity for an electronic point-of-care decision support tool for patients with penicillin allergy. We additionally provide a comparative analysis between our electronic tool and two other published risk-stratification tools in pregnant populations.