Reliability, validity, and responsiveness of the Rhinitis Control Assessment Test in patients with rhinitis (2025)

Study designs and samples

The qualitative study to assess the content validity of the RCAT items consisted of individual interviews with 58 patients. Participants were (1) 18 years of age or older, (2) given a diagnosis of either allergic rhinitis or NAR based on clinical history and responses to skin prick tests, (3) experiencing rhinitis symptoms within the past 12 months, (4) taking either over-the-counter or prescription medications to treat their rhinitis symptoms, and (5) able to provide informed consent and read and understand English. Content validity was assessed through patient interviews conducted by 2 experienced interviewers with a standard interview guide. Patients were asked to describe their rhinitis symptoms and identify the symptoms that were most bothersome. After these questions, patients were asked to provide feedback on the 6 RCAT items while describing their thought process out loud. Interviewers then asked questions about the way in which patients interpreted the items and thoughts about potential responses.

The longitudinal validation study was a no-treatment study conducted at 29 investigational sites in the United States. All investigators were allergists, otolaryngologists, or both. Central (Great Lakes College of Clinical Medicine Institutional Review Board) or local institutional review board approval was obtained. The study consisted of 2 clinician visits occurring 15 to 29 days apart. Included participants (1) provided written informed consent; (2) were treatable on an outpatient basis; (3) had not seen a specialist regarding rhinitis symptoms in at least 3 years; (4) were at least 12 years of age; (5) received a diagnosis of noninfectious seasonal allergic rhinitis (SAR), perennial allergic rhinitis (PAR), or NAR based on a clinical history and skin test results; (6) were able to comply with study procedures; and (7) were literate. Participants were excluded if there was (1) evidence of rhinitis medicamentosa, (2) bacterial or viral respiratory tract infection at the time of the study visit, (3) a physical impairment that prevented the subject from participating, (4) clinical evidence of a Candida species infection, (5) severe obstruction of nasal passages caused by a deviated septum or nasal polyp, (6) a positive or inconclusive pregnancy test result for female patients, (7) evidence of acute or chronic sinusitis, (8) a history of psychiatric disease or dementia, or (9) use of allergy medications within 3 days before visit 1.

Data from qualified participants who had a Total Nasal Symptom Score (TNSS) of 2 or greater at visit 1 were included in the analysis. The TNSS is a daily symptom severity score that rates nasal congestion, rhinorrhea, nasal itching, sneezing, and postnasal drip on a 0- to 3-point scale. Stratified sampling was used to ensure appropriate sample sizes and equal representation of the 3 rhinitis types across 2 levels of symptom severity (TNSS 0-5=mild and TNSS 6-15=moderate/severe), with a goal of 400 eligible subjects: 150subjects in each of the PAR and SAR groups and 100 subjects in the NAR group, with half of each category of symptom severity in each diagnosis group.

Reliability

The reliability of the RCAT was evaluated by using internal consistency and test-retest reliability methods. For internal consistency, the Cronbach α value13 was computed from the 6 RCAT items at both study visits. Test-retest reliability was examined by calculating the intraclass correlation coefficient14 between RCAT scores at both visits among patients who self-reported no change in their rhinitis symptoms during the visit interval.

Convergent validity

Convergent validity was examined by computing Spearman rank-order correlations between RCAT scores and criterion measures, including TNSSs, patients' ratings of frequency and improvement in symptoms, and physicians' ratings of disease severity and rhinitis control. It was hypothesized that RCAT scores would correlate14 with each of these criterion measures with a Spearman r value of at least 0.3.

Discriminant validity

Discriminant validity was investigated by using known-groups validity15 in which criterion groups were defined that differed in ways that affect rhinitis control. We tested the ability of the RCAT to discriminate among groups of patients who differed on each of the criterion measures of physicians' ratings of disease severity (mild, moderate, and severe), TNSSs (mild=0-5, moderate=6-10, and severe=11-15), and physicians' ratings of rhinitis control (uncontrolled=not controlled at all, poorly controlled, somewhat controlled, and controlled=well-controlled or completely controlled) by using mean RCAT scores and ANOVA to test for significant differences. It was hypothesized that the groups of patients with more severe physician-rated disease, higher TNSSs, and uncontrolled rhinitis symptom status would have lower (worse) RCAT scores than patients with less severe disease, lower TNSSs, and controlled rhinitis symptom status.

Responsiveness

The responsiveness of the RCAT was evaluated by analyzing mean changes in RCAT scores between visits across groups of patients who changed on the criterion measures of (1) physicians' ratings of rhinitis symptom control, (2) TNSS severity categories, and (3) self-rating of change in rhinitis symptom control status, as reported at visit 2, by using ANOVA to test for significant differences. It was hypothesized that RCAT scores would increase (improve) among groups of patients whose symptoms improved and decrease among groups of patients whose symptoms worsened on each of the criterion measures.

Screening accuracy

The accuracy of the RCAT as a tool for screening patients with rhinitis symptom control problems was assessed by means of logistic regression and receiver operating characteristic (ROC) curve analyses by using visit 1 data and the criterion measure of the physician's rating of rhinitis symptom control. Patients were categorized either as having control problems if the physician's rating was “not controlled at all,” “poorly controlled,” or “somewhat controlled” or having controlled rhinitis if the physician's rating was “well controlled” or “completely controlled.” Aseparate analysis was conducted for each cut-point score from 14 to 25 on the RCAT scale. Results were summarized in terms of sensitivity, specificity, percentage of patients correctly classified, positive and negative predictive values, and the area under the ROC curve.

MID

Distribution-based and anchor-based approaches were used to determine the MID of the RCAT. For the distribution-based approach, the score equivalents of ½ SD (moderate effect size) and 1 SEM were considered as MIDs of the RCAT. The score equivalents of ½ SD16 and 1 SEM17,18 have both been considered thresholds for determining the MID on patient-reported outcome (PRO) tools.

The anchor-based approach examines the relationship between scores on the target instrument (RCAT) and some independent measure (anchor).19 Clinically relevant benchmarks likely to be conceptually related to the RCAT were used as anchors: (1) TNSS; (2) the physician's rating of rhinitis control; (3) the physician's rating of rhinitis severity; and (4) the patient's self-report of change in rhinitis symptom control. Differences in mean RCAT scores were computed between groups that differed on each criterion measure at baseline, and differences in mean changes in RCAT scores were computed between groups of patients that differed in the level of change on each criterion measure.

Patient samples

Table I provides demographic characteristics of patients who participated in the qualitative study to evaluate the content validity of the RCAT. The majority were female (67%) and white (74%), with a mean age of 48 years (range, 20-71 years). All patients had at least a high school education, and 77% had annual incomes of less than $75,000. Most patients self-reported allergic rhinitis (64%).

Table I also provides demographic characteristics of participants in the longitudinal study. Four hundred forty-nine patients were screened for the study, and 402 patients with rhinitis were randomized (12% screen failure rate) and administered the RCAT survey at visit 1 (150 with PAR, 152 with SAR, and 100 with NAR), of whom 396 (99%) also completed the survey at visit 2 (147 with PAR, 150 with SAR, and 99 with NAR). The average age was 37 years (SD, 15 years), and the majority of patients were female (66%) and white (81%). There were no significant differences in age, sex, or race across disease types. The mean baseline TNSS for the 402 randomized patients was 7.26 (SD, 3.24).

Content validity

Overall, patients found the RCAT items to be “relevant,” “concise,” and “easy to understand.” Most patients found the 1-week recall period appropriate for assessing the symptoms they experienced. All patients believed that they could accurately remember their rhinitis symptoms over the past week. The majority of patients found the response options of the RCAT items easy to understand and noted that each option was distinct from the adjacent options. Patients believed that the RCAT items appropriately covered most of the symptoms they experienced. However, roughly two thirds of the patients believed that several important symptoms, including itchy eyes, headache, and itchy throat, were not captured by the RCAT. Lastly, during the cognitive debriefing, patients indicated that all RCAT items were clear and easy to understand and relevant to their condition, and all patients interpreted the items similarly.

Reliability

The internal consistency reliability of the RCAT in the total sample was 0.77 at visit 1 and 0.84 at visit 2 (Table II) and was very similar across the 3 disease types. The test-retest reliability of the RCAT among patients who indicated no change in their rhinitis condition between visits 1 and 2 was 0.78 for the total sample (Table II). By disease type, test-retest reliability was highest among patients with PAR (0.84) compared with that among patients with SAR (0.78) and patients with NAR (0.72).

Convergent validity

Correlations between the RCAT and the other criterion measures (total TNSS and individual symptom scores, the patient's global rating of severity, and physicians' ratings of severity and control) were generally in the 0.3 to 0.6 range in the total sample and by disease type (Table III). The correlation between the RCAT and the physician's rating of rhinitis control for patients with SAR was smaller (0.24) compared with the other measures.

Discriminant validity

Mean RCAT scores differed significantly (P<.001) across groups of patients known to differ in TNSSs (F=72.7), physicians' ratings of rhinitis symptom control (F=28.6), and physicians' ratings of disease severity (F=34.1, Table IV). As hypothesized, mean RCAT scores were lowest among patients in the most severe group and highest among patients in the least severe group on each criterion measure. The discriminant validity of the RCAT was reproduced in each type of rhinitis (patients with PAR, SAR, and NAR).

Responsiveness

The responsiveness of the RCAT to changes in the criterion measures is presented in Table V. As shown, mean changes in RCAT scores differed significantly (P<.001) across patients, differing in their change in TNSS (F=50.2), physician's rating of control (F=23.7), physician's severity rating (F=36.5), and patient's self-assessment of change (F=63.3). As hypothesized, mean changes in RCAT scores increased (improved) significantly among those patients categorized as “improved” on each of the criterion measures. Mean changes in RCAT scores among patients categorized as the “same” or “worse” on each criterion measure were generally small and insignificant. Results were fairly consistent by disease type.

Screening accuracy

Table VI summarizes the performance of the RCAT in screening for rhinitis symptom control problems by using the criterion measure of the physician's rating of rhinitis symptom control. Classification statistics are presented for scores on the RCAT scale between 14 and 25. Scores greater than and less than these cut points are not presented because of poor classification statistics. Lower cut-point scores yielded lower sensitivity and higher specificity (fewer false-positive results), whereas higher cut-point scores yielded higher sensitivity and lower specificity (higher false-positive rate). The highest area under the ROC curve (0.689) was observed at a cut-point score of 21 or less (Fig 2), at which the sensitivity and specificity of the RCAT was 83% and 55%, respectively, and 79% were correctly classified.

MID

The SEM and ½ SD distribution approaches each yielded an MID estimate of roughly 2 points (Table VII). Results of the anchor-based approach yielded estimates ranging from 0.6 to 4.0 across criterion measures, with 8 (57%) of 14 between-group comparisons yielding MID estimates of between 2 and 3 points (average and median, 2.4 points). For comparisons between populations, the data suggest that a mean difference in RCAT scores between groups of greater than 2.4 would be clinically significant. For changes in individual patients over time, in which score changes are in integers, the value of 2.4 would be rounded up to a recommended MID of 3 points.