Rasch Measurement Theory (RMT) Analyses of the Huntington’s Disease Everyday Functioning (Hi-DEF) to Evaluate Item Fit and Performance

Study population and sampling

An observational validation study was conducted virtually at 9 HD centers of excellence (COEs) across the US. The COEs helped in recruitment and assessment of participants for this study.

Inclusion criteria for participants included: (i) ability to read and respond to questionnaire items in English, (ii) age range of 25 to 65 years of age, (iii) huntingtin (HTT) mutation gene positive carrier, and (iv) investigator-confirmed cognitive changes or Total Functional Capacity (TFC) score ≥8 [23]. The exclusion criteria included an ongoing neurological condition other than HD that, in the opinion of the investigator, may have impacted the participants’ current cognitive or motor symptoms, or had a history of neurosurgical intervention or significant head injury. Participants received compensation of $150 for participation. This study received ethical approval from Advarra IRB (ref# Pro00048743).

Participants completed a demographic and medical history form, two PRO instruments, the Hi-DEF scale and HD-PRO-Triad (measuring cognitive, behavioral, and motor symptoms in HD) [16], and a battery of cognitive assessments (Spatial Span, One Touch Stockings of Cambridge, Spatial Working Memory, Emotion Recognition Task, and Paired Associate Learning). All of the above instruments were administered to participants on the Cambridge Neuropsychological Test Automated Battery (CANTAB) Connect platform, a cognitive assessment research software, via the web on a laptop or computer screen either at their home or during an onsite visit.

The study visit was completed either at the clinical site or at the participant’s home. The visit consisted of PRO and demographic questionnaire completion (20 min), a practice cognitive assessment session (30 min), a mandatory break (60 min), and then the scored cognitive assessment session (30 min). Participant medical history, including date of genetic testing, CAG repeats, and Unified Huntington’s Disease Rating Scale (UHDRS) TFC score were entered into the platform by study coordinators.

Hi-DEF description

The Hi-DEF is a PRO instrument developed to measure the impact of cognitive impairment on daily living in early-stage HD. The draft 47 items concerned daily life difficulties related to cognitive functioning challenges at ‘Home’ (17 items), ‘At work’ (13 items), and while ‘Driving’ (9 items), all of which were scored on a 5-level difficulty scale (1 = No difficulty, 2 = A little difficulty, 3 = Some difficulty, 4 = A lot of difficulty, 5 = Cannot do this anymore, N/A = Didn’t have the opportunity to do this in the past week). In addition, cognitive functioning challenges related to ‘Communicating’ (8 items) were scored on a 4-level frequency scale (1 = Never, 2 = Sometimes, 3 = Often, 4 = Almost always, N/A = Didn’t have the opportunity to do in the past week). The items were completed online and required a response on each item, resulting in no true missing data. Hi-DEF responses 1–5 were rescored as 0–4 for analysis; ‘N/A’ responses were recorded as missing.

Psychometric analysis method: Rasch measurement theory

Rasch measurement theory (RMT) analyses were used to examine the extent to which observed raw scores on the Hi-DEF meet the scores expected by the Rasch model and subsequently indicating the extent to which the summing of scale items results in rigorous measurement [24–26]. RMT analysis has three broad aims relating to evaluating the extent to which the sample-to-scale targeting is adequate, the measurement continuum has been constructed successfully, and the sample was valid. To this effect, seven RMT-based psychometric properties were examined (Table 1), results of which were interpreted with reference to published guidelines wherever possible [27].

To facilitate interpretation, measures (scores) were transformed into an accessible and intuitively meaningful metric, ranging from 0 (no difficulty) to 100 (maximum difficulty).

RMT analysis process

RMT analysis was performed using RUMM2030 software [28] in two stages. At the first stage, a comprehensive psychometric analysis of the measurement performance of the draft 47-item Hi-DEF scale was performed in line with methods [29–31] described above and in Table 1. Results were reviewed, and any necessary revisions were made to the item content and/or response scale. The revised item content and response scales, comprising the final Hi-DEF, were evaluated using the same RMT methods. Additional construct validity analyses were performed on the final 40-item Hi-DEF using the cognitive assessments and HD-PRO-TRIAD™ administered to patients; those analyses will be described in a separate manuscript.

Demographics

A total of 151 individuals with HD participated in this study. Recruitment was over a period of 12 months, although delays from academic IRB approvals reduced the time available for recruitment for some sites. The majority of participants (n = 117; 77.5%) completed the assessment within the expected duration of 3 hours.

The sample was representative of a population in early stages of HD (Table 2): 59% female, with a mean age of 47.3 years (SD 11.6; range 25 to 65), and a mean TFC score of 11.4 (range 8 to 13; score 8, n = 15; score 9, n = 15; score 10, n = 13; score 11, n = 21; score 12, n = 30; score 13, n = 55). A third of participants did not self-report any psychiatric conditions on the demographic and health information form (n = 50, 33.1%); however, the majority of participants self-reported at least one psychiatric condition: depression (n = 72, 47.7%), anxiety (n = 79, 52.3%), mood swings (n = 26, 17.2%), obsessive-compulsive symptoms (n = 16, 10.6%), and substance or alcohol use disorder (n = 4, 2.6%).

The majority of participants were highly educated, with either a bachelor’s (34%; n = 51) or postgraduate (28%; n = 42) degree, and most participants were working full-time (41%; n = 62) or part-time (11%; n = 16), while 24% (n = 36) were on disability related to HD (Table 2). In total, one-fifth (n = 30) of participants had a TFC = 13, were working full or part time, and had an associate’s, bachelor’s, or post-graduate degree. This subgroup of the validation sample provides the opportunity for insight into the performance of the measure at the upper end of functioning. However, further work is needed to assess the broader range of the measure and potential decline over time.

Step 1: RMT analysis of draft 47-items

For full results of the draft instrument, see Table 3. In summary, all subscales with the exception of ‘Driving’ had high reliability, minimal issues with person fit, and were found to be stable across gender and age groups (i.e., no differential item functioning [DIF]). In terms of sample-to-scale targeting, scales had adequate to good coverage ranging from 61% to 76% for all subscales except for ‘Driving’, which showed sub-optimal coverage (51%). In terms of item fit, items were largely cohesive, and item dependency was minimal across the four subscales, although there was some dependency in the full scale. Based on the results, 7 items were removed (‘Home’: 2, ‘At work’: 2, ‘Driving’: 1, ‘Communicating’: 2) to produce the final 40-item Hi-DEF, which consists of the following subscales: ‘Home’ (15 items), ‘At work’ (11 items), ‘Driving’ (8 items), and ‘Communicating’ (6 items). Supplementary Table 1 summarizes the items removed and corresponding rationale.

Step 2: RMT analysis of the final 40-items

Targeting & scale coverage

Targeting and scale coverage indicates whether the Hi-DEF items measure the full range of HD patients, from low to high cognitive impairment/daily functioning difficulties [32, 33]. A higher coverage rate means that more participants are measured. Sample-to-scale targeting, of the 40-item Hi-DEF was consistent with the draft version across all subscales and the full scale (Table 3). Scale coverage remained sub-optimal for the ‘Driving’ subscale (49%), adequate for the ‘At work’ subscale (60%), and good for the ‘Home’, ‘Communicating’, and the full scale with coverage ranging from 68% to 72%, respectively (Figs. 1 and 2). As Fig. 1 illustrates, some Hi-DEF items target high levels of difficulties where no patients from the current sample are located; these items are mostly from the ‘Driving’ and ‘Home’ subscales as well as two items from the ‘At Work’ subscale (Fig. 2).

Fig. 1

Hi-DEF scale (total score) targeting plot. The upper histogram represents the sample distribution for the Hi-DEF scale total score whereas the lower histogram represents the scale item threshold distribution plotted on the same linear measurement continuum. This allows a comparison between the range of cognitive functioning difficulties reported in the sample (upper histogram) and the range of cognitive functioning difficulties measured by the items of the Hi-DEF (lower histogram). Overlap between the ranges of the sample and item threshold distributions indicates the instrument is well-matched and able to measure the construct (cognitive functioning difficulties) within the sample accurately. The curve above the upper histogram represents an inverse function of the standard error associated with each person measurement (the peak of the curve indicating the best point of measurement).

Fig. 2

Targeting plots of Hi-DEF subscales: 1) Home; 2) At work; 3) Driving; 4) Communicating. The upper histogram represents the sample distribution for each Hi-DEF subscale (Home, Work, Driving, and Communicating) whereas the lower histograms represent the item threshold distribution for each subscale plotted on the same linear measurement continuum. This allows a comparison between the range of cognitive functioning difficulties reported in the sample (upper histogram) and the range of cognitive functioning difficulties measured by the items of the Hi-DEF (lower histogram) for each subscale. Overlap between the ranges of the sample and item threshold distributions indicates the instrument is well-matched and able to measure the construct (cognitive functioning difficulties) within the sample accurately. The curve over the upper histogram represents an inverse function of the standard error associated with each person measurement (the peak of the curve indicating the best point of measurement). These figures illustrate the targeting for each of the instrument subscales. The Hi-DEF subscales capture everyday functioning in different environments and may not be applicable to everyone.

Targeting was also examined with the sample divided by TFC score. On average, the higher the TFC score, the lower the Hi-DEF mean score (indicating fewer impacts of cognitive difficulties on daily functioning), as displayed by the mean logits on Fig. 3. Participants with TFC 9 reported slightly more difficulties on average than those with TFC 8; however, both these groups had a small sample size (n = 15 for both). Participants with TFC 10 on average reported more difficulties than TFC 11 or 12, although this was the smallest group of participants (n = 13). The mean scores may also have been affected by the participants who reported higher functioning than expected according to their TFC score (located more than one SD below the TFC score group mean): n = 2 participants with TFC 10; n = 3 participants with TFC 9; and n = 3 with TFC 8. Although some variation within participants with the same TFC score may be expected, it is possible that these participants could have under-reported their difficulties on the Hi-DEF due to a lack of insight into their cognitive impairment and related functioning. All other participants with TFC scores of 10 or lower were located at the lower-functioning end of the continuum (within one SD from the TFC score group mean) which suggests that, other than these eight participants (n = 8; 5.3% of the sample), participants’ Hi-DEF scores aligned with their TFC scores. This indicates agreement between participants’ self-reported cognitive functioning difficulties (Hi-DEF scores) and clinician ratings of their functioning (TFC scores).

Fig. 3

Hi-DEF full scale targeting plot by TFC score. The upper histogram (variable blocks: colors and patterns indicate TFC score, ranging from TFC 13 to TFC 8, as shown in the key) represents the sample distribution for the Hi-DEF scale, whereas the lower histogram represents the scale item threshold distribution plotted on the same linear measurement continuum.

Reliability

Reliability assesses the extent to which items can detect differences in cognitive impairment/daily functioning difficulties within a sample, and detect changes over time [32, 33]. Good reliability (the higher the Person Separation Index (PSI) value, the better) indicates a smaller proportion of measurement error in data collected through the Hi-DEF. Reliability of the 40-item Hi-DEF was also consistent with the draft version, with estimated PSI scores ranging between 0.53 (‘Driving’ subscale) and 0.87 (‘Home’ subscale), suggesting reliability varied from reasonable to excellent (Table 3). The full Hi-DEF had an estimated PSI of 0.92 (including extreme scores), suggesting excellent reliability.

Item thresholds

Item scoring is justified when patients can discriminate between all response categories (i.e., no difficulty, a little difficulty, a lot of difficulty, etc.) and perceive them as ordered as intended. Along the continuum of cognitive impairment/daily functioning difficulties, there are thresholds where two adjacent response categories are equally likely to be chosen. Ordered thresholds indicate that response options measure distinct categories [34].

The revised 40-item Hi-DEF improved the performance of the item thresholds but did not resolve some of the identified issues, as some item thresholds remained disordered. Specifically, findings indicate the 5-level difficulty response scale worked as intended for 93% (14/15 items) of the ‘Home’ subscale, 82% (9/11 items) of the ‘At work’ subscale, and 25% (2/8 items) of the ‘Driving’ subscale. This suggests participants could not consistently distinguish between the five suggested levels of difficulty for only 1–2 items of the ‘Home’ and ‘At work’ subscales, and for 6 items of the ‘Driving’ subscale. The ‘Driving’ subscale showed the weakest performance in terms of item thresholds, which was expected since not all of the sample drive. The 4-level frequency response scale of the ‘Communicating’ subscale displayed no disordered item response thresholds, indicating participants were able to distinguish between the four implied levels of frequency. In terms of the full Hi-DEF, 75% of the items (30/40 items) displayed ordered item thresholds.

Item fit

Item fit assesses whether Hi-DEF items work together clinically and statistically to form a cohesive continuum of cognitive impairment/daily functioning difficulties. Fit residuals within +/–2.5 and non-significant chi-square estimates indicate good item fit, which endorses the appropriateness of summing single item responses to obtain a total score [32, 35]. The revised 40-item Hi-DEF improved the cohesiveness of the ‘Home,’ ‘At work,’ and ‘Communication’ subscales while introducing minimal misfit for the ‘Driving’ subscale, leaving the overall item fit of the full scale unchanged (Table 3). Between 0 and 17% of items in each subscale displayed fit residuals outside the recommended range (+/–2.5), while review of chi-square only demonstrated marginal misfit in the ‘Driving’ subscale and the full scale (one item within each scale, corresponding to 12% and 2% of the items in each scale, respectively, which displayed misfit).

Item dependency

Local independence implies that responses to Hi-DEF items are only related due to a shared relationship with the same underlying latent variable, cognitive impairment/daily functioning difficulties, as measured by the Hi-DEF. Any additional relationships between any pair of Hi-DEF items means there is local dependence between those items. Residual correlations > 0.3 are strong evidence of local dependence as this reflects > 9% shared variance, indicating a higher chance that item responses are biased by each other [36, 37]. Item dependency in the 40-item Hi-DEF was consistent for the total and the ‘Home’ subscale score, with 3% (22/780 item pairs) and 1% (1/105 item pairs) of item pairs, respectively, demonstrating high residual correlations, suggesting minimal dependency. Item pairs of the ‘Driving’ subscale remained consistently locally independent while item dependency for the ‘At work’ and ‘Communicating’ subscales improved with no item pairs demonstrating high residual correlation in the 40-item version (Table 3).

Differential item functioning (DIF)

Item stability (DIF) assesses the extent to which Hi-DEF items are stable and work psychometrically in the same way across subgroups. Non-significant analyses of variance (ANOVAs) indicate no DIF, which endorses that measurements are objective and comparable between subgroups [38]. In the Hi-DEF full scale and subscales, items were stable and are interpreted in the same way across gender and age groups. Sample Hi-DEF items are presented in Table 4.

Person fit

Person fit indicates the extent to which participant response patterns are statistically consistent with the RMT model. Good person fit (up to 5% underfitting) ensures the measurement is valid. The 40-item Hi-DEF demonstrated consistent person fit. The percentage of person misfit for the subscales ranged between 1–5% with marginally higher misfit for the full scale (7%), which improved from the original draft version. These findings are supportive of the validity of sample measurement.