An evaluation of the inter-rater and intra-rater reliability of OccuPro’s functional capacity evaluation

4.1Design

It is important that research performed on medical testing is done independently from the developer of the medical test to reduce any potential bias. The developer of OccuPro’s FCE methodology, Jim Mecham, MSIE, OTR/L, CPE, provided standard FCE performance training at Concordia University of Wisconsin to students in the Occupational Therapy and Rehabilitation Science Programs who were working towards their Master’s degrees. All three research projects summarized in this publication where carried out from beginning to thesis exclusively by Occupational and Physical Therapy students at Concordia University Wisconsin, overseen by Carrie Scheel, EdD, OTR/L, CPE. All three research projects where approved by Concordia University Wisconsin’s Institutional Review Board prior to data collection.

In study one, Buckley, Ferracane, and Pickerill analyzed the inter-rater reliability of upper extremity testing within the OccuPro FCE and whether an FCE examiner could consistently determine occasional, frequent or constant return to work levels [13]. In study two, Devaguptapu used an exploratory design investigating the intra-rater reliability of occasional and frequent material handling testing within the OccuPro FCE methodology [10]. In the third study, Gunda investigated the inter-rater reliability of occasional and frequent material handling also within the OccuPro FCE [11].

4.2Participants

Study one, consisted of 27 subjects who were recruited from a local physical therapy clinic by treating Occupational Therapists who asked their existing upper extremity patients if they were interested in participating in a research project [13]. They all were identified as having an orthopedic or musculoskeletal disorder affecting the upper extremities. Subjects that underwent recent surgery (within 8 weeks) or an acute injury were excluded from the study. All participants were between the ages of 18 and 75. Seven subjects were unable to complete all three data collections. Reasoning for exclusion of these subjects included missed sessions (vacation, illness, etc.) and 2 subjects chose not to participate after reading the consent form. The final study sample consisted of 20 subjects.

Study two and three respectively included 31 healthy adults with no history of current or previous illness or injury [10, 11]. Seventeen females and 14 males were chosen from enrolled students at Concordia University Wisconsin using convenience sampling. Participants were between 20 and 70 years of age. Exclusion criteria for these studies included: participants not within the age range, injured workers, and participants with medical impairments. Three subjects were unable to complete the study for various reasons. The final study sample consisted of 28 subjects.

4.3Procedures

In study one, Buckley, Ferracane, and Pickerill analyzed upper extremity subtests of the OccuPro FCE [13]. Prior to testing, three separate raters from the Master of Occupational Therapy Program at Concordia University Wisconsin were trained in a four-hour FCE procedure course focusing on the upper extremity subtests of the OccuPro FCE. All testing was completed by these trained raters. Upper extremity subtests included evaluation of simple and firm grasp, pinch, gross motor, and fine motor. Five participants were randomly assigned to each rater during the first session. The participants then rotated to a different rater during the next two sessions, allowing each participant to be individually tested by each of the trained raters over a two-week period. Each session took about 30 minutes to complete. This process was completed a total of three times over a six week period of time. Between data collection sessions, the participants performed their normal daily activities.

The four OccuPro FCE upper extremity subtests were conducted using different measurement tools. In order to reduce incidents of fatigue and irritation of current injury, participants were offered a one minute break between subtests. The grasp strength assessment was measured using a grip strength dynamometer. Three measurements were taken for each hand and the mean was compared with the Mathiowetz age and gender norms. Pinch strength was measured using a pinch dynamometer. Three different pinch measurements were assessed including lateral pinch, palmar pinch, and tip pinch. Three measurements were taken for each hand and the mean was compared with the Mathiowetz age and gender norms. Gross motor was measured using the Box and Blocks assessment. Both hands were assessed and a total number of blocks were counted for each trial and compared to Matheowitz age and gender norms. Fine motor was assessed using the Purdue Peg Board assessment. Both hands were assessed with four total trials and compared to the Purdue Peg Board norms. Biomechanics were analyzed, a functional pain score gathered, pain behaviors noted, and a comparison to their demonstrated functional ability was performed after each of the subtests. Table 1 shows the decision-making process using OccuPro’s decision making algorithms.

In study two, Devaguptapu had healthy subjects perform occasional and frequent lifting and carrying testing [10]. The material handling tests were assessed by the same rater each time to establish intra-rater-reliability with the second rating being performed within two weeks of the first rating. Each subject was asked to perform an occasional and frequent bilateral squat lift, power lift and carry using a kinesiophysical approach. Occasional material handing testing was performed starting at low weight levels with one repetition at each weight increment and progressing up to a level where the subject exhibited biomechanical deficits. The weight level was then lowered by 5 or 10 pounds to the safe occasional lift/carry. Frequent material handling was tested starting at a low weight levels and progressed using a five-repetition approach and progressing up to the subject’s safe biomechanical change (single substitution pattern) level. The weight level was then lowered by 5 or 10 pounds to establish the safe frequent material handling level. The subject’s heart rate and rating of perceived exertion were measured with a heart rater monitoring strap and Borg Rating of Perceived Exertion Scale at the end of each task. Subjects were informed to stop the test at any time if they felt unsafe or their maximum capacity performance was reached prior to the rater stopping the test.

In study three, Gunda had healthy subjects perform occasional and frequent lifting and carrying testing with two different raters to establish inter-rater reliability with the second rater performing their data collection within two weeks for the first rater’s data collection [11]. Each subject was asked to perform an occasional and frequent bilateral squat lift, power lift and carry using a kinesiophysical approach. Occasional material handing testing was performed starting at low weight levels with one repetition at each weight increment and progressing up to a level where the subject exhibited biomechanical deficits. The weight level was then lowered by 5 or 10 pounds to the safe occasional lift/carry. Frequent material handling was tested starting at a low weight levels and progressed using a five-repetition approach and progressing up to the subject’s safe biomechanical change (single substitution pattern) level. The weight level was then lowered by 5 or 10 pounds to establish the safe frequent material handling level. The subject’s heart rate and rating of perceived exertion were measured with a heart rater monitoring strap and Borg Rating of Perceived Exertion Scale at the end of each task. Subjects were informed to stop the test at any time if they felt unsafe or their maximum capacity performance was reached prior to the rater stopping the test.

4.4Data analysis

In all three studies the Intraclass Correlation Coefficient (ICC) was used to measure the consistency of therapist ratings for FCE subtests and was calculated using SPSSv.20. Koo and Li in 2016 classified the ICC as follows: values less than 0.50 are considered poor reliability, 0.50 to 0.75 are considered moderate reliability, 0.75 to 0.90 is considered good reliability and ICC’s greater than 0.90 is considered excellent reliability [12].

In study one, Buckley, Ferracane, and Pickerill studied the inter-rater reliability of the upper extremity subtests of the OccuPro FCE [13]. Data collected included whether the person should avoid the upper extremity subtest or if they could perform the subtests occasionally, frequently or constantly as it relates to work.

In study two, Devaguptapu measured the intra-rater reliability of the OccuPro FCE by measuring intra-class correlation coefficients (ICC) between 28 subjects [10]. The assessments included 6 subtests that were measured by the same rater twice over a period of 2 weeks. There are six models for calculating the ICC. Model 3 was chosen for this research study because each subject was assessed by each of the two raters and these were the only raters available. Because there were only two raters, random selection of raters was not applicable. ICC scores range from 0.00 to 1.00, with values above 0.75 representing good to excellent reliability, and values below 0.75 representing poor to moderate reliability.

In study three, Gunda measured the inter-rater reliability of the OccuPro FCE by completing the intra-class correlation coefficient (ICC) between 28 subjects [11]. The assessment included 6 subtests that were measured by two different raters. The second model of the ICC was used to assess inter-rater reliability.

6.1Main findings

In study one, Buckley, Ferracane, and Pickerill, established the inter-rater reliability of the upper extremity subtests of the OccuPro FCE [13]. One of the challenges for FCE’s as a whole has been an examiner making a decision of the ability of the worker to perform at an occasional, frequent or constant level following a standardized test of grasping, pinching, fine motor coordination and gross motor coordination. The results of study one show that the firm grasp, simple grasp, fine motor, and gross motor subtests have good to excellent inter-rater reliability and that multiple testers can have the confidence that on the same patient they would come to the same decisions on whether this patient can perform grasping and coordination on an occasional, frequent or constant basis in regards to work. The results of the pinch subtest showed moderate inter-rater reliability and was noted to have the lowest ICC score within the first study. The decision-making procedures within the OccuPro FCE has been praised by users throughout the world as a means of having a brand new FCE examiner and an experienced FCE examiner come to similar consistent conclusions related to return-to-work decisions. The pinch testing inter-rater reliability was moderate which shows the return to work decision-making procedures for the pinch testing subtests could be defined better so the FCE examiner can make consistent return to work decisions.

Devaguptapu, in study two showed good to excellent intra-rater reliability when studying occasional and frequent lifting and carrying [10]. One of the primary decisions when determining return to work comes from the physical demand level of occasional, frequent, or constant material handling. It is tied tightly into the US Department of Labor Physical Demand Categories (PDC) which is defined in the Dictionary of Occupational Titles. Within the various FCE testing methodologies on the market there are slightly different methodologies to determine safe occasional and frequent material handling levels. FCE examiners using the OccuPro FCE testing method can have the confidence that the same examiner will make reliable decisions on the same patient if asked to perform multiple FCE’s on a patient.

Study three performed by Gunda showed moderate to good inter-rater reliability between occasional and frequent lifting and carrying [11]. It could be argued that inter-rater reliability, where two separate raters come to the same conclusion on one patient, has a higher level of importance within an FCE. It is noted that floor to waist lifting and bilateral carrying have good reliability which again tend to be the primary decision making parameters to determine the Physical Demand Category (PDC) in an FCE. The 12 inch to waist lift, both occasionally and frequently, had moderate reliability. This level of reliability allows FCE examiners to have confidence that separate raters make reliable decisions about occasional and frequent material handling and subsequently reliable Sedentary, Light, Medium, Heavy or Very Heavy PDC level decisions.

6.2Limitations

In study one, a limitation was the sample size secondary to a high attrition rate. Twenty seven subjects were originally recruited for the study but only 20 participated in all three data collection sessions. A larger sample size would increase the confidence in the results and allow for greater precision [13]. A further limitation was the written decision-making directions provided for the pinch and fine motor subtests. Following the performance of a functional test, the decision-making procedures are read by the FCE examiner and there was some ambiguity associated with pain ratings. Following a functional test, a testing subject reports a level of pain on a 0–10 functionally based pain scale and some raters scored the examinee higher and some scored them lower in regards to return to work ability secondary to the written directions. An improvement to the pain level verbiage in the decision-making procedures may help the reliability of pinch testing and fine motor coordination. Some further limitations outlined by the researchers included the examinees having lotion on their hands and fatigue as testing progressed.

In study two and three, a limitation to each of these studies was that all participants in the study were un-injured, healthy individuals and a sample of convenience. Although several research projects within FCE literature performed reliability research on healthy subjects, there is more value in FCE research when the study is performed on injured subjects since this has more similarity to an actual FCE. A further limitation to study two and three was that healthy university students were tested performing maximal lifting and carrying. Many of them reported that on session one they performed at their maximum level as requested and were sore the following day. This is like the soreness a patient might experience from an actual FCE in a clinic. However, the examinees during the second round of material handling testing verbally reported they did not want to lift and carry as much as they did on the first day of data collection as they did not want to be as sore as they were the first time. The examiners, as would be standard in an FCE, allowed the university students to stop the test if they chose to stop the test.

A limitation associated with the compilation of studies in this singular body of work include the separate and distinct variables that were measured in each study. Combining the three separate studies into one body of work limited the ability to combine variables which would have enhanced statistical power. Furthermore, the authors were unable to perform higher level statistical analysis as one overall study due to the different variables measured in each of the three studies.

Future research on the OccuPro FCE should include additional components of the FCE subtests, use of injured subjects, and larger sample sizes to improve the statistical power of the research.