Longitudinal Cohort Study of Verbatim-Reported Postural Instability Symptoms as Outcomes for Online Parkinson’s Disease Trials
Abstract
Background:
The Parkinson’s Disease Patient Report of Problems (PD-PROP) captures the problems and functional impact that patients report verbatim. Online research participation and advances in language analysis have enabled longitudinal collection and classification of symptoms as trial outcomes.
Objective:
Analyze verbatim reports longitudinally to examine postural-instability symptoms as 1) precursors of subsequent falling and 2) newly occurring symptoms that could serve as outcome measures in randomized controlled trials.
Methods:
Problems reported by >25,000 PD patients in their own words were collected online in the Fox Insight observational study and classified into symptoms by natural language processing, clinical curation, and machine learning. Symptoms of gait, balance, falling, and freezing and associated reports of having fallen in the last month were analyzed over three years of longitudinal observation by a Cox regression model in a cohort of 8,287 participants. New onset of gait, balance, falling, and freezing symptoms was analyzed by Kaplan-Meier survival techniques in 4,119 participants who had not previously reported these symptoms.
Results:
Classified verbatim symptoms of postural instability were significant precursors of subsequent falling among participants who were older, female, and had longer PD duration. New onset of symptoms steadily increased and informed sample size estimates for clinical trials to reduce the onset of these symptoms.
Conclusion:
The tools to analyze symptoms reported by PD patients in their own words and capacity to enroll large numbers of research participants online support the feasibility and statistical power for conducting randomized clinical trials to detect effects of therapeutic interventions.
INTRODUCTION
Postural instability is a therapeutically intractable feature of Parkinson’s disease (PD), associated with decreased quality of life, risk of falling, and adverse consequences on morbidity and mortality [1–4]. Gait, balance, and falling are common postural instability problems reported by PD patients, but these symptoms have not been obtained in patients’ own words nor measured systematically as clinical trial outcomes of interest. Direct patient reporting may also benefit earlier identification of research participants who are at risk of falls, particularly in the absence of in-person clinical assessment.
The Fox Insight online research platform (https://foxinsight.michaeljfox.org) was developed by the Michael J. Fox Foundation for Parkinson’s Research (MJFF) to collect longitudinal information from consenting participants who self-identified that they had been diagnosed with PD [5]. The study captures directly what participants feel and experience, and the manifold variables that affect illness. Collectively and over time, these data will inform a patient-reported natural history of PD and the design and conduct of clinical trials.
Emergent technology to analyze what patients say in their own words, using the Parkinson’s Disease Patient Report of Problems (PD-PROP), provides the opportunity to examine what participants report verbatim about their problems and related functional consequences. Beginning in 2017, PD-PROP data have been collected in Fox Insight at baseline in >25,000 PD patients and longitudinally over periods up to three years. Clinically curated verbatim replies from a Fox Insight longitudinal cohort of 1,557 participants over 2.5 years of observation have demonstrated the utility of the PD-PROP in characterizing and classifying symptoms referable to postural instability. When compared with traditional PD patient-reported categorical measures, the PD-PROP added predictive information and compared favorably with categorical scales in foretelling the self-reported occurrence of falling in the last month [6]. The current study extends the application of the PD-PROP to determine longitudinally the time-dependent relationships of postural instability symptoms as precursors of having fallen in the last month and the new onset of these symptoms as potential clinical trial outcomes.
METHODS
The recruitment plan and research protocol for the Fox Insight online study and platform, which includes the PD-PROP research module, has received prior approval by the New England/WCG Institutional Review Board, and informed consent was obtained online from each research participant [5].
Research participants were invited by the MJFF to complete initial and follow-up questionnaires at a study visit frequency of about six months. Each visit included a series of categorical questionnaires (e.g., Have you experienced falling in the last month?), and the PD-PROP that included two open-ended questions: 1) What is the most bothersome problem for you due to your Parkinson’s disease? and 2) In what way does this problem bother you by affecting your everyday functioning or ability to accomplish what needs to be done? Using keyboard entry, respondents reported in their own words, self-prioritized up to their fifth most bothersome problem. De-identified verbatim replies were analyzed as described below. There was no word limit for the keyboard-entered verbatim replies that were largely phrases or 1–2 sentences. Rarely, a full paragraph was entered. Regardless, the curation techniques were applied feasibly and uniformly in classifying the verbatims into symptoms.
Clinical curation of verbatim reports
Keyboard-entered verbatim reports from participants were analyzed by natural language processing (NLP) [7] and machine learning techniques [8] to generate the PD-PROP dataset for curation and classification into PD clinical domains and symptoms, as described previously [6]. In brief, random samples of text displays of concatenated problem and functional consequence were reviewed collectively by a team of clinical research and patient experience experts who then independently classified the reports into broad clinical domains (‘Autonomic Dysfunction’, ‘Bradykinesia’, ‘Cognition’, ‘Fatigue’, ‘Mood’, ‘Pain’, ‘Postural Instability’, ‘Rigidity’, ‘Sleep’, and ‘Tremor’) and specific PD symptoms.
Classifications were validated for specificity and sensitivity, which were highly concordant comparing human curators with the machine output [9, 10]. For the postural instability domain, the curators classified four major symptoms (sensitivity/specificity): Balance (97.0/95.4), Gait (100.0/96.5), Falling (100.0/95.2), and Freezing (100.0/91.7) [6]. De-identified symptom classification data for this report are available to researchers for independent analysis through the Fox Data Exploration Network (Fox DEN) (https://foxden.michaeljfox.org). Researchers should be able to independently replicate the analyses and reported results through (free) access to the Fox DEN symptom classifications and data dictionary and following the described and referenced methods, including the data flow diagrams in Fig. 1. Individual verbatim transcriptions are not available on Fox DEN to protect personal privacy.
Fig. 1
Participant datasets created for longitudinal regression and survival analyses
Participant selection
English-speaking adults who were diagnosed with PD by a physician or other health care professional were eligible to enroll using 24-hour online access to the Fox Insight research portal. Recruitment was carried out through social media, search engine marketing, e-mail newsletters, public research events, and clinician referrals [5].
After the data were filtered to eliminate the small proportion of outlier participants whose age was >100 years or age at diagnosis <15 years, two parallel datasets (Fig. 1) of research participants were examined.
The first dataset comprised the participants who had at least two visits reporting PD-PROP symptoms in their own words. The second dataset comprised the participants who had at least a baseline PD-PROP visit and who answered on at least two occasions the question Have you experienced falling in the last month? (‘NMS-Quest PD’ label on FoxDEN). This yes-no reply is distinguished from the PD-PROP symptoms, such as Falling, formed by verbatim reports of problems and their functional consequences. The analysis was limited to an observation period of 30 months, resulting in N = 13,892 participants in the PD-PROP dataset and N = 14,446 in the dataset ‘having experienced falling (yes-no) in the last month’ (Fig. 1).
The final analytic dataset (N = 8287) included those subjects in both these datasets who had the same baseline visit date for PD-PROP symptoms and the reply to having fallen in the last month; participants who reported at their baseline visit that they had fallen in the last month were excluded. To ascertain the new onset of any specific symptom in the survival analysis, the N = 8,287 dataset was further filtered to omit participants who reported any of the four postural instability symptoms at baseline, yielding a longitudinal dataset of N = 4,119 participants. In these analyses, participants verbatim replies about their bothersome problems and functional consequences are referred to as problems. The clinically curated problems are referred to as symptoms.
Statistical analyses
Postural instability symptoms referable to Balance, Gait, Falling, Freezing and their inter-relationships were analyzed by an UpSet plot [11] (“Company They Keep”). A time-dependent Cox analysis examined the relationship over time of precursors antedating the question (yes-no) ‘have you experienced falling in the last month.’ Results were expressed as hazard ratios and 95% confidence intervals. Age (≤65, >65), duration of PD as measured by years since diagnosis (YSD <3, 3–5, >5) and sex were included in the model. Time to first report of postural instability symptoms was analyzed by Kaplan-Meier survival methods [12] omitting participants who reported any of the four Postural Instability symptoms at baseline, and calculated as the number of days between the baseline visit and the first symptom report. Total sample size requirements were calculated using the method of George and Desu [13].
RESULTS
Between March 2017 and February 2020, more than 25,000 self-identified PD patients consented and enrolled online on the Fox Insight online research platform [5], representing an average monthly enrollment rate of about 800 participants.
The cohort N = 8,287 used for the regression analysis (Fig. 1) was characterized by mean age 65.6 years (range 23–93), sex 46% women, and YSD averaging 4.0 years with a distribution of 48% <3 years, 26% 3–5 years, 26% >5 years, 3.3 average visits/participant, and was similar to the cohort of N = 4,119 used for the survival analysis (Fig. 1) (mean age 64.9 years (range 23–92), sex 44% women, and YSD averaging 3.7 years with distribution of 51% <3 years, 27% 3–5 years, 22% >5 years, 3.3 average visits/participant). This English-speaking cohort was well educated (65.6% had ≥16 years of education), predominantly white Caucasian (95.6%), and from the United States (83.3%) and Canada (5.3%).
Among the 8,287 participants who completed the PD-PROP, the three-year aggregated proportions who reported any postural instability symptom were 50% for Gait, 29% for Balance, 18% for Falling, and 7% for Freezing. These symptoms were reported singularly (35%), but also occurred in various combinations (Fig. 2). Gait was the most common singular symptom (N = 2,002), followed by co-reporting of Gait and Balance symptoms (N = 878).
Fig. 2
UpSet plot (“Company They Keep”) of PD-PROP symptoms of Balance, Gait, Falling and Freezing. UpSet plot (“Company They Keep”) illustrates the number of PD-PROP symptoms of Balance, Gait, Falling and Freezing occurring prior to first affirmation of having fallen in the last month or end of observation period. Dots denote whether symptoms were reported alone (single dot) or in combination. Horizontal bars display total symptom reporting for each of the four symptoms.
Table 1 summarizes the frequencies of PD-PROP symptoms of Balance, Gait, Falling, and Freezing (including baseline reports) occurring prior to the first affirmative report of having fallen in the last month or the end of the observation period. Among the 8,287 participants in the survival analysis (Fig. 1), a total of 2,131 (26%) affirmed having fallen in the last month.
Table 1
Postural instability symptoms as precursors to subsequent falling
| Total N (8,287) | Did not fall 6,156 (74%) | Affirmed falling 2,131 (26%) | P | |
| Age | ||||
| ≤65 | 3,756 (45%) | 2,905 (77%) | 851 (23%) | <0.0001 |
| >65 | 4,531 (55%) | 3,251 (72%) | 1,280 (28%) | |
| Sex | ||||
| Male | 4,438 (54%) | 3,369 (76%) | 1,069 (24%) | 0.0003 |
| Female | 3,849 (46%) | 2,787 (72%) | 1,062 (28%) | |
| YSD | ||||
| <3 | 3,988 (48%) | 3,157 (79%) | 831 (21%) | <0.0001 |
| 3 – 5 | 2,164 (26%) | 1,620 (75%) | 544 (25%) | |
| >5 | 2,135 (26%) | 1,379 (65%) | 756 (35%) | |
| Balance | ||||
| Not reported | 5,924 (71%) | 4,553 (77%) | 1,371 (23%) | <0.0001 |
| Reported | 2,363 (29%) | 1,603 (68%) | 760 (32%) | |
| Gait | ||||
| Not reported | 4,171 (50%) | 3,184 (76%) | 987 (24%) | <0.0001 |
| Reported | 4,116 (50%) | 2,972 (72%) | 1,144 (28%) | |
| Falling | ||||
| Not reported | 6,815 (82%) | 5,193 (76%) | 1,622 (24%) | <0.0001 |
| Reported | 1,472 (18%) | 963 (65%) | 509 (35%) | |
| Freezing | ||||
| Not reported | 7,748 (93%) | 5,812 (75%) | 1,936 (25%) | <0.0001 |
| Reported | 539 (7%) | 344 (64%) | 195 (36%) |
PD-PROP symptoms including baseline reports of Balance, Gait, Falling, and Freezing, prior to first affirmative report to “Have you experienced falling in the last month” or the end of observation period. p-values represent chi-square or Cochran-Armitage tests comparing not having fallen to having fallen in the last month.
These data indicate that the baseline variables of age, sex, and YSD as well as the reported/not reported PD-PROP symptoms of Balance, Gait, Falling, and Freezing were all significantly different with respect to the yes/no report of having fallen in the last month. For example, among subjects who reported Balance problems, 32% reported having fallen in the last month, compared with 23% among those who did not report Balance problems.
Table 2 shows the time-dependent Cox proportional hazards analysis of time after baseline for affirming having fallen in the last month.
Table 2
Time-dependent impact of demographic features and postural instability symptoms
| Parameter | Hazard Ratio | 95% confidence limits | p |
| Age>65 | 1.259 | 1.153, 1.375 | <0.0001 |
| Sex female | 1.145 | 1.051, 1.247 | 0.0020 |
| YSD 3–5 | 1.052 | 0.944, 1.173 | 0.3573 |
| YSD>5 | 1.593 | 1.441, 1.762 | <0.0001 |
| Balance | 1.426 | 1.295, 1.569 | <0.0001 |
| Gait | 1.170 | 1.070, 1.279 | 0.0006 |
| Falling | 1.441 | 1.295, 1.604 | <0.0001 |
| Freezing | 1.494 | 1.283, 1.738 | <0.0001 |
Hazard ratios and 95% confidence limits from the time-dependent regression model. The outcome variable is the time after baseline to the first affirmative report of having fallen in the last month. The p-value for the overall two degrees of freedom test for years since diagnosis (YSD) is p < 0.0001.
The model describes the relationship between the incidence of having fallen in the last month and the various covariates in a single model. Balance, Gait, Falling, and Freezing are all significant precursor symptoms of having fallen in the last month after adjusting for the effects of the baseline variables of age, sex, and YSD. The effect of Balance, for example, adjusting for all other variables, shows a hazard ratio of 1.4, meaning that participants reporting balance symptoms are 40% more likely to subsequently affirm having fallen in the last month.
Figure 3 shows the Kaplan-Meier survival analysis of time to first report for each of the four postural instability symptoms (Balance, Gait, Falling, Freezing), after excluding participants who reported any of these symptoms at baseline. The four survival plots are overlaid on a single graph covering up to 2.5 years of longitudinal observation. The periodic pattern of the Kaplan-Meier curves parallels the approximately 6-month cadence of participant visits.
Fig. 3
Kaplan-Meier survival analysis of the time to the new occurrence of postural instability symptoms. Kaplan-Meier survival analysis of the time to the new occurrence of postural instability PD-PROP symptoms of Gait, Balance, Falling, and Freezing. Sample size calculations are in the table insert. Total number of participants at each time point are listed along the x-axis.
Survival analyses examining the time to first affirmation of having experienced falling in the last month showed trajectories to be very similar to the onset of PD-PROP postural instability symptoms (data not shown).
Sample size estimates on the insert table in (Fig. 3) are based on survival analysis data covering up to 1.0, 1.5, and 2.0 years of observation. These estimates can be applied to a randomized clinical trial comparing an active treatment with placebo over these durations for the new onset of PD-PROP symptoms of Gait, Balance, Falling, or Freezing as an outcome. Sample size calculations are based on risk (hazard ratio) reductions of 20%, 30%, and 50% using the logrank test with alpha = 5% (two-sided) and power of 80%. No allowance is made for dropouts.
The cumulative incidence of a new symptom is shown in the first three columns for trial durations of 1.0, 1.5, and 2.0 years. Using these survival data, for a trial lasting 1.5 years of observation, new symptom reporting is projected as Balance 36%, Gait 22%, Falling 16%, and Freezing 4%. For example, based on the foregoing sample size assumptions, and for an experimental treatment with 1:1 active:placebo allocation that is expected to exert a 30% reduction in risk of the new onset of these symptoms, total sample size (N) requirements would be Balance 686, Gait 1,125, Falling 1,521, and Freezing 5,490.
DISCUSSION
Among approximately 25,000 Fox Insight research participants, 13,892 (∼55%) provided two or more PD-PROP replies referable to the symptom domain of postural instability within an observation period up to three years. Within this timeframe, PD-PROP symptoms of Gait, Balance, Falling, and Freezing were significant precursors of having experienced falling in the last month. The longitudinal time-dependent measures differ in that participants who reported Falling as a PD-PROP symptom may or may not have experienced falling in the last month. Prior analysis has demonstrated that the cumulative occurrence of having fallen in the last month was highest for participants whose duration of PD was greater than five years since diagnosis and who prioritized the PROP symptom of falling as their most bothersome problem [6]. The current longitudinal analysis of postural instability symptoms as precursors of having experienced falling is also in keeping with the predictive value of these verbatim-reported symptoms compared with categorical scales such as the UPDRS part 2 [6].
While age and duration of PD as measured by years since diagnosis were expected to be predictors of having fallen in the last month, the disproportionate reporting by women is not as understandable. This sex difference could in part be related to greater reporting tendencies by women as well as greater vulnerabilities to falling [14, 15]. Contemporaneous confirmation of falls and their consequences should help clarify the factors accounting for the greater propensity of women to report falling [16].
Symptoms referable to postural instability and cognitive impairment are the two most therapeutically intractable features of PD [1, 2] and represent meaningful and detectable patient-reported outcomes for interventional trials. Postural instability symptoms arise relatively early in PD compared with cognitive symptoms that generally surface later. New-onset postural instability symptoms among all participants were reported anew by 1.5 and 2.0 years respectively for Gait (36% and 47%), Balance (22% and 28%), and Falling (16% and 23%). Freezing symptoms (4% and 7%), while troublesome, did not develop at the steady pace of Gait, Balance, and Falling.
The parallel analysis indicating similar trajectories for falling in the last month as an outcome is not surprising given the highly significant predictive and precursor impact of PD-PROP symptoms on the risk of falling in the last month. While the yes-no question may be suited as an outcome in online trials, it alone does not capture the underlying clinically meaningful symptoms of Gait, Balance, Falling, and Freezing. Similar longitudinal analyses are underway of self-reported PD-PROP cognitive symptoms that likely contribute to the emergence and impact of motor symptoms such as gait, balance, and falling [17]. Analyses of contributions of PD medications, clinical care considerations, other PD symptoms, and comorbidities (as accessible to researchers via FoxDEN) are also in progress. In the setting of randomized trials, it is expected that eligibility criteria, treatment randomization, and stratification of key variables will control and account for the impact of these potential contributors.
Lessening emerging symptoms of gait, balance, and falling would represent clinically meaningful benefit and a welcome therapeutic advance for people who are now affected by PD. The present findings may have direct clinical applications in identifying PD patients who are at high risk of falling and provide a research framework for examining behavioral, device, and pharmacological interventions. Capturing and analyzing what patients say in their own words can also clarify the meaning of terms such as ‘fear of falling’ and the utility of measures for balance confidence [18, 19].
While the estimated sample-size requirements may seem daunting for traditional in-person multicenter randomized trials, the online potential for large-scale enrollment, relative low participant burden, and cost efficiencies may surmount these concerns. Based on the observed average monthly accrual rate of about 800 new PD-PROP participants in the Fox Insight online study, sample size requirements estimated from longitudinal data are attainable in a timely fashion. The online approach of classifying directly reported symptoms may be particularly suitable for examining experimental treatments with known safety profile, such as behavioral, device or drug interventions.
Randomized clinical trials conducted remotely have already found a niche in the experimental therapeutics of COVID-19 [20, 21]. Importantly, the signals and insights generated by the online trial of fluvoxamine to treat symptomatic ambulatory COVID patients and forestall hospitalization and other serious outcomes [21] were recently confirmed in a larger trial [22]. The PD-PROP and human-in-the-loop curation provide the means to examine online therapeutic interventions that deserve the rigor of a randomized clinical trial using the outcome of symptoms that patients report directly. Important issues in online trials such as adherence monitoring, adverse effect reporting, laboratory surveillance, and misdiagnosis can be accomplished at home by in-person or telehealth visits involving research staff, nurses, and physicians.
Study limitations
While the large sample size and long period of observation of this study are remarkable, interpretation of the Fox Insight PD-PROP dataset is limited by several considerations. PD was self-reported by Fox Insight participants, perhaps questioning diagnostic validity, but studies have shown that what PD patients report compares favorably with medical records and clinician assessments [17, 23, 24]. There are real concerns about the generalizability of online reported symptoms due to sampling bias from limited access to broadband, computer, and devices. Efforts are growing to improve participation and diversity in online trials by expanded availability of broadband, less expensive devices, and skills training. Enrollment of an average of 800 PD patients monthly in Fox Insight is evidence of the appeal and interest to participate in online clinical research.
The Fox Insight online study cohort applies primarily to English-speaking participants who are predominantly Caucasian, educated and have basic computer skills. The high education level, averaging about two years of college, is in line with the educational attainment of research participants in PD multi-center trials [25, 26]. A cross-sectional study of 790 PD patients in the United Kingdom, where limited demographic information was collected, showed motor and non-motor classifications of verbatim reports that were very similar to the PD-PROP analyses [27]. The Fox Insight online cohort over-represents participants within three years of PD diagnosis, but this sampling is in keeping with requirements for most early PD trials. The cohort also spans 10 years since diagnosis, thereby affording longitudinal analysis of the emergence of symptoms. The sample estimates calculated in this report are based on online Fox Insight observational data and assume that online randomized clinical trials will have placebo outcomes that perform according to observational data trajectories. For in-person trial settings, placebo groups often perform better than informed observational data, thus limiting the accuracy of the sample size estimates. It remains to be seen whether this concern applies to online trials.
Despite these limitations, the patient history, what patients say in their own words, remains a cornerstone of clinical care. The PD-PROP tool has facilitated patient reporting of prioritized problems using unvarnished text that is free of categorical restraints and the imposing influence of clinicians. Maturing technologies to process language, clinical curation to afford clinical meaning, and machine learning techniques to classify verbatim reports enable interpretation and quantification of what patients feel and experience for application to clinical research. The advent of online research platforms, such as Fox Insight, has further empowered patients to participate in research in the comfort and privacy of home and at an extraordinary scale of data capture and analysis.
Conclusions
Postural instability and symptoms referable to gait disorder, balance, and falling developed anew in substantial proportions of PD patients over three years of observation. Online PD-PROP reporting in patients’ own words can be analyzed and practically applied as clinically meaningful outcomes in trials aimed at ameliorating or forestalling the onset of the therapeutically intractable postural instability symptoms of PD.
ACKNOWLEDGMENTS
The authors thank Amy Amara MD, PhD (University of Alabama Birmingham), Karen E. Anderson MD (Georgetown University), Clare Bale MSc (Parkinson’s UK), Lana Chahine MD PhD (University of Pittsburgh), Daniel Kinel JD (University of Rochester), Sneha Mantri MD (Duke University), Connie Marras MD, PhD (University of Toronto), Soania Mathur MD (Toronto, Canada), David Standaert MD, PhD (University of Alabama Birmingham), Benjamin Stecher BA (Toronto, Canada), Caroline Tanner MD, PhD (University of California San Francisco), and Daniel Weintraub MD (University of Pennsylvania) for their expert clinical curation of verbatim reports in Fox Insight and Carol Christopher PhD (Grey Matter Technologies) for technical and administrative support. We extend our special gratitude to the thousands of Fox Insight research participants and to the Michael J. Fox Foundation for Parkinson’s Research (MJFF) for supporting and enabling this remarkable online research study. Research reported in this publication was supported by the MJFF by Computational Science Awards to Grey Matter Technologies, The Pritzker Prize to Ira Shoulson MD, and the Fellowship in Regulatory Science of Parkinson’s Disease to the University of Rochester.
CONFLICT OF INTEREST
Dr. Shoulson is the founder and President of Grey Matter Technologies Inc (GMT) from which he receives compensation and has equity interests. Dr. Shoulson also receives compensation as Professor of Neurology at the University of Rochester (UR). Ms. Arbatti and Mr. Hosamath are employees of GMT. Ms. Eberly and Dr. Oakes are faculty members in the Department of Biostatistics and Computational Biology at the UR. This research is supported by computational science grants, the Pritzker Prize, and a fellowship in the Regulatory Science of Parkinson Disease from the Michael J Fox Foundation for Parkinson’s Research (NY), respectively to GMT, Dr. Shoulson and the UR.
DATA AVAILABILITY
The classified symptoms and associated data on Fox Insight are available to researchers through the Fox Data Exploration Network (DEN) (https://foxden.michaeljfox.org). Validation metrics for curation of postural instability symptoms have been published [6].
REFERENCES
[1] | Bloem BR , Beckley DJ , van Dijk JG , Zwinderman AH , Remler MP , Roos RA ((1996) ) Influence of dopaminergic medication on automatic postural responses and balance impairment in Parkinson’s disease. Mov Disord 11: , 509–521. |
[2] | Grimbergen YAM , Schrag A , Mazibrada G , Borm GF , Bloem BR ((2013) ) Impact of falls and fear of falling on health-related quality of life in patients with Parkinson’s disease. J Parkinsons Dis 3: , 409–413. |
[3] | Latt MD , Lord SR , Morris JGL , Fung VSC ((2009) ) Clinical and physiological assessments for elucidating falls risk in Parkinson’s disease. Mov Disord 24: , 1280–1289. |
[4] | Rahman S , Griffin HJ , Quinn NP , Jahanshahi M ((2008) ) Quality of life in Parkinson’s disease: The relative importance of the symptoms. Mov Disord 23: , 1428–1434. |
[5] | Smolensky L , Amondikar N , Crawford K , Neu S , Kopil CM , Daeschler M , Riley L , 23andMe Research Team, Brown E , Toga AW , Tanner C ((2020) ) Fox Insight collects online, longitudinal patient-reported outcomes and genetic data on Parkinson’s disease. Sci Data 7: , 67. |
[6] | Javidnia M , Arbatti L , Hosamath A , Eberly SW , Oakes D , Shoulson I ((2021) ) Predictive value of verbatim Parkinson’s disease patient-reported symptoms of postural instability and falling. J Parkinsons Dis 11: , 1957–1964. |
[7] | Nadkarni PM , Ohno-Machado L , Chapman WW ((2011) ) Natural language processing: An introduction. J Am Med Inform Assoc 18: , 544–551. |
[8] | MonkeyLearn, Guide to Text Classification with Machine Learning & NLP. |
[9] | Wu X , Xiao L , Sun Y , Zhang J , Ma T , He L ((2021) ) A survey of human-in-the-loop for machine learning. ArXiv210800941 Cs. |
[10] | Zhang S , He L , Dragut E , Vucetic S ((2019) ) How to invest my time: Lessons from human-in-the-loop entity extraction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining Association for Computing Machinery, New York, NY, USA, pp. 2305–2313. |
[11] | Conway J , Gehlenborg N ((2019) ) UpSetR: A More Scalable Alternative to Venn and Euler Diagrams for Visualizing Intersecting Sets. |
[12] | Cox DR , Oakes D ((2017) ) Analysis of Survival Data, Chapman and Hall/CRC, Boca Raton. |
[13] | George SL , Desu MM ((1974) ) Planning the size and duration of a clinical trial studying the time to some critical event. J Chronic Dis 27: , 15–24. |
[14] | Chou KL , Elm JJ , Wielinski CL , Simon DK , Aminoff MJ , Christine CW , Liang GS , Hauser RA , Sudarsky L , Umeh CC , Voss T , Juncos J , Fang JY , Boyd JT , Bodis-Wollner I , Mari Z , Morgan JC , Wills A-M , Lee SL , Parashos SA , NINDS NET-PD Investigators ((2017) ) Factors associated with falling in early, treated Parkinson’s disease: The NET-PD LS1 cohort. J Neurol Sci 377: , 137–143. |
[15] | Parashos SA , Bloem BR , Browner NM , Giladi N , Gurevich T , Hausdorff JM , He Y , Lyons KE , Mari Z , Morgan JC , Post B , Schmidt PN , Wielinski CL ((2018) ) What predicts falls in Parkinson disease? Observations from the Parkinson’s Foundation registry. Neurol Clin Pract 8: , 214–222. |
[16] | Schneider RB , Omberg L , Macklin EA , Daeschler M , Bataille L , Anthwal S , Myers TL , Baloga E , Duquette S , Snyder P , Amodeo K , Tarolli CG , Adams JL , Callahan KF , Gottesman J , Kopil CM , Lungu C , Ascherio A , Beck JC , Biglan K , Espay AJ , Tanner C , Oakes D , Shoulson I , Novak D , Kayson E , Ray Dorsey E , Mangravite L , Schwarzschild MA , Simuni T , Parkinson Study Group AT-HOME PD Investigators ((2021) ) Design of a virtual longitudinal observational study in Parkinson’s disease (AT-HOME PD). Ann Clin Transl Neurol 8: , 308–320. |
[17] | Zolfaghari S , Thomann AE , Lewandowski N , Trundell D , Lipsmeier F , Pagano G , Taylor KI , Postuma RB ((2022) ) Self-report versus clinician examination in early Parkinson’s disease. Mov Disord 37: , 585–597. |
[18] | Curtze C , Nutt JG , Carlson-Kuhta P , Mancini M , Horak FB ((2016) ) Objective gait and balance impairments relate to balance confidence and perceived mobility in people with Parkinson disease. Phys Ther 96: , 1734–1743. |
[19] | Lee HK , Altmann LJP , McFarland N , Hass CJ ((2016) ) The relationship between balance confidence and control in individuals with Parkinson’s disease. Parkinsonism Relat Disord 26: , 24–28. |
[20] | Boulware DR , Pullen MF , Bangdiwala AS , Pastick KA , Lofgren SM , Okafor EC , Skipper CP , Nascene AA , Nicol MR , Abassi M , Engen NW , Cheng MP , LaBar D , Lother SA , MacKenzie LJ , Drobot G , Marten N , Zarychanski R , Kelly LE , Schwartz IS , McDonald EG , Rajasingham R , Lee TC , Hullsiek KH ((2020) ) A randomized trial of hydroxychloroquine as postexposure prophylaxis for Covid-19. N Engl J Med 383: , 517–525. |
[21] | Lenze EJ , Mattar C , Zorumski CF , Stevens A , Schweiger J , Nicol GE , Miller JP , Yang L , Yingling M , Avidan MS , Reiersen AM ((2020) ) Fluvoxamine vs placebo and clinical deterioration in outpatients with symptomatic COVID-19: A randomized clinical trial. JAMA 324: , 2292–2300. |
[22] | Reis G , Moreira-Silva dos EA S , Silva DCM , Thabane L , Milagres AC , Ferreira TS , Santos dos CVQ , Campos de VH S , Nogueira AMR , Almeida de APFG , Callegari ED , Neto de AD F , Savassi LCM , Simplicio MIC , Ribeiro LB , Oliveira R , Harari O , Forrest JI , Ruton H , Sprague S , McKay P , Glushchenko AV , Rayner CR , Lenze EJ , Reiersen AM , Guyatt GH , Mills EJ ((2021) ) Effect of early treatment with fluvoxamine on risk of emergency care and hospitalisation among patients with COVID-19: The TOGETHER randomised, platform clinical trial. Lancet Glob Health 10: , e42–e51. |
[23] | Dorsey ER , Omberg L , Waddell E , Adams JL , Adams R , Ali MR , Amodeo K , Arky A , Augustine EF , Dinesh K , Hoque ME , Glidden AM , Jensen-Roberts S , Kabelac Z , Katabi D , Kieburtz K , Kinel DR , Little MA , Lizarraga KJ , Myers T , Riggare S , Rosero SZ , Saria S , Schifitto G , Schneider RB , Sharma G , Shoulson I , Stevenson EA , Tarolli CG , Luo J , McDermott MP ((2020) ) Deep phenotyping of Parkinson’s disease. J Parkinsons Dis 10: , 855–873. |
[24] | Dorsey ER , Wagner JD , Bull MT , Rizzieri A , Grischkan J , Achey MA , Sherer T , Chowdhury S , Meunier C , Cappelletti L , Rocker C , Richard IH , Schwarz H , Kang G , Ahmad SH , Biemiller RA , Biglan KM ((2015) ) Feasibility of virtual research visits in Fox Trial Finder. J Parkinsons Dis 5: , 505–515. |
[25] | ((2020) ) Isradipine versus placebo in early Parkinson disease a randomized trial. Ann Intern Med 172: , 591–598. |
[26] | The Parkinson Study Group SURE-PD3 Investigators ((2021) ) Effect of urate-elevating inosine on early Parkinsondisease progression: The SURE-PD3 Randomized Clinical Trial. JAMA 326: , 926–939. |
[27] | Port RJ , Rumsby M , Brown G , Harrison IF , Amjad A , Bale CJ ((2021) ) People with Parkinson’s disease: What symptoms do they most want to improve and how does this change with disease duration? J Parkinsons Dis 11: , 715–724. |
