5 November 2019
Busting the myth that pen-and-paper is better than electronic outcome assessments
Clinical outcome assessments administered with pen-and-paper tasks have long been the assessment method of choice for measuring treatment effects on patients. However, with the advent of electronic alternatives - how do these two assessment methods compare?
The patient-perspective has become an integral part of the drug development and approval process1. Over the course of four years (2006-2010) the European Medicines Agency (EMA) and Food and Drug Administration (FDA) issued almost 50 guidance documents which recommended the collection of patient-reported outcomes in clinical trials (EMA = 34, FDA = 15; Marquis et al., 2011)2. The patient perspective is clearly a primary concern for the regulatory bodies – but how best to capture this?
What is a clinical outcome assessment?
The patient-perspective on the effects, and effectiveness, of clinical trials is commonly referred to as a patient-reported outcome (PRO). Outcomes can be reported by: the patient (PRO) directly, their clinician (ClinRO), or an observer (ObsRO). Clinical outcome assessments (COAs) use reported outcomes and task performance (PerfO) to capture and measure the effects of treatment on patient functioning3.
Electronic clinical outcome assessments: what are the facts?
COAs have traditionally been administered with pen-and-paper tasks. However, with the widespread availability of electronic devices (tablets, smartphones), and fast-paced internet connections, the electronic delivery of COAs has become a credible alternative. Nevertheless, the following myths about pen-and-paper continue to propagate:
- Better quality
- More secure
- Preferred by patients
Here we will tackle each of these myths to show that electronic COAs (eCOAs) deliver high-quality outcome metrics which are comparable to pen-and-paper tasks4, and actually offer more operational efficiency and cost effectiveness.
Whilst it’s true that the set-up costs of electronic assessments are typically higher than pen-and-paper, the down-stream savings are substantial. The feasibility of conducting clinical trials over the internet was investigated by McAlindon, Formica, Kabbara, LaValley, and Lehmer (2003). The study found that the trial cost per participant was half as expensive when conducted online ($914) than in the hospital ($1925)5. The researchers suggested that the significant differences in cost were driven by the substantial overheads associated with hospital-based testing, such as: clinician time, clinical space, participants’ travel costs, and time taken to enter the data 5. Furthermore, these hospital-based costs would likely be exacerbated in a multi-site trial. In contrast, web-based testing additional participants places minimal additional costs on the trial6. The operational-overheads associated with pen-and-paper methods makes it difficult to achieve high-quality, multi-site clinical trials for an acceptable price7.
Using pen-and-paper methodologies seems simple and speedy. However, this manual approach becomes increasingly costly, on time and resources, as the trial develops. In contrast, eCOAs offer operational efficiency at every stage of the process. At the data collection stage, completing a questionnaire online should take about half as long as answering the same questions in a telephone interview8. The inherently automated nature of eCOAs dramatically reduces data-entry time. The down-stream benefits of this include: expediting the transition from data collection to analysis and generating results, as well as minimising the burden on researchers’ time, thus reducing overall study spend. Furthermore, eCOAs automatically upload data in the desired format, which reduces the risk of data-entry and transformation errors.
A myth which surrounds electronic testing is that high-quality, paper-based assessments cannot be meaningfully replicated in an electronic format. However, there is a large amount of evidence to suggest otherwise. For example, an epidemiological study asked 157 volunteers to complete an anthropometric questionnaire (weight, height etc.) online and on paper9. The study found that not only was the web-based version preferred by participants (92.2%), but both formats provided the same high-quality test9. Furthermore, several clinical assessments which were originally developed for pen-and-paper administration, have now been validated for use online, such as the Edinburgh Depression Scale10, and the Obsessive Compulsive Inventory11.
Automation mediates the risk of human error, which enhances the overall quality of the data. Taking the epidemiological study conducted by Touvier et al. (2010) again, data gathered online showed no errors in data entry because the process is automated. In contrast, the pen-paper-format showed human-error in the form of: data entry mistakes, missing, inconsistent and abnormal values9. Although, the number of errors as an overall percentage of the data was low (0.1%-1.5%), it takes expensive researcher time to resolve these queries. This is on top of the researcher time spent on the initial data collection and entry. There are further concerns that during data collection, within- and between-rater bias can introduce variability into the data12. A rater may score a participant differently on different days, or differently from another rater. In contrast, eCOAs always follow an automated procedure which provides consistency.
The risk of cyber-attacks is what often compels researchers to reach for the pen-and-paper. Indeed, cybersecurity was a pressing concern at the advent of web-based testing. However, because of this, electronic data protection is now heavily regulated. For example, the EU General Data Protection Regulation (GDPR) was passed in 2018 to provide cohesive conventions for the protection of people’s data across the EU, and enforce large fines from companies who are non-compliant13. Although, the web-based nature of eCOAs may appear risky, there is also safety in how the data is frequently backed-up, whereas paper is inherently perishable.
Preferred by patients?
Patient-centred care is essential to engage and retain patients in a trial; so the most important factor to consider when comparing pen-and-paper and electronic COAs is patient preference. A recent systematic review of studies which used electronic and paper-based PROs was conducted to compare the validity of the measures, and investigate if patients preferred one format over the other4. The review found that 78% (of 55 articles) showed equivalence between the formats in terms of quality, and 87% (of 30 articles) showed that patients preferred electronic formats4.
Patients show a preference for electronic PROs across therapeutic areas: from oncology14 to rheumatoid arthritis15. In the rheumatoid arthritis study, despite 67.8% of patients having no prior experience of computers, a staggering majority (86%) preferred the electronic format to the paper format (2%)15. This study sampled 87 patients aged 34-83 years, subverting the myth that “older adults don’t like technology”. A 2018 survey in Australia further challenged this myth. Of 547 older adults sampled (aged 55-96 years old), over half preferred computerised assessments16. The crux of this preference seems to be that web-based testing offers participants the option to be tested at home. Interestingly, only 3.6% of the older adults expressed a desire to be accompanied by a nurse during home-based testing16. This finding would suggest that eCOAs are not only tolerated by older adults but, for many, are favoured over pen-and-paper formats.
- Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: An ad hoc task force report of the patient-reported outcomes (PRO) Harmonization Group meeting at the food and drug administration, February 16, 2001. In: Value in Health. Vol 6. Elsevier; 2003:522-531. doi:10.1046/j.1524-4733.2003.65309.x.
- Marquis P, Caron M, Emery M-P, Scott JA, Arnould B, Acquadro C. The Role of Health-Related Quality of Life Data in the Drug Approval Processes in the US and Europe. Pharmaceut Med. 2011;25(3):147-160. doi:10.1007/BF03256856.
- FDA. Drug Development Tools Qualification Programs - Clinical Outcome Assessment Qualification Program. https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm284077.htm. Published 2017. Accessed January 11, 2018.
- Campbell N, Ali F, Finlay AY, Salek SS. Equivalence of electronic and paper-based patient-reported outcome measures. Qual Life Res. 2015;24(8):1949-1961. doi:10.1007/s11136-015-0937-3.
- McAlindon T, Formica M, Kabbara K, LaValley M, Lehmer M. Conducting clinical trials over the internet: feasibility study. BMJ. 2003;327(7413):484-487. doi:10.1136/bmj.327.7413.484
- Dillman DA, Smyth JD, Christian LM, Dillman DA. Internet, Mail, and Mixed-Mode Surveys : The Tailored Design Method. Wiley & Sons; 2009.
- Ekman A, Litton J-E. New times, new needs; e-epidemiology. Eur J Epidemiol. 2007;22(5):285-292. doi:10.1007/s10654-007-9119-0.
- Coyne KS, Sexton CC, Kopp ZS, et al. Rationale for the study methods and design of the epidemiology of lower urinary tract symptoms (EpiLUTS) study. BJU Int. 2009;104(3):348-351. doi:10.1111/j.1464-410X.2009.08425.x.
- Touvier M, Méjean C, Kesse-Guyot E, et al. Comparison between web-based and paper versions of a self-administered anthropometric questionnaire. Eur J Epidemiol. 2010;25(5):287-296. doi:10.1007/s10654-010-9433-9.
- Spek V, Nyklícek I, Cuijpers P, Pop V. Internet administration of the Edinburgh Depression Scale. J Affect Disord. 2008;106(3):301-305. doi:10.1016/j.jad.2007.07.003.
- Coles ME, Cook LM, Blake TR. Assessing obsessive compulsive symptoms and cognitions on the internet: Evidence for the comparability of paper and Internet administration. Behav Res Ther. 2007;45(9):2232-2240. doi:10.1016/j.brat.2006.12.009.
- Khan A, Yavorsky WC, Liechti S, et al. Assessing the Sources of Unreliability (Rater, Subject, Time-Point) in a Failed Clinical Trial Using Items of the Positive and Negative Syndrome Scale (PANSS). J Clin Psychopharmacol. 2013;33(1):109-117. doi:10.1097/JCP.0b013e3182776ebe.
- The European Parliament and The Council of the European Union. REGULATION (EU) 2016/ 679 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL - of 27 April 2016 - on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/ 46/ EC (General Data Protection Regulation).; 2016. http://ec.europa.eu/justice/data-protection/reform/files/regulation_oj_en.pdf. Accessed January 16, 2018.
- Ring AE, Cheong KA, Watkins CL, Meddis D, Cella D, Harper PG. A Randomized Study of Electronic Diary versus Paper and Pencil Collection of Patient-Reported Outcomes in Patients with Non-Small Cell Lung Cancer. Patient. 2008;1(2):105-113. http://www.ncbi.nlm.nih.gov/pubmed/22272807. Accessed December 1, 2017.
- Salaffi F, Gasparini S, Grassi W. The use of computer touch-screen technology for the collection of patient-reported outcome data in rheumatoid arthritis: Comparison with standardized paper questionnaires. Clin Exp Rheumatol. 2009;27(3):459-468. doi:2627 [pii].
- Earl JK, Gerrans P, Hunter M. Better Ways of Assessing Cognitive Health. Brisbane; 2017. https://nationalseniors.com.au/system/files/09172697PAR_CognitiveHealth_Report_FNREV_WEB.pdf. Accessed November 22, 2017.
By Sally Jennings
Tags : ecoa