Ecological momentary assessment versus standard assessment instruments for measuring mindfulness, depressed mood, and anxiety among older adults
Graphical abstract
Introduction
Ecological momentary assessment (EMA) is a data capture technique that involves repeated sampling of thoughts, feelings, or behaviors as close in time to the experience as possible in the naturalistic environment (Shiffman et al., 2008). Among the purported advantages of EMA is the mitigation of biases inherent in retrospective self-reports, such as the concern that the participant's reporting of subjective experiences in the past may be influenced by their current state (Axelson et al., 2003, Ebner-Priemer and Trull, 2009, Granholm et al., 2008, Johnson et al., 2009, Moskowitz and Young, 2006, Shiffman et al., 2008, Trull and Ebner-Priemer, 2009). Among older adults, memory impairment and unfamiliarity with questionnaire formats may further limit the validity of assessment tools that require the participant to recall their experience over the past week or month (Lenze and Wetherell, 2009). Assessing symptoms such as depressed mood or anxiety, or psychological constructs such as mindfulness, with retrospective self-report measures is particularly problematic given their variability within and between days (Baer et al., 2009, Bishop et al., 2004, Lau et al., 2006, Orsillo, 2005, Starr and Davila, 2012). EMA queries about present moment experiences in real time multiple times throughout the day, which could create more stable estimates of phenomena that fluctuate over time compared to single time-point measurement. For some internal experiences, such as mindfulness, in-the-moment questions may better enable sampling of experiences without the retrospective judgments that are inherent in global self-reports.
With the emergence of smartphones, there is unprecedented capacity to obtain EMA data in naturalistic environments. Even with the ‘digital divide’ in older adults' comfort and experience with technology, on average, relative to younger adults, a number of studies support the feasibility and acceptability of EMA techniques assessing multiple patient-reported outcomes with older adults (Cain et al., 2009). However, although much cross-sectional data support the feasibility and construct validity of EMA relative to traditional paper-and-pencil patient-reported outcomes, little is known about the sensitivity of EMA-based measures to change in clinical trials. The great majority of prior studies employing EMA have been observational studies and have not employed EMA in the context of detecting the effect of interventions. A number of authors have suggested that EMA could provide a useful approach to gathering patient-reported outcome measures and better representing the patient's experience over time during treatment (Gwaltney et al., 2008). Measurement error known to be associated with traditional paper-and-pencil measures can result in low assay sensitivity and potentially smaller intervention effect sizes of clinical trials (Cain et al., 2009, Collins et al., 2003, Slater and Bick, 1994). However “head-to-head” comparisons addressing sensitivity to change with identical point-in-time paper-and-pencil measures have, to our knowledge, not been performed. There is non-trivial participant training, burden, and expense in implementing EMA, and so its use as an outcome measurement tool would need to be justified by evidence of increased reliability, validity, and sensitivity to change over traditional self-reports. The added challenges posed by EMA implementation may be more substantial in older adults, who may require more training and support in using EMA.
In this study, we examined the psychometric properties and sensitivity of EMA in contrast to paper-and-pencil measures among older adults who participated in a randomized controlled trial examining Mindfulness-Based Stress Reduction (MBSR) vs. a health education control group. Identical EMA and paper-and-pencil measures of depression, anxiety (derived from Patient Reported Outcome Management System [PROMIS] Short-Form), and mindfulness (derived from the CAMS-R; Feldman et al., 2007) were administered at baseline and post-treatment, affording us the opportunity to contrast the reliability, concordance, and ability to detect changes over the study period. This is the first study, to our knowledge, to examine sensitivity to change of EMA methods in contrast to paper-and-pencil measures, and among the first to measure sensitivity to change in mindfulness as assessed via EMA. Comparing these two assessment methods is important because ultimately mindfulness-based interventions needs to show efficacy for clinical outcomes if it is to be a treatment for late-life mental disorders; this requires reliable measurement of clinical outcomes (Bierman et al., 2005). We hypothesized that 1) EMA would be associated with greater internal consistency and item-total correlations than paper-and-pencil measures, 2) changes in EMA would be associated with larger effect sizes than paper-and-pencil measures.
Section snippets
Participants and design
This multisite study was conducted at Washington University in St. Louis and the University of California, San Diego, and was approved by both sites’ institutional review boards. This study represents a secondary aim of a randomized clinical trial in which participants with anxiety or depressive disorders and subjective cognitive complaints were randomized to either participate in MBSR or health education. Therefore, the study was statistically powered to detect change in anxious and depressive
Participant characteristics
As seen in Table 1, participants in both treatment conditions did not significantly differ on any demographic variables; therefore, none of these variables were included as covariates in the analyses. Moreover, no significant differences were observed between the two treatment conditions on mindfulness measured with either the paper-and-pencil measure or EMA. However, baseline group differences were observed for depression and anxiety as measured with EMA, with more severe symptomatology
Discussion
This study compared the sensitivity to change of clinical symptoms among psychologically distressed older adults across two different assessment methods: ecological momentary assessment (EMA) versus traditional paper-and-pencil measures. Results indicated greater improvement in mindfulness, depression, and anxiety in the MBSR intervention than the control intervention when symptoms were measured via EMA, but these effects were not seen for depression and mindfulness on the corresponding
Submission declaration
The authors assert that this work has not been published previously, this it is not under consideration for publication elsewhere, that its publication is approved by all authors, and that, if accepted, it will not be published elsewhere including electronically in the same form, in English or in any other language, without the written consent of the copyright-holder.
Contributors
Raeanne Moore carried out the statistical analyses and wrote the paper. Colin Depp supervised statistical analyses and assisted with writing the paper. Julie Wetherell and Eric Lenze conceptualized the design of the study, supervised data collection, and assisted with writing the paper. All authors contributed to and have approved the final manuscript.
Role of funding source
This work was supported primarily by National Institutes of Health (EJL, grant number R34AT007064 and AG049369, JLW, grant number R34AT007070, RCM, grant number K23MH107260, and CAD, grant number MH100417). Additional funding came from the Taylor Family Institute for Innovative Psychiatric Research (EJL).
Conflicts of interest
EJL receives (past or present) non-governmental research support from the McKnight Brain Research Foundation, Taylor Family Institute for Innovative Psychiatric Research, Barnes-Jewish Foundation, Roche, Lundbeck, and Sidney R Baer Foundation.
References (35)
- et al.
Effects of anxiety versus depression on cognition in later life
Am. J. Geriatr. Psychiatry
(2005) - et al.
Ecological momentary assessment in aging research: a critical review
J. Psychiatric Res.
(2009) - et al.
Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review
Value Health
(2008) - et al.
Mobile technologies in the elderly detection of cogntive decline
PLoS One
(2014) - et al.
Measuring mood and complex behavior in natural environments: use of ecological momentary assessment in pediatric affective disorders
J. Child Adolesc. Psychopharmacol.
(2003) - et al.
Assessment of Mindfulness
(2009) - et al.
Mindfulness: a proposed operational definition
Clin. Psychol. Sci. Pract.
(2004) - et al.
Difference in method of administration did not significantly impact item response: an IRT-based analysis from the patient-reported out- comes measurement information system (PROMIS) initiative
Qual. Life Res.
(2013) - et al.
The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years
Med. Care
(2007) - et al.
The feasibility of using cellular phones to collect ecological momentary assessment data: application to alcohol consumption
Exp. Clin. Psychopharmacol.
(2003)
A Pilot study of mood ratings captured by mobile phone versus paper-and-pencil mood charts in bipolar disorder
J. Dual Diagn.
Ambulatory assessment: an innovative and promising approach for clinical psychology
Eur. Psychol.
Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis
Psychol. Methods
Mindfulness and emotion regulation: the development and initial validation of the cognitive and affective mindfulness scale revised (CAMS-R)
J. Psychopathol. Behav. Assess.
Statistical inference for coefficient alpha
Appl. Psychol. Meas.
How to obtain NNT from Cohen's d: comparison of two methods
PLoS One
Meditation programs for psychological stress and well-being: a systematic review and meta-analysis
JAMA Intern. Med.
Cited by (162)
Current and Emerging Technologies to Address the Placebo Response Challenge in CNS Clinical Trials: Promise, Pitfalls, and Pathways Forward
2024, Innovations in Clinical NeurosciencePhysical Activity, Sedentary Time, and Psychosocial Functioning among Adults with Cancer: A Scoping Review
2024, International Journal of Environmental Research and Public Health