Elsevier

Journal of Psychiatric Research

Volume 75, April 2016, Pages 116-123
Journal of Psychiatric Research

Ecological momentary assessment versus standard assessment instruments for measuring mindfulness, depressed mood, and anxiety among older adults

https://doi.org/10.1016/j.jpsychires.2016.01.011Get rights and content

Highlights

  • Paper-and-pencil measures are traditionally used to measure treatment outcomes.

  • Emotionally distressed older adults participated in a randomized controlled trial.

  • We compared traditional vs ecological momentary assessment methods on study outcomes.

  • Study outcomes greatly differed depending on which measurement method was used.

  • Effect sizes for ecological momentary assessment methods were significantly higher.

Abstract

As mobile data capture tools for patient-reported outcomes proliferate in clinical research, a key dimension of measure performance is sensitivity to change. This study compared performance of patient-reported measures of mindfulness, depression, and anxiety symptoms using traditional paper-and-pencil forms versus real-time, ambulatory measurement of symptoms via ecological momentary assessment (EMA). Sixty-seven emotionally distressed older adults completed paper-and-pencil measures of mindfulness, depression, and anxiety along with two weeks of identical items reported during ambulatory monitoring via EMA before and after participation in a randomized trial of Mindfulness-Based Stress Reduction (MBSR) or a health education intervention. We calculated effect sizes for these measures across both measurement approaches and estimated the Number-Needed-to-Treat (NNT) in both measurement conditions. Study outcomes greatly differed depending on which measurement method was used. When EMA was used to measure clinical symptoms, older adults who participated in the MBSR intervention had significantly higher mindfulness and significantly lower depression and anxiety than participants in the health education intervention at post-treatment. However, these significant changes in symptoms were not found when outcomes were measured with paper-and-pencil measures. The NNT for mindfulness and depression measures administered through EMA were approximately 25–50% lower than NNTs derived from paper-and-pencil administration. Sensitivity to change in anxiety was similar across administration modes. In conclusion, EMA measures of depression and mindfulness substantially outperformed paper-and-pencil measures with the same items. The additional resources associated with EMA in clinical trials would seem to be offset by its greater sensitivity to detect change in key outcome variables.

Introduction

Ecological momentary assessment (EMA) is a data capture technique that involves repeated sampling of thoughts, feelings, or behaviors as close in time to the experience as possible in the naturalistic environment (Shiffman et al., 2008). Among the purported advantages of EMA is the mitigation of biases inherent in retrospective self-reports, such as the concern that the participant's reporting of subjective experiences in the past may be influenced by their current state (Axelson et al., 2003, Ebner-Priemer and Trull, 2009, Granholm et al., 2008, Johnson et al., 2009, Moskowitz and Young, 2006, Shiffman et al., 2008, Trull and Ebner-Priemer, 2009). Among older adults, memory impairment and unfamiliarity with questionnaire formats may further limit the validity of assessment tools that require the participant to recall their experience over the past week or month (Lenze and Wetherell, 2009). Assessing symptoms such as depressed mood or anxiety, or psychological constructs such as mindfulness, with retrospective self-report measures is particularly problematic given their variability within and between days (Baer et al., 2009, Bishop et al., 2004, Lau et al., 2006, Orsillo, 2005, Starr and Davila, 2012). EMA queries about present moment experiences in real time multiple times throughout the day, which could create more stable estimates of phenomena that fluctuate over time compared to single time-point measurement. For some internal experiences, such as mindfulness, in-the-moment questions may better enable sampling of experiences without the retrospective judgments that are inherent in global self-reports.

With the emergence of smartphones, there is unprecedented capacity to obtain EMA data in naturalistic environments. Even with the ‘digital divide’ in older adults' comfort and experience with technology, on average, relative to younger adults, a number of studies support the feasibility and acceptability of EMA techniques assessing multiple patient-reported outcomes with older adults (Cain et al., 2009). However, although much cross-sectional data support the feasibility and construct validity of EMA relative to traditional paper-and-pencil patient-reported outcomes, little is known about the sensitivity of EMA-based measures to change in clinical trials. The great majority of prior studies employing EMA have been observational studies and have not employed EMA in the context of detecting the effect of interventions. A number of authors have suggested that EMA could provide a useful approach to gathering patient-reported outcome measures and better representing the patient's experience over time during treatment (Gwaltney et al., 2008). Measurement error known to be associated with traditional paper-and-pencil measures can result in low assay sensitivity and potentially smaller intervention effect sizes of clinical trials (Cain et al., 2009, Collins et al., 2003, Slater and Bick, 1994). However “head-to-head” comparisons addressing sensitivity to change with identical point-in-time paper-and-pencil measures have, to our knowledge, not been performed. There is non-trivial participant training, burden, and expense in implementing EMA, and so its use as an outcome measurement tool would need to be justified by evidence of increased reliability, validity, and sensitivity to change over traditional self-reports. The added challenges posed by EMA implementation may be more substantial in older adults, who may require more training and support in using EMA.

In this study, we examined the psychometric properties and sensitivity of EMA in contrast to paper-and-pencil measures among older adults who participated in a randomized controlled trial examining Mindfulness-Based Stress Reduction (MBSR) vs. a health education control group. Identical EMA and paper-and-pencil measures of depression, anxiety (derived from Patient Reported Outcome Management System [PROMIS] Short-Form), and mindfulness (derived from the CAMS-R; Feldman et al., 2007) were administered at baseline and post-treatment, affording us the opportunity to contrast the reliability, concordance, and ability to detect changes over the study period. This is the first study, to our knowledge, to examine sensitivity to change of EMA methods in contrast to paper-and-pencil measures, and among the first to measure sensitivity to change in mindfulness as assessed via EMA. Comparing these two assessment methods is important because ultimately mindfulness-based interventions needs to show efficacy for clinical outcomes if it is to be a treatment for late-life mental disorders; this requires reliable measurement of clinical outcomes (Bierman et al., 2005). We hypothesized that 1) EMA would be associated with greater internal consistency and item-total correlations than paper-and-pencil measures, 2) changes in EMA would be associated with larger effect sizes than paper-and-pencil measures.

Section snippets

Participants and design

This multisite study was conducted at Washington University in St. Louis and the University of California, San Diego, and was approved by both sites’ institutional review boards. This study represents a secondary aim of a randomized clinical trial in which participants with anxiety or depressive disorders and subjective cognitive complaints were randomized to either participate in MBSR or health education. Therefore, the study was statistically powered to detect change in anxious and depressive

Participant characteristics

As seen in Table 1, participants in both treatment conditions did not significantly differ on any demographic variables; therefore, none of these variables were included as covariates in the analyses. Moreover, no significant differences were observed between the two treatment conditions on mindfulness measured with either the paper-and-pencil measure or EMA. However, baseline group differences were observed for depression and anxiety as measured with EMA, with more severe symptomatology

Discussion

This study compared the sensitivity to change of clinical symptoms among psychologically distressed older adults across two different assessment methods: ecological momentary assessment (EMA) versus traditional paper-and-pencil measures. Results indicated greater improvement in mindfulness, depression, and anxiety in the MBSR intervention than the control intervention when symptoms were measured via EMA, but these effects were not seen for depression and mindfulness on the corresponding

Submission declaration

The authors assert that this work has not been published previously, this it is not under consideration for publication elsewhere, that its publication is approved by all authors, and that, if accepted, it will not be published elsewhere including electronically in the same form, in English or in any other language, without the written consent of the copyright-holder.

Contributors

Raeanne Moore carried out the statistical analyses and wrote the paper. Colin Depp supervised statistical analyses and assisted with writing the paper. Julie Wetherell and Eric Lenze conceptualized the design of the study, supervised data collection, and assisted with writing the paper. All authors contributed to and have approved the final manuscript.

Role of funding source

This work was supported primarily by National Institutes of Health (EJL, grant number R34AT007064 and AG049369, JLW, grant number R34AT007070, RCM, grant number K23MH107260, and CAD, grant number MH100417). Additional funding came from the Taylor Family Institute for Innovative Psychiatric Research (EJL).

Conflicts of interest

EJL receives (past or present) non-governmental research support from the McKnight Brain Research Foundation, Taylor Family Institute for Innovative Psychiatric Research, Barnes-Jewish Foundation, Roche, Lundbeck, and Sidney R Baer Foundation.

References (35)

  • E. Bierman et al.

    Effects of anxiety versus depression on cognition in later life

    Am. J. Geriatr. Psychiatry

    (2005)
  • A.E. Cain et al.

    Ecological momentary assessment in aging research: a critical review

    J. Psychiatric Res.

    (2009)
  • C. Gwaltney et al.

    Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review

    Value Health

    (2008)
  • M. Allard et al.

    Mobile technologies in the elderly detection of cogntive decline

    PLoS One

    (2014)
  • D. Axelson et al.

    Measuring mood and complex behavior in natural environments: use of ecological momentary assessment in pediatric affective disorders

    J. Child Adolesc. Psychopharmacol.

    (2003)
  • R.A. Baer et al.

    Assessment of Mindfulness

    (2009)
  • S. Bishop et al.

    Mindfulness: a proposed operational definition

    Clin. Psychol. Sci. Pract.

    (2004)
  • J. Bjorner et al.

    Difference in method of administration did not significantly impact item response: an IRT-based analysis from the patient-reported out- comes measurement information system (PROMIS) initiative

    Qual. Life Res.

    (2013)
  • D. Cella et al.

    The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years

    Med. Care

    (2007)
  • R. Collins et al.

    The feasibility of using cellular phones to collect ecological momentary assessment data: application to alcohol consumption

    Exp. Clin. Psychopharmacol.

    (2003)
  • C.A. Depp et al.

    A Pilot study of mood ratings captured by mobile phone versus paper-and-pencil mood charts in bipolar disorder

    J. Dual Diagn.

    (2012)
  • U. Ebner-Priemer et al.

    Ambulatory assessment: an innovative and promising approach for clinical psychology

    Eur. Psychol.

    (2009)
  • A. Feingold

    Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis

    Psychol. Methods

    (2009)
  • G. Feldman et al.

    Mindfulness and emotion regulation: the development and initial validation of the cognitive and affective mindfulness scale revised (CAMS-R)

    J. Psychopathol. Behav. Assess.

    (2007)
  • L.S. Feldt et al.

    Statistical inference for coefficient alpha

    Appl. Psychol. Meas.

    (1987)
  • T.A. Furukawa et al.

    How to obtain NNT from Cohen's d: comparison of two methods

    PLoS One

    (2011)
  • M. Goyal et al.

    Meditation programs for psychological stress and well-being: a systematic review and meta-analysis

    JAMA Intern. Med.

    (2014)
  • Cited by (162)

    • Physical Activity, Sedentary Time, and Psychosocial Functioning among Adults with Cancer: A Scoping Review

      2024, International Journal of Environmental Research and Public Health
    View all citing articles on Scopus
    View full text