TY - JOUR
T1 - Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions
AU - Mesa-Eguiagaray, Ines
AU - Böhning, Dankmar
AU - McLean, Chris
AU - Griffiths, Peter
AU - Bridges, Jackie
AU - Pickering, Ruth M.
N1 - Funding Information:
The analysis presented here is based on data collected during research funded by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care Wessex, the NHS South Central through funding of a clinical lecturer internship undertaken by CMcL and the National Institute for Health Research Health Services and Delivery Research programme. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Health Services and Delivery Research programme, NIHR, NHS or the Department of Health.
Publisher Copyright:
© 2016 The Author(s).
PY - 2016/12/7
Y1 - 2016/12/7
N2 - Background: Recent studies of the quality of in-hospital care have used the Quality of Interaction Schedule (QuIS) to rate interactions observed between staff and inpatients in a variety of ward conditions. The QuIS was developed and evaluated in nursing and residential care. We set out to develop methodology for summarising information from inter-rater reliability studies of the QuIS in the acute hospital setting. Methods: Staff-inpatient interactions were rated by trained staff observing care delivered during two-hour observation periods. Anticipating the possibility of the quality of care varying depending on ward conditions, we selected wards and times of day to reflect the variety of daytime care delivered to patients. We estimated inter-rater reliability using weighted kappa, Kw, combined over observation periods to produce an overall, summary estimate, Kw. Weighting schemes putting different emphasis on the severity of misclassification between QuIS categories were compared, as were different methods of combining observation period specific estimates. Results: Estimated Kw did not vary greatly depending on the weighting scheme employed, but we found simple averaging of estimates across observation periods to produce a higher value of inter-rater reliability due to overweighting observation periods with fewest interactions. Conclusions: We recommend that researchers evaluating the inter-rater reliability of the QuIS by observing staffinpatient interactions during observation periods representing the variety of ward conditions in which care takes place, should summarise inter-rater reliability by Kw, weighted according to our scheme A4. Observation period specific estimates should be combined into an overall, single summary statistic Kw random, using a random effects approach, with Kw random, to be interpreted as the mean of the distribution of Kw across the variety of ward conditions. We draw attention to issues in the analysis and interpretation of inter-rater reliability studies incorporating distinct phases of data collection that may generalise more widely.
AB - Background: Recent studies of the quality of in-hospital care have used the Quality of Interaction Schedule (QuIS) to rate interactions observed between staff and inpatients in a variety of ward conditions. The QuIS was developed and evaluated in nursing and residential care. We set out to develop methodology for summarising information from inter-rater reliability studies of the QuIS in the acute hospital setting. Methods: Staff-inpatient interactions were rated by trained staff observing care delivered during two-hour observation periods. Anticipating the possibility of the quality of care varying depending on ward conditions, we selected wards and times of day to reflect the variety of daytime care delivered to patients. We estimated inter-rater reliability using weighted kappa, Kw, combined over observation periods to produce an overall, summary estimate, Kw. Weighting schemes putting different emphasis on the severity of misclassification between QuIS categories were compared, as were different methods of combining observation period specific estimates. Results: Estimated Kw did not vary greatly depending on the weighting scheme employed, but we found simple averaging of estimates across observation periods to produce a higher value of inter-rater reliability due to overweighting observation periods with fewest interactions. Conclusions: We recommend that researchers evaluating the inter-rater reliability of the QuIS by observing staffinpatient interactions during observation periods representing the variety of ward conditions in which care takes place, should summarise inter-rater reliability by Kw, weighted according to our scheme A4. Observation period specific estimates should be combined into an overall, single summary statistic Kw random, using a random effects approach, with Kw random, to be interpreted as the mean of the distribution of Kw across the variety of ward conditions. We draw attention to issues in the analysis and interpretation of inter-rater reliability studies incorporating distinct phases of data collection that may generalise more widely.
KW - Averaging
KW - Collapsing
KW - QuIS
KW - Random effects meta-analysis
KW - Weighted kappa
UR - http://www.scopus.com/inward/record.url?scp=85002291959&partnerID=8YFLogxK
U2 - 10.1186/s12874-016-0266-4
DO - 10.1186/s12874-016-0266-4
M3 - Article
C2 - 27927178
AN - SCOPUS:85002291959
SN - 1471-2288
VL - 16
SP - 1
EP - 12
JO - BMC Medical Research Methodology
JF - BMC Medical Research Methodology
IS - 1
M1 - 171
ER -