Evaluating the performance of a video expert panel in assessing respiratory rate from recorded videos for the diagnosis of non-severe paediatric pneumonia

OBJECTIVE: To describe the process of interpreting chest movement videos for respiratory rate (RR) assessment in children aged 0-59 months with suspected non-severe pneumonia by a video expert panel (VEP) and to evaluate the panel's performance.

METHODS: Six physicians trained and standardised to RR assessment from recorded videos formed a VEP. The panel interpreted RR from recorded videos collected from health facilities in Bangladesh. The videos were distributed using a web-based system among the panel members. Each member was evaluated by assessing inter-reader agreement among the primary readers, each reader's agreement with the final panel reading and intrareader agreement using per cent agreement. Agreement was defined as the video being interpretable and the difference in RR count between two readers being ≤2 breaths per minute. The panel's performance was evaluated using Bland-Altman analysis against a panel of paediatricians who reviewed a randomly selected 10% subset of videos.

RESULTS: Among the 605 videos, the VEP classified 89.9% (n=544) of them as interpretable with RR count difference ≤2 breaths per minute, 2.0% (n=12) as interpretable with RR count difference >2 breaths per minute and 8.1% (n=49) as uninterpretable. For the video interpretability, inter-reader agreement among primary readers ranged from 88% to 95%, each reader's agreement with the final panel was 90%-95%, and intrareader agreement was 98%-100%. For the RR measurement, five out of six VEP members demonstrated satisfactory inter-reader agreement (80%-83%), agreement with final panel reading (92%-97%) and intrareader agreement (92%-100%). One outlier reader showed lower agreement levels of 55%, 61% and 75%, respectively. The mean difference in RR counts between the VEP and paediatrician panel was -0.3 breaths per minute, and limits of agreement were -3.0-2.3 breaths per minute.

CONCLUSION: The performance of VEP was satisfactory for evaluating RR counting. The study provided valuable insights into the development and evaluation of the VEP as a potential reference standard.