Improving Science Communication through Scenario-based Role-plays

We report on a two-year project to develop, validate and evaluate a complex scenario-based role-play (SBRP) exercise. Using an earthquake scenario, our aim was to assist participants (students in geology, engineering, and hazard and disaster management, as well as emergency management professionals) to improve their science communication skills.
The project was split into two phases. The first phase focused on the development of the scenario itself, and the communication tasks for the participants. This phase comprised iterations 0-3 (n=30 students, n=11 instructors) and we used a mixture of classroom observations, and student and instructor feedback to make iterative improvements to optimise the flow and look-and-feel of the SBRP. In the second phase of the study (iterations 4-6; n=44), we developed and validated two instruments to measure communication experience (CE) and perceptions of crisis communication (PCC). Combined with an existing instrument measuring communication confidence (SPCC), we were able to assess a variety of factors influencing communication performance. For the scenario, we chose an earthquake event affecting Greymouth, on the South Island of New Zealand; which is affected by local seismicity and regional seismicity from a possible Alpine Fault event. Supportive technology was also developed. This focused on providing supporting scientific and infrastructure data, as well as modern communication tools, delivered through Google Earth, file sharing, and social media.
SBRP participants are assigned to realistic roles and responsibilities (e.g., the Group Controller, GNS Seismologist, or Public Information Manager) and work within realistic teams modelled after the New Zealand Civil Defence and Emergency Management organisational protocols and structures. These are the Science Advisory Group (SAG) which provides science advice, and the Civil Defence and Emergency Management (CDEM) team which manages the crisis. Participants experience several authentic crisis communication tasks: town-hall meetings, developing an information pamphlet, media releases, radio bulletins, a press conference, a panel discussion and a debrief. Results from phase 1 showed that the final SBRP was a robust and flexible tool to meet a variety of learning goals.
Using the SPCC, we found that students from the US had statistically higher pre-scores than NZ students (78% versus 69%, unpaired t-test, t=2.39, p=0.03). This indicates that a student’s background may influence their communication confidence. Overall (on average) experience resulted in positive changes with most students achieving positive changes with a mean change of 2.6 ± 4.3. A paired t-test of pre- and post-SPCC scores resulted in statistically significant differences (t=-3.00, p=0.006) indicating that our SBRP is successful at positive changes in communication confidence. These changes were independent of pre-score, meaning that the SBRP is effective in changing confidence levels regardless of previous levels of confidence. The largest positive shifts are observed in the public speaking, meeting, and stranger (i.e., unknown member of the public) categories of the instrument, which is encouraging as these dimensions are explicitly emphasised in the SBRP.
Using the PCC, we compared participants’ perceptions of crisis communication to those of experts (i.e., academics, emergency managers, and science communicators). Participants showed statistically significant positive shifts (i.e. more agreement with expert perceptions; paired t-test comparing pre- and post-scores; t=-7.76, p<0.001, Cohen’s d=-1.00). In addition, one group achieved higher changes than others (iteration 4 participants mean change =10.7 ± 6.0; iteration 6 participants mean change = 5.7± 3.4; unpaired t-test, t=2.38, p=0.03; Cohen’s d=1.02). Several factors appeared to influence the amount of changes achieved, such as nationality, their year of degree programme, and the team (i.e., CDEM vs. SAG) the participants were in during the SBRP.
Analysis of the individual statements on the PCC (49 in total) showed that there were items in which most student groups agreed with experts (‘high perceptions’) and others which they disagreed with experts (‘low perceptions’). More importantly there were several topics which experts struggled with (i.e., resulted in predominantly ‘neutral’ responses, but with distributions leaning more towards agree or disagree) that also resulted in mixed and low perceptions from the student participants. Notably, the topics of: comprehensiveness, showing the scientist’s emotions, political influence/agenda, use of formal language, and use of graphs and plots. There were also statements which resulted in emergency management professionals disagreeing with the student groups: the ‘why’ of the crisis, discussing past crisis scenarios (i.e., context), and the communication of probabilities.
To investigate any relationships between the communication proxies, we compared the scores of the communication experience (CE), confidence (SPCC), and perceptions (PCC) to one another. Though there were no statistically significant associations between pre- or post-scores within the instruments, but we did find a positive relationship between the changes achieved in confidence and perceptions, indicating that students who experience positive shifts in confidence, also experienced positive shifts in perceptions. This means that for some participants, the SBRP was duly effective in both dimensions resulting in an overall, challenging but highly beneficial learning experience.
Lessons learned from this project include: 1) the buy-in from instructors of courses is crucial for a successful integration in the curricula; 2) the amount of flexibility in the design of the SBRP is not as large as we thought it would be; and 3) the value of having a full-time postdoctoral fellow working on the project, rather than multiple part-time employees. In the near future, we plan to collaborate with various stakeholders to bring the SBRP to professional development opportunities for practicing emergency managers in New Zealand.
Future research will include a more detailed characterisation of the phase 2 pre and post qualitative data to more clearly link the communication proxies to the experiences in the SBRP and attempt to make causal inferences of the proxies to the communication performance of participants.