Coming up next week at LAK21, the latest work from the PhD of Gloria Fernandez-Nieto…
Gloria Milena Fernandez-Nieto, Roberto Martinez-Maldonado, Kirsty Kitto, and Simon Buckingham Shum. 2021. Modelling Spatial Behaviours in Clinical Team Simulations using Epistemic Network Analysis: Methodology and Teacher Evaluation. In LAK21: 11th International Learning Analytics and Knowledge Conference (LAK21), April 12–16, 2021, Irvine, CA, USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3448139.3448176 [open access eprint]
Abstract: In nursing education through team simulations, students must learn to position themselves correctly in coordination with colleagues. However, with multiple student teams in action, it is difficult for teachers to give detailed, timely feedback on these spatial behaviours to each team. Indoor-positioning technologies can now capture student spatial behaviours, but relatively little work has focused on giving meaning to student activity traces, transforming low-level x/y coordinates into language that makes sense to teachers. Even less research has investigated if teachers can make sense of that feedback. This paper therefore makes two contributions. (1) Methodologically, we document the use of Epistemic Network Analysis (ENA) as an approach to model and visualise students’ movements. To our knowledge, this is the first application of ENA to analyse human movement. (2) We evaluated teachers’ responses to ENA diagrams through qualitative analysis of video-recorded sessions. Teachers constructed consistent narratives about ENA diagrams’ meaning, and valued the new insights ENA offered. However, ENA’s abstract visualisation of spatial behaviours was not intuitive, and caused some confusions. We propose, therefore, that the power of ENA modelling can be combined with other spatial representations such as a classroom map, by overlaying annotations to create a more intuitive user experience.
2020 will be remembered for many things… but amidst the disruption, it’s been a year of consolidation for the exciting, emerging field of Quantitative Ethnography (the book by David Williamson Shaffer; my review for Jnl. Learning Analytics).
The newly launched International Society for QE has been coordinating virtual events to strengthen professional ties across the globe, upskill researchers in the new tools and techniques, and the 2nd international conference is in Feb 2021. ISQE are to be congratulated on this progress, and in particular, the Epistemic Analytics Lab at U. Wisconsin-Madison are doing an awesome job in generously sharing their expertise, and making their work available through free analytical tools.
I was honoured to be asked to help design and chair the monthly webinar series which is building a library of examples how QE methods can be applied in diverse contexts. That’s proven to be a fascinating experience, and our own work (based on Vanessa Echeverria’s PhD) wrapped this up earlier this month (more coming in 2021!).
Abstract: Collocated, face-to-face teamwork remains a pervasive mode of working and learning, which is hard to replicate online. In team-based situations, learners’ embodied, multimodal interaction with each other and with digital and material resources has been studied by researchers, but due to its complexity, has remained opaque to automated analysis. The ready availability of sensors makes it increasingly affordable to instrument work spaces to automatically capture activity traces to study teamwork and groupwork. Yet, a key challenge is the enrichment of these multiple and intertwined quantitative data streams with the qualitative insights needed to make sense of them. In this seminar, we will discuss our inroads into giving meaning to multimodal group data. We have followed a human-centred approach to design meaningful end-user interfaces that convert multimodal data into data stories. Based on Quantitative Ethnography principles, we developed a modelling technique, termed the Multimodal Matrix, to grounding quantitative data in the semantics derived from a qualitative interpretation of the context from which it arises. We will present practical examples in the context of high-fidelity clinical simulations in which multimodal data (physiological, positioning, and logged actions) have been transformed into learning analytics interfaces that support teachers’ and learners’ reflection.
This project is a collaboration between UTS:CIC and the University of Melbourne SWARM Project. The successful candidate will be based in Sydney, also spending time in Melbourne.
Visit the CIC PhD Scholarships page for full details. Please email us to express interest, ask any questions, and if we can see a potential fit we’ll advise you on writing your proposal.
The Challenge
The real world challenge: improving collaborative EBR
The challenges facing society are so complex that multiple expertises are needed. Consider security, science, law, health, policy-making, finance. The teamis the ubiquitous organisational unit, but the quality of its reasoning, especially under pressure, can vary dramatically. Problems are not provided in neat, well-defined packages: a team must frame problems in creative ways that lead to insights, and resolve uncertainties around possible responses, making the best possible use of evidence, plus their own judgement. Studies of how teams engage in such “sensemaking” highlight the blinkers that can blindside teams, and how the ways that the problem is expressed and visually represented can help or hinder (Weick, 1995). We will term this whole process Evidence-Based Reasoning (EBR). (We note of course that politics and social dynamics are unavoidable whenever people come together, and effective team members learn how to navigate these dynamics effectively.)
Improving collaborative EBR is an interesting scientific and design challenge. A successful support system (i.e. ways of working + enabling tools) must respect the principles of good reasoning, as determined by fields such as logic, argumentation and epistemology, and the domain-specific knowledge (i.e. emergency response, engineering, social work, counter-intelligence, etc.). At the same time, it must accommodate the strengths, weaknesses and vagaries of human reasoners, which is the terrain of cognitive and social psychologists. If part of the support system is interactive software, then it must have a good user interface and a solid underlying architecture. Assessing the resulting performance is a difficult evaluation problem. Building such systems is therefore inherently multidisciplinary.
How do we better equip teams for collaborative EBR? From an educational perspective, teamwork, problem solving and critical thinking skills are now among the most in demand ‘transferable competencies’ (Fiore et al 2018). The challenge of assessing and equipping graduates in these is at the heart of the learning and teaching strategies at UTS and U. Melbourne.
The technology support challenge:
While in some fields, there are specialist tools for modelling and simulation that assist analysts by managing constraints in the problem, but even with machine intelligence, the agency typically rests with the human analysts to decide how much weight to give to the machine’s output. Most other fields, however, do not have such tools: collaborative EBR is typically supported by general-purpose information technologies such as word processors, spreadsheets, databases, and project planners to help with managing information and producing reports. Similarly, generic communication tools dominate, such as email, chat, video conferencing, phone. In most cases, the reasoning itself is typically left wholly to the human reasoners themselves.
There have been remarkably few attempts to provide direct technological support for the processes of inference and judgement that are at the heart of collaborative EBR, and moreover, those attempts have had little impact on the way it is actually conducted in most places (van Gelder, 2012). There are methods and software tools for facilitating group processes and visualising team reasoning, but these require quite an advanced facilitation and software skillset (e.g. Culmsee and Awati, 2013; Okada, et al., 2008; Selvin et al, 2012).
Our interest is in developing computer-support to improve the collaborative EBR of geographically and often temporally distributed teams, that does not require specialist skills to start using beyond using what are now familiar collaboration tools. SWARM is an online platform emerging from an ongoing research project to improve the kind of collaborative EBR undertaken by intelligence analysts making sense of complex sets of qualitative and quantitative information or varying reliability. However, these are the conditions under which most other domains operate, and we hypothesise that it has broader potential, and specifically in this project, for education and training. SWARM is based on three design principles: cultivating user engagement, exploiting natural expertise, and supporting rich collaboration (van Gelder et al, 2018). Central to its approach is the upskilling of team members to equip them with different EBR skills (see in particular the Lens Kit).
Figure: The SWARM workspace
Recent large scale empirical evaluations, in which teams of analysts tackled complex challenges with or without SWARM, indicated that the quality of the reports produced by SWARM teams was significantly better than reports produced by analysts using normal methods (van Gelder et al, In Prep). In a follow-up project, “super-teams” on the platform produced reasoning so good it would plausibly be called “super-reasoning” (van Gelder & de Rozario, 2017) analogous to “super-forecasting” (Tetlock & Gardner, 2015).
This CIC seminar is a great introduction to the work so far:
Learning Analytics for SWARM
The encouraging evidence of SWARM’s effectiveness makes it an attractive candidate platform for use in educational/training contexts. While evaluation of final reports (i.e. the team’s product) is a conventional measure of team performance, and certainly one that educators will be interested in, this is not the only possible indicator of improvement. The emergence of data science, activity-based analytics and visualisation opens new possibilities for tracking the process that teams are following. Learning Analytics connects such techniques to what is known about the teaching and learning of teamwork, and could make the assessment of team performance more rigorous, and more cost effective.
This PhD is therefore focusing on inventing and validating new forms of automated team analytics for collaborative EBR, to provide insights into both process and product. Such analytics might enable not only coaches and researchers to gain insights into a team’s effectiveness, but the teams themselves to monitor their work in real time, or critically review their project on completion. Further, real-time analytics can be used to shape the collaborative environment itself, resulting in better collaboration and better outputs. Some prototype analytics have already been developed to summarise participants’ contributions and interactions. This PhD will build on this work, synthesise the literature, plus insights from the SWARM team and educators, in order to define, design, implement and evaluate automated analytics in different contexts, spanning education and training, research, and potentially more authentic deployments with professional teams.
Figure: Early version of the SWARM group dynamics dashboard. Upper diagrams shows levels of interaction among team members working on a particular problem.
Relevant analytics techniques include, but are not limited to:
Text analysis to identify significant contributions to the team communications and the report they are producing
Social network analysis to identify significant interaction patterns among team members
Process mining to identify significant sequences in the actions that individuals engage in, within or between sessions
Statistical techniques to identify significant differences between teams
Candidates
In addition to the broad skills and dispositions that we are seeking in all candidates (see CIC’s PhD homepage), you should have:
A Masters degree, Honours distinction or equivalent with at least above-average grades in computer science, mathematics, statistics, or equivalent
Analytical, creative and innovative approach to solving problems
Strong interest in designing and conducting quantitative, qualitative or mixed-method studies
Strong programming skills in at least one relevant language (e.g. R, Python)
Experience with web log analysis, statistics and/or data science tools.
It is advantageous if you can evidence:
Design and Implementation of user-centred software, especially data/information visualisations
Skill in working with non-technical clients to involve them in the design and testing of software tools
Knowledge and experience of natural language processing/text analytics
Familiarity with the scholarship in a relevant areas (e.g. high performance teams; collective intelligence; collaborative problem solving)
We will discuss your ideas with you to help sharpen up your proposal, which will be competing with others for a scholarship. Please follow the application procedure for the submission of your proposal.
Fiore, S. M., Graesser, A., & Greiff, S. (2018). Collaborative problem-solving education for the twenty-first-century workforce. Nature Human Behaviour, 2(6), 367–369.
van Gelder, T., & de Rozario, R. (2017). Pursuing Fundamental Advances in Human Reasoning. In T. Everitt, B. Goertzel, & A. Potapov (Eds.), Artificial General Intelligence(Vol. 10414, pp. 259–262). Cham: Springer International Publishing.
As a PhD student from 1988 at the University of York HCI group and Rank Xerox Cambridge EuroPARC (as it was called then), I found my intellectual community and cut my teeth at the British HCI conference, and ACM CHI. I then spun off into various other orbits, seeing HCI as my bedrock but enjoying smaller, more focused conferences (e.g. Hypertext, CSCW, Semantic Web, OER and then ed-tech). However, my current desire to see Learning Analytics become more human-centred in its design processes, and working with Roberto Martinez-Maldonado, has looped me back into the HCI community again, and I’m thoroughly enjoying reconnecting with old and new faces!
So, here’s our latest work, building on our CHI19 paper, which is for me a very satisfying convergence of multimodal analytics, collocated teamwork, visual analytics, pedagogy and my longstanding interest in narrative. It incorporates the doctoral work of Vanessa Echeverria (who has just submitted her thesis and is now at CMU HCII) and Gloria Fernandez-Nieto (who just passed her first year with flying colours).
The teaching and learning challenge is to give instant feedback to nursing students on how well they performed as a team in treating a patient in a simulation. The research question is how to make streams of multimodal data intelligible. Enjoy!
Martinez-Maldonado, R., Echeverria, V., Fernandez-Nieto, G. & Buckingham Shum, S. (2020). From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. Proc. ACM CHI 2020: Human Factors in Computing Systems (April 25–30, 2020, Honolulu, HI, USA), Paper 21, pp.1-15. https://doi.org/10.1145/3313831.3376148 [Open Access Eprint]
Abstract: Significant progress to integrate and analyse multimodal data has been carried out in the last years. Yet, little research has tackled the challenge of visualising and supporting the sensemaking of multimodal data to inform teaching and learning. It is naïve to expect that simply by rendering multiple data streams visually, a teacher or learner will be able to make sense of them. This paper introduces an approach to unravel the complexity of multimodal data by organising it into meaningful layers that explain critical insights to teachers and students. The approach is illustrated through the design of two data storytelling prototypes in the context of nursing simulation. Two authentic studies with educators and students identified the potential of the approach to create learning analytics interfaces that communicate insights on team performance, as well as concerns in terms of accountability and automated insights discovery.
Sadly I missed the inaugural International Conference on Quantitative Ethnography for family reasons, but by all accounts it was a resounding success, with the 2020 conference already set for LA (Oct. 25-27).
The only silver lining of not making it in person was I wanted to make a screencast of our presentation, so here you go. We consider how QE principles have helped inform our efforts to model multimodal data streams from collocated nursing teamwork, in order to generate meaningful feedback.
Buckingham Shum, S., Echeverria, V. & Martinez-Maldonado, R. (2019). The Multimodal Matrix as a Quantitative Ethnography Methodology. In: Eagan B., Misfeldt, M. & Siebert-Evenstone, A. (Eds.), Advances in Quantitative Ethnography. Communications in Computer and Information Science, Vol. 1112. Springer: Cham, pp.26-40. DOI: https://doi.org/10.1007/978-3-030-33232-7_3. [Paper][Slides]
Abstract: This paper seeks to contribute to the emerging field of Quantitative Ethnography (QE) by demonstrating its utility to solve a complex challenge in Learning Analytics: the provision of timely feedback to collocated teams and their coaches. We define two requirements that extend the QE concept in order to operationalise such a design process, namely, the use of co-design methodologies, and the availability of automated analytics workflow to close the feedback loop. We introduce the Multimodal Matrix as a data modelling approach that can integrate theoretical concepts about teamwork with contextual insights about specific work practices, enabling the analyst to map between higher order codes and low-level sensor data, with the option add the results of manually performed analyses. This is implemented in software as a workflow for rapid data modelling, analysis and interactive visualisation, demonstrated in the context of nursing teamwork simulations. We propose that this exemplifies how a QE methodology can underpin collocated activity analytics, at scale, with in-principle applications to embodied, collocated activities beyond our case study.
Keywords: multimodal, learning analytics, teamwork, CSCL, sense making
For several years in CIC, we’ve been prototyping multimodal learning analytics in partnership with our colleagues in the UTS Faculty of Health, with the ambition to generate instant feedback for debriefing after simulation exercises with mannikin patients. I’m looking forward to presenting this new work at CHI in May (in Glasgow, no less, where I grew up!). Congratulations to Vanessa on her great PhD work, and to Roberto for leading the multimodal learning analytics research program.
Echeverria, V., Martinez-Maldonado, R. and Buckingham Shum, S. (2019). Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. In Proceedings of ACM CHI Conference (CHI’19). ACM, New York, NY, USA, Paper 39, 16 pages. https://doi.org/10.1145/3290605.3300269 [PDF Reprint]
ABSTRACT: Collocated, face-to-face teamwork remains a pervasive mode of working, which is hard to replicate online. Team members’ embodied, multimodal interaction with each other and artefacts has been studied by researchers, but due to its complexity, has remained opaque to automated analysis. However, the ready availability of sensors makes it increasingly affordable to instrument work spaces to study teamwork and groupwork. The possibility of visualising key aspects of a collaboration has huge potential for both academic and professional learning, but a frontline challenge is the enrichment of quantitative data streams with the qualitative insights needed to make sense of them. In response, we introduce the concept of collaboration translucence, an approach to make visible selected features of group activity. This is grounded both theoretically (in the physical, epistemic, social and affective dimensions of group activity), and contextually (using domain-specific concepts). We illustrate the approach from the automated analysis of healthcare simulations to train nurses, generating four visual proxies that fuse multimodal data into higher order patterns.
Here’s an extended version of the CHI talk, presented at U. Sydney CHAI research group [PDF slides]:
