Sensemaking / Education / Analytics / AI
Last Friday we wrapped up the First International Symposium on Educating for Collective Intelligence, which I had been working with my co-chairs towards for half the year.
Follow the links to learn more about the rationale, the amazing cast of speakers who showed up to share their thoughts, with their papers and talks. But here are the quick links and video playlist you can browse, or just binge the entire thing for 3.5 hours 🙂
The CRADLE International Symposium ‘How could generative AI change work-integrated learning?’ sought to unpack current and timely research questions surrounding artificial intelligence and its role and impact with higher education and work. AI is starting to fundamentally change the nature of both work and learning. What about learning through work? Many questions present themselves in a climate of simultaneous opportunities, dilemmas, and hazards:
Watch the public panel to see some of key ideas to emerge from two days of rich conversation.
My own contribution was to pitch the design provocation of “the WIL-bot”, which was then refined in the fire of expertise they had gathered! “WIL-bot” — a design provocation is the co-authored summary of where we got to.
Hoping this sparks further design thinking and prototyping…
10 years ago today, the UTS Connected Intelligence Centre (CIC) launched. In fact on that day, I was in the air en route London to Sydney, not only to start the new job, but on a pressing mission to find a home and school for the rest of the family, who would land in 6 weeks time! Jenna Price and John made me very welcome in their home, and who better than two wonderful journalists to give me a crash orientation course to life in Sydney and AUS – a special time!
Shirley Alexander, UTS VP/DVC for Education & Students, had brought me over to help think through the implications of the tech revolution de jour — remember Big Data?! — and launch the university’s first transdisciplinary data science program, the Master of Data Science in Innovation (MDSI), inspired by the groundbreaking success of the transdisciplinary Bachelor of Creative Intelligence & Innovation. Given my work in Learning Analytics, the specific challenge was to invent, pilot and scale the use of data and analytics to improve the learning experience, as well as help units across UTS understand the potential of data science to support their work. The most recent account of how we’ve navigated that quest is in Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom. A decade on, we face a new revolution with the explosive arrival of generative AI, which has dominated the last 18 months.
So thank you EVERYONE who has passed through CIC’s doors, who’s walked this last decade with me, plus of course SO MANY colleagues in UTS, and Australia. Apart from thanking Shirley, whose vision underpinned this, I simply dare not start listing you all since I know I will instantly regret missing names. But I do want to extend a special thank you to Gabrielle Gardiner who was here long before I arrived, working with Shirley on the Connected Intelligence Strategy, and from 1 August 2014, was always there supporting, advising, and making stuff happen!
To everyone else — academics, MDSI and PhD students, tech coders and architects, learning designers, strategists, directors, senior leaders, and more — you know who you are! Thank You, it’s been a joy. Here’s to 1st August, 2034 🙂
And thank you everyone for all the kind words on LinkedIn!
I’ve just spent last week with PhD students at the European Joint Technology-Enhanced Learning Summer School. This was no less than the 18th such event, and it’s clear why this has established itself as an annual fixture in so many doctoral researchers’ and mentors’ calendars. They’ve got a winning mix of pechakucha intros, workshops led by both senior and PhD researchers, keynotes, speed-mentoring, great local food, scenic trips, and lots of time for all those random conversations that go unexpected places!
I was delighted to be invited to give Monday evening’s informal keynote (less a regular talk, more an opportunity to reflect on how you’re developing as a researcher). I shared some of my work-in-progress thinking, as I try to make sense of why the intersecting crises in our newsfeeds barely penetrate the ed-tech academic bubble, and whether the poly/perma/meta-crisis should shape our priorities. As the ridiculous title indicates, the challenges are almost too huge to frame coherently, but if you’re curious, here are the slides (abstract below), where I hope you’ll find at least one interesting thinker to chase down.
It’s always hard to know how such a provocation will go down, so I was delighted with the appetite to wrestle with these questions in many follow-up chats. I loved being immersed in such a cultural melting pot for a week, and given the topic, an added edge was meeting students from countries including Syria, Ukraine and Israel, who have lived/are living the daily hell the rest of us watch on screens.
Kudos to the lead team who orchestrated so effectively, everyone who created such a vibrant atmosphere, and sincere thanks for welcoming me into the special JTELSS community!
Abstract. I don’t think it’s overstating matters to say that humanity finds itself at an inflection point. The interlocking crises can feel overwhelming (ecological; political; financial; technological; medical; spiritual…). And I don’t know about you, but I’m finding it increasingly surreal attending conferences where these are not mentioned, and seem to have zero impact on our work. Or is this just ridiculous ranting? Why indeed would irreversible ecosystem collapse (for example) change how we think about TEL, pedagogy, analytics or AI? Sure, it’s really sad, but does it make sense to ask how this impacts our research? So, while it’s an exhilarating time to be working on TEL given all the AI advances, the societal challenges are daunting, and I find myself reflecting increasingly on whether this brings a responsibility to those of us who invent the future of TEL. How do we go about wrestling with this? How do we stay hopeful? I invite you to hear my thoughts-in-progress, and disagree with anything I say! We have a whole week to discuss and sort this out…
Last week I had the pleasure of being welcomed into the Team-Based Learning community, at their Asia Pacific Community Symposium. While everyone is doing “team-based learning” of some sort, I was not familiar with TBL as a set of specific learning design patterns which have been refined across multiple contexts, with a dedicated and passionate community advocating and researching it, and dedicated platforms scaffolding the process.
They invited me to share how UTS has responded to GenAI, to which (given the community) I added a few pointers to work from my other work on Collective Intelligence and Teamwork Analytics. Slides below and as PDF.
I was honoured to join a TEQSA/CRADLE panel yesterday, the 3rd in a series on the implications of ChatGPT (or GenAI more broadly) for higher education. Nearly 3000 people registered, with >1200 joining live, reflecting either the gravity of the situation now facing us — or the consequences of AI and assessment becoming mainstream media fodder! It’s both in fact.
In the 2nd panel in March, in my 8min slot I flagged the absence (at that early stage) of any evidence about whether students have the capacity to engage critically with ChatGPT. So many people were proposing to do interesting, creative things with students — but we didn’t know how it would turn out.
But 3 months on, we now have:
Educators can now articulate the range of critical engagement that their students are displaying, and I share what we’re learning at UTS from some of our leading educators who have been introducing assessments integrating ChatGPT. We now need to track how well these, and other interesting proposals, for AI-informed learning and assessment translate across diverse contexts.
I also urge us to harness the diverse brilliance of our student community in navigating this system shock, sharing what we’re learning from our Student Partnership in AI.
Here are my slides, and the full replay below (jumps to my 12min talk, but watch the whole panel!)
Here’s the symposium paper and session replay from the International Conference on Quantitative Ethnography, where a group of us reflected on the prospects for moving forward the concept of Participatory Quantitative Ethnography.
[Update! which has led to the creation of a PQE SIG]
Buckingham Shum, S., Arastoopour Irgens, G., Moots, H., Phillips, M., Shah, M., Vega, H. & Wooldridge, A. (2021). Participatory Quantitative Ethnography. In: Barbara Wasson & Szilvia Zörgő (Eds.), Third International Conference on Quantitative Ethnography: Proceedings Supplement. [Eprint]
Abstract: This symposium proposes that Participatory Quantitative Ethnography (PQE) is an important new strand of research for the QE community to develop. This paper introduces the participatory research values motivating PQE, outlines contributions from symposium speakers explaining the importance of PQE from different perspectives, before closing with a set of research questions that motivate a research agenda. It is hoped that this symposium may spark fruitful conversations and collaborations that advance PQE concepts, methodologies and tools.
2020 will be remembered for many things… but amidst the disruption, it’s been a year of consolidation for the exciting, emerging field of Quantitative Ethnography (the book by David Williamson Shaffer; my review for Jnl. Learning Analytics).
The newly launched International Society for QE has been coordinating virtual events to strengthen professional ties across the globe, upskill researchers in the new tools and techniques, and the 2nd international conference is in Feb 2021. ISQE are to be congratulated on this progress, and in particular, the Epistemic Analytics Lab at U. Wisconsin-Madison are doing an awesome job in generously sharing their expertise, and making their work available through free analytical tools.
I was honoured to be asked to help design and chair the monthly webinar series which is building a library of examples how QE methods can be applied in diverse contexts. That’s proven to be a fascinating experience, and our own work (based on Vanessa Echeverria’s PhD) wrapped this up earlier this month (more coming in 2021!).
Abstract: Collocated, face-to-face teamwork remains a pervasive mode of working and learning, which is hard to replicate online. In team-based situations, learners’ embodied, multimodal interaction with each other and with digital and material resources has been studied by researchers, but due to its complexity, has remained opaque to automated analysis. The ready availability of sensors makes it increasingly affordable to instrument work spaces to automatically capture activity traces to study teamwork and groupwork. Yet, a key challenge is the enrichment of these multiple and intertwined quantitative data streams with the qualitative insights needed to make sense of them. In this seminar, we will discuss our inroads into giving meaning to multimodal group data. We have followed a human-centred approach to design meaningful end-user interfaces that convert multimodal data into data stories. Based on Quantitative Ethnography principles, we developed a modelling technique, termed the Multimodal Matrix, to grounding quantitative data in the semantics derived from a qualitative interpretation of the context from which it arises. We will present practical examples in the context of high-fidelity clinical simulations in which multimodal data (physiological, positioning, and logged actions) have been transformed into learning analytics interfaces that support teachers’ and learners’ reflection.
Video: Transcript
Papers:
Presentation: Slides
The All Party Parliamentary Group on AI is a multi-year initiative to help anticipate the widespread impacts that AI could have on society. They convene Evidence Sessions on different themes, in which Lords and MPs have the opportunity to hear from, and question, diverse experts.As noted in the introduction to one of their recent reports,
“The evidence APPG AI has been gathering since 2017 shows education at the heart of both the opportunities and the risks in the narratives forming around AI.”
“[…] In 2019, APPG AI launched the Education Pillar to tackle some of these multi-faceted questions over the next two years. We will focus on:
- how AI can be used as a tool to improve learning,
- what skills we need to prioritise as a society,
- how school curriculums need to transform,
- and what the role of ethics in education should be.”
Reports (access requires a free signup) on the education theme to date have collated evidence on:
The latest meeting focused on Designing Fair & Robust AI-based Assessment Systems. This brought together a very interesting set of people, to which I was honoured to be invited.
The meeting switched from the House of Lords to Zoom (so a distinct loss of oak panelling and leather upholstery there!) but it meant many others could tune in live.
The guiding questions they set were:
With just 5 minutes/speaker, it was an interesting challenge to decide what and how to present. Here’s the video and the underpinning written statement which includes the sources I mention (with thanks to several colleagues for their input).
To see all the contributions, here’s the full meeting replay and final Parliamentary Brief.
Dec 10-11, 2020 • Online Conference + Local Events
We warmly welcome you to join this public conversation on how Australia can equip its citizens to engage productively with societal infrastructure powered by data, analytics and AI.
As you can see from the schedule, we have a great line-up of world leading keynotes, and plenary panels, which will be mixed with local meetings at state level to spark the conversations between stakeholders.
Audience: The conference will be of interest to individuals with all levels of AI expertise, from beginner to advanced. If your interests involve how data, analytics and AI will shape the future of learning, this open conference is for you!
One of the privileges of becoming a professor is to choose your title. Exciting but a challenge: encapsulate everything you’re passionate about in just a few words, which aren’t going to date too fast as thinking moves on.
I thought hard about this in 2014 when I was at The Open University UK. Learning Analytics was the hot new thing, but who knew how that was going to pan out? (very well as it happens!). But it seemed too early to nail all my colours to this mast.
There was a bigger picture, but what was its name? My home-base was Human-Computer Interaction, with the ACM CHI and BCS HCI conferences my stamping ground as a PhD student and early postdoc. But I’d moved into a range of other communities since, and at the OU the focus was now firmly on the role of knowledge media in shaping the future of learning. Human-Centred Computing was too broad, so how about Human-Centred Educational Technologies? Knowledge Media? Learning Technologies?
I reflected on which movements in HCI best expressed the richness of perspective that I found so exciting. And there it was staring me in the face: Informatics.
That definition comes from Kristen Nygaard‘s invited address to the 1986 World Computer Congress, entitled Program Development as a Social Activity. Informatics was a longstanding term in Europe, and was spreading in the US and elsewhere (perhaps in part as an extension of the move to creating broad, rich iSchools — someone more familiar than me with that history might comment on this).
So, I married Learning + Informatics. With the launch last year of the Learning Informatics Lab at University of Minnesota, I was delighted to be invited by Bodong Chen to give this talk (but sadly that trip was cancelled). However, we finally put that right this week, and here it is: why in my view Learning Informatics offers the depth and breadth we need to design learning analytics and AI in truly human-centred ways.
Dedicated to the extraordinary life and work of Kristen Nygaard! You will see in his reflections on the shaping of participatory design methods with trades unions and management, and definition of informatics, prescient ideas that are as vital now as then.
Learning Informatics: AI • Analytics • Accountability • Agency
Slides [pdf]
Abstract: “Health Informatics”. “Urban Informatics”. “Social Informatics”. Informatics offers systemic ways of analyzing and designing the interaction of natural and artificial information processing systems. In the context of education, I will describe some Learning Informatics lenses and practices which we have developed for co-designing analytics and AI with educators and students. We have a particular focus on closing the feedback loop to equip learners with competencies to navigate a complex, uncertain future, such as critical thinking, professional reflection and teamwork. En route, we will touch on how we build educators’ trust in novel tools, our design philosophy of “embracing imperfection” in machine intelligence, and the ways that these infrastructures embody values. Speaking from the perspective of leading an institutional innovation centre in learning analytics, I hope that our experiences spark productive reflection around as the UMN Learning Informatics Lab builds its program.
The role of automated feedback systems in creating feedback-rich environments
Last week I hosted a 2 day dialogue, Designing Automated Feedback for Impact (DAFFI 2020). The original concept was to bring the editors and authors from The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners (Eds. Henderson, Ajjawi, Boud & Molloy) to the UTS Connected Intelligence Centre, to spend 2 days in a workshop. The pandemic shifted this online, but the goals remain the same and moving online enabled us to more easily bring in additional participants.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
We call for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI.
Perhaps we can learn from each other:
Over two days, the book’s authors shared examples of this reconceptualisation of designing feedback for impact, and learning analytics researchers showed what is now possible with automated feedback. The extended dialogue was very rich, and we look forward to sharing the fruit from that as we reflect on how to take this forward.
Program and resources below…
Simon Buckingham Shum (UTS)
Dragan Gaševic (Monash)
This talk explores how learning analytics can help educators design impactful feedback processes and support learners to identify the impact of feedback information, both across time and at scale. In doing so, it offers current examples of how learning analytics could guide policy and educational designs and be usefully employed to support learners to direct their own learning and study habits. This chapter also highlights how learning analytics can help individuals understand and optimise learning, and the environments in which the learning occurs.
11.40 Break
Roberto Martinez-Maldonado, Gloria Fernandez Nieto, Jurgen Schulte, Simon Buckingham Shum (UTS)
Our work with colleagues in Health focuses on how sensors and multimodal analytics enable automated feedback to nursing teams on embodied, collocated activity. Work with Science has used movement tracking to prototype automated feedback to educators on their use of teaching spaces.
12.55 Lunch
Michael Henderson (Monash) and Rola Ajjawi (Deakin)
The challenges and opportunities of identifying, influencing and assessing feedback impact.
3.00 Break
David Boud (Deakin)
The challenge of building feedback literacy in students and staff
Carless, D. and Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback, Assessment and Evaluation in Higher Education, 43, 8, 1315-1325. DOI: 10.1080/02602938.2018.1463354
Molloy, E., Boud, D. and Henderson, M. (2020) Developing a learner-centred framework for feedback literacy, Assessment and Evaluation in Higher Education, 45, 4, 527-540. DOI: 10.1080/02602938.2019.1667955
Jaclyn Broadbent (Deakin)
This is a practice-based discussion of feedback design at scale in a context involving 1500 students. Discussion touches on improving understanding of standards, scaffolded assessment, high-quality audio feedback with feedforward aspects. This practice-based discussion will also mention the use of a tool known as Intelligent Agents which send automated feedback to students based on their digital activity as a way for staff and students to connect.
11.30 Break
Lisa Lim & Abelardo Pardo (UniSA)
An OLT consortium has designed and is now piloting a platform called OnTask which enables an educator to design personalised feedback messages for hundreds of students at a time, based on their digital activity. Evidence is now emerging regarding the student and educator experience of such tools, and how their effectiveness can be judged.
Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction. doi:10.1016/j.learninstruc.2019.04.003
Lim, L.-A., Dawson, S., Gašević, D., Joksimović, S., Pardo, A., Fudge, A., & Gentili, S. (2020). Students’ perceptions of, and emotional responses to, personalised LA-based feedback: An exploratory study of four courses. Assessment & Evaluation in Higher Education. doi:10.1080/02602938.2020.1782831
12.45 Lunch
Emerging themes, overlapping interests, next steps…
Further reading…
Core source: The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners
Significant work has focused on how we co-design automated feedback on writing with educators and students, leading to the refinement and release of an (open source) web tool called AcaWriter (orientation website for staff and students). This uses natural language processing to identify ‘rhetorical moves’ that are hallmarks of different genres of academic writing, in order to generate formative feedback.
Orientation for staff and students: https://uts.edu.au/acawriter
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á., Kitto, K., Liu, M., Mogarkar, R. & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12, (1), 141-186. (Published online 12 April 2020). DOI: https://doi.org/10.17239/jowr-2020.12.01.06
Antonette Shibani, Simon Knight and Simon Buckingham Shum (2020). Educator perspectives on learning analytics in classroom practice. The Internet and Higher Education, Volume 46. Available online 20 February 2020. https://doi.org/10.1016/j.iheduc.2020.100730
We co-designed and are now piloting a platform called OnTask which enables an educator to design personalised feedback messages or portals for hundreds of students at a time. Other institutions are embedding this or similar platforms (like ECoach and SRES), and evidence is now emerging regarding the student and educator experience of such tools, and how their effectiveness can be judged.
Introductions to OnTask and EClass: see these workshop videos
Lisa-Angelique Lim, Shane Dawson, Dragan Gašević, Srecko Joksimović, Abelardo Pardo, Anthea Fudge & Sheridan Gentili (2020) Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: an exploratory study of four courses, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2020.1782831
Hamideh Iraj, Anthea Fudge, Margaret Faulkner, Abelardo Pardo, and Vitomir Kovanović. 2020. Understanding students’ engagement with personalised feedback messages. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (LAK ’20). Association for Computing Machinery, New York, NY, USA, 438–447. DOI: https://doi.org/10.1145/3375462.3375527
Pardo, A., Bartimote, K., Buckingham Shum, S., Dawson, S., Gao, J., Gašević, D., Leichtweis, S., Liu, D., Martínez-Maldonado, R., Mirriahi, N., Moskal, A. C. M., Schulte, J., Siemens, G. and Vigentini, L. (2018). OnTask: Delivering Data-Informed, Personalized Learning Support Actions. Journal of Learning Analytics, 5(3), 235-249. doi:https://doi.org/10.18608/jla.2018.53.15
We are working with colleagues in Health on how sensors and multimodal analytics enable automated feedback on embodied, collocated activity. Work with Science has used movement tracking to prototype automated feedback to educators on Classroom Proxemics — their use of teaching spaces.
Roberto Martinez-Maldonado, Vanessa Echeverria, Gloria Fernandez Nieto, and Simon Buckingham Shum. 2020. From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–15. DOI: https://doi.org/10.1145/3313831.3376148
Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C. and Buckingham Shum, S. (2020). Teacher Tracking with Integrity: What Indoor Positioning Can Tell About Instructional Proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (UBICOMP): to appear (Accepted Jan. 2020).
Broader perspectives that could help illuminate how we design Analytics/AI-augmented “feedback rich environments”.
Buckingham Shum, S.J. and Luckin, R. (2019), Learning analytics and AI: Politics, pedagogy and practices. British Journal of Educational Technology, 50, (6), pp.2785-2793. http://dx.doi.org/10.1111/bjet.12880
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-Centred Learning Analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1
Kitto, K., Buckingham Shum, S., & Gibson, A. (2018). Embracing imperfection in learning analytics. Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Association for Computing Machinery, New York, NY, USA, pp.451–460. DOI: https://doi.org/10.1145/3170358.3170413