Could conversational AI serve as “thinking partners” to stretch our reasoning, deepen reflection, and foster the intellectual agility needed to navigate these turbulent times?
Session Description: As overlapping global crises reshape our world, education faces a profound challenge – how do we prepare students to navigate complexity, uncertainty, and systemic collapse? In this webinar, Prof. Simon Buckingham Shum argues that conversational AI offers more than efficiency and that it can serve as a “thinking partner” to stretch our reasoning, deepen reflection, and foster the intellectual agility needed to navigate turbulent times. Drawing on his recent work, Prof. Buckingham Shum will explore how dialogical AI tools might help educators and learners engage with the deep uncertainties of the polycrisis era. Join us and don’t miss this opportunity to rethink what it means to think with AI in challenging times.”
ABSTRACT: Until recently, qualitative data analysis (QDA), such as the deductive and inductive coding of textual data, was considered the preserve of human researchers. The nuanced judgements required to apply a complex coding scheme, or to discern themes that evolve into a coding scheme, were beyond algorithms. However, the emergence and mainstream availability of large language models (LLMs: e.g., GPT, Gemini, Claude, Llama) has catalysed rigorous research into their ability to perform such QDA in minutes. This is accompanied by healthy debate on whether this could lead to the full automation of certain kinds of analysis, or the augmentation of their work through productive, hybrid analysis with a new generation of interactive QDA tools. Using LLMs hosted by privacy-respecting, secure, university instances, we have been testing LLMs for both inductive and deductive coding, and welcome your thoughts on how we address important considerations including:
How can we translate a theory-grounded codebook into a system prompt guiding the LLM?
How do we evaluate the quality of the coding compared to human researchers?
Since (like humans) LLMs are intrinsically variable in their coding, how do we understand and manage this variability?
How can an LLM provide a transparent account of its inductive coding of a corpus so humans can understand it?
How will human and machine analysts work together in the future, harnessing their respective strengths?
What concerns do researchers have about automated coding, and can these be addressed?
Publications for the details…
Bakharia, A., Shibani, A., Lim, L.-A., McCluskey, T., & Buckingham Shum, S. (2025). From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models. Proceedings of Workshop From Data to Discovery: LLMs for Qualitative Analysis in Education, LAK25: 15th International Conference on Learning Analytics & Knowledge, Dublin, IRE, pp. 1-10. https://ceur-ws.org/Vol-3995/LLMQUAL_paper1.pdf
Ramanathan, S., Lim, L.-A., Mottaghi, Nazanin R., & Buckingham Shum, S. (2025). When the Prompt Becomes the Codebook: Grounded Prompt Engineering (GROPROE) and its Application to Belonging Analytics. Proceedings LAK25: 15th International Conference on Learning Analytics & Knowledge, Dublin, IRE. https://doi.org/10.1145/3706468.3706564
Follow the links to learn more about the rationale, the amazing cast of speakers who showed up to share their thoughts, with their papers and talks. But here are the quick links and video playlist you can browse, or just binge the entire thing for 3.5 hours 🙂
Coming up for 2 years since OpenAI launched ChatGPT, it’s worth a pause for thought on the rollercoaster in education. In this update I note some signals of where we may be heading, and pose some provocations on where we should be going.
Reflections on the GenAI Rollercoaster: Glimpses into Our Future [slides]
Abstract: Generative artificial intelligence is scoring high on the university Richter scale, with aftershocks accompanying each upgrade. We’re witnessing the largest rollout of AI in educational history, powered by unprecedented tech investment, extraordinary engineering advances and ubiquitous marketing. TEQSA considers this not only a profound disruption to higher education, but also a transformational opportunity to improve assessment practice and deepen student learning, with every institution now developing its action plan in response to Assessment Reform for Age of AI. I’ll share some reflections on the rollercoaster so far, and show some examples of GenAI that I find provocative for imagining the future.
While my day-job is immersed in analytics/AI-enabled ed-tech in higher ed — the co-design of tools, practices and policy — I’m increasingly compelled to step back and survey the bigger picture: as a species, we face overwhelming, interlocking crises — and we seem to be paralysed. I’m asking whether, and if so how, this should more strongly frame and shape my work and that of the communities I’m in. I’m drawing much inspiration from an exciting neuropsychological account of how we attend to/construct the world (Iain McGilchrist’s The Matter With Things), and the increasingly urgent call for education to equip students to create a more equitable society (Henry Giroux’s work on critical pedagogy).
I was honoured to receive invitations to speak at two recent events focused in different but connected ways on the future of education, in the context of current debates about university futures in the age of AI, and the social context for platforms enabling learning at scale. These gave me opportunities to share and get feedback on how this preliminary thinking helps frame these pressing issues. Here are my Universitas 21 and ACM Learning@Scale keynotes — your feedback most welcome.
Universitas 21
Universitas 21 is an international network of research-intensive universities, committed to sharing insights. In 2014 they invited me to share my thoughts on the toddler field that was Learning Analytics, as part of their focus on personalised learning (an interesting flashback to watch that talk!). I had barely set foot in Australia, but had lots of ideas about what would be possible in my new job at UTS. So in June, it was a pleasure to reconnect, and reflect on that journey. They invited me to their Educational Innovation Symposium:
“U21’s Educational Innovation Symposium, titled ‘Scoping the Future in Higher Education: Transition or Transformation?’ brought together delegates from across the network to tackle some of the big questions currently facing university educators. The symposium, held at McMaster University, explored issues arising from swiftly advancing technologies such as Artificial Intelligence, which affects many areas of educational practice. This includes curriculum development, the way in which teaching and learning are delivered, assessment practices, digital ethics and, significantly, how students can be part of the conversation.”
Transition or transformation? In my abstract, I propose that what we have learnt on our journey at UTS running CIC provides some assurance that universities can transition into the effective, ethical use of AI, since we’ve been inventing, piloting, evaluating and scaling analytics/AI-powered ed-tech since 2015. Conversations with diverse stakeholders are at the heart of this process: Boardroom, Staff room, Server room, Classroom. The talk summarises my take on what we’re seeing in the GenAI-for-Education frenzy, examples from my own work (Bing Chat for argument analysis), and unpacks how we have been responding at UTS in the last 6 months since the GenAI rollercoaster launched, to support faculty academics and students. Human-centred design and Deliberative Democracy are important pieces of this jigsaw puzzle.
However, flipping the order in the abstract, before diving into that detail, in the talk I decided to engage with the bigger picture — the transformation question posed to the symposium. This is where the work of Giroux and McGilchrist has important contributions to make, as introduced below.
Buckingham Shum, S. (2023). Learning, Analytics, AI, Trust (and the future of universities). Keynote address, Universitas 21 Educational Innovation Symposium, (29 June, 2023, McMaster University, Hamilton, Canada). [abstract/replay/slides/reflection]
Thanks to U21 for engaging the talented Emma Richard who created this artful graphic recording (click to zoom)
Learning@Scale
Last month I presented the opening keynote to the 10th ACM Conference on Learning@Scale in Copenhagen. For those not familiar with the L@S community, the conference first emerged amidst the excitement (and data deluge) triggered by Massive Open Online Courses. As an ACM conference L@S started with a strong computational flavour, and while maintaining data science, educational data mining and AI, there is also qualitative attention to the critical human dimensions in all forms of large scale learning. The focus for this year:
“The theme of this year’s conference is the learning futures that the L@S community aims to develop and support in the coming decades. Of special interest this year are contributions that examine the design and the deployment of large-scale systems for the future of learning at scale. We are especially welcoming works targeting not only learners but also educators, educational institutions and other stakeholders involved in the design, use and evaluation of large-scale learning systems. Moreover, we welcome qualitative and mixed-methods contributions, as well as studies that are not at scale themselves but about scaled learning phenomena/environments. Finally, we welcome submissions focusing on the role of culture and cultural values in the implementation and evaluation of large-scale systems.”
Given the intersecting crises now confronting us, I took these opportunities to share some of my current thinking on a question that has increasingly troubled me: What difference, if any, should the climate crisis should make to ed-tech research, especially involving analytics/AI? This is of course just one of the interlocking dilemmas we now face, in what some have termed the “meta-crisis”, but this one comes with an hourglass running down all too fast.
Buckingham Shum, S. (2023). Trust, Sustainability and Learning@Scale. In Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S ’23). Association for Computing Machinery, New York, NY, USA, pp. 1–2. https://doi.org/10.1145/3573051.3593375. [abstract/replay/slides]
Diagnosing our collective paralysis
In the talks, I propose that a plausible diagnosis of our current paralysis — whether or not it proves terminal — is failure to learn. We are simply not learning fast enough and deeply enough. No doubt that is a partial diagnosis, but as people passionate about education and lifelong learning, we can hardly wash our hands of any responsibility when we survey the blasted landscape that is our planet, and the dysfunctional state of civic discourse in so many democracies.
I might have added failure to remember: urgently, we need to re-engage with First Nations people’s knowledge systems. This comes up in the talk later, inspired by Iain McGilchrist, and I also point briefly to the work of Angie Abdilla (Indigenous AI protocols) and Tyson Yunkaporta (Sand Talk). I need and want to go much deeper into this in future.
So, at L@S I asked — intentionally rhetorically — given this massive failure to learn@scale, how should the learning@scale community respond? And to U21, is there anything new to say about the kinds of graduates universities should be cultivating?
Dispositions: how we attend to the world
Knowledge and skills are important, and an ever-changing landscape given cognitive automation. I focus instead on dispositions — ways of attending to the world that are short in supply, and seem particularly salient in these times. I draw on two diagnoses of our collective paralysis — Iain McGilchrist’s neuropsychology work on how we attend to the world (notably his acclaimed new book, The Matter With Things), and Henry Giroux’s work on critical pedagogy, continuing the work of Paulo Freire (Giroux is at McMaster University, and we had a spirited and enjoyable hour in his office!). There is much to read and watch online, but to get a flavour of their work, try Giroux’s keynote to this year’s International Society for the Learning Sciences, and McGilchrist’s keynote to the AI World Summit.
I see McGilchrist and Giroux converging in their calls to resist dehumanising, decontextualizing, extremist ways of representing issues, people and nature. Both challenge us to use technology to help nurture citizens who can think differently, and not merely fuel the mindset that has brought us to the precipice. Both call us to engage with the world in a way that honours relationships, context and justice. Both call for defiant, educated hope as a form of resistance in dark times.
In case this slide is misunderstood, the argument is not that “right-wing politics has a neuroscience basis”. It is that extremism of any sort, of any political persuasion, is black and white thinking, erasing nuance, humility, context, empathy, dehumanising, objectifying, and seeking to manipulate. That has all the hallmarks of how the left hemisphere attends to the world so carefully documented by McGilchrist, when not under the balancing disposition of the right hemisphere’s mode of attention. The polarisation we see now in the culture wars is extremist mindsets of all flavours. But since I’m drawing on Giroux, we’re concerned in this case with right-wing extremism as it threatens educational freedom, the marketisation of universities more broadly, and hence threats to democracy when universities are not playing their role in developing graduates with critical consciousness to fight for a more just society.
Worked example: Belonging Analytics
I don’t think this translates into direct implications for all ed-tech research, but I suggest they pose important provocations for any educator to reflect on, especially those of us immersed in educational data, analytics and AI. Descending from high altitude to practices on the ground, I describe how at UTS we build trust in our automated feedback platforms by democratizing the design and governance processes. And in the L@S talk, I take as a worked example an approach that we’ve termed “Belonging Analytics”, to show how data-informed platforms can be aligned with some of the values championed by Giroux and McGilchrist.
What do you think?
I had encouraging feedback at both conferences, helpful ideas on how I might craft a stronger narrative, and some critical questioning of the arguments. There is so much more to learn, better ways to make the case — and the clock is ticking. I’m looking for intellectual soul mates, and welcome your honest feedback.
Connecting the fields of student belonging and learning analytics: “Belonging Analytics”
As with every purposeful human endeavour, motivation for learning and becoming a professional within a discipline is enhanced when individuals feel a sense of belonging. In the context of education, belonging refers to students’ subjective feeling of being a valued member of the learning community, that comes from a sense of connection with others as well as to the course of study. This affective dimension of engagement has notable links with many positive learning outcomes, including transition, retention, success and well-being. The importance of belonging has been underscored by the recent COVID-19 pandemic and subsequent increase in online, remote learning, and more students found themselves learning in isolation. The issue is further compounded for students from equity or disadvantaged groups, who already feel a lower sense of belonging.
Notwithstanding the pandemic, students’ experiences of belonging is dynamic and contextual, which presents challenges for tracking and supporting students in a timely manner. Traditional research methodologies such as surveys and interviews, may be useful sources of data for understanding student belonging, however these are difficult to scale and repeat over multiple episodes.
In response to the urgency of student belonging, CIC researchers Lisa-Angelique Lim and Simon Buckingham Shum are collaborating with belonging experts Peter Felten and Jennifer Uno (Elon University, USA), to conceptualise a scalable approach to this important issue.
In a new paper, we explore the possibility of harnessing learning analytics to monitor and support belonging in timely and personalised ways. Drawing on examples of where learning analytics has been used for personalising feedback to students, we propose a framework for “belonging analytics”, based on the dimensions of agents, data, and feedback mechanisms.
Overall, our framework suggests approaches that leverage a range of quantitative and qualitative data to monitor and support student belonging over time and at scale. Clearly, as with other substantial learning concepts, much care is needed to ensure that any approach drawing on learning data to inform belonging must be firmly grounded in theory, and that analytical approaches do not foster inequity. We conclude the paper with further questions to explore in this new field.
Watch: We recently presented these ideas in a webinar at the recent Indiana University Learning Analytics Summit.
Engage: Finally, just as belonging is inherently relational, we invite educators and researchers with a shared interest in this topic, to be part of a new Belonging Analytics community on LinkedIn. We look forward to building this community together with you.
So far much of my year has been dominated by the widespread availability of generative AI apps, especially ChatGPT given my work in writing analytics. It’s been hectic but interesting connecting across the university, working closely with Kylie Readman (VP Education & Students) and my IML colleagues, to help prepare briefings and policy.
If you’re helping your institution develop responses to this, or are wondering as a researcher in EdTech/Learning Analytics/AIED how to engage, then you may be interested in:
Simon Buckingham Shum is a Professor of Learning Informatics & Director, Connected Intelligence Centre.
Baki Kocaballi is a Senior Lecturer in the School of Computer Science. He is actively researching Conversational Interfaces and Human-AI Interaction.
Shibani Antonette is a Lecturer in the TD School, and actively researching Automated Writing Feedback and AI tools for education.
Generative AI (GenAI) is being hailed as a tipping point in AI, but let’s be clear: when it comes to educational technologies (EdTech), we have not just landed on “terra nullius”. While apps such as ChatGPT and DALL-E were never developed explicitly as EdTech, like so many other interactive tools we use every day, that doesn’t mean they have no educational value when used well. It’s too early to have peer-reviewed evidence of ChatGPT’s educational effectiveness, but prior research in related areas offers both theory, evidence and practice. So in this session, we’ll locate ChatGPT in the broader research landscape. Experts in two key fields will share their work on Automated Writing Evaluation, and Conversational Interfaces, where ChatGPT sits right at the intersection. Sharing brief glimpses of this work, we aim to spark ideas around how we can build on such foundations, to promote effective, ethical engagement with GenAI and avoid going down dead-ends that are already known from pre-GenAI research. [slides]
We just wrapped up the 2nd International Conference on Quantitative Ethnography (open access proceedings from Springer), postponed from October due to the pandemic, in the hope that we might all yet meet up in February — alas it was not to be. However, the organisers did a really great job designing the program with a lot of informal interaction time, and the delegates threw themselves into it with a fantastic spirit, with everyone out to help everyone, as the crew figures out how to sail this recently launched ship!
If QE is new to you, it springs from the foundational work at the University of Wisconsin-Madison’s Epistemic Analytics Lab, led by the inspirational David Williamson Shaffer. The team’s publications are the source point, specifically, David’s QE book, which impressed me so much and drew me into this vision of how quant+qual could come together. In fact, I first met David in 2013 after I received a very hot tip that he was doing amazing work, relevant to a discourse analytics workshop I was chairing. His keynote was a revelation to me of his team’s long term research program, but it’s taken a while to figure out if and how to bring it into my own work.
Well, the first conference (ICQE 2019 / proceedings) was a great success, and so it was a real honour to be asked to give one of the keynote talks at this year’s conference. However, following the brilliant 2019 keynotes by Jim Gee, Dragan Gašević and Gol Arastoopour Irgens, I accepted with some trepidation to be honest, since I am far from a QE expert compared to those blazing this new trail. At the time of being invited, my team had not done any work with the main QE analysis approach, Epistemic Network Analysis, though we had drawn inspiration from its data modelling methodology in our multimodal learning analytics work.
So I thought long and hard about what I might bring to the party, with several false starts, which might have gone deeper into QE and Learning Analytics, or QE and Algorithmic Accountability. I decided in the end to go back to my roots — all the way back to my PhD in fact, focusing on the cognitive affordances of semiformal graphical representations, and what I’ve learnt since about what it takes to wield such tools in participatory design with fluency, developing open source visual hypermedia software for 20 years, and more recent work on data storytelling. What was particularly fun was bringing that into dialogue with what I was seeing in the QE webinars last year, specifically, how the community is telling its stories with visualizations. It was really enjoyable thinking what my journey might have to say to the QE community, the talk seemed to go down well, and I’m looking forward to seeing if/how these ideas take deeper root.
I am indebted to so many colleagues who have shaped those ideas, noting in particular, the Knowledge Art research and practice of my PhD student, colleague and friend, Al Selvin, tragically taken from us, far too early.
Quantitative Ethnography Visualizations as Tools for Thinking [pdf slides]
Abstract: All research must give form to data and insights. Visualizations serve as cognitive extensions that assist researchers not only in exploring their data, but in communicating findings to colleagues and broader audiences. Especially in data-intensive fields, widely used software tools define, and are defined by, research communities; you can’t fully participate in a community until you can wield its tools responsibly. In an emerging field like Quantitative Ethnography (QE), inventing its own tools, how we model and map the world are therefore defining characteristics, and merit critical reflection.
QE’s principles currently find fullest expression in Epistemic Network Analysis (ENA). It’s fair to say that the interest in ENA is attributable not only to the power of its data modelling and analysis, but also to the engaging, interactive visualizations it generates. Inspired by the ways I see ENA used, in this talk I bring my background in Human-Computer Interaction and the design of tools for working with conceptual structures, as a lens on ENA and other QEgenerated visuals. When we consider in detail how external representations serve as personal and shared cognitive tools, this illuminates current and future techniques for presenting QE analyses. A data-storytelling lens asks how the audience will engage with our insights, while participatory methods ask whether we cast them as passive recipients or active agents in validating those narratives. Moreover, as QE analyses begin to underpin new tools designed for people other than QE researchers, human-centred design should give voice to non-technical stakeholders. These lenses could point to a future in which visualization tools evolve to scaffold more participatory forms of sensemaking as an important hallmark of how QE models and narrates the world.
2020 will be remembered for many things… but amidst the disruption, it’s been a year of consolidation for the exciting, emerging field of Quantitative Ethnography (the book by David Williamson Shaffer; my review for Jnl. Learning Analytics).
The newly launched International Society for QE has been coordinating virtual events to strengthen professional ties across the globe, upskill researchers in the new tools and techniques, and the 2nd international conference is in Feb 2021. ISQE are to be congratulated on this progress, and in particular, the Epistemic Analytics Lab at U. Wisconsin-Madison are doing an awesome job in generously sharing their expertise, and making their work available through free analytical tools.
I was honoured to be asked to help design and chair the monthly webinar series which is building a library of examples how QE methods can be applied in diverse contexts. That’s proven to be a fascinating experience, and our own work (based on Vanessa Echeverria’s PhD) wrapped this up earlier this month (more coming in 2021!).
Abstract: Collocated, face-to-face teamwork remains a pervasive mode of working and learning, which is hard to replicate online. In team-based situations, learners’ embodied, multimodal interaction with each other and with digital and material resources has been studied by researchers, but due to its complexity, has remained opaque to automated analysis. The ready availability of sensors makes it increasingly affordable to instrument work spaces to automatically capture activity traces to study teamwork and groupwork. Yet, a key challenge is the enrichment of these multiple and intertwined quantitative data streams with the qualitative insights needed to make sense of them. In this seminar, we will discuss our inroads into giving meaning to multimodal group data. We have followed a human-centred approach to design meaningful end-user interfaces that convert multimodal data into data stories. Based on Quantitative Ethnography principles, we developed a modelling technique, termed the Multimodal Matrix, to grounding quantitative data in the semantics derived from a qualitative interpretation of the context from which it arises. We will present practical examples in the context of high-fidelity clinical simulations in which multimodal data (physiological, positioning, and logged actions) have been transformed into learning analytics interfaces that support teachers’ and learners’ reflection.
The All Party Parliamentary Group on AI is a multi-year initiative to help anticipate the widespread impacts that AI could have on society. They convene Evidence Sessions on different themes, in which Lords and MPs have the opportunity to hear from, and question, diverse experts.
As noted in the introduction to one of their recent reports,
“The evidence APPG AI has been gathering since 2017 shows education at the heart of both the opportunities and the risks in the narratives forming around AI.”
“[…] In 2019, APPG AI launched the Education Pillar to tackle some of these multi-faceted questions over the next two years. We will focus on:
how AI can be used as a tool to improve learning,
what skills we need to prioritise as a society,
how school curriculums need to transform,
and what the role of ethics in education should be.”
Reports (access requires a free signup) on the education theme to date have collated evidence on:
The latest meeting focused on Designing Fair & Robust AI-based Assessment Systems. This brought together a very interesting set of people, to which I was honoured to be invited.
The meeting switched from the House of Lords to Zoom (so a distinct loss of oak panelling and leather upholstery there!) but it meant many others could tune in live.
The guiding questions they set were:
What are the benefits and challenges of different types of AI-based assessment systems in education?
How can it be guaranteed that they will deliver reliable and fair results?
How might AI-based assessment systems change the teacher-student relationship?
How will these technologies affect students’ motivation and trust in a fair evaluation of their performance?
How to prepare students, teachers, and parents before implementing AI-based assessment technologies in education?
How do AI and human understandings of assessment differ?
With just 5 minutes/speaker, it was an interesting challenge to decide what and how to present. Here’s the video and the underpinning written statement which includes the sources I mention (with thanks to several colleagues for their input).
To see all the contributions, here’s the full meeting replay and final Parliamentary Brief.
One of the privileges of becoming a professor is to choose your title. Exciting but a challenge: encapsulate everything you’re passionate about in just a few words, which aren’t going to date too fast as thinking moves on.
I thought hard about this in 2014 when I was at The Open University UK. Learning Analytics was the hot new thing, but who knew how that was going to pan out? (very well as it happens!). But it seemed too early to nail all my colours to this mast.
There was a bigger picture, but what was its name? My home-base was Human-Computer Interaction, with the ACM CHI and BCS HCI conferences my stamping ground as a PhD student and early postdoc. But I’d moved into a range of other communities since, and at the OU the focus was now firmly on the role of knowledge media in shaping the future of learning. Human-Centred Computing was too broad, so how about Human-Centred Educational Technologies? Knowledge Media? Learning Technologies?
I reflected on which movements in HCI best expressed the richness of perspective that I found so exciting. And there it was staring me in the face: Informatics.
That definition comes from Kristen Nygaard‘s invited address to the 1986 World Computer Congress, entitled Program Development as a Social Activity. Informatics was a longstanding term in Europe, and was spreading in the US and elsewhere (perhaps in part as an extension of the move to creating broad, rich iSchools — someone more familiar than me with that history might comment on this).
So, I married Learning + Informatics. With the launch last year of the Learning Informatics Lab at University of Minnesota, I was delighted to be invited by Bodong Chen to give this talk (but sadly that trip was cancelled). However, we finally put that right this week, and here it is: why in my view Learning Informatics offers the depth and breadth we need to design learning analytics and AI in truly human-centred ways.
Dedicated to the extraordinary life and work of Kristen Nygaard! You will see in his reflections on the shaping of participatory design methods with trades unions and management, and definition of informatics, prescient ideas that are as vital now as then.
Learning Informatics: AI • Analytics • Accountability • Agency
Abstract: “Health Informatics”. “Urban Informatics”. “Social Informatics”. Informatics offers systemic ways of analyzing and designing the interaction of natural and artificial information processing systems. In the context of education, I will describe some Learning Informatics lenses and practices which we have developed for co-designing analytics and AI with educators and students. We have a particular focus on closing the feedback loop to equip learners with competencies to navigate a complex, uncertain future, such as critical thinking, professional reflection and teamwork. En route, we will touch on how we build educators’ trust in novel tools, our design philosophy of “embracing imperfection” in machine intelligence, and the ways that these infrastructures embody values. Speaking from the perspective of leading an institutional innovation centre in learning analytics, I hope that our experiences spark productive reflection around as the UMN Learning Informatics Lab builds its program.
The role of automated feedback systems in creating feedback-rich environments
Last week I hosted a 2 day dialogue, Designing Automated Feedback for Impact (DAFFI 2020). The original concept was to bring the editors and authors from The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners (Eds. Henderson, Ajjawi, Boud & Molloy) to the UTS Connected Intelligence Centre, to spend 2 days in a workshop. The pandemic shifted this online, but the goals remain the same and moving online enabled us to more easily bring in additional participants.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
We call for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI.
Perhaps we can learn from each other:
designers of automated feedback (for students or educators) are challenged on how they could build more robustly on principles of good feedback design;
researchers and educators working on feedback design are challenged as to whether automated feedback opens up new possibilities not taken into account in prior research;
potentially, new concepts may emerge that provide important language to clarify a changing design space for “feedback-rich environments” (as the book terms them);
new opportunities for learning analytics to tackle obstacles to the uptake of better feedback design practices;
identify topics for future events, and potential next steps.
Over two days, the book’s authors shared examples of this reconceptualisation of designing feedback for impact, and learning analytics researchers showed what is now possible with automated feedback. The extended dialogue was very rich, and we look forward to sharing the fruit from that as we reflect on how to take this forward.
10.30Identifying the Impact of Feedback Over Time and at Scale: Opportunities for Learning Analytics [slides]
Dragan Gaševic (Monash)
This talk explores how learning analytics can help educators design impactful feedback processes and support learners to identify the impact of feedback information, both across time and at scale. In doing so, it offers current examples of how learning analytics could guide policy and educational designs and be usefully employed to support learners to direct their own learning and study habits. This chapter also highlights how learning analytics can help individuals understand and optimise learning, and the environments in which the learning occurs.
30mins: Progress and challenges [Key ref: Book Chapter 12]
30mins: Questions and commentary / General discussion
11.40Break
11.55Automated Feedback on Collocated Teamwork & Classroom Proxemics [slides]
Our work with colleagues in Health focuses on how sensors and multimodal analytics enable automated feedback to nursing teams on embodied, collocated activity. Work with Science has used movement tracking to prototype automated feedback to educators on their use of teaching spaces.
30mins: Questions and commentary / General discussion
12.55Lunch
2.00Role of automated feedback in generating feedback-rich environment [slides]
Michael Henderson (Monash) and Rola Ajjawi (Deakin)
The challenges and opportunities of identifying, influencing and assessing feedback impact.
30mins: Overview – Feedback Research & Practice Challenges [Key ref: Book Chapters 2, 14 and 15] and Rola Ajjawi & David Boud (2018) Examining the nature and effects of feedback dialogue, Assessment & Evaluation in Higher Education, 43:7, 1106-1119, DOI: 10.1080/02602938.2018.1434128
30mins: Questions and commentary / General discussion
3.00Break
3.15Redesigning feedback involves addressing the feedback literacy of students and staff [slides]
David Boud (Deakin)
The challenge of building feedback literacy in students and staff
30mins: Overview Key refs Book Chapter 4 and:
Carless, D. and Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback, Assessment and Evaluation in Higher Education, 43, 8, 1315-1325. DOI: 10.1080/02602938.2018.1463354
Molloy, E., Boud, D. and Henderson, M. (2020) Developing a learner-centred framework for feedback literacy, Assessment and Evaluation in Higher Education, 45, 4, 527-540. DOI: 10.1080/02602938.2019.1667955
30mins: Questions and commentary / General discussion
4.15Reflections on Day 1
4.30Close
Wed 9 Sept (all times AEST)
10.15Fresh Croissants & Reflections for those who want to join early
10.30 Assessment and feedback design at scale [replay][slides]
Jaclyn Broadbent (Deakin)
This is a practice-based discussion of feedback design at scale in a context involving 1500 students. Discussion touches on improving understanding of standards, scaffolded assessment, high-quality audio feedback with feedforward aspects. This practice-based discussion will also mention the use of a tool known as Intelligent Agents which send automated feedback to students based on their digital activity as a way for staff and students to connect.
30mins: Questions and commentary / General discussion
11.30Break
11.45Examining impact and sense-making of personalised feedback messages using OnTask [replay][slides]
Lisa Lim & Abelardo Pardo (UniSA)
An OLT consortium has designed and is now piloting a platform called OnTask which enables an educator to design personalised feedback messages for hundreds of students at a time, based on their digital activity. Evidence is now emerging regarding the student and educator experience of such tools, and how their effectiveness can be judged.
30mins: Examining impact and sense-making of personalised feedback messages using OnTask
Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction. doi:10.1016/j.learninstruc.2019.04.003
Lim, L.-A., Dawson, S., Gašević, D., Joksimović, S., Pardo, A., Fudge, A., & Gentili, S. (2020). Students’ perceptions of, and emotional responses to, personalised LA-based feedback: An exploratory study of four courses. Assessment & Evaluation in Higher Education. doi:10.1080/02602938.2020.1782831
30mins: Questions and commentary / General discussion
12.45Lunch
2.00Where have we got to?
Emerging themes, overlapping interests, next steps…
Significant work has focused on how we co-design automated feedback on writing with educators and students, leading to the refinement and release of an (open source) web tool called AcaWriter (orientation website for staff and students). This uses natural language processing to identify ‘rhetorical moves’ that are hallmarks of different genres of academic writing, in order to generate formative feedback.
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á., Kitto, K., Liu, M., Mogarkar, R. & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12, (1), 141-186. (Published online 12 April 2020). DOI: https://doi.org/10.17239/jowr-2020.12.01.06
Antonette Shibani, Simon Knight and Simon Buckingham Shum (2020). Educator perspectives on learning analytics in classroom practice. The Internet and Higher Education, Volume 46. Available online 20 February 2020. https://doi.org/10.1016/j.iheduc.2020.100730
Automated feedback on online engagement (any platform)
We co-designed and are now piloting a platform called OnTask which enables an educator to design personalised feedback messages or portals for hundreds of students at a time. Other institutions are embedding this or similar platforms (like ECoach and SRES), and evidence is now emerging regarding the student and educator experience of such tools, and how their effectiveness can be judged.
Introductions to OnTask and EClass: see these workshop videos
Lisa-Angelique Lim, Shane Dawson, Dragan Gašević, Srecko Joksimović, Abelardo Pardo, Anthea Fudge & Sheridan Gentili (2020) Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: an exploratory study of four courses, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2020.1782831
Hamideh Iraj, Anthea Fudge, Margaret Faulkner, Abelardo Pardo, and Vitomir Kovanović. 2020. Understanding students’ engagement with personalised feedback messages. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (LAK ’20). Association for Computing Machinery, New York, NY, USA, 438–447. DOI: https://doi.org/10.1145/3375462.3375527
Pardo, A., Bartimote, K., Buckingham Shum, S., Dawson, S., Gao, J., Gašević, D., Leichtweis, S., Liu, D., Martínez-Maldonado, R., Mirriahi, N., Moskal, A. C. M., Schulte, J., Siemens, G. and Vigentini, L. (2018). OnTask: Delivering Data-Informed, Personalized Learning Support Actions. Journal of Learning Analytics, 5(3), 235-249. doi:https://doi.org/10.18608/jla.2018.53.15
Roberto Martinez-Maldonado, Vanessa Echeverria, Gloria Fernandez Nieto, and Simon Buckingham Shum. 2020. From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–15. DOI: https://doi.org/10.1145/3313831.3376148
Critical, human-centred, design of learning analytics
Broader perspectives that could help illuminate how we design Analytics/AI-augmented “feedback rich environments”.
Buckingham Shum, S.J. and Luckin, R. (2019), Learning analytics and AI: Politics, pedagogy and practices. British Journal of Educational Technology, 50, (6), pp.2785-2793. http://dx.doi.org/10.1111/bjet.12880
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-Centred Learning Analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1
Kitto, K., Buckingham Shum, S., & Gibson, A. (2018). Embracing imperfection in learning analytics. Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Association for Computing Machinery, New York, NY, USA, pp.451–460. DOI: https://doi.org/10.1145/3170358.3170413
I am indebted to so many colleagues who have shaped those ideas, noting in particular, the Knowledge Art research and practice of my PhD student, colleague and friend, 
The 
