Category: Papers & Talks

Universitas21 & Learning@Scale keynotes

Summary

While my day-job is immersed in analytics/AI-enabled ed-tech in higher ed — the co-design of tools, practices and policy — I’m increasingly compelled to step back and survey the bigger picture: as a species, we face overwhelming, interlocking crises — and we seem to be paralysed. I’m asking whether, and if so how, this should more strongly frame and shape my work and that of the communities I’m in. I’m drawing much inspiration from an exciting neuropsychological account of how we attend to/construct the world (Iain McGilchrist’s The Matter With Things), and the increasingly urgent call for education to equip students to create a more equitable society (Henry Giroux’s work on critical pedagogy).

I was honoured to receive invitations to speak at two recent events focused in different but connected ways on the future of education, in the context of current debates about university futures in the age of AI, and the social context for platforms enabling learning at scale. These gave me opportunities to share and get feedback on how this preliminary thinking helps frame these pressing issues. Here are my Universitas 21 and ACM Learning@Scale keynotes — your feedback most welcome.

Universitas 21

Universitas 21 is an international network of research-intensive universities, committed to sharing insights. In 2014 they invited me to share my thoughts on the toddler field that was Learning Analytics, as part of their focus on personalised learning (an interesting flashback to watch that talk!). I had barely set foot in Australia, but had lots of ideas about what would be possible in my new job at UTS. So in June, it was a pleasure to reconnect, and reflect on that journey. They invited me to their Educational Innovation Symposium:

“U21’s Educational Innovation Symposium, titled ‘Scoping the Future in Higher Education: Transition or Transformation?’ brought together delegates from across the network to tackle some of the big questions currently facing university educators. The symposium, held at McMaster University, explored issues arising from swiftly advancing technologies such as Artificial Intelligence, which affects many areas of educational practice.  This includes curriculum development, the way in which teaching and learning are delivered, assessment practices, digital ethics and, significantly, how students can be part of the conversation.”

Transition or transformation? In my abstract, I propose that what we have learnt on our journey at UTS running CIC provides some assurance that universities can transition into the effective, ethical use of AI, since we’ve been inventing, piloting, evaluating and scaling  analytics/AI-powered ed-tech since 2015. Conversations with diverse stakeholders are at the heart of this process: Boardroom, Staff room, Server room, Classroom. The talk summarises my take on what we’re seeing in the GenAI-for-Education frenzy, examples from my own work (Bing Chat for argument analysis), and unpacks how we have been responding at UTS in the last 6 months since the GenAI rollercoaster launched, to support faculty academics and students. Human-centred design and Deliberative Democracy are important pieces of this jigsaw puzzle.

However, flipping the order in the abstract, before diving into that detail, in the talk I decided to engage with the bigger picture — the transformation question posed to the symposium. This is where the work of Giroux and McGilchrist has important contributions to make, as introduced below.

Buckingham Shum, S. (2023). Learning, Analytics, AI, Trust (and the future of universities). Keynote address, Universitas 21 Educational Innovation Symposium, (29 June, 2023, McMaster University, Hamilton, Canada). [abstract/replay/slides/reflection]

Thanks to U21 for engaging the talented Emma Richard who created this artful graphic recording (click to zoom)

Learning@Scale

Last month I presented the opening keynote to the 10th ACM Conference on Learning@Scale in Copenhagen. For those not familiar with the L@S community, the conference first emerged amidst the excitement (and data deluge) triggered by Massive Open Online Courses. As an ACM conference L@S started with a strong computational flavour, and while maintaining data science, educational data mining and AI, there is also qualitative attention to the critical human dimensions in all forms of large scale learning. The focus for this year:

“The theme of this year’s conference is the learning futures that the L@S community aims to develop and support in the coming decades. Of special interest this year are contributions that examine the design and the deployment of large-scale systems for the future of learning at scale. We are especially welcoming works targeting not only learners but also educators, educational institutions and other stakeholders involved in the design, use and evaluation of large-scale learning systems. Moreover, we welcome qualitative and mixed-methods contributions, as well as studies that are not at scale themselves but about scaled learning phenomena/environments. Finally, we welcome submissions focusing on the role of culture and cultural values in the implementation and evaluation of large-scale systems.”

Given the intersecting crises now confronting us, I took these opportunities to share some of my current thinking on a question that has increasingly troubled me: What difference, if any, should the climate crisis should make to ed-tech research, especially involving analytics/AI? This is of course just one of the interlocking dilemmas we now face, in what some have termed the “meta-crisis”, but this one comes with an hourglass running down all too fast.

Buckingham Shum, S. (2023). Trust, Sustainability and Learning@Scale. In Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S ’23). Association for Computing Machinery, New York, NY, USA, pp. 1–2. https://doi.org/10.1145/3573051.3593375. [abstract/replay/slides]

Diagnosing our collective paralysis

In the talks, I propose that a plausible diagnosis of our current paralysis — whether or not it proves terminal — is failure to learn. We are simply not learning fast enough and deeply enough. No doubt that is a partial diagnosis, but as people passionate about education and lifelong learning, we can hardly wash our hands of any responsibility when we survey the blasted landscape that is our planet, and the dysfunctional state of civic discourse in so many democracies.

I might have added failure to remember: urgently, we need to re-engage with First Nations people’s knowledge systems. This comes up in the talk later, inspired by Iain McGilchrist, and I also point briefly to the work of Angie Abdilla (Indigenous AI protocols) and Tyson Yunkaporta (Sand Talk). I need and want to go much deeper into this in future.

So, at L@S I asked — intentionally rhetorically — given this massive failure to learn@scale, how should the learning@scale community respond? And to U21, is there anything new to say about the kinds of graduates universities should be cultivating?

Dispositions: how we attend to the world

Knowledge and skills are important, and an ever-changing landscape given cognitive automation. I focus instead on dispositions — ways of attending to the world that are short in supply, and seem particularly salient in these times. I draw on two diagnoses of our collective paralysis — Iain McGilchrist’s neuropsychology work on how we attend to the world (notably his acclaimed new book, The Matter With Things), and Henry Giroux’s work on critical pedagogy, continuing the work of Paulo Freire (Giroux is at McMaster University, and we had a spirited and enjoyable hour in his office!). There is much to read and watch online, but to get a flavour of their work, try Giroux’s keynote to this year’s International Society for the Learning Sciences, and McGilchrist’s keynote to the AI World Summit.

I see McGilchrist and Giroux converging in their calls to resist dehumanising, decontextualizing, extremist ways of representing issues, people and nature. Both challenge us to use technology to help nurture citizens who can think differently, and not merely fuel the mindset that has brought us to the precipice. Both call us to engage with the world in a way that honours relationships, context and justice. Both call for defiant, educated hope as a form of resistance in dark times.

In case this slide is misunderstood, the argument is not that “right-wing politics has a neuroscience basis”. It is that extremism of any sort, of any political persuasion, is black and white thinking, erasing nuance, humility, context, empathy, dehumanising, objectifying, and seeking to manipulate. That has all the hallmarks of how the left hemisphere attends to the world so carefully documented by McGilchrist, when not under the balancing disposition of the right hemisphere’s mode of attention. The polarisation we see now in the culture wars is extremist mindsets of all flavours. But since I’m drawing on Giroux, we’re concerned in this case with right-wing extremism as it threatens educational freedom, the marketisation of universities more broadly, and hence threats to democracy when universities are not playing their role in developing graduates with critical consciousness to fight for a more just society.

Worked example: Belonging Analytics

I don’t think this translates into direct implications for all ed-tech research, but I suggest they pose important provocations for any educator to reflect on, especially those of us immersed in educational data, analytics and AI. Descending from high altitude to practices on the ground, I describe how at UTS we build trust in our automated feedback platforms by democratizing the design and governance processes. And in the L@S talk, I take as a worked example an approach that we’ve termed “Belonging Analytics”, to show how data-informed platforms can be aligned with some of the values championed by Giroux and McGilchrist.

What do you think?

I had encouraging feedback at both conferences, helpful ideas on how I might craft a stronger narrative, and some critical questioning of the arguments. There is so much more to learn, better ways to make the case — and the clock is ticking. I’m looking for intellectual soul mates, and welcome your honest feedback.

“Belonging Analytics”?

Connecting the fields of student belonging and learning analytics: “Belonging Analytics”

As with every purposeful human endeavour, motivation for learning and becoming a professional within a discipline is enhanced when individuals feel a sense of belonging. In the context of education, belonging refers to students’ subjective feeling of being a valued member of the learning community, that comes from a sense of connection with others as well as to the course of study. This affective dimension of engagement has notable links with many positive learning outcomes, including transition, retention, success and well-being. The importance of belonging has been underscored by the recent COVID-19 pandemic and subsequent increase in online, remote learning, and more students found themselves learning in isolation. The issue is further compounded for students from equity or disadvantaged groups, who already feel a lower sense of belonging.

Notwithstanding the pandemic, students’ experiences of belonging is dynamic and contextual, which presents challenges for tracking and supporting students in a timely manner.  Traditional research methodologies such as surveys and interviews, may be useful sources of data for understanding student belonging, however these are difficult to scale and repeat over multiple episodes.

In response to the urgency of student belonging, CIC researchers Lisa-Angelique Lim and Simon Buckingham Shum are collaborating with belonging experts Peter Felten and Jennifer Uno (Elon University, USA), to conceptualise a scalable approach to this important issue.

In a new paper, we explore the possibility of harnessing learning analytics to monitor and support belonging in timely and personalised ways. Drawing on examples of where learning analytics has been used for personalising feedback to students, we propose a framework for “belonging analytics”, based on the dimensions of agents, data, and feedback mechanisms.

Overall, our framework suggests approaches that leverage a range of quantitative and qualitative data to monitor and support student belonging over time and at scale. Clearly, as with other substantial learning concepts, much care is needed to ensure that any approach drawing on learning data to inform belonging must be firmly grounded in theory, and that analytical approaches do not foster inequity. We conclude the paper with further questions to explore in this new field.

Learn more…

Read: Needed Now: Belonging@Scale blog and a longer read: Lim, L.-A., Buckingham Shum, S., Felten, P. and Uno, J. (2023). “Belonging Analytics”: A Proposal. Learning Letters, Vol. 1, Article 4, 1-12.

Watch: We recently presented these ideas in a webinar at the recent Indiana University Learning Analytics Summit.

Engage: Finally, just as belonging is inherently relational, we invite educators and researchers with a shared interest in this topic, to be part of a new Belonging Analytics community on LinkedIn. We look forward to building this community together with you.

Understanding skilled use of open automated feedback tools as teacher feedback literacy

Summary: a new paper forges a bridge between data-driven, open automated feedback platforms, and teacher feedback literacy competences: 

Buckingham Shum, S., Lim, L.-A., Boud, D., Bearman, M. & Dawson, P. (2023). A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy. International Journal of Educational Technology in Higher Education, 20:40 (12 July 2023). https://doi.org/10.1186/s41239-023-00410-9 

The mass availability of generative AI continues to reshape thinking about the future of work and learning. Conversational apps can now give instant feedback to learners about their work — but the educational question is how effective this interaction is. A new design space has opened up for tuning generative AI to give high quality feedback to learners about their work. We are not in uncharted waters here: there is a growing body of knowledge on what “effective feedback” means in higher education, and how to create the conditions for this. It goes far beyond comments accompanying an assignment, with a shift towards “feedback rich ecosystems” in which both teachers and students exercise far greater agency and sensemaking competencies.

In 2019, an exciting book came out: The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners (Eds. Henderson, Ajjawi, Boud & Molloy):

“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”

In 2020, I conceived a symposium to bring the editors and authors to UTS to spend 2 days in dialogue with CIC and other researchers developing automated-feedback tools using Learning Analytics/AI. We called for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI. The pandemic shifted this online, but the goals remained the same, and moving online enabled us to more easily bring in additional participants, resulting in DAFFI 2020: Designing Automated Feedback for Impact whose presentations I commend to you.

I’m now delighted to share one of the fruit from this, a collaboration between CIC (Lisa Lim and myself) and our colleagues at Deakin University’s Centre for Research in Assessment and Digital Learning (CRADLE). The focus of the paper is not on generative, conversational AI (which did not exist when we started this work), but on technically less complicated, but correspondingly far more transparent platforms that use simple rules authored by teachers themselves.

“In contrast to closed AF tools, we define “open” AF tools as enabling the educator to specify some or all of the following key parameters in the tool’s behaviour:

  1. the student activity data that the system analyses;

  2. the algorithms that analyse that data;

  3. the feedback information the teacher wishes the software to compile for students;

  4. the modalities via which feedback information is communicated by teachers;

  5. the student-driven feedback processes that are afforded.”

What does it mean to do this skillfully? We demonstrate that Boud & Dawson’s  teacher feedback literacy competency framework can be applied very usefully to analysing teaching practices with data-driven, automated feedback platforms. A next step will be to think through what this means for tuning large language models for educational contexts.

A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy

Simon Buckingham Shuma, Lisa-Angelique Lima, David Bouda,b,c, Margaret Bearmanb, Phillip Dawsonb

a University of Technology Sydney, AUS
b Deakin University, AUS
c Middlesex University, UK

Effective learning depends on effective feedback, which in turn requires a set of skills, dispositions and practices on the part of both students and teachers which have been termed feedback literacy. A previously published teacher feedback literacy competency framework has identified what is needed by teachers to implement feedback well. While this framework refers in broad terms to the potential uses of educational technologies, it does not examine in detail the new possibilities of automated feedback (AF) tools, especially those that are open by offering varying degrees of transparency and control to teachers. Using analytics and artificial intelligence, open AF tools permit automated processing and feedback with a speed, precision and scale that exceeds that of humans. This raises important questions about how human and machine feedback can be combined optimally and what is now required of teachers to use such tools skillfully. The paper addresses two research questions: Which teacher feedback competencies are necessary for the skilled use of open AF tools? and What does the skilled use of open AF tools add to our conceptions of teacher feedback competencies? We conduct an analysis of published evidence concerning teachers’ use of open AF tools through the lens of teacher feedback literacy, which produces summary matrices revealing relative strengths and weaknesses in the literature, and the relevance of the feedback literacy framework.  We conclude firstly, that when used effectively, open AF tools exercise a range of teacher feedback competencies. The paper thus offers a detailed account of the nature of teachers’ feedback literacy practices within this context. Secondly, this analysis reveals gaps in the literature, signalling opportunities for future work. Thirdly, we propose several examples of automated feedback literacy, that is, distinctive teacher competencies linked to the skilled use of open AF tools.

Your comments most welcome…

ChatGPT: What have we learnt, what do we need to learn next?

ChatGPT: What have we learnt?
What do we need to learn next?

I was honoured to join a TEQSA/CRADLE panel yesterday, the 3rd in a series on the implications of ChatGPT (or GenAI more broadly) for higher education. Nearly 3000 people registered, with >1200 joining live, reflecting either the gravity of the situation now facing us — or the consequences of AI and assessment becoming mainstream media fodder! It’s both in fact.

In the 2nd panel in March, in my 8min slot I flagged the absence (at that early stage) of any evidence about whether students have the capacity to engage critically with ChatGPT. So many people were proposing to do interesting, creative things with students — but we didn’t know how it would turn out.

But 3 months on, we now have:

  • myriad demos of GPT’s capabilities given the right prompts
  • a few systematic evaluations of that capability
  • myriad proposals for how this can enable engaging student learning
  • and a small but growing stream of educators’ stories from the field
  • with peer reviewed research about to hit the streets.

Educators can now articulate the range of critical engagement that their students are displaying, and I share what we’re learning at UTS from some of our leading educators who have been introducing assessments integrating ChatGPT. We now need to track how well these, and other interesting proposals, for AI-informed learning and assessment translate across diverse contexts.

I also urge us to harness the diverse brilliance of our student community in navigating this system shock, sharing what we’re learning from our Student Partnership in AI.

Here are my slides, and the full replay below (jumps to my 12min talk, but watch the whole panel!)

Framing Generative AI as EdTech

So far much of my year has been dominated by the widespread availability of generative AI apps, especially ChatGPT given my work in writing analytics. It’s been hectic but interesting connecting across the university, working closely with Kylie Readman (VP Education & Students) and my IML colleagues, to help prepare briefings and policy.

If you’re helping your institution develop responses to this, or are wondering as a researcher in EdTech/Learning Analytics/AIED how to engage, then you may be interested in:

Framing Generative AI as EdTech

1 hr UTS webinar, 23 February 2023

Simon Buckingham Shum is a Professor of Learning Informatics & Director, Connected Intelligence Centre.

Baki Kocaballi is a Senior Lecturer in the School of Computer Science. He is actively researching Conversational Interfaces and Human-AI Interaction.

Shibani Antonette is a Lecturer in the TD School, and actively researching Automated Writing Feedback and AI tools for education.

Generative AI (GenAI) is being hailed as a tipping point in AI, but let’s be clear: when it comes to educational technologies (EdTech), we have not just landed on “terra nullius”. While apps such as ChatGPT and DALL-E were never developed explicitly as EdTech, like so many other interactive tools we use every day, that doesn’t mean they have no educational value when used well. It’s too early to have peer-reviewed evidence of ChatGPT’s educational effectiveness, but prior research in related areas offers both theory, evidence and practice. So in this session, we’ll locate ChatGPT in the broader research landscape. Experts in two key fields will share their work on Automated Writing Evaluation, and Conversational Interfaces, where ChatGPT sits right at the intersection. Sharing brief glimpses of this work, we aim to spark ideas around how we can build on such foundations, to promote effective, ethical engagement with GenAI and avoid going down dead-ends that are already known from pre-GenAI research. [slides]

HuCETA: Human-Centered Embodied Teamwork Analytics

After about 7 years working on multimodal teamwork analytics (specifically in nursing simulations) with Roberto Martinez-Maldonado, and then PhDs with Vanessa Echeverria & Gloria Fernandez — and more recently in an ARC-funded project with Dragan Gasevic, Lixiang (Jimmie) Yan, Linxuan Zhao — we are moving towards theoretically grounded, open source infrastructure for analysing collocated teamwork. We’ve distilled the essence of all that we’ve learnt into a new conceptual framework called HuCETA:

Echeverria, V., Martinez-Maldonado, R., Yan, L., Zhao, L., Fernandez-Nieto, G., Gasevic, D., & Buckingham Shum, S. (2022). HuCETA: A Framework for Human-Centered Embodied Teamwork Analytics. IEEE Pervasive Computing, 1-11. https://doi.org/10.1109/MPRV.2022.3217454 [Open Access Eprint]

Abstract: Collocated teamwork remains a pervasive practice across all professional sectors. Even though live observations and video analysis have been utilized for understanding embodied interaction of team members, these approaches are impractical for scaling up the provision of feedback that can promote developing high-performance teamwork skills. Enriching spaces with sensors capable of automatically capturing team activity data can improve learning and reflection. Yet, connecting the enormous amounts of data such sensors can generate with constructs related to teamwork remains challenging. This article presents a framework to support the development of human-centered embodied teamwork analytics by 1) enabling hybrid human–machine multimodal sensing; 2) embedding educators’ and experts’ knowledge into computational team models; and 3) generating human-driven data storytelling interfaces for reflection and decision making. This is illustrated through an in-the-wild study in the context of healthcare simulation, where predictive modeling, epistemic network analysis, and data storytelling are used to support educators and nursing teams.

What could Learning Analytics learn from HCI theory?

It’s good to share this chapter that’s been brewing for about a year now, with helpful feedback from quite a few colleagues en route, gratefully acknowledged. I go back to my roots and share some viewpoints from HCI on my current field of Learning Analytics. This preprint will appear (subject to minor production edits) in the forthcoming book:

Buckingham Shum, S. (In Press). What could Learning Analytics learn from Human-Computer Interaction theory? In: Kathryn Bartimote, Sarah Howard & Dragan Gašević (Eds.), Theory Informing and Arising from Learning Analytics. Springer Nature

Abstract: The design of Learning Analytics (LA) tools is an example of the general problem of designing interactive tools, which is the focus of Human-Computer Interaction (HCI) research and design practice. LA as a field must understand how to embed LA into organisations and the design of effective, trustworthy human-computer systems is where HCI theory and practice have much to offer. Consequently, this chapter argues that LA can learn from (i) the way that theory has evolved in HCI, (ii) the field’s methods for evaluating interactive systems at different scales, and (iii) HCI debates how established scientific theories and methods relate to design theories and methods. As a highly interdisciplinary applied field, LA (like HCI) faces the challenge of maintaining academic standards in the conduct and review of research from many disciplinary traditions. I propose that HCI offers inspiration for researchers seeking rigorous methods to design and evaluate LA in authentic contexts, including principles to maintain their intellectual rigour, which will also be of interest to LA journals and conferences seeking to maintain peer review standards.

[Update 11 Nov 2024] The book is due out soon, and includes in conversation chapters and podcasts:

“Theory Informing and Arising from Learning Analytics delves into the dynamic intersection of learning theory and educational data analysis within the field of Learning Analytics (LA). This groundbreaking book illuminates how theoretical insights can revolutionize data interpretation, reshape research methodologies, and expand the horizons of human learning and educational theory. Organized into three distinct sections, it offers a comprehensive introduction to the role of theory in LA, features contributions from leading scholars who apply diverse theoretical frameworks to their research, and explores cutting-edge topics where new theories are emerging. A standout feature is the inclusion of three “in conversation” chapters, where expert panels dive into the topics of ethics, self-regulated learning, and qualitative computation, enriched by accompanying podcasts that provide fresh, thought-provoking perspectives. This book is an invaluable resource for researchers, sparking debates on the evolving role of theory in LA and challenging conventional epistemological views. Published by Springer, it is an essential read for both aspiring and seasoned scholars eager to engage with the forefront of LA research.”

Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom

Here’s the open access preprint of a chapter to appear in a forthcoming book edited by Olga Viberg and Åke Grönlund. I was grateful to be invited to contribute to this, and it was an enjoyable reflective journey figuring out how to tell the story. I hope you enjoy exploring the four rooms!

Buckingham Shum, S. (2022). Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom. In Viberg, O. and  Grönlund, Å. (Eds.), Practicable Learning Analytics, SpringerNature.

Abstract: In this chapter, I describe and reflect on the last 8 years at an Australian public university, inventing, piloting and evaluating Learning Analytics tools, specifically focused on data-driven personalised feedback, leading in some cases to integration with the institution’s learning technology ecosystem, and accompanied by staff training and support. I will summarise this as conversations in the Boardroom, the Staff Room, the Server Room and the Classroom, reflecting the different levels of influence, partnership and adaptation required to introduce and sustain novel technologies in the complex system that constitutes a university, or indeed, any educational institution. This chapter is pragmatic, documenting aspects of our work that are typically not the focus in research papers, intending to make a practice contribution.

Keywords: Organisational Strategy, Innovation Diffusion, Personalised Feedback

Dec’22 update: I was invited by the Leiden-Delft-Erasmus Universities Centre for Education and Learning to share this work with them at their annual conference so here’s the replay [slides], together with the fab live drawing  by Mark van Huystee!

Live drawing by Mark van Huystee

ICQE21 Participatory Quantitative Ethnography Symposium

Here’s the symposium paper and session replay from the International Conference on Quantitative Ethnography, where a group of us reflected on the prospects for moving forward the concept of Participatory Quantitative Ethnography.

[Update! which has led to the creation of a PQE SIG]

Buckingham Shum, S., Arastoopour Irgens, G., Moots, H., Phillips, M., Shah, M., Vega, H. & Wooldridge, A. (2021). Participatory Quantitative Ethnography. In: Barbara Wasson & Szilvia Zörgő (Eds.),  Third International Conference on Quantitative Ethnography: Proceedings Supplement. [Eprint]

Abstract: This symposium proposes that Participatory Quantitative Ethnography (PQE) is an important new strand of research for the QE community to develop. This paper introduces the participatory research values motivating PQE, outlines contributions from symposium speakers explaining the importance of PQE from different perspectives, before closing with a set of research questions that motivate a research agenda. It is hoped that this symposium may spark fruitful conversations and collaborations that advance PQE concepts, methodologies and tools. 

Ethics of AI in Education: Towards a Community-Wide Framework

With the growing activity in both academia and mainstream journalism on Fairness, Accountability, Transparency, and Ethics (FATE) as they pertain to machine reasoning in society, it’s a good signal that a Special Issue of the International Journal of Artificial Intelligence in Education is coming out shortly, entitled: The FATE of AI in Education: Fairness, Accountability, Transparency, and Ethics.

While one would not expect education — as one of society’s most important public functions — to be exempt from wider AI-FATE concerns in society, we need to understand in detail how the problems and solutions play out specifically with particular stakeholders, organisations, laws, policies/politics, practices and platforms in education.

Moreover, is there nothing distinctive about teaching and learning, that sets it apart from (say) health, policing and shopping as a phenomenon? One would hope so (and here’s a tutorial exploring this in more depth).

I’ll update this blog when the issue is officially published, but meantime, here’re the results of interviewing the community about the challenges…

Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Buckingham Shum, S., Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I. and Koedinger, K. R. (2021), Ethics of AI in Education: Towards a Community-Wide Framework. International Journal of Artificial Intelligence in Education (Published online 2021/04/09) https://doi.org/10.1007/s40593-021-00239-1 (Open Access)

Abstract: While Artificial Intelligence in Education (AIED) research has at its core the desire to support student learning, experience from other AI domains suggest that such ethical intentions are not by themselves sufficient. There is also the need to consider explicitly issues such as fairness, accountability, transparency, bias, autonomy, agency, and inclusion. At a more general level, there is also a need to differentiate between doing ethical things and doing things ethically, to understand and to make pedagogical choices that are ethical, and to account for the ever-present possibility of unintended consequences. However, addressing these and related questions is far from trivial. As a first step towards addressing this critical gap, we invited 60 of the AIED community’s leading researchers to respond to a survey of questions about ethics and the application of AI in educational contexts. In this paper, we first introduce issues around the ethics of AI in education. Next, we summarise the contributions of the 17 respondents, and discuss the complex issues that they raised. Specific outcomes include the recognition that most AIED researchers are not trained to tackle the emerging ethical questions. A well-designed framework for engaging with ethics of AIED that combined a multidisciplinary approach and a set of robust guidelines seems vital in this context.

ICQE20 keynote: QE Visualizations as tools for thinking

We just wrapped up the 2nd International Conference on Quantitative Ethnography (open access proceedings from Springer), postponed from October due to the pandemic, in the hope that we might all yet meet up in February — alas it was not to be. However, the organisers did a really great job designing the program with a lot of informal interaction time, and the delegates threw themselves into it with a fantastic spirit, with everyone out to help everyone, as the crew figures out how to sail this recently launched ship!

If QE is new to you, it springs from the foundational work at the University of Wisconsin-Madison’s Epistemic Analytics Lab, led by the inspirational David Williamson Shaffer. The team’s publications are the source point, specifically, David’s QE book, which impressed me so much and drew me into this vision of how quant+qual could come together. In fact, I first met David in 2013 after I received a very hot tip that he was doing amazing work, relevant to a discourse analytics workshop I was chairing. His keynote was a revelation to me of his team’s long term research program, but it’s taken a while to figure out if and how to bring it into my own work.

Well, the first conference (ICQE 2019 / proceedings) was a great success, and so it was a real honour to be asked to give one of the keynote talks at this year’s conference. However, following the brilliant 2019 keynotes by Jim Gee, Dragan Gašević and Gol Arastoopour Irgens, I accepted with some trepidation to be honest, since I am far from a QE expert compared to those blazing this new trail. At the time of being invited, my team had not done any work with the main QE analysis approach, Epistemic Network Analysis, though we had drawn inspiration from its data modelling methodology in our multimodal learning analytics work.

So I thought long and hard about what I might bring to the party, with several false starts, which might have gone deeper into QE and Learning Analytics, or QE and Algorithmic Accountability. I decided in the end to go back to my roots — all the way back to my PhD in fact, focusing on the cognitive affordances of semiformal graphical representations, and what I’ve learnt since about what it takes to wield such tools in participatory design with fluency, developing open source visual hypermedia software for 20 years, and more recent work on data storytelling. What was particularly fun was bringing that into dialogue with what I was seeing in the QE webinars last year, specifically, how the community is telling its stories with visualizations. It was really enjoyable thinking what my journey might have to say to the QE community, the talk seemed to go down well, and I’m looking forward to seeing if/how these ideas take deeper root.

I am indebted to so many colleagues who have shaped those ideas, noting in particular, the Knowledge Art research and practice of my PhD student, colleague and friend, Al Selvin, tragically taken from us, far too early.

Quantitative Ethnography Visualizations as Tools for Thinking [pdf slides]

Abstract: All research must give form to data and insights. Visualizations serve as cognitive extensions that assist researchers not only in exploring their data, but in communicating findings to colleagues and broader audiences. Especially in data-intensive fields, widely used software tools define, and are defined by, research communities; you can’t fully participate in a community until you can wield its tools responsibly. In an emerging field like Quantitative Ethnography (QE), inventing its own tools, how we model and map the world are therefore defining characteristics, and merit critical reflection.

QE’s principles currently find fullest expression in Epistemic Network Analysis (ENA). It’s fair to say that the interest in ENA is attributable not only to the power of its data modelling and analysis, but also to the engaging, interactive visualizations it generates. Inspired by the ways I see ENA used, in this talk I bring my background in Human-Computer Interaction and the design of tools for working with conceptual structures, as a lens on ENA and other QEgenerated visuals. When we consider in detail how external representations serve as personal and shared cognitive tools, this illuminates current and future techniques for presenting QE analyses. A data-storytelling lens asks how the audience will engage with our insights, while participatory methods ask whether we cast them as passive recipients or active agents in validating those narratives. Moreover, as QE analyses begin to underpin new tools designed for people other than QE researchers, human-centred design should give voice to non-technical stakeholders. These lenses could point to a future in which visualization tools evolve to scaffold more participatory forms of sensemaking as an important hallmark of how QE models and narrates the world.

Collaborative Learning Analytics (CSCL Handbook chapter)

Delighted to share this preprint of a chapter that I’ve been working on with Alyssa Wise and Simon Knight. It’s due out in the exciting new CSCL Handbook this year!

Wise, A., Knight, S., Buckingham Shum, S. (In Press) Collaborative Learning Analytics. In: Cress, U., Rosé C., Wise, A. & Oshima, J. (Eds.), International Handbook of Computer-Supported Collaborative Learning (Springer). Preprint: PDF

Abstract

The use of data from computer-based learning environments has been a longstanding feature of CSCL. Learning analytics can enrich this established work in CSCL. This chapter outlines synergies and tensions between the two fields. Drawing on examples, we discuss established work to use learning analytics as a research tool (analytics of collaborative learning – ACL). Beyond this potential though, we discuss the use of analytics as a mediational tool in CSCL – Collaborative Learning Analytics (CLA). This shift raises important challenges regarding the role of the computer – and analytics –in supporting and developing human agency and learning. LA offers a new tool for CSCL research. CSCL offers important contemporary perspectives on learning for a knowledge society, and as such is an important site of action for learning analytics research that both builds our understanding of collaborative learning, and support that learning.