Category: News-Archives

UTS:CIC (Aug. 2014 – Aug. 2026)

The University of Technology Sydney’s Connected Intelligence Centre launched 1stAugust 2014, when I moved down under, after close on 19 years at The Open University’s Knowledge Media Institute. Two years ago, I marked CIC’s 10th Birthday, but today this chapter came to a close. I’ve had the privilege of 12 years leading this amazing team in a research-active innovation centre within the VP/DVC Academic’s Education Portfolio. My thanks to Shirley Alexander for conceiving CIC, and Kylie Readman who continued to recognise our work.  I now return to my substantive position as an academic, moving out from a central unit to a new academic home. But not just any home…

Those of you who’ve been in Sydney or closer to developments at UTS over the last few years will be aware of all the organisational drama, which I won’t rehearse here. Suffice to say we are reconfiguring for the future. As such processes always are, this has been costly to us all, those who have left, and those still here.

If we glance in the rear-view mirror, I think it’s not too much to  conclude that CIC made its mark within UTS, and well beyond. The organisational dynamics of getting a hybrid R&D&Service centre like CIC to work are interesting. In some of my writing I’ve tried to capture the DNA of our human-centred approach to co-designing advanced data science and generative AI tools used by many tens of thousands of students, upskilling academics who aligned their learning design and assessments to integrate the tools into a coherent student experience. A very partial snapshot:

    • Launching the Master of Data Science & Innovation, the transdisciplinary data science program that CIC coordinated from 2015-18. The Victorian Blackfriars building that was our home for many years became a very special incubator of the next generation of data scientists now embedded in so many teams across diverse sectors.
    • Instant writing feedback since 2016 using the pre-GenAI technologies of the day, and the world’s first feedback on reflective writing.
    • Customised feedback emails to each student in sometimes large cohorts of hundreds, tailored to what they had accomplished, their goals, or their learning pathway.
    • Embodied teamwork analytics augmenting our simulation wards with sensors to pick up who’s doing what and where in a nursing team exercise, with automated visualisations ready to support the immediate debrief.
    • Multiple GenAI-powered applications for educators, students and researchers, developed in close partnership with our technology, analytics and AI colleagues.
    • All of them pedagogically-grounded tools, deployed in our degree programs, with empirical evaluations. Rigorous research published in the leading journals and conferences that put UTS on the international research and practice map in Learning Analytics and AI in Education.
    • Pioneered the use of Deliberative Democracy for student/staff consultation on AI/EdTech Ethics
    • Listening not only to educators on the challenges and opportunities of GenAI, but also to the diversity of student voices.
    • PhD alumni who’ve conducted Design-Based Research in partnership with faculties to develop co-design approaches, iterate next generation feedback tools, and advance conceptual frameworks and methods.
    • And so much more — see the website and news stories.

To all the students, academics, and professional staff who’ve been part of the CIC journey — your dedication, expertise and teamwork made the magic happen — Thank You! CIC has been a significant chapter in many lives 🙂

But now the page is turning, and we embark on the next chapter. I’m delighted to announce that I and my PhD students are crossing the few steps from the UTS Tower over Alumni Green, to join the Transdisciplinary School. TD School is where you’ll find the most extraordinarily eclectic group on campus, inhabiting the edges and liminal spaces where disciplines rub shoulders (not always comfortably), working with stakeholders on their most pressing societal challenges, breaking new ground pedagogically with students, and advancing knowledge about a world that isn’t carved into disciplinary boxes. “Transdisciplinarity is… when great minds don’t think alike”.

Some of you will have registered a shift in my work over the last few years which makes this move particularly significant. While the warnings of planetary overshoot have been there for decades, for those ready to listen (Limits to Growth anyone?), it’s becoming quite clear every passing month how serious the disruption to life as we know it is going to be. This is not just a climate crisis, but a deeply entangled set of ecological, societal, political, technological, psychological (and hence unavoidably educational) systems interactions. The global polycrisis is just one of the latest names to bring deep systems thinking and change strategies to our attention. The move to TD School opens fresh conversations and trajectories, as I figure out the work I am called to do, and who to do it with. I already work closely with some TD colleagues and PhDs (indeed two CIC alumni are now faculty: Simon Knight and Antonette Shibani), and so I very much look forward to seeing what emerges next.

And if you wish, do please share a CIC memory/reflection…

ALASI 2022 – special call to schools!

I’m delighted to say we are hosting the Australian Learning Analytics Summer Institute for a day and a half 8-9 December.

Learning Analytics has been dominated to date by higher education, but this is changing as ed-tech products get more data-intensive, and now AI-enabled, and business intelligence services and trained staff become part of school life. So this year we are particularly encouraging school leadership and teachers to join us, to share progress in their contexts, and learn from peers within and beyond the school sector. This also includes those in government education departments at state and federal level.

Learning Analytics is particularly focused on how we can harness the power of data science to make sense of student data in a timely manner, to close the feedback loop — either to the student, or to help their teachers make suitably differentiated interventions in time to make a difference to the outcome. (This is in contrast to what has been termed Academic Analytics to help school leaders reflect on cohort-level outcomes with more conventional school data, but not at the speed and detail needed to help  individual learners on a weekly basis.) Some examples of LA in schools are in this “LA in schools” collection, several papers in this collection on LA for 21st century competencies, this lit review on LA for knowledge creation/inventing, and this work on researcher/teacher partnerships to design LA for a school LMS.

Critical questions arise, of course, around the responsible use of educational data — whether what is being counted and visualised on all those glossy dashboards is educationally valuable, who gets to design it, who gets to interpret it, and what actions may flow. To that, we might add concerns from some quarters about the growing role of data-intensive companies in public education (especially with the pandemic-driven rush online), and the privacy tradeoffs we are willing to negotiate for cloud services.

So we are inviting you to participate in a School Show & Tell Panel…

  • We are hoping school leadership and teachers will be interested in sharing their experiences with learning analytics, platforms and their own data practices with the community.
  • In the Show and Tell portion, leadership and teachers will have an opportunity to demonstrate a tool or approach they are using in their school and discuss their experience. Each presenter will be given a 10-15 minute time slot.
  • Presentations may take the form of a digital poster, short series of slides, or a demonstration. Presenters are also encouraged to present works in progress, questions or issues a school may be working on around the use of learning analytics.
  • All school presenters would then participate in a panel discussing their work, experience and critical insights into the use of learning analytics in schools.
  • The ALASI deadline for submissions is 20 October, but we are happy to extend proposals for this schools panel to 1 November. Please contact Sarah Howard and me if you have any questions, or suggestions of hot topics you’d love to see covered: Simon.BuckinghamShum@uts.edu.au;  sahoward@uow.edu.au 

Longer sessions: In addition, if anyone would like a longer slot for a deeper dive interactive session, there are 45min and 90min slots available. See the website for the interactive formats you might use. Deadline 20 Oct.

We want to create an attractive day for the schools community, and hope this is a pre-cursor to a larger event in 2023 bringing school and higher education learning analytics practitioners together.

Framing Professional Learning Analytics as Reframing Oneself

It’s been rewarding working with close colleagues on this, weaving our ideas together over the last year. Here’s the Open Access Preprint (final version has some minor edits). It will appear later this year in what should be a really interesting special issue on “Designing Technologies to Support Professional & Workplace Learning for Situated Practice”.

Buckingham Shum, S., Littlejohn, A., Kitto, K. & Crick, R. (2022). Framing Professional Learning Analytics as Reframing Oneself. IEEE Transactions on Learning Technologies, 15(5), pp.634-649. https://doi.org/10.1109/TLT.2022.3190055

Abstract: Central to imagining the future of technology-enhanced professional learning is the question of how data are gathered, analyzed, and fed back to stakeholders. The field of learning analytics (LA) has emerged over the last decade at the intersection of data science, learning sciences, human-centered and instructional design, and organizational change, and so could in principle inform how data can be gathered and analyzed in ways that support professional learning. However, in contrast to formal education where most research in LA has been conducted, much work-integrated learning is experiential, social, situated, and practice-bound. Supporting such learning exposes a significant weakness in LA research, and to make sense of this gap, this article proposes an adaptation of the Knowledge-Agency Window framework. It draws attention to how different forms of professional learning locate on the dimensions of learner agency and knowledge creation. Specifically, we argue that the concept of “reframing oneself” holds particular relevance for informal, work-integrated learning. To illustrate how this insight translates into LA design for professionals, three examples are provided: first, analyzing personal and team skills profiles (skills analytics); second, making sense of challenging workplace experiences (reflective writing analytics); and third, reflecting on orientation to learning (dispositional analytics). We foreground professional agency as a key requirement for such techniques to be used effectively and ethically.

Human/AI exam proctoring with integrity?

Enjoyed convening this SoLAR Panel with some very knowledgeable colleagues…

The emergence of online exam proctoring (aka remote invigilation) in higher education may be seen as a function of multiple interacting drivers, including:

  • the rise of online learning
  • emergency exam measures required by the pandemic
  • cloud computing and the increasing availability of data for training machine learning classifiers
  • university assessment regimes
  • rising concerns around student cheating
  • accountability pressures from accrediting bodies

Commercial proctoring services claiming to automate the detection of potential cheating are among the most complicated forms of AI deployed at scale in higher education, requiring various combinations of image, video and keystroke analysis, depending on the services. Moreover, due to the pandemic, they were introduced in great haste in many institutions in order to permit students to graduate, with far less time for informed deliberation than would have been expected. Consequently, there was significant controversy around this form of automation, with protests at some universities seeing withdrawal of the services, and research beginning to clarify the ethical issues, and produce new empirical evidence.

However, numerous institutions are satisfied that the services they procured met the emergency need, and are continuing with them, which would make this one of the ‘new normal’ legacies of the pandemic. Critics ask, however, whether this should become ‘business as usual’. Regardless of one’s views, the rapid introduction of such complex automation merits ongoing critical reflection.

SoLAR was delighted to host this panel, which brought together expertise from multiple quarters to explore a range of questions, arguments, and what the evidence is telling us, such as…

  • This is just exams and invigilation in new clothes, right? They’re not perfect, but universities aren’t about to drop them anytime soon, so let’s all get on with it…
  • Are there quite distinct approaches to the delivery of such services that we can now articulate, to help people understand the choices they need to make?
  • What ethical issues do we now recognise that were perhaps poorly understood 2 years ago — or simply couldn’t afford to engage with in the emergency, but which we must address now?
  • What evidence is there about the effectiveness of remote proctoring — automated, or human-powered — at reducing rates of cheating?
  • What answers are there to the question, “Should we trust the AI?” Are we now over (yet another) AI hype curve, and ready for a reality check on what “human-AI teaming” looks like for online proctoring to function sustainably and ethically?
  • What (new?) alternatives to exams are there for universities to deliver trustworthy verification of student ability, and what are the tradeoffs?
  • Who might be better or worse off as a result of the introduction of proctoring?

This panel brought rich experience on the frontline of practice, business and academia:

Phillip Dawson is a Professor and the Associate Director of the Centre for Research in Assessment and Digital Learning, Deakin University. Phill researches assessment in higher education, focusing on feedback and cheating, predominantly in digital learning contexts. His 2021 book “Defending Assessment Security in a Digital World” explores how cheating is changing and what educators can do about it.

Jarrod Morgan is an inspiring entrepreneur, award-winning business leader, keynote speaker, and chief strategist for the world’s leading online testing company. Jarrod founded ProctorU in 2008, and in 2020 led the company through its merger and evolution into Meazure Learning. In his role as chief strategy officer, he is a frequent speaker for the Online Learning Consortium (OLC), the Association of Test Publishers (ATP), Educause, and many others. He has appeared on PBS and the Today Show, and has been covered by the Wall Street Journal, The New York Times, and is a columnist with Fast Company through their Executive Board program.

Jeannie Paterson is Professor of Law and Co-Director of the Centre for AI and Digital Ethics, University of Melbourne. She teaches and researches in the fields of consumer protection law, consumer credit and banking law, and AI and the law. Jeannie’s research covers three interrelated themes: The relationship between moral norms, ethical standards and law; Protection for consumers experiencing vulnerability; Regulatory design for emerging technologies that are fair, safe, reliable and accountable. She recently co-authored “Good Proctor or “Big Brother”? Ethics of Online Exam Supervision Technologies”.

Lesley Sefcik is a Senior Lecturer and Academic Integrity Advisor at Curtin University. She provides university-wide teaching, advice, and academic research within the field of academic integrity. She is a Homeward Bound Fellow and a Senior Fellow of the Higher Education Academy. Dr. Sefcik’s professional background is situated in Assessment and Quality Learning within the domain of Learning and Teaching. Current projects include the development, implementation and management of remote invigilation for online assessment, and academic integrity related programs for students and staff at Curtin. She co-authored “An examination of student user experience (UX) and perceptions of remote invigilation during online assessment”.

(Chair) Simon Buckingham Shum is Professor of Learning Informatics and Director of the Connected Intelligence Centre, University of Technology Sydney, where his team researches, deploys and evaluates Learning Analytics/AI-enabled ed-tech tools. He has helped to develop Learning Analytics as an academic field for the last decade, and has served two terms as SoLAR Vice-President. His background in ergonomics and human-computer interaction always draws his attention to how the human and technical must be co-designed to work together to create sustainable work practices. He recently coordinated the UTS “EdTech Ethics” Deliberative Democracy Consultation in which online exam proctoring was an example examined by students and staff.

Further resources shared during the webinar:

Explainable AI in Education

A new piece (open access), with thanks to Hassan Khosravi for coordinating this…

Hassan Khosravi, Simon Buckingham Shum, Guanliang Chen, Cristina Conati, Yi-Shan Tsai, Judy Kay, Simon Knight, Roberto Martinez-Maldonado, Shazia Sadiq, Dragan Gašević (2022). Explainable Artificial Intelligence in Education, Computers and Education: Artificial Intelligence, Vol. 3, 2022, 100074, ISSN 2666-920X. DOI: https://doi.org/10.1016/j.caeai.2022.100074

There are emerging concerns about the Fairness, Accountability, Transparency, and Ethics (FATE) of educational interventions supported by the use of Artificial Intelligence (AI) algorithms. One of the emerging methods for increasing trust in AI systems is to use eXplainable AI (XAI), which promotes the use of methods that produce transparent explanations and reasons for decisions AI systems make. Considering the existing literature on XAI, this paper argues that XAI in education has commonalities with the broader use of AI but also has distinctive needs. Accordingly, we first present a framework, referred to as XAI-ED, that considers six key aspects in relation to explainability for studying, designing and developing educational AI tools. These key aspects focus on the stakeholders, benefits, approaches for presenting explanations, widely used classes of AI models, human-centred designs of the AI interfaces and potential pitfalls of providing explanations within education. We then present four comprehensive case studies that illustrate the application of XAI-ED in four different educational AI tools. The paper concludes by discussing opportunities, challenges and future research needs for the effective incorporation of XAI in education.

Is “The Matter With Things” also what’s the matter with Learning Analytics?

I enjoyed sharing some early reflections yesterday at the LAK22 Workshop on the Philosophy of Learning Analytics…

Is “The Matter With Things” also what’s the matter with Learning Analytics?

Abstract: I will share some very preliminary thoughts on the potential implications for learning analytics of “The Matter With Things” by Iain McGilchrist (2021). We’re all familiar with the classic differences between on the one hand, a science and technology approach to understanding the world, and on the other, a more poetic, artistic, literary approach. While ultimately McGilchrist seeks to transcend such dichotomies, he argues that it is no accident that these different dispositions have emerged repeatedly through history, but in fact, the brain sciences now trace them back to stark differences in the ways that the left and right hemispheres attend to and hence construct reality. In one sentence, the left hemisphere (LH) is tuned to apprehend the world, seeing “static, isolated, fragmentary elements that can be manipulated easily, are decontextualised, abstracted, detached, disembodied, mechanical […] a bureaucrat’s dream.” In sharp contrast, to the RH tuned to comprehend the world, “all is flowing and changing, provisional, and complexly interconnected.” “Here, wholes are different from the sum of the parts, and beauty and morality, along with empathy and emotional depth, help us to intuit meaning […] a bureaucrat’s nightmare.”

I suggest that we see these two dispositions to the world playing out in LA all the time, with the quantifying, modelling worldview required to develop functioning LA software, in “creative dialogue” (sometimes open conflict) with the more qualitative disciplines, such as education, sociology, learning sciences. While I am far from grasping McGilchrist’s work, I will tentatively glimpse implications for how we think about learning analytics.

For those interested in previewing some of his ideas, see:

Iain McGilchrist (2021), The Matter With Things: Our Brains, Our Delusions, and the Unmaking of the World. Perspectiva Press.

Slides [PDF]

What capabilities do learners need for an AI world?

I took part in an enjoyable and stimulating exercise led by my colleague ‘up the road’, the fabulous Lina Markauskaite, in which she orchestrated a “polylogue” among authors on how we envisioned the capabilities that learners will increasingly need in an AI-infused society. We each responded independently to a common set of prompt questions, and then began to comment on each others’ work, moving to a discussion and synthesis. This produces a different kind of article. See what you think (open access)…

L. Markauskaite, R. Marrone, O. Poquet, S. Knight, R. Martinez-Maldonado, S. Howard, J. Tondeur, M. De Laat, S. Buckingham Shum, D. Gašević, and G. Siemens (2022), Rethinking the entwinement between artificial intelligence and human learning: What capabilities do learners need for a world with AI? Computers and Education: Artificial Intelligence, Vol.3, 100056. https://doi.org/10.1016/j.caeai.2022.100056

The proliferation of AI in many aspects of human life—from personal leisure, to collaborative professional work, to global policy decisions—poses a sharp question about how to prepare people for an interconnected, fast-changing world which is increasingly becoming saturated with technological devices and agentic machines. What kinds of capabilities do people need in a world infused with AI? How can we conceptualise these capabilities? How can we help learners develop them? How can we empirically study and assess their development? With this paper, we open the discussion by adopting a dialogical knowledge-making approach. Our team of 11 co-authors participated in an orchestrated written discussion. Engaging in a semi-independent and semi-joint written polylogue, we assembled a pool of ideas of what these capabilities are and how learners could be helped to develop them. Simultaneously, we discussed conceptual and methodological ideas that would enable us to test and refine our hypothetical views. In synthesising these ideas, we propose that there is a need to move beyond AI-centred views of capabilities and consider the ecology of technology, cognition, social interaction, and values.

Deliberative Democracy for EdTech Ethics

What principles should govern UTS’ use of analytics and artificial intelligence to improve teaching and learning for all, while minimising the possibility of harmful outcomes?

This was the challenge we set a team of 20 people – students, casual tutors and full-time academics. And 5 intensive workshops later, they had delivered their response! A draft set of ethical principles to govern the use of these fast-changing technologies in UTS. How did we manage this? Below is the executive summary from the report on the EdTech Ethics website.

Executive Summary

This report has been written to document a novel community consultation process, using the principles and methods of Deliberative Democracy to consult with the UTS community on the following brief:

What principles should govern UTS use of analytics and artificial intelligence to improve teaching and learning for all, while minimising the possibility of harmful outcomes?

We’re sharing this to assist colleagues in UTS and beyond who are seeking more participatory models for community deliberation, with (in this case) specific application to the responsible use of educational technology that is powered by analytics and artificial intelligence. This is not a research paper, seeking to argue conceptual or empirical contributions to academic fields, although research is underway analysing and evaluating this process. We do hope, however, that this represents an interesting and novel ‘data point’ that others will find useful.

Deliberative Democracy (DD) is a movement in response to the crisis in confidence in how typical democratic systems engage citizens in decision making. DD works by creating a Deliberative Mini-Public (DMP). DMPs can be convened at different scales (organisation; community; region; nation) and can take many forms.

A DMP of 20 was selected through stratified sampling from UTS students, casual tutors and academics, who engaged in a series of five online workshops over seven weeks, due to Covid-19 conditions. With little to no prior knowledge among most members, they learned about the topic, worked well together, and converged on a set of principles that they felt reflected their shared values. The university experts who were involved in the workshops recognised the quality of the progress made in such a short period. UTS now has a plausibly representative expression of the community’s values, interests and concerns, in response to the brief. The principles can be viewed in Appendix 1: Draft Ethics Principles.

The raison d’etre for the initiative is to build trust within the university that these technologies are being deployed responsibly. The DMP process delivered on its promise to build engagement and trust across diverse stakeholders. The recording of the final briefing (18 mins, below) conveys the passion and commitment that the DMP invested in the process and outcome, reinforced by the preliminary themes emerging from interviews with students, educators and senior leaders.

Deliberative Democracy, even when conducted wholly online, would appear to offer educational institutions an approach to address the urgent need for meaningful student/staff consultation on the ethical implications of introducing Learning Analytics and Artificial Intelligence into teaching and learning. The implementation process is now beginning, which we will be studying with equal interest.

 

AIED2021 Human-Centred Design session

The International Conference on Artificial Intelligence in Education has been running online all week. The theme this year is Mind the Gap: AIED for Equity and Inclusion, reflecting justified concerns in society at large, about the potential for data, analytics and AI to exacerbate societal inequities. Given the call for the community to work on “diversity, equity, and inclusion practices”, I was delighted to be asked to host a discussion session on Human-Centred Design.

Here we are in gather.town (worked nicely, and note the Covid-safe seating plan!) — below are the key points and links I posted to the chat during the session, which I’ve touched up slightly to make them intelligible… 

Firstly, from my perspective, designing AIED is a specific instance of the broader challenge of designing interactive systems that people value and use. So we can and should draw on the wealth of knowledge out there on how to do this well.

Secondly, I want to flag that HCD is about far more than nice user interfaces! Depending on the scale of lens you want to use, for me, we’re talking about understanding how socio-technical infrastructures get embedded into daily life.

So Informatics provides us with the broad lens needed to integrate computational artifacts into human ecosystems. Hence I frame my work as “Learning Informatics” https://simon.buckinghamshum.net/2020/09/why-learning-informatics

Human-Computer Interaction (HCI) aka Human-Centred Informatics provide a wealth of methodologies to study how people engage with interactive tools.

  • Dan Russell’s opening keynote emphasised the vital importance of an HCI skillset for designing effective AIED, especially when that intelligence is imperfect
  • Great book: “Ways of Knowing in HCI” edited by Judy Olson & Wendy Kellogg https://www.springer.com/gp/book/9781493903771
  • A great book on how the meanings and roles of “theory” have evolved in HCI (which might spark thoughts wrt how AIED is evolving) “HCI Theory: Classical, Modern, and Contemporary” by Yvonne Rogers https://doi.org/10.2200/S00418ED1V01Y201205HCI014
    • Note in particular that when HCI started out (the first ACM CHI conference was in the early 80s) we sought theory to get a grip on the dominant computing paradigm: individual in front of a computer. Cognitive psychologists believed they brought the concepts and tools needed to design and evaluate user interfaces, but that things have come a long way since.

The current important interest in FATE of AIED (https://doi.org/10.1007/s40593-021-00239-1) boils down to the trustworthiness of LA/AIED systems. This is about a lot more than opening black boxes. A diverse set of arguments underpins the claim that a system should be considered trustworthy https://simon.buckinghamshum.net/2019/11/black-box-learning-analytics

HCI is now coming into dialogue with Learning Analytics & AIED:

  • BJET special issue “Learning Analytics and AI: Politics, Pedagogy and Practices” https://onlinelibrary.wiley.com/toc/14678535/2019/50/6
  • Jnl Learning Analytics special issue “Human-Centred Learning Analytics”  https://learning-analytics.info/index.php/JLA/issue/view/463
    • Note: in the editorial we discuss briefly whether there are any features of education that make HCD different from other domains. I mentioned this:
      “In most HCI design contexts, stakeholders are treated as authoritative sources on how their work should be performed. Current work practices are studied to ensure that the envisaged software system does not inadvertently disrupt the human ecology of formal and informal activity. In sharp contrast, for HCLA, while learners are obviously able to speak with authority about their experiences of studying, they are

      • not expert learners whose work practices should necessarily be worked around;
      • not experts in the subject matter;
      • not expert educators whose views —e.g., about the design of a course, what counts as good feedback, or what analytics will help learning — can be treated as authoritative.”

There was some good discussion about whether there is anything distinctive about AIED systems that requires the invention of special HCD techniques to aid, for example, rapid prototyping.

  • We noted the use of Wizard of Oz
  • Paper prototyping, but ensuring that workshop participants are empowered to co-construct the designs
  • The tuning of existing techniques specifically for our domain, examples below

Specific examples of human-centred design for LA/AIED in recent CIC PhDs — Antonette Shibani, Vanessa Echeverria and Carlos Prieto-Alvarez:

 

We’re translating this into HCD training/resources for researchers and educators:

Backing out to the bigger picture, as researchers, we’re transitioning education into a new kind of “knowledge infrastructure” — the system of systems that interoperate technically and socially to generate, sanction and maintain knowledge about a field  https://simon.buckinghamshum.net/2018/06/icls2018-keynote