Category: CIC News

News syndicated to the UTS Connected Intelligence Centre

Award! CIC and faculties co-design personalised feedback with OnTask to build student belonging

UTS awards CIC’s multi-year, multidisciplinary faculty collaboration, using the OnTask tool to scale personalised feedback and build student belonging.

Since 2017, CIC has been supporting academics to embed OnTask messaging in learning design, offering workshops and consultations. When Lisa-Angelique Lim joined CIC in 2021, she brought deep expertise from her PhD at UniSA, and took this work to a new level, leading a learning community to implement OnTask across a wider range of subjects, thereby enabling personalised feedback and support for students in multiple faculties. 

So it’s a true mark of recognition to see that CIC, in partnership with UTS academics from multiple faculties, received an award in recognition for our sustained efforts in the Student Experience category, for fostering students’ belonging in multiple disciplines through personalised feedback. This accolade was part of the Vice-Chancellor’s 2024 Learning and Teaching Awards and Citations, and was celebrated at the 2025 Learning and Teaching Awards Ceremony at UTS on 4 April 2025.

Our large, multidisciplinary team was led by Dr Lisa-Angelique Lim (CIC), with Associate Professor Amanda White (Business), Dr Amara Atif (FEIT), Chris Croese (Law), Associate Professor James Wakefield (Business), Dr Keith Heggart (FASS), Associate Professor Nicole Sutton (Business), Ram Ramanathan (CIC PhD student), Dr Rina Dhillon (Business), Dr Simone Faulkner (Business), and Professor Simon Buckingham Shum (CIC).

Addressing the challenges of student belonging with personalised feedback using OnTask

Belonging is a cornerstone of the student experience, critically impacting engagement, retention, and overall success. At UTS, the Student Experience Framework places a strong emphasis on belonging. However, the challenge of nurturing a sense of belonging at the classroom level is significant, given the diverse and large student population, particularly in first-year core subjects. Personalised feedback has emerged as a potent tool to address this challenge, serving as a form of ‘relational pedagogy’ that builds self-efficacy and fosters greater engagement and thriving at university.

To tackle the challenge of belonging, our team leveraged OnTask — a tool designed to support students through personalised communication and feedback across multiple disciplines, based on their data. OnTask uses learning analytics to provide tailored messages that help students stay on track with their studies, understand their progress, and feel supported throughout their academic journey. Check out this short animation on how OnTask works.

Quick teaser video of our academics’ perspectives of their implementation of personalised feedback in their context:

 

What this looks like in practice

Here, we highlight a few examples of the work by our team.

  • The UTS Business First and Further Year Experience (FFYE) team, led by A/Prof James Wakefield and Dr Simone Faulkner, used OnTask to personalize orientation communications for both undergraduate and postgraduate commencing students, leading to a significant increase in orientation registrations.
  • A/Prof Amanda White used OnTask in the Accounting for Business Decision A (ABDA) subject to tailor emails based on students’ progress, encouraging higher engagement and leading to improved exam grades.
  • Dr Keith Heggart created personalised video messages tailored to students’ confidence levels in the fully online Graduate Certificate in Learning Design, significantly enhancing student engagement and perceptions of support.
  • The positive impacts of OnTask are well-documented. For instance, in the large first-year subject, 22208 Accounting, Business and Society (ABS), led by Dr Rina Dhillon and A/Prof Nicole Sutton, personalised messages based on weekly quiz results led to improved pass rates and enhanced feelings of being valued among at-risk students.
  • Similarly, personalised feedback messages sent by Chris Croese to his students in large Law subjects kept students on track and motivated, correlating with better final grades. More stories of how academics at UTS have used OnTask, with research papers, can be found on CIC’s OnTask page.

Towards future partnerships to enhance student belonging

With this award, we celebrate the collaborative efforts and innovative approaches taken by our team to foster a sense of belonging among students through personalised feedback. Our journey over these past three years demonstrates the sustainability of this practice and its potential to influence and enhance teaching and learning widely.

We are honoured to receive this recognition, and look forward to continuing strong partnerships, to make a positive impact on the student experience at UTS.

CIC is 10!

10 years ago today, the UTS Connected Intelligence Centre (CIC) launched. In fact on that day, I was in the air en route London to Sydney, not only to start the new job, but on a pressing mission to find a home and school for the rest of the family, who would land in 6 weeks time! Jenna Price and John made me very welcome in their home, and who better than two wonderful journalists to give me a crash orientation course to life in Sydney and AUS – a special time!

Shirley Alexander, UTS VP/DVC for Education & Students, had brought me over to help think through the implications of the tech revolution de jour — remember Big Data?! — and launch the university’s first transdisciplinary data science program, the Master of Data Science in Innovation (MDSI), inspired by the groundbreaking success of the transdisciplinary Bachelor of Creative Intelligence & Innovation. Given my work in Learning Analytics, the specific challenge was to invent, pilot and scale the use of data and analytics to improve the learning experience, as well as help units across UTS understand the potential of data science to support their work. The most recent account of how we’ve navigated that quest is in Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom. A decade on, we face a new revolution with the explosive arrival of generative AI, which has dominated the last 18 months.

So thank you EVERYONE who has passed through CIC’s doors, who’s walked this last decade with me, plus of course SO MANY colleagues in UTS, and Australia. Apart from thanking Shirley, whose vision underpinned this, I simply dare not start listing you all since I know I will instantly regret missing names. But I do want to extend a special thank you to Gabrielle Gardiner who was here long before I arrived, working with Shirley on the Connected Intelligence Strategy, and from 1 August 2014, was always there supporting, advising, and making stuff happen!

To everyone else — academics, MDSI and PhD students, tech coders and architects, learning designers, strategists, directors, senior leaders, and more — you know who you are! Thank You, it’s been a joy. Here’s to 1st August, 2034 🙂

And thank you everyone for all the kind words on LinkedIn!

CHI2020: Layered Storytelling for Multimodal Learning Analytics

As a PhD student from 1988 at the University of York HCI group and Rank Xerox Cambridge EuroPARC (as it was called then), I found my intellectual community and cut my teeth at the British HCI conference, and ACM CHI. I then spun off into various other orbits, seeing HCI as my bedrock but enjoying smaller, more focused conferences (e.g. Hypertext, CSCW, Semantic Web, OER and then ed-tech). However, my current desire to see Learning Analytics become more human-centred in its design processes, and working with Roberto Martinez-Maldonado, has looped me back into the HCI community again, and I’m thoroughly enjoying reconnecting with old and new faces!

So, here’s our latest work, building on our CHI19 paper, which is for me a very satisfying convergence of multimodal analytics, collocated teamwork, visual analytics, pedagogy and my longstanding interest in narrative. It incorporates the doctoral work of Vanessa Echeverria (who has just submitted her thesis and is now at CMU HCII) and Gloria Fernandez-Nieto (who just passed her first year with flying colours).

The teaching and learning challenge is to give instant feedback to nursing students on how well they performed as a team in treating a patient in a simulation. The research question is how to make streams of multimodal data intelligible. Enjoy!

Martinez-Maldonado, R., Echeverria, V., Fernandez-Nieto, G. & Buckingham Shum, S. (2020). From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. Proc. ACM CHI 2020: Human Factors in Computing Systems (April 25–30, 2020, Honolulu, HI, USA), Paper 21, pp.1-15. https://doi.org/10.1145/3313831.3376148 [Open Access Eprint]

Abstract: Significant progress to integrate and analyse multimodal data has been carried out in the last years. Yet, little research has tackled the challenge of visualising and supporting the sensemaking of multimodal data to inform teaching and learning. It is naïve to expect that simply by rendering multiple data streams visually, a teacher or learner will be able to make sense of them. This paper introduces an approach to unravel the complexity of multimodal data by organising it into meaningful layers that explain critical insights to teachers and students. The approach is illustrated through the design of two data storytelling prototypes in the context of nursing simulation. Two authentic studies with educators and students identified the potential of the approach to create learning analytics interfaces that communicate insights on team performance, as well as concerns in terms of accountability and automated insights discovery.

SoLAR webinar: Learning Analytics as Educational Knowledge Infrastructure

With thanks to SoLAR for inviting me to kick off this new series, this webinar was a deeper dive into the concept of Learning Analytics as Educational Knowledge Infrastructure, building on the thoughts I first shared at ICLS 2018. I very much welcome follow-up discussion in the webinar’s Google Group thread.

Here’s the replay and the slides [PDF].

Evaluating ML for Pharmacy Student Reflection

UTS will be represented at the 20th International Conference on Artificial Intelligence in Education by CIC research fellow in writing analytics, Ming Liu, who leads a new paper from our ongoing collaboration with Cherie Lucas (UTS School of Pharmacy), now joined by her pharmacy colleague Efi Mantzourani (Cardiff University).

Building on our previous work in the Academic Writing Analytics project, which uses a rule-based implementation of Ágnes Sándor’s concept matching framework, this is our first paper to investigate the potential of machine learning approaches to the detection of reflective statements in student writing about their work placements.

Liu, M., Buckingham Shum, S., Mantzourani, E. and Lucas, C. (2019). Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. Proceedings AIED2019: 20th International Conference on Artificial Intelligence in Education, June 25th – 29th 2019, Chicago, USA. Lecture Notes in Computer Science & Artificial Intelligence: Springer. 

Abstract. Reflective writing is widely acknowledged to be one of the most effective learning activities for promoting students’ self-reflection and critical thinking. However, manually assessing and giving feedback on reflective writing is time consuming, and known to be challenging for educators. There is little work investigating the potential of automated analysis of reflective writing, and even less on machine learning approaches which offer potential advantages over rule-based approaches. This study reports progress in developing a machine learning approach for the binary classification of pharmacy students’ reflective statements about their work placements. Four common statistical classifiers were trained on a corpus of 301 statements, using emotional, cognitive and linguistic features from the Linguistic Inquiry and Word Count (LIWC) analysis, in combination with affective and rhetorical features from the Academic Writing Analytics (AWA) platform. The results showed that the Random-forest algorithm performed well (F-score=0.799) and that AWA features, such as emotional and reflective rhetorical moves, improved performance.

CHI’19: Towards Collaboration Translucence

For several years in CIC, we’ve been prototyping multimodal learning analytics in partnership with our colleagues in the UTS Faculty of Health, with the ambition to generate instant feedback for debriefing after simulation exercises with mannikin patients. I’m looking forward to presenting this new work at CHI in May (in Glasgow, no less, where I grew up!). Congratulations to Vanessa on her great PhD work, and to Roberto for leading the multimodal learning analytics research program.

Echeverria, V., Martinez-Maldonado, R. and Buckingham Shum, S. (2019). Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data. In Proceedings of ACM CHI Conference (CHI’19). ACM, New York, NY, USA, Paper 39, 16 pages. https://doi.org/10.1145/3290605.3300269 [PDF Reprint]

ABSTRACT: Collocated, face-to-face teamwork remains a pervasive mode of working, which is hard to replicate online. Team members’ embodied, multimodal interaction with each other and artefacts has been studied by researchers, but due to its complexity, has remained opaque to automated analysis. However, the ready availability of sensors makes it increasingly affordable to instrument work spaces to study teamwork and groupwork. The possibility of visualising key aspects of a collaboration has huge potential for both academic and professional learning, but a frontline challenge is the enrichment of quantitative data streams with the qualitative insights needed to make sense of them. In response, we introduce the concept of collaboration translucence, an approach to make visible selected features of group activity. This is grounded both theoretically (in the physical, epistemic, social and affective dimensions of group activity), and contextually (using domain-specific concepts). We illustrate the approach from the automated analysis of healthcare simulations to train nurses, generating four visual proxies that fuse multimodal data into higher order patterns.

Here’s an extended version of the CHI talk, presented at U. Sydney CHAI research group [PDF slides]:

 

Open source release of Academic Writing Analytics

When I arrived at UTS, I spent the first 6 months or so assessing the state of play with respect to student writing. It became clear through consultations across faculties that student writing was a strategically important area for UTS teaching and learning (and indeed, for most other educational institutions). Academic writing is hard to learn, hard to teach, academics don’t necessarily want to play that role, and there is never enough capacity to give detailed feedback on submissions (never mind drafts). The possibility of providing instant, personalised, actionable feedback to students about their drafts, 24/7, was a compelling one.

We initiated the Academic Writing Analytics (AWA) project in 2015. To deliver on this vision requires integrated expertise including natural language processing, linguistics, academic language pedagogy, learning design, feedback design, user experience, and cloud computing. Truly a transdisciplinary effort, which has been enormously stimulating.

We’ve passed milestones such as establishing the technical platform, first deployments with students, first publications of what we’ve learned, and now we hit a critical one for the future development of the technical infrastructure, as well as the educational community who need to be driving this. CIC has released the AWA infrastructure open source, with developer resources, and educational resources. I am indebted to the many people who have contributed to this, both in UTS, other universities, and our key partners at Naver Labs Europe. The ATN-funded Higher Education Text Analytics (HETA) project is now collaborating around the platform, and investigating other Higher Ed applications in addition to writing.

Learn more and join the community 🙂

Analytical research writing example [learn more]

Reflective practitioner writing [learn more]

 

New PostDoc Fellowship & PhD Scholarships in Learning Analytics

It’s been an exciting 3.5 years here at UTS, and CIC is about to move into a new phase, with some team changes, and new openings.

It’s always a delight to see my team move on to the next level, but of course I’m sorry to lose from the immediate team Simon Knight as he takes up a Lectureship in the new UTS Faculty of Transdisciplinary Innovation, and Andrew Gibson as he takes up a Lectureship at Queensland University of Technology. Congratulations to both of them on securing these posts, and they both know how grateful I am for their many contributions to our joint work. We will still be collaborating of course. . .

So this opens up a new 4 year post doc position in Writing Analytics which went live today — please get in touch if you want to discuss informally. We also have two 3-year PhD scholarships to continue building probably the world’s fist doctoral training program dedicated to Learning Analytics.

Postdoctoral Research Fellow: Writing Analytics (4 year post to start ASAP, Salary AUD $100,923 — $115,486 pa: deadline 11.59pm Mon 12th February 2018 (AEDT).

An exciting opportunity has opened up to join CIC, and advance the Academic Writing Analytics R&D program. You bring expertise in text analytics, great interpersonal skills to work with academics, students and partner researchers, a strong research track record, and a commitment to invent and validate 24/7 actionable feedback on writing to all students.

2 PhD Scholarships (3 year positions to start by August 2018, AUD $35,000/pa plus additional work opportunities)

CIC conducts distinctive research into the use of analytics to nurture in learners the creative, critical, sensemaking qualities needed for lifelong learning, employment and citizenship in a complex, data-saturated society. You will join four current doctoral students in this new program, the first in the world dedicated to Learning Analytics, in a university committed to making the most of Data Science.

Addressing educators’ concerns about Learning Analytics

It was a pleasure to spend Tuesday afternoon in Melbourne at the Assessment Research Centre (Director, Sandra Milligan) and Centre for the Study of Higher Education (Director, Gregor Kennedy). There is such a breadth and depth of work in these centres, and I met an extraordinary range of researchers.

They invited me to address the concerns that many educators have around “Learning Analytics” — the application of data science to educational data, so here goes…

Teaching, Assessment and Learning Analytics: Time to Question Assumptions

This will be a non-technical talk accessible to a broad range of educational practitioners and researchers, designed to provoke a conversation that provides time to question assumptions. The field of Learning Analytics sits at the convergence of two fields: Learning (including learning technology, educational research and learning/assessment sciences) and Analytics (statistics; visualisation; computer science; data science; AI). Many would add Human-Computer Interaction (e.g. participatory design; user experience; usability evaluation) as a differentiator from related fields such as Educational Data Mining, since the Learning Analytics community attracts many with a concern for the sociotechnical implications of designing and embedding analytics in educational organisations.

Learning Analytics is viewed by many educators with the same suspicion they reserve for AI or “learning management systems”. While in some cases this is justified, I will question other assumptions with some learning analytics examples which can serve as objects for us to think with. I am curious to know what connections/questions arise when these are shared..

(PDF download)

Visiting the IARPA SWARM Collective Intelligence project

As society’s problems get only more complex (not just more complicated), there is growing interest in how we can blend “the cloud with the crowd” — the best of machine intelligence and human intelligence. Specifically, complex, wicked problems are never resolvable by individuals analysing a problem and announcing the solution. Multiple perspectives are needed, and engaging stakeholders in helping to define the problem, never mind deciding what might count as an acceptable solution, is critical.

Collective Intelligence (CI) is one of the current names given to efforts to demonstrate the when done well, groups or even casts of hundreds/thousands of citizens, can work more effectively on a problem than individuals. Internet platforms provide cost effective ways to harness the crowd. As we speak, teams are competing in the IARPA CREATE competition, to build the best online platform that can harness the expertise of a team of volunteer citizens.

I was delighted to be invited by Tim van Gelder, who co-leads the University of Melbourne’s IARPA SWARM Project (Smartly-assembled, Wiki-style Argument Marshalling), to spend the day with them, discussing how they are tackling this challenge. Here’s a nice news story introducing Tim and the project.

For over 20 years, I’ve been exploring the design of a specific class of CI platform, which I call Contested Collective Intelligence (CCI). These are designed around the principle that people will disagree as much as they agree, and it is important for computer systems to be able to work with that. Read more on the CIC Knowledge Cartography page (or go deeper into CCI). Tim and I go way back due to our common interest in visualising argumentation (see Tim’s track record developing argument mapping tools), and the lessons we’ve learned about both the benefits of ‘seeing what you’re saying’, as well as the adoption obstacles to helping people think more critically.

The SWARM team’s approach looks to be very promising, and could have exciting applications in educational and training contexts. A finely balanced design mix of social platform with very lightweight semantics, and analytics on the roadmap. I look forward to seeing the word spread as they put it out there. The good news is that the platform will be released open source. Track their news and sign up if you want to participate.

Meantime, here is my distillation of 20 years work into a series of dilemmas and (partial) solutions, which I’ve been blogging over the years under the tags collective intelligence and argument mapping.

MDSI students shine (again) in NSW Gov Data Challenge

Microsoft WordScreenSnapz003

Students studying the Master of Data Science & Innovation (MDSI) have once again emerged victorious in the second NSW government’s Data Analytics Centre invitational hackathon. Tackling three rounds of a challenge set by Fire and Rescue NSW to differentiate true fires from false alarms, the winning team enriched the dataset provided by adding weather and Twitter data, and generated additional features such as building type. Excitingly, their work may now influence data policy. A combined team from FEIT and UNSW took silver.

Having won the TransportNSW challenge in 2015, this advances UTS reputation as delivering outstanding, work-ready students. The MDSI teams have defeated other universities and professional companies, not only due to their technical ability, but with high performance teamwork, creativity and their ability to pitch their work to the client.

A consequence of fielding teams in these and other hackathons is that we have come to recognise them as a vehicle for very authentic learning. In close partnership with NSW Data Analytics Centre, MDSI has integrated their data challenges into the 2016 curriculum, providing a core experience for all students.

Algorithmic Accountability for Learning Analytics

sna-aa

Update 26.11.19: An extended version of this talk is now available as a webinar)

JISC in the UK is providing the education sector with a valuable service through its Effective Learning Analytics programme. What caught my eye recently was Niall Sclater’s excellent blog with podcasts from his interviews with leading UK practitioners on the ethical dimensions to analytics.

There are many insights to gain from playing these podcasts, but I was particularly tuned to any mention of making the algorithms underpinning analytics intelligible, and to whom. This cropped up a few times when the interviewees discussed to what extent students should be shown analytics, and how to explain their inner workings in a helpful way to them, the teaching staff expected to trust these new tools, and analytics researchers keen to know the inner workings of, for instance, a new analytics product. If handing over an SQL export or full LMS log aren’t considered helpful, what is the right level of detail, and summarised in what ways, for us to be “transparent”? Listening to this, it struck me that in fact this turns out to be a technology-enhanced learning design problem: how to engage non-expert audiences with very complex material to deliver quality ‘learning outcomes’? There’s a few PhDs in that. (I note in passing the Open Learner Models research strand from AIED which is now in dialogue with learning analytics).

It turns out from Niall’s interviews that students aren’t actually very curious, which is in my view a reflection of the data illiteracy in society at large. I certainly intend to make my students very curious about the analytics we run on them, but then, they’re data science students. It would seem that some vendors of predictive models are banking on customers not asking too many questions, because in my interactions with them, they have yet to develop any conception of a service to help a client tune the algorithms to their context.

Back to the JISC interviews. I see the material here, and work on the ethics of learning analytics (e.g. Pardo & Siemens 2014; Prinsloo & Slade 2015) as coming at the problem from one angle, namely ethics/legal compliance/student support/educational institutional processes. Another related but slightly different angle is to approach the problem is to ask what would it mean for a learning analytics system to be accountable to its stakeholders?

This issue is by no means restricted to learning analytics of course. Education is — as ever — slow out of the blocks compared to other sectors that have been transformed by technology. What is encouraging is that as algorithms pervade societal life, they are moving from the sorts of things that only computer scientists and mathematicians would discuss, to becoming the object of far wider academic and indeed media attention [try a web news search on algorithms]. As we (and we might ask, who is we?) delegate machines with increasing powers to make judgements about fuzzy human qualities such as ’employability’, ‘credit worthiness’, or ‘likelihood of committing a crime’, many are now asking how the behaviour of algorithms can be made more transparent and accountable. But in what senses  are they opaque and to whom? What is meant by “accountable”?

The learning analytics community can learn something from our colleagues in other fields as they wrestle with these questions. I love the provocation piece for the  Governing Algorithms conference, and the sparkling set of videos. Reflect on Tarleton Gillespie’s analysis of Google’s and Apple’s algorithms. Check out Paul Dourish’s recent lecture on the Social Lives of Algorithms. I learnt a lot from Solon Barocas’ tutorial on the ways that machine learning can replicate structural injustice if deployed unethically for recruitment purposes. Watch Frank Pasquale on The Promise (and Threat) of Algorithmic Accountability in the Black Box Society, and be afraid…

pasquale-blackbox

In a series of talks* I am test flying my thoughts as I get to grips with this work. I propose a set of lenses that we can bring to bear on a given learning analytics system to define “accountability” at multiple levels from multiple angles. It turns out that algorithmic accountability may be the wrong focus — or rather, just one piece in the jigsaw puzzle. Intriguingly, even if you can look inside the algorithmic ‘black box’, which is imagined to lie in the system’s code, there may be little of use there. I suggest that a human-centred informatics approach is an appropriate one to embrace when considering “the system” wholistically, where the aggregate quality we are after might be dubbed Analytic System Integrity. I conclude by working through a couple of worked examples from current projects as a form of ‘Analytic System Integrity audit’, to show where one can identify weaknesses.

May 6 update: The following replay is from a talk at the UCL Institute of Education (Knowledge Lab) joint with UCL Interaction Centre. It is v2 of the talk, updating the one I posted earlier from University of South Australia Digital Learning Week.

* My thanks to colleagues for hosting these events: Kirsty Kitto (Queensland University of Technology, Institute for Future Environments), Shane Dawson (University of South Australia, Digital Learning Week), UCL (Manolis Mavrikis), and The Open University (Rebecca Ferguson).