UTS awards CIC’s multi-year, multidisciplinary faculty collaboration, using the OnTask tool to scale personalised feedback and build student belonging.
Since 2017, CIC has been supporting academics to embed OnTask messaging in learning design, offering workshops and consultations. When Lisa-Angelique Lim joined CIC in 2021, she brought deep expertise from her PhD at UniSA, and took this work to a new level, leading a learning community to implement OnTask across a wider range of subjects, thereby enabling personalised feedback and support for students in multiple faculties.
So it’s a true mark of recognition to see that CIC, in partnership with UTS academics from multiple faculties, received an award in recognition for our sustained efforts in the Student Experience category, for fostering students’ belonging in multiple disciplines through personalised feedback. This accolade was part of the Vice-Chancellor’s 2024 Learning and Teaching Awards and Citations, and was celebrated at the 2025 Learning and Teaching Awards Ceremony at UTS on 4 April 2025.
Our large, multidisciplinary team was led by Dr Lisa-Angelique Lim (CIC), with Associate Professor Amanda White (Business), Dr Amara Atif (FEIT), Chris Croese (Law), Associate Professor James Wakefield (Business), Dr Keith Heggart (FASS), Associate Professor Nicole Sutton (Business), Ram Ramanathan (CIC PhD student), Dr Rina Dhillon (Business), Dr Simone Faulkner (Business), and Professor Simon Buckingham Shum (CIC).
Addressing the challenges of student belonging with personalised feedback using OnTask
Belonging is a cornerstone of the student experience, critically impacting engagement, retention, and overall success. At UTS, the Student Experience Framework places a strong emphasis on belonging. However, the challenge of nurturing a sense of belonging at the classroom level is significant, given the diverse and large student population, particularly in first-year core subjects. Personalised feedback has emerged as a potent tool to address this challenge, serving as a form of ‘relational pedagogy’ that builds self-efficacy and fosters greater engagement and thriving at university.
To tackle the challenge of belonging, our team leveraged OnTask — a tool designed to support students through personalised communication and feedback across multiple disciplines, based on their data. OnTask uses learning analytics to provide tailored messages that help students stay on track with their studies, understand their progress, and feel supported throughout their academic journey. Check out this short animation on how OnTask works.
Quick teaser video of our academics’ perspectives of their implementation of personalised feedback in their context:
What this looks like in practice
Here, we highlight a few examples of the work by our team.
The UTS Business First and Further Year Experience (FFYE) team, led by A/Prof James Wakefield and Dr Simone Faulkner, used OnTask to personalize orientation communications for both undergraduate and postgraduate commencing students, leading to a significant increase in orientation registrations.
A/Prof Amanda White used OnTask in the Accounting for Business Decision A (ABDA) subject to tailor emails based on students’ progress, encouraging higher engagement and leading to improved exam grades.
Dr Keith Heggart created personalised video messages tailored to students’ confidence levels in the fully online Graduate Certificate in Learning Design, significantly enhancing student engagement and perceptions of support.
The positive impacts of OnTask are well-documented. For instance, in the large first-year subject, 22208 Accounting, Business and Society (ABS), led by Dr Rina Dhillon and A/Prof Nicole Sutton, personalised messages based on weekly quiz results led to improved pass rates and enhanced feelings of being valued among at-risk students.
Similarly, personalised feedback messages sent by Chris Croese to his students in large Law subjects kept students on track and motivated, correlating with better final grades. More stories of how academics at UTS have used OnTask, with research papers, can be found on CIC’s OnTask page.
Towards future partnerships to enhance student belonging
With this award, we celebrate the collaborative efforts and innovative approaches taken by our team to foster a sense of belonging among students through personalised feedback. Our journey over these past three years demonstrates the sustainability of this practice and its potential to influence and enhance teaching and learning widely.
We are honoured to receive this recognition, and look forward to continuing strong partnerships, to make a positive impact on the student experience at UTS.
10 years ago today, the UTS Connected Intelligence Centre (CIC) launched. In fact on that day, I was in the air en route London to Sydney, not only to start the new job, but on a pressing mission to find a home and school for the rest of the family, who would land in 6 weeks time! Jenna Price and John made me very welcome in their home, and who better than two wonderful journalists to give me a crash orientation course to life in Sydney and AUS – a special time!
Shirley Alexander, UTS VP/DVC for Education & Students,had brought me over to help think through the implications of the tech revolution de jour — remember Big Data?! — and launch the university’s first transdisciplinary data science program, the Master of Data Science in Innovation (MDSI), inspired by the groundbreaking success of the transdisciplinary Bachelor of Creative Intelligence & Innovation. Given my work in Learning Analytics, the specific challenge was to invent, pilot and scale the use of data and analytics to improve the learning experience, as well as help units across UTS understand the potential of data science to support their work. The most recent account of how we’ve navigated that quest is in Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom. A decade on, we face a new revolution with the explosive arrival of generative AI, which has dominated the last 18 months.
So thank you EVERYONE who has passed through CIC’s doors, who’s walked this last decade with me, plus of course SO MANY colleagues in UTS, and Australia. Apart from thanking Shirley, whose vision underpinned this, I simply dare not start listing you all since I know I will instantly regret missing names. But I do want to extend a special thank you to Gabrielle Gardinerwho was here long before I arrived, working with Shirley on the Connected Intelligence Strategy, and from 1 August 2014, was always there supporting, advising, and making stuff happen!
To everyone else — academics, MDSI and PhD students, tech coders and architects, learning designers, strategists, directors, senior leaders, and more — you know who you are! Thank You, it’s been a joy. Here’s to 1st August, 2034 🙂
I particularly enjoy writing these blog posts, celebrating the publication of a doctoral thesis. The blood, sweat and tears by doctoral researchers that goes into this moment is always immense, when one thinks about the learning curve they go through, the highs and the lows, and the balancing act of juggling this with the rest of life over 3-4 years. But here we are again, and I’m delighted to add a new one to these team posts!
Well done Carlos Prieto-Alvarez, as today sees the publication of one of the first PhDs in Learning Analytics devoted to the contributions of human-centred design methods (specifically, co-design) to give non-technical stakeholders such as educators and students a voice in shaping Learning Analytics. My thanks to co-supervisor Roberto Martinez-Maldonado, and also to Theresa Anderson who was on the team for the first year or so. Carlos is the third graduate from the CIC Learning Analytics PhD Program launched in 2016!
Prieto-Alvarez, C.G. (2020), Engaging Stakeholders in the Learning Analytics Design Process. Doctoral Dissertation, Connected Intelligence Centre, University of Technology Sydney, AUS. http://hdl.handle.net/10453/142525
The abstract and full dissertation are below, read the CIC news stories over the years from his work, and replay his final year thesis presentation:
Engaging Stakeholders in the Learning Analytics Design Process
ABSTRACT:
Learning Analytics (LA) is a new promising field that is attracting the attention of education providers and a range of stakeholders including teachers, learning designers academic directors and data scientists. Researchers and practitioners are interested in learning analytics as it can provide insights from student data about learning processes, learners who may need more help, and learners’ behaviours and strategies. However, problems such as low educator satisfaction, steep learning curves, misalignment between the analytics and pedagogical approaches, lack of engagement with learning technologies and other barriers to learning analytics development have already been reported. From a human-centred design perspective, these problems can be explained due to the lack of stakeholders’ involvement in the design of the LA tools. In particular, learners and teachers are commonly not considered as active agents of the LA design process. Including teachers, learners, developers and other stakeholders as collaborators in the co-design of LA innovations can bring promising benefits in democratising the LA design process, aligning analytics and pedagogy, and meeting stakeholders’ expectations. Yet, working in collaboration with stakeholders to design LA innovations opens a series of questions that are addressed in this thesis in order to contribute to closing the gap for effective co-design of LA innovations. The questions addressed in this thesis are the following:
How can co-design techniques assist in the integration of diverse stakeholders in the LA design process?
What are the roles of the co-design practitioner/researcher in the LA design process?
What are the challenges in engaging stakeholders in the LA design process?
Based on co-design principles, and following a Design-Based Research process, this thesis explores the critical challenge of engaging educators and students, the non-technical stakeholders who are often neglected, but who should ultimately be the main beneficiaries of LA innovations. In this research work, three case studies have been used to test, analyse and verify various co-design techniques in diverse learning contexts across a university to generate a co-design toolkit and recommendations for other co-design practitioners: i) learners and educators engaged in simulation-based healthcare scenarios, ii) learners, educators and other stakeholders in a Data Science Masters program, and iii) educators interested in providing personalised feedback at scale.
This thesis presents three contributions to knowledge for effectively collaborating with educational stakeholders in the LA co-design process:
Inspired by archetypal challenges reported in classic and contemporary co-design literature, and in current LA research, the thesis identifies, exemplifies and reflects on five key challenges for LA co-design: power relationships, surveillance, learning design dependencies, asymmetric teaching/learning expertise, and data literacy.
By adopting and adapting well established co-design techniques, across the three case studies, the thesis provides empirical evidence of how these techniques can be used in LA co-design, reflecting on their affordances, and providing guidance on their usage. These detailed findings are distilled into a Learning Analytics Co-design Playbook, published under an open license to assist adoption and improvements.
Recognising the importance of the co-design practitioner in ensuring that the design process is participatory, the thesis documents and discusses the key functions and skills that this position requires. The role is further complicated when the practitioner is not only a facilitator serving a project, but also a researcher of co-design. This motivates guidelines on the role of the co-design practitioner/researcher when working with stakeholders, and simultaneously studying the LA co-design process, tools and methods.
See Carlos’ ResearchGate site for full-text papers.
Conference Papers
Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK2020), Frankfurt, Germany, March 2020, ACM, New York, NY, USA. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476
Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2018). Mapping Learner-Data Journeys: Evolution of a Visual Co-design Tool. Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI ’18), Melbourne, Australia, Dec. 2018, ACM, New York, NY, USA, pp. 205–214. DOI: https://doi.org/10.1145/3292147.3292168
Prieto-Alvarez, C.G, et al. (2018). Collaborative Personas for Crafting Learners Stories for Learning Analytics Design. Workshop Participatory Design for Learning Analytics, International Conference on Learning Analytics and Knowledge LAK’18. Sydney, Australia, ACM: 647-652. ISBN: 978-1-4503-6400-3
Book Chapter
Prieto-Alvarez, C.G., Martinez-Maldonado, R. Anderson, T. (2018). Co-designing learning analytics tools with learners. Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers, Taylor & Francis Groups: 93-110.
Workshops
Carlos G. Prieto-Alvarez et al (2018). Learning Analytics Design Cards (LA-DECK): Unpacking inter stakeholder co-design through strategic cards. Australian Learning Analytics Summer Institute. Melbourne, Australia. Website: http://ladeck.utscic.edu.au/events.html
Carlos G. Prieto-Alvarez et al (2018). Participatory design of learning analytics. International Conference on Learning Analytics and Knowledge LAK’18. Sydney, Australia, ACM. Website: http://pdlak.utscic.edu.au.
I regularly receive emails from hopeful students wanting to do a PhD, and I frequently decline with similar feedback. Supervisors differ of course in what they look for, but I thought I’d blog these points in case it helps you decide whether a PhD is the right thing (whether or not with me!).
Demonstrate that you understand our work, and can position your interest in relation to this. If you’re seriously proposing to work on a project for 3-4 years of your life, show me how interesting you find it — so interesting that in fact, you’ve read some of the key literature, and specifically, our work.
Research groups are normally looking to build a research program in which each person’s work adds a new piece of the jigsaw to an emerging picture. Your PhD experience will also be massively enhanced when surrounded by colleagues interested in your work, and able to connect you to key ideas and people that will help you.
Attempt to propose a research question. This may well be over-ambitious, and in fact you’ll spend your first year (possibly longer) reviewing this, but a PhD is driven by an RQ, often broken down into 2-3 sub-questions. These will run as a “red thread” throughout the whole thesis: everything you do will be tied to these RQs. To help you do this…
What do you think your contribution will be? In other words, indicate what you think answers to the RQs might look like. This is your first attempt to articulate your potential “contributions to knowledge” — which is what earn you a PhD.
Do you have any idea what a PhD thesis looks like? Before deciding you want to set sail, why not check out the different kinds of PhD I’ve supervised recently, and see what it is you’re promising to deliver? :-). [See also the SoLAR PhD Thesis Hub for more learning analytics PhDs]
Clarity of argument. English may not be your first language, but I’m looking for clarity in your thinking, even if the words are not perfect. However, your thinking will be judged by your writing, so good academic English will be important.
There are free resources online now that can help with your academic English, such as Academic Phrasebank.
I strongly recommend you take this one hour UTS Open taster, an interactive tutorial on how to write a research abstract.This explains the key building blocks in an archetypal abstract — basically: Why should we care about this topic? What don’t we understand yet? What are you going to do about it? If you make these moves in your proposal, it will have a sound structure. This also enables you to try out AcaWriter, one of the instant feedback tools CIC’s developed, on your own writing.
Apply in time to revise your proposal. Check the deadlines, and ideally, get in touch months ahead to start the conversation, because…
If your proposal has promise, I’ll give feedback. Take this feedback on board, or argue back if you disagree. I’m assessing your readiness to take criticism, as well as your readiness to take a stand.
Clever coding ≠ a PhD. When a technically proficient developer applies, they sometimes think quality = developing some extremely cool functionality. A PhD may well include exciting implementations as a proof of the concept, and as a way to evaluate the idea with stakeholders — but at least in my field, it’s the conceptual contributions advancing the field that count.
Recommended books. 3 books to consider, and note that the first two major on the “non-academic” dimensions of doing a PhD:
It is with enormous pleasure that I applaud Vanessa Echeverria Barzola for completing her PhD here at UTS:CIC!
Moving with her family from Ecuador in 2017, Vanessa joined as one of our first doctoral researchers when we launched the Learning Analytics PhD Program, and has wrapped it up within her four years, demonstrating huge personal resilience on the way.
Read her thesis below, related CIC News stories, and browse her website for the extensive publications generated by her work. We wish her every success as she develops her academic career, and can’t wait to see what emerges!
Vanessa Echeverria (2020). Designing Feedback for Collocated Teams using Multimodal Learning Analytics. Doctoral Dissertation, Connected Intelligence Centre, University of Technology Sydney, Australia. http://hdl.handle.net/10453/140936
Abstract: The ability to communicate, be an effective team or group member and collaborate face-to-face are critical skills for employability in the 21st century workplace. Previous research suggests that learning to collaborate effectively requires practice, awareness of group dynamics and reflection upon past activities. However, although having a teacher closely supervising and providing detailed feedback to each group would be ideal, it may be unrealistic in practice. A promising way to approach this challenge could be to capture behavioural traces from group interactions in order to generate comprehensible and actionable feedback to support team reflection. In this sense, Multimodal Learning Analytics (MMLA) is a promising field, offering the potential to track learners’ activity across digital and collocated contexts, using emerging sensing and pervasive computing technologies. Most of the research in MMLA has been conducted in lab conditions, to help researchers validate learning theories or generate more comprehensive learner models. However, one of the most underexplored aspects of MMLA has been the generation of feedback to support teaching and learning, and moreover, in authentic locations and activities.
This thesis reports progress in tackling this challenge by designing and validating computer-based feedback, by means of visual representations and narrative, to support effective, guided reflection using multimodal learning analytics evidence. To achieve this, three contributions are presented. The first contribution is a human-centred design method to translate the informal outputs of co-design sessions with teachers and students, into more meaningful group work constructs with clear MMLA design requirements. The second contribution is a modelling approach to add meaning to low-level multimodal group data based on the characteristics of the context (domain expertise, theory, and the learning design). Finally, the third contribution is an approach for augmenting visual representations with data storytelling elements to facilitate the interpretation of group dynamics insights by educators and students. This thesis is developed in the context of two distinct, collocated group work settings, in the domains of collaborative database design and healthcare simulation. Using a Design-Based Research process, a set of interfaces (i.e. interfaces that communicate insights) was designed and validated with teachers and students. The thesis provides timely and necessary groundwork for researchers and practitioners to design visual representations capable of communicating actionable insights, using multimodal data in complex and authentic collaboration scenarios.
This project is a collaboration between UTS:CIC and the University of Melbourne SWARM Project. The successful candidate will be based in Sydney, also spending time in Melbourne.
Visit the CIC PhD Scholarships page for full details. Please email us to express interest, ask any questions, and if we can see a potential fit we’ll advise you on writing your proposal.
The Challenge
The real world challenge: improving collaborative EBR
The challenges facing society are so complex that multiple expertises are needed. Consider security, science, law, health, policy-making, finance. The teamis the ubiquitous organisational unit, but the quality of its reasoning, especially under pressure, can vary dramatically. Problems are not provided in neat, well-defined packages: a team must frame problems in creative ways that lead to insights, and resolve uncertainties around possible responses, making the best possible use of evidence, plus their own judgement. Studies of how teams engage in such “sensemaking” highlight the blinkers that can blindside teams, and how the ways that the problem is expressed and visually represented can help or hinder (Weick, 1995). We will term this whole process Evidence-Based Reasoning (EBR). (We note of course that politics and social dynamics are unavoidable whenever people come together, and effective team members learn how to navigate these dynamics effectively.)
Improving collaborative EBR is an interesting scientific and design challenge. A successful support system (i.e. ways of working + enabling tools) must respect the principles of good reasoning, as determined by fields such as logic, argumentation and epistemology, and the domain-specific knowledge (i.e. emergency response, engineering, social work, counter-intelligence, etc.). At the same time, it must accommodate the strengths, weaknesses and vagaries of human reasoners, which is the terrain of cognitive and social psychologists. If part of the support system is interactive software, then it must have a good user interface and a solid underlying architecture. Assessing the resulting performance is a difficult evaluation problem. Building such systems is therefore inherently multidisciplinary.
How do we better equip teams for collaborative EBR? From an educational perspective, teamwork, problem solving and critical thinking skills are now among the most in demand ‘transferable competencies’ (Fiore et al 2018). The challenge of assessing and equipping graduates in these is at the heart of the learning and teaching strategies at UTS and U. Melbourne.
The technology support challenge:
While in some fields, there are specialist tools for modelling and simulation that assist analysts by managing constraints in the problem, but even with machine intelligence, the agency typically rests with the human analysts to decide how much weight to give to the machine’s output. Most other fields, however, do not have such tools: collaborative EBR is typically supported by general-purpose information technologies such as word processors, spreadsheets, databases, and project planners to help with managing information and producing reports. Similarly, generic communication tools dominate, such as email, chat, video conferencing, phone. In most cases, the reasoning itself is typically left wholly to the human reasoners themselves.
There have been remarkably few attempts to provide direct technological support for the processes of inference and judgement that are at the heart of collaborative EBR, and moreover, those attempts have had little impact on the way it is actually conducted in most places (van Gelder, 2012). There are methods and software tools for facilitating group processes and visualising team reasoning, but these require quite an advanced facilitation and software skillset (e.g. Culmsee and Awati, 2013; Okada, et al., 2008; Selvin et al, 2012).
Our interest is in developing computer-support to improve the collaborative EBR of geographically and often temporally distributed teams, that does not require specialist skills to start using beyond using what are now familiar collaboration tools. SWARM is an online platform emerging from an ongoing research project to improve the kind of collaborative EBR undertaken by intelligence analysts making sense of complex sets of qualitative and quantitative information or varying reliability. However, these are the conditions under which most other domains operate, and we hypothesise that it has broader potential, and specifically in this project, for education and training. SWARM is based on three design principles: cultivating user engagement, exploiting natural expertise, and supporting rich collaboration (van Gelder et al, 2018). Central to its approach is the upskilling of team members to equip them with different EBR skills (see in particular the Lens Kit).
Figure: The SWARM workspace
Recent large scale empirical evaluations, in which teams of analysts tackled complex challenges with or without SWARM, indicated that the quality of the reports produced by SWARM teams was significantly better than reports produced by analysts using normal methods (van Gelder et al, In Prep). In a follow-up project, “super-teams” on the platform produced reasoning so good it would plausibly be called “super-reasoning” (van Gelder & de Rozario, 2017) analogous to “super-forecasting” (Tetlock & Gardner, 2015).
This CIC seminar is a great introduction to the work so far:
Learning Analytics for SWARM
The encouraging evidence of SWARM’s effectiveness makes it an attractive candidate platform for use in educational/training contexts. While evaluation of final reports (i.e. the team’s product) is a conventional measure of team performance, and certainly one that educators will be interested in, this is not the only possible indicator of improvement. The emergence of data science, activity-based analytics and visualisation opens new possibilities for tracking the process that teams are following. Learning Analytics connects such techniques to what is known about the teaching and learning of teamwork, and could make the assessment of team performance more rigorous, and more cost effective.
This PhD is therefore focusing on inventing and validating new forms of automated team analytics for collaborative EBR, to provide insights into both process and product. Such analytics might enable not only coaches and researchers to gain insights into a team’s effectiveness, but the teams themselves to monitor their work in real time, or critically review their project on completion. Further, real-time analytics can be used to shape the collaborative environment itself, resulting in better collaboration and better outputs. Some prototype analytics have already been developed to summarise participants’ contributions and interactions. This PhD will build on this work, synthesise the literature, plus insights from the SWARM team and educators, in order to define, design, implement and evaluate automated analytics in different contexts, spanning education and training, research, and potentially more authentic deployments with professional teams.
Figure: Early version of the SWARM group dynamics dashboard. Upper diagrams shows levels of interaction among team members working on a particular problem.
Relevant analytics techniques include, but are not limited to:
Text analysis to identify significant contributions to the team communications and the report they are producing
Social network analysis to identify significant interaction patterns among team members
Process mining to identify significant sequences in the actions that individuals engage in, within or between sessions
Statistical techniques to identify significant differences between teams
Candidates
In addition to the broad skills and dispositions that we are seeking in all candidates (see CIC’s PhD homepage), you should have:
A Masters degree, Honours distinction or equivalent with at least above-average grades in computer science, mathematics, statistics, or equivalent
Analytical, creative and innovative approach to solving problems
Strong interest in designing and conducting quantitative, qualitative or mixed-method studies
Strong programming skills in at least one relevant language (e.g. R, Python)
Experience with web log analysis, statistics and/or data science tools.
It is advantageous if you can evidence:
Design and Implementation of user-centred software, especially data/information visualisations
Skill in working with non-technical clients to involve them in the design and testing of software tools
Knowledge and experience of natural language processing/text analytics
Familiarity with the scholarship in a relevant areas (e.g. high performance teams; collective intelligence; collaborative problem solving)
We will discuss your ideas with you to help sharpen up your proposal, which will be competing with others for a scholarship. Please follow the application procedure for the submission of your proposal.
Fiore, S. M., Graesser, A., & Greiff, S. (2018). Collaborative problem-solving education for the twenty-first-century workforce. Nature Human Behaviour, 2(6), 367–369.
van Gelder, T., & de Rozario, R. (2017). Pursuing Fundamental Advances in Human Reasoning. In T. Everitt, B. Goertzel, & A. Potapov (Eds.), Artificial General Intelligence(Vol. 10414, pp. 259–262). Cham: Springer International Publishing.
Ming Liu and Simon Buckingham Shum (UTS:CIC), Cherie Lucas (UTS:Pharmacy) The supervision team for this PhD is a partnership between CIC and the School of Pharmacy, who together have pioneered reflective writing analytics.
Visit the CIC PhD Scholarships page for full details. Please email us to express interest, ask any questions, and if we can see a potential fit we’ll advise you on writing your proposal.
The Challenge
The societal challenge:
“We do not learn from experience… we learn from reflecting on experience.”
(paraphrasing John Dewey)
The problems now confronting society place an unprecedented urgency on learning from experience. Such is the pace of change that before we can plan for them, citizens and professionals in all sectors find themselves immersed in novel, complex problems. Moreover in education, the learning sciences tell us that crafting authentic experiences is a powerful trigger for learning.
“White water is the new normal”, as they say. But as Dewey noted, critical to this is our capacity to reflect on the experience of shooting those rapids. If we can’t learn how we could do better next time — individually and collectively — we are in deep trouble. From school age students, through higher education, and into professional leadership, we have to make sense of challenging experiences, recognise how we were challenged, how we are changing, and how we can improve.
At the heart of deep learning is our sense of identity. People rarely shift from entrenched positions by force of argument alone. However, when we undergo challenging experiences that force us to question assumptions and worldviews at the heart of our identity, this can indeed be transformational if we are assisted in making sense of this, and can emerge with our identify intact but now under reconstruction. Without such shifts, it’s hard to see how we will move beyond current polarisations around how we relate to each other, and the planet. Given our current political and cultural climate, applied research to help people reflect on how they adjust to threatening transitions is both timely, and of first order importance.
So, we need to get better at deep reflection, and clearly, there’s nothing as valuable as detailed feedback to provoke further reflection. But this is a scarce skillset and very labour-intensive. The practical consequence is that most students and leaders do not understand what good reflective writing is, and do not receive good feedback. For these reasons, there’s interest in educational and professional sectors in the potential of automated techniques to deliver real-time, personalised coaching.
In sum, this PhD is fundamentally about harnessing computational intelligence to deepen human learning in contexts spanning formal education, professional practice, and community transformation.
The writing challenge:
Effective written communication is an essential skill which promotes educational success for university students. However, far too many students have never had the features of good rhetorical moves explained well to them, and most educators are subject matter experts, not skilled writing coaches (Lucas, Gibson & Buckingham Shum, 2018). CIC initiated its Academic Writing Analytics (AWA) project in 2015, as it became clear through consultations across faculties that student writing was a strategically important area for UTS teaching and learning (and indeed, for most other educational institutions). The goal is to more effectively teach the building blocks of good academic writing by providing instant, personalised, actionable feedback to students about their drafts (Knight, Buckingham Shum, Ryan, Sándor, & Wang, 2018).
To deliver on this vision requires integrated expertise including natural language processing, linguistics, academic language pedagogy, learning design, feedback design, user experience, and cloud computing. This is truly a transdisciplinary effort, which has been enormously stimulating. To date, we have worked on critical, argumentative, analytical writing of the sort typically found in literature reviews, persuasive essays and research articles, as well as reflective writing, in which learners make sense of their workplace experiences, try to integrate this with their academic understanding, and share their own uncertainties, emotions and sense of personal challenge/growth (Gibson, Aitken, Sándor, Buckingham Shum, Tsingos-Lucas, & Knight, 2017).
Learning Analytics tools are most effective when co-designed with effective Learning Designs: the features constructed by the analytics align with the assessment criteria, and the tool is coherently embedded in authentic student learning tasks. Our program has demonstrated how this can be accomplished (Knight, Shibani & Buckingham Shum, 2018; Shibani, Knight, Buckingham Shum & Ryan, 2017).
Depending on your interests and skillset, critical advances that this PhD might range across technical and pedagogical contributions to educational technology:
Technical:
Integration between rule-based modelling and machine learning approaches
Accelerated customisation of the parsers to different disciplinary domains and genres of writing
Definition of new computational proxies that can serve as indicators of deep reflection
User experience and machine learning to enable user feedback that teaches the tool when it make errors
Curation of text corpora to advance the field
Pedagogical:
Design and validation of analytics-augmented learning design patterns
Radical improvements in the user experience of automated feedback, e.g. through novel educator/student co-design processes, or user interfaces
Analytics Approaches
We currently implement the underlying concept matching model using a rule-based grammar and human-curated lexicons, which for those not familiar with this kind of work, brings both pros and cons (Buckingham Shum, Sándor, Goldsmith, & McWilliams 2017; Ullmann, 2017). The rules are grounded in scholarly literature on the features of academic research writing, and have been tested on diverse texts by the team through close manual analysis. The lexicons can be edited to tune them to the language used in different disciplines and subjects. This relatively traditional AI approach provides familiar intellectual credentials when introducing the system to educators, and when we’re testing it, the underlying behaviour is easier to explain, and errors can be diagnosed very precisely. However, it brings the limitations associated with any rule-based approach: given the richness of open-ended reflective writing, there are exception cases to debug, and improvements to the system’s performance require manual edits to the rules and lexicon.
We are now beginning work to investigate if a machine learning approach can augment the current infrastructure (Liu, et al. 2019; Ullmann, 2019). Recent years, with the availability of “big data”, such as large question answer banks (Rajpurkar, Zhang, Lopyrev, & Liang, 2016), and effective machine learning algorithms, e.g. deep neural networks (Lecun, Bengio, & Hinton, 2015), data driven approaches based on new data processing architectures have attracted a great attention in natural language processing tasks, such as neural text summarization (Liu & Manning, 2017) and neural machine translation (Bahdanau, Cho, & Bengio, 2014), mainly because these approaches do not require human defined rules and have good generalization power. However, such data driven approaches require a large amount of data, and some statistical learning models such as deep neural networks are not easy to comprehend. Preliminary results are reported
Therefore, we invite your proposals as to which techniques might be best suited to this challenge. What’s more, the creation of a corpus for writing raises ethical challenges, and we invite your thoughts on what these are, and how we might address them.
You will work in close collaboration with one or more academics from other faculties/units in UTS, using co-design methods with academics, potentially external partners, with opportunities for synergy with existing projects and tools as described on the CIC website. For more information about ongoing research in this area, please visit the Academic Writing Analytics homepage and the Writing Analytics blog.
Resources to help you understand the current state of the technology and its educational applications include the references cited, plus:
We’re looking for the broad skills and dispositions that we are seeking in all candidates (see CIC’s PhD homepage). In addition, we envisage that applicants will either come from strong technical backgrounds and are passionate to see these make a difference in education, or from strong educational backgrounds seeking to shape the design of analytics/AI.
Core strengths that we expect from applicants with technical backgrounds:
A Masters degree, Honours distinction or equivalent with above-average grades in computer science, mathematics, statistics, or equivalent
Analytical, creative and innovative approach to solving problems
Strong interest in designing and conducting quantitative, qualitative or mixed-method studies
Knowledge and experience of natural language processing/text analytic
Strong programming skills in at least one relevant language (e.g. C++, .NET, Java, Python)
Experience with statistical and data mining, deep learning, or data science tools (e.g. R, Weka, Tensorflow, ProM, RapidMiner).
Core strengths that we expect from applicants with educational backgrounds:
A Masters degree, Honours distinction or equivalent with above-average grades in teaching, educational theory, instructional design, learning sciences (ideally experience in the pedagogy and scholarship of writing)
Knowledge and experience in Design-Based Research, or a related methodology for authentic design and evaluation of educational technologies
Qualitative and quantitative data collection and analysis skills
It is advantageous if you can evidence:
Skill in working with non-technical clients to involve them in the design and testing of software tools
Peer-reviewed publications
Design and Implementation of user-centred software
Interested candidates should contact the team to open a conversation: Ming.Liu@uts.edu.au; Simon.BuckinghamShum@uts.edu.au; Cherie.Lucas@uts.edu.au We will discuss your ideas to help you sharpen up your proposal, which will be competing with others for a scholarship. Please follow the application procedure for the submission of your proposal.
Buckingham Shum, S., Sándor, Á.,Goldsmith, R.,Bass R.,and McWilliams M.(2017). Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results. Journal of Learning Analytics, 4, (1), 58–84.
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C.,and Knight, S. (2017). Reflective Writing Analytics for Actionable Feedback. In Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge.
I’m delighted to announce that Antonette Shibani has become the first graduate from CIC’s PhD program! She passed with flying colours, and has secured a Lectureship in the UTS Faculty of Transdisciplinary Innovation.
Well done Shibani – here’s the link to the thesis, and check out her blog to learn more!
Augmenting pedagogic writing practice with contextualizable learning analytics
ABSTRACT: Academic writing is a key skill that contributes to essential learning outcomes for higher education students. Despite its importance, students often lack proficiency in writing and find it challenging to learn. While previous research suggests that students’ writing skills are enhanced through formative feedback, the time-consuming nature of providing formative feedback on individual student drafts, especially in large cohorts, makes it impractical for educators to provide detailed writing support in this way. A promising approach, therefore, is the use of writing analytics to provide automated formative feedback on writing. This particular form of learning analytics, using computational techniques and natural language processing, provides timely, immediate, and consistent automated feedback to help students improve their writing. However, for such tools to work effectively in pedagogic settings, and be adopted by practitioners, academics need to feel a sense of ownership over how the tool fits into their practice. This recognition motivates an increased emphasis on aligning learning analytics applications with learning design, so that analytics-driven feedback is congruent with the pedagogy and assessment regime. The thesis investigates how writing practice can be augmented with a writing analytics tool called ‘AcaWriter’ by aligning it with learning design. The approach is evaluated across two disciplines in authentic higher educational settings using a design-based research approach. Mixed methods and multiple data sources are used to examine how students perceive and interact with automated feedback, and revise their writing. Based on this analysis, the thesis provides empirical evidence that students found the writing intervention and automated feedback from AcaWriter useful, and improved their subject-related writing skills, thus validating its applicability in writing contexts. It identifies varied levels of student engagement with automated feedback and ways to scaffold its application for effective use. Cross-fertilizing research and practice, the key insights gained from these design iterations are formalised as theContextualizable Learning Analytics Design model. The model clarifies how the features, feedback and learning activities around AcaWriter can be tuned for different pedagogical contexts and assessment regimes, by co-designing them with educators. The thesis also studies the perspectives of educators, who play a key role in implementing such learning analytics innovations in their classrooms. The thesis advances theory and practice in the development of flexible learning analytics applications, capable of providing meaningful, contextualized support that enhances learning, and adoption by practitioners in authentic practice.
The start of the new year is a good moment to distill the key ingredients of the 2 year collaboration around writing analytics (automated feedback to students on their reflective writing), that CIC has built with UTS academic Cherie Lucas from our School of Pharmacy. As with other collaborations (such as with Pip Ryan on her students’ legal writing), it exemplifies the co-design process that we initiate with academics, in which we iteratively seek to design an automated feedback tool that students can use to improve their drafts, prior to submission:
distill key insights from the scholarship into the teaching and learning of good reflective writing, to design a formal, implementable model that – in principle – should be applicable to a wide range of reflective writing contexts (learn more)
understand the academic’s specific educational challenge in context (e.g. students are struggling to produce good reflective writing about their work experience placements)
establish the mapping to the features that our text analytics tool is able to detect
evaluate the performance of the parser (in close partnership with the academic)
In the video, Cherie Lucas describes the nature of the challenge, and what AcaWriter contributes to the learning experience. The interface looks like this, with the text editor frame on the left, and the Reflective Report annotation of the writing on the right, generated after a few seconds on clicking Get Feedback:
Zooming in on the automatically annotated student writing in the Reflective Report:
Not shown in the video is the Feedback Tab providing encouragement when there appear to be good features in the text, and actionable feedback for improvement, e.g.
For us, one of the hallmarks of a successful collaboration is that our academic partners’ own disciplinary community of educators recognise the advance they’ve made. Here’s a brief, very helpful introduction that Cherie wrote for her peers. Her work has excited significant interest with colleagues around the world, who are now initiating their own projects to install our software for piloting with their students.
Learn more about how we design Writing Activities with Writing Analytics through the integration of learning design, analytics, educator and student resources, and evaluation evidence…
Dive deeper…
Gibson A., Aitken A., Sándor Á., Buckingham Shum S., Tsingos-Lucas C. and Knight S. (2017), Reflective writing analytics for actionable feedback. Proceedings of LAK17: 7th International Conference on Learning Analytics and Knowledge, March 13-17, 2017, Vancouver, CA (ACM Press: NY). [Video] (AWARDED BEST PAPER)
Tsingos-Lucas C, Aitken A, Gibson A, Buckingham Shum S. (2017). Utilisation of a Novel Online Educational Toolto Assist Pharmacy Students to Self- Critique Reflective Writing Tasks. Proceedings of the 9th Pharmacy Education Symposium, Prato, Italy, 9-12th July 2017. (AWARDED BEST TEACHING INNOVATION POSTER)
It’s been an exciting 3.5 years here at UTS, and CIC is about to move into a new phase, with some team changes, and new openings.
It’s always a delight to see my team move on to the next level, but of course I’m sorry to lose from the immediate team Simon Knight as he takes up a Lectureship in the new UTS Faculty of Transdisciplinary Innovation, and Andrew Gibson as he takes up a Lectureship at Queensland University of Technology. Congratulations to both of them on securing these posts, and they both know how grateful I am for their many contributions to our joint work. We will still be collaborating of course. . .
So this opens up a new 4 year post doc position in Writing Analytics which went live today — please get in touch if you want to discuss informally. We also have two 3-year PhD scholarships to continue building probably the world’s fist doctoral training program dedicated to Learning Analytics.
An exciting opportunity has opened up to join CIC, and advance the Academic Writing Analytics R&D program. You bring expertise in text analytics, great interpersonal skills to work with academics, students and partner researchers, a strong research track record, and a commitment to invent and validate 24/7 actionable feedback on writing to all students.
2 PhD Scholarships (3 year positions to start by August 2018, AUD $35,000/pa plus additional work opportunities)
CIC conducts distinctive research into the use of analytics to nurture in learners the creative, critical, sensemaking qualities needed for lifelong learning, employment and citizenship in a complex, data-saturated society. You will join four current doctoral students in this new program, the first in the world dedicated to Learning Analytics, in a university committed to making the most of Data Science.
Many congratulations to Duygu Bektik (née Simsek) whose PhD is now available! Duygu passed with flying colours — truly a landmark in one’s life 🙂 Thank you to Rebecca Ferguson, Denise Whitelock and Ágnes Sándor (Xerox) for supporting her through to the end.
This is gonna be the best day of my liiiife li-i-i-i-i-i-fe! PhD viva is over & I passed with very tiny teeny minor corrections ❤️ #phdlifepic.twitter.com/IoqMTeqL9a
Effective written communication is an essential skill which promotes educational success for undergraduates. Argumentation is a key requirement of successful writing, which is the most common genre that undergraduates have to write particularly in the social sciences. Therefore, when assessing student writing academic tutors look for students’ ability to present and pursue well-reasoned and strong arguments through scholarly argumentation, which is articulated by meta-discourse.
Today, there are some natural language processing systems which automatically detect authors’ rhetorical moves in scholarly texts. Hence, when assessing their students’ essays, educators could benefit from the available automated textual analysis which can detect meta-discourse. However, previous work has not shown whether these technologies can be used to analyse student writing reliably. The aim of this thesis therefore has been to understand how automated analysis of meta-discourse in student writing can be used to support tutors’ essay assessment practices. This thesis evaluates a particular language analysis tool, the Xerox Incremental Parser (XIP) as an exemplar of this type of automated technology.
The studies presented in this thesis investigates how tutors define the quality of undergraduate writing and suggests key elements that make for good quality student writing in the social sciences, where XIP seems to work best. This thesis also sets out the changes that needs to be made to the XIP and proposes in what ways its output can be delivered to tutors so that they make use of this output to give feedback on student essays.
The findings reported also show problems that academic tutors experience in essay assessment, which potentially could be solved by automated support. However, tutors have preconceptions about the use of automated support.
The study revealed that tutors want to be assured that they retain the ‘power’ themselves in any decision of using automated support to overcome these preconceptions.
Congratulations to Dr Simon Knight! Simon passed his PhD viva just before Christmas, with a great performance. The examiners Professor Sten Ludvigsen (University of Oslo) and Professor Allison Littlejohn (OU) commended Simon’s thesis: “the research takes on board some of the more challnging problems facing learning analytics, moving analytics from the ‘management’ of learning assessment towards addressing cognitive dimensions of learning”. Simon is now a research fellow with me at UTS:CIC, where he is advancing our work on writing analytics.
Simon’s thesis is entitled Developing Learning Analytics for Epistemic Commitments in a Collaborative Information Seeking Environment. This was co-supervised with Prof. Karen Littleton and Dr. Bart Rientes — to whom I am indebted for their continued support for Simon after I moved to Sydney! The PhD focuses on how we can quantify aspects of the epistemic cognition that students bring to their learning, using collaborative information seeking and sensemaking as a task context, grounded in socio-cultural theory. The PhD exemplifies how learning analytics research can combine statistical with qualitative methodologies.
The full thesis will be available shortly, but meantime here’s the abstract, snapshots of the work from his publications, and you can track his work on his blog.
By Katie Chan (Own work)
CC-BY-SA-3.0 via Wikimedia Commons
Abstract: Learning analytics sits at the confluence of learning, information, and computer sciences. Using a distinctive account of learning analytics as a form of assessment, I first argue for its potential in pedagogically motivated learning design, suggesting a particular construct – epistemic cognition in literacy contexts – to probe using learning analytics. I argue for a recasting of epistemic cognition as ‘epistemic commitments’ in collaborative information tasks drawing a novel alignment between information seeking and multiple document processing (MDP) models, with empirical and theoretical grounding given for a focus on collaboration and dialogue in such activities. Thus, epistemic commitments are seen in the ways students seek, select, and integrate claims from multiple sources, and the ways in which their collaborative dialogue is brought to bear in this activity. Accordingly, the empirical element of the thesis develops two pedagogically grounded literacy based tasks: a MDP task, in which pre-selected documents were provided to students; and a collaborative information seeking task (CIS), in which students could search the web. These tasks were deployed at scale (n > 500) and involved writing an evaluative review, followed by a pedagogically supported peer assessment task. Assessment outcomes were analysed in the context of a new epistemic commitments-oriented set of trace data, and psychometric data regarding the participants’ epistemic cognition. Demonstrating the value of the methodological and conceptual approach taken, qualitative analyses indicate clear epistemic activity, and stark differences in behaviour between groups, the complexity of which is challenging to model computationally. Despite this complexity, quantitative analyses indicate that up to 30% of variance in output scores can be modelled using behavioural indicators. The explanatory potential of behaviourally-oriented models of epistemic commitments grounded in tool-interaction and collaborative dialogue is demonstrated. The thesis provides an exemplification of theoretically positioned analytic development, drawing on interdisciplinary literatures in addressing complex learning contexts.
I particularly enjoy writing these blog posts, celebrating the publication of a doctoral thesis. The blood, sweat and tears by doctoral researchers that goes into this moment is always immense, when one thinks about the learning curve they go through, the highs and the lows, and the balancing act of juggling this with the rest of life over 3-4 years. But here we are again, and I’m delighted to add a new one to these 
It is with enormous pleasure that I applaud 
I’m delighted to announce that Antonette Shibani has become the first graduate from CIC’s PhD program! She passed with flying colours, and has secured a Lectureship in the UTS Faculty of Transdisciplinary Innovation.
