Simon.BuckinghamShum.net

In 2015 I blogged about the emergence of what I dubbed “Writing Analytics” as a stream within the wider Learning Analytics field. Five years on, we see regular Writing Analytics workshops at LAK and ALASI (see the Events menu on that link), publications appearing in top tier conferences and journals (e.g. see below), and even the launch of a new conference and journal in the last two years.

This blog is to mark the emergence of Reflective Writing Analytics as a sub-stream. Reflective writing is quite different from the more widely used forms of academic writing (literature review, persuasive essay, research paper; etc.), but is growing in importance as we seek to place learners in authentic learning contexts (e.g. internships; work placements; or simulated teams), specifically so that they experience something of the complexity of real workplaces.

Honest reflection can make the writer vulnerable, as they reflect on their uncertainties, failings, and how they are changing as a learner/professional. They are often deeply personal, connecting to different threads across the many areas of their life. That’s almost the opposite of the other genres of writing that dominate students’ and professionals’ lives, which emphasise rational distance, mastery of the material, and confident rhetoric. Deep reflection can even share transformational moments in someone’s life and learning. Speaking personally, I find some student reflections profound, and even moving — an inspirational reminder of why we do what we do. Yet reflecting is not something that people are always (or even often) given the opportunity to learn how to do well.

As noted in a recent paper:

“Helping people make sense of their thoughts, feelings, reactions and approaches when stretched out of their comfort zones is core business for educators and coaches. Suitably supported, honest reflection makes it safe to question assumptions and consider change, but we also know that this is often difficult to teach, and challenging to learn.

[…] Reflective Writing is a strategy used in education and many professions to help learners, professionals and leaders make sense of challenging experiences, and prepare for the future. It integrates “head and heart”: valuing not only technical/academic knowledge, but how this interplays with experiential/professional ways of knowing, and recognising the fact that learning and working engage our emotions and feelings.

[…] However, while we know there is nothing as valuable as detailed coaching feedback to build this capacity, this is a scarce, costly skillset and labor-intensive. The practical consequence is that most students and leaders do not understand how to reflect deeply, and do not receive good feedback.” (Buckingham Shum & Lucas, 2020)

The desire to provide people with useful and timely feedback on reflection on a widespread basis has led to interest in developing Reflective Writing Analytics — broadly, the use of natural language processing and automated feedback methods to understand and support the process of reflection. This is a nascent area. There are not many people (that we know of) working on the challenge of providing automated analysis of, and feedback on, reflective writing, so it was a delight to convene a post-LAK20 call last week with teams from the USA, UK and AUS, when we spent 2 hours comparing notes on what we’re wrestling with, and how we might collaborate to move the field forward. 

Examples of current work are below to help you get up to speed with this emerging field, the different emphases within it, and the scholarly communities who participate. There’s a significant existing body of work outside the field of  analytics on the nature of reflection, and how to teach reflective writing (reviewed in the papers). The intriguing challenge is to translate that, with integrity, into the world of text analytics and automated feedback. As we noted during our call, it’s a really exciting nexus of the cognitive, social, affective, pedagogical, ethical, user experience and technical. 

Do get in touch if you want to join forces — everyone is most welcome, and we’re sure there must be more people out there doing this we haven’t met! 

Thanks to the kickoff videoconference participants for co-authoring this blog: Alyssa Wise (NYU), Andrew Gibson (QUT), Huda Alrashidi (Warwick), Ming Liu (UTS), Qiujie Li (NYU), Sameen Reza (NYU), Thomas Ullmann (OU), Yeonji Jung (NYU) 

New York University  (Lead: Alyssa Wise)

Cui, Y., Wise, A. F., & Allen, K. L. (2019). Developing Reflection Analytics for Health Professions Education: A Multi-dimensional Framework to Align Critical Concepts with Data Features. Computers in Human Behavior, 100, 305-324. https://doi.org/10.1016/j.chb.2019.02.019

Jung, Y. and Wise, A.F. (2020). How and How Well Do Students Reflect?: Multi-Dimensional Reflection Assessment in Health Professions Education. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK’20). ACM, New York, NY, USA, pp.595-604. https://doi.org/10.1145/3375462.3375528 [Preprint]

Wise, A. F., & Cui, Y. (2019,). Top Concept Networks of Professional Education Reflections. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK’19). ACM, New York, NY, USA pp. 260-264. https://dl.acm.org/doi/pdf/10.1145/3303772.3303840

Wise, A.F., Reza, S. & Han, R. J. (2020). Becoming a Dentist: Tracing Professional Identity Development through Mixed-Methods Data Mining of Student Reflections. Proceedings of ICLS’20: International Conference of the Learning Sciences. Nashville, TN: ISLS. [Preprint]

Queensland University of Technology (Lead: Andrew Gibson)

Work in progress with RWA and GoingOK: http://goingok.org

Willis, J., and Gibson, A (2020). The Emotional Work of Being an Assessor: A Reflective Writing Analytics Inquiry into Digital Self-assessment. In Fox, J., Alexander, C., Aspland,T.(Eds.) Teacher Education in Globalised Times. (Sringer).  https://doi.org/10.1007/978-981-15-4124-7 [pre-order]

Gibson, Andrew P. (2017) Reflective writing analytics and transepistemic abduction. PhD Thesis, Queensland University of Technology. https://doi.org/10.5204/thesis.eprints.106952 [Preprint]

Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing AnalyticsFor ActionableFeedback. Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver, BC, Canada. (ACM Press), pp.153-162. http://dx.doi.org/10.1145/3027385.3027436 [Preprint] [Replay]

The Open University & Warwick University (Lead: Thomas Ullmann)

Ullmann, T. D. (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257. https://doi.org/10.1007/s40593-019-00174-2 [Preprint]

Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing Automatically Detected Reflective Texts with Human Judgements. 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning. CEUR-WS.org. http://ceur-ws.org/Vol-931/paper8.pdf 

Try ReflectR – an online tool to classify sentences regarding reflection: http://qone.eu/reflectr  

Alrashidi, Huda; Ullmann, Thomas; Ghounaim, Samiah and Joy, Mike (2020). A Framework For Assessing Reflective Writing Produced Within the Context of Computer Science Education. In: Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20, 24/03/2020, Frankfurt, Germany). [Preprint]

University of Technology Sydney (Lead: Simon Buckingham Shum)

AcaWriter automated feedback tool: AcaWriter orientation for staff and students • Video intro to the reflective module and embedding in Pharmacy Masters program

A blog post on the challenge of sharing reflective writing datasets

Buckingham Shum, S., Á. Sándor, R. Goldsmith, R. Bass and M. McWilliams (2017). Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results. Journal of Learning Analytics, 4, (1), 58–84. https://doi.org/10.18608/jla.2017.41.5 (Open Access)

Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing AnalyticsFor ActionableFeedback. Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver, BC, Canada. (ACM Press), pp.153-162. http://dx.doi.org/10.1145/3027385.3027436 [Preprint] [Replay]

Liu, M., Buckingham Shum, S., Mantzourani, E.,Lucas, C. (2019). Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. In: Isotani S., Millán E., Ogan A., Hastings P., McLaren B., Luckin R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science, vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_19 [Preprint]

Buckingham Shum, S. and Lucas, C. (2020). Learning to Reflect on Challenging Experiences: An AI Mirroring Approach. Proceedings of ACM CHI 2020 Workshop on Detection and Design for Cognitive Biases in People and Computing Systems, April 25, 2020 (online). [Preprint]

Categories: Events, Papers & Talks, Projects
Tags: learning analytics, reflection, writing analytics