Theme Explorer: LLM-augmented Inductive Coding

In a previous post I shared work on developing a rigorous process for automating deductive coding with an LLM (GPT-4). Next, here’s a snapshot of where we’ve got to with LLM-augmented inductive coding, resulting in the Theme Explorer interactive web app.

This work comes out of the Australian Student Voices on AI in Higher Ed project, with the code development led by the brilliant Aneesha Bakharia at UQ, shaped through a multidisciplinary, cross-institutional team. We’ll present this next week at the LAK25 workshop From Data to Discovery: LLMs for Qualitative Analysis in Education. As the title and full-day program signal, we’re witnessing an explosion in interest in what it means to harness LLMs for qualitative research, education being our specific interest, but this is just one of many domains spanning arts and social sciences, as well as STEM.

Naturally, this extraordinary acceleration of coding, by what machines see in a text, is a development regarded with great scepticism and concern by some in the QDA community, so I hope that we can convene productive dialogues.

For me, as ever, the exciting opportunity is to “augment human intellect” (thanks Doug Engelbart) by enabling analyses that would otherwise be impractical (in terms of human resources) or impossible (beyond human capability). Hence our design  requirements were:

  • Requirement 1: To maintain the integrity of coded textual extracts: (i) verify against the source data that quotes are verbatim and not hallucinated, and (ii) verify that they are meaningfully classified under the assigned code.
  • Requirement 2: To maintain the transparency of the coding: (i) explain the rationale for each code, and (ii) trace every code, whatever level of abstraction, back to its source data.

This motivated a workflow:

…with Step 5 generating an interactive Sankey Flow Diagram:

…below which is an interface to support Reqt 2 traceability — selecting a top level category to explore displays:

  • its rationale and keywords
  • the themes from which it was derived
  • quotes from each student transcript (fictional student names):

We are working towards an open source release, meantime enjoy the the paper!

Aneesha Bakharia, Antonette Shibani, Lisa-Angelique Lim, Trish McCluskey and Simon Buckingham Shum (2025). From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models. Proceedings of LAK25 Workshop: From Data to Discovery: LLMs for Qualitative Analysis in Education (Dublin, IRE, 4 March 2025), 10 pages. https://ceur-ws.org [Open Access Eprint]

From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models

Aneesha Bakharia, Antonette Shibani, Lisa-Angelique Lim, Trish McCluskey, Simon Buckingham Shum

We present a novel workflow that leverages Large Language Models (LLMs) to advance qualitative analysis within Learning Analytics, addressing the limitations of existing approaches that fall short in providing theme labels, hierarchical categorization, and supporting evidence, creating a gap in effective sensemaking of learner-generated data. Our approach uses LLMs for inductive analysis from open text, enabling the extraction and description of themes with supporting quotes and hierarchical categories. This trustworthy workflow allows for researcher review and input at every stage, ensuring traceability and verification, key requirements for qualitative analysis. Applied to a focus group dataset on student perspectives on generative AI in higher education, our method demonstrates that LLMs are able to effectively extract quotes and provide labeled interpretable themes compared to traditional topic modeling algorithms. Our proposed workflow provides comprehensive insights into learner behaviors and experiences and offers educators an additional lens to understand and categorize student-generated data according to deeper learning constructs, which can facilitate richer and more actionable insights for Learning Analytics.

Grounded Prompt Engineering for researchers

This year has been one of rapid learning for us, regarding the use of large language models, not to power educational chatbots to promote deeper learning (also of great interest), but as textual analysis tools for deductive coding — a task that until recently was really beyond machines.

In this paper we document some progress in the research led by doctoral researcher Ram Ramanathan, co-supervised with Lisa Lim, with prompt engineering expertise from Naz Rezazadeh Mottaghi.

Ram’s PhD is focused on Belonging Analytics — the use of data, analytics and AI to understand university students’ sense of belonging. He’ll present this in a few weeks at the 15th International Learning Analytics and Knowledge Conference (LAK25)

Sriram Ramanathan, Lisa-Angelique Lim, Nazanin Rezazadeh Mottaghi, and Simon Buckingham Shum. 2025. When the Prompt becomes the Codebook: Grounded Prompt Engineering (GROPROE) and its application to Belonging Analytics. In LAK25: The 15th International Learning Analytics and Knowledge Conference (LAK 2025), March 03–07, 2025, Dublin, Ireland. (ACM, New York, NY, USA, 13 pages). https://doi.org/10.1145/3706468.3706564 [Eprint]

Abstract: With the emergence of generative AI, the field of Learning Analytics (LA) has increasingly embraced the use of Large Language Models (LLMs) to automate qualitative analysis. Deductive analysis requires theoretical or other conceptual grounding to inform coding. However, few studies detail the process of translating the literature into a codebook, and then into an effective LLM prompt. In this paper, we introduce Grounded Prompt Engineering (GROPROE) as a systematic process to develop a literature-grounded prompt for deductive analysis. We demonstrate our GROPROE process on a dataset of 860 written reflections, coding for students’ affective engagement and sense of belonging. To evaluate the quality of the coding we demonstrate substantial human/LLM Inter-Annotator Reliability (IAR). To evaluate the consistency of LLM coding, a subset of the data was analysed 60 times using the LLM Quotient showing how this stabilized for most codes. We discuss the dynamics of human-AI interaction when following GROPROE, foregrounding how the prompt took over as the iteratively revised codebook, and how the LLM provoked codebook revision. The contributions to the LA field are threefold: (i) GROPROE as a systematic prompt-design process for deductive coding grounded in literature, (ii) a detailed worked example showing its application to Belonging Analytics, and (iii) implications for human-AI interaction in automated deductive analysis.

 

2 years since ChatGPT launched: edu reflections + future glimpses

Coming up for 2 years since OpenAI launched ChatGPT, it’s worth a pause for thought on the rollercoaster in education. In this update I note some signals of where we may be heading, and pose some provocations on where we should be going.

This is yesterday evening’s guest lecture to the UTS Graduate Certificate in Learning Design and related Crunch: Learning Analytics for Performance Improvement microcredential.  With thanks to the leads John Vulic & Lisa Lim for inviting me to speak with the students.

Reflections on the GenAI Rollercoaster: Glimpses into Our Future [slides]

Abstract: Generative artificial intelligence is scoring high on the university Richter scale, with aftershocks accompanying each upgrade. We’re witnessing the largest rollout of AI in educational history, powered by unprecedented tech investment, extraordinary engineering advances and ubiquitous marketing. TEQSA considers this not only a profound disruption to higher education, but also a transformational opportunity to improve assessment practice and deepen student learning, with every institution now developing its action plan in response to Assessment Reform for Age of AI. I’ll share some reflections on the rollercoaster so far, and show some examples of GenAI that I find provocative for imagining the future.

Your comments welcomed via LinkedIn…