AQR Hackathon Talk: how to get the best from analysis using AI tools

Last week I spoke at the AQR's ’s second AI Hackathon having been asked to talk about getting the most out of GenAI tools for qualitative analysis.

View my slides here

Following Mike Stevens' talk that mapped what's changed in the AI for research space in 2025, I began putting Generative-AI capabilities for qualitative data analysis (QDA) in context by commenting on “traditional-AI” tools, such as topic modelling, sentiment analysis, automatic document classification, and so on that have been around for a long time. For more on the historical context see here

The newer capabilities have led to increasing interest in and discussion about the role of technology in the practice of QDA. In the almost 30 years I've been engaged in the field, I've not witnessed as much discussion about the implications as were seeing now. There is a lot of excitement and hype (not all of it accurate) and also a lot of scepticism and resistance from various quarters about what Generative-AI means for the profession, implications that vary across contexts and sectors.

Tools for QDA

As a transition into the practical aspects of the hackathon that came after my talk, I incorporated a little bit of audience participation, starting off asking the audience what tools they'd used for QDA before Generative-AI came into the space. The answers reflect what I usually hear in response to this question. Most had used manual tools (post it notes, coloured pens, scissors, bits of ribbons and stuff all over the place), some had used general-purpose digital tools (e.g. Excel and Word), and a few had used bespoke qualitative analysis tools, such as NVivo.

This discussion led to reflecting on the fact that the kinds of tools we now have available for QDA, as a result of Generative-AI, are very different in terms of what they can actually do.

And so my talk was partly about engaging with how we can make the most of these new tools. It’s important not to use them just because they are there, but because they add something of value. Because researchers who use these tools need to ensure they’re not only doing so appropriately, but also getting the maximum potential out of them.

This is important whatever the context. For example, in industry it's important in terms of how researchers communicate to clients about the role and use of GenAI. For government researchers it's important in ensuring research outcomes upon which policy is based, is robust. In academic contexts, its important to ensure the next generation of researchers are adequately and appropriately equipped.

None of this is actually anything new, it's just that what these tools can do is so different from what has previously been possible: the level and type of assistance Generative-AI tools for qualitative analysis  looks very, very different. So what we need to think about is how to leverage that in a way that improves analysis.

The importance of strategies driving tactics

Over the past 30 years, from my experiences using, teaching and researching computer-assisted qualitative analysis, the one thing that has struck me most is the importance of strategies driving tactics.

To illustrate the point, I used one of my favourite analogies, about choosing an appropriate mode of transport (tactic) when traveling from London to Paris, depending on whether the aim is to get there as quickly as possible or with the least environmental impact (strategy). This is always a fun discussion, with Londoners usually expressing differing opinions about whether the Eurostar is quicker than flying.

Translating this principle into the context of Qual analysis, Generative-AI is just one of many tactics we can choose between to operationalise our analytic strategies. Qual projects are not all the same, therefore neither are the tools that are appropriate. There is no one-size-fits-all.

For this talk I was asked to speak about how to get the most out of Generative-AI and at the conceptual level this is the one most important piece of advice I can give. What constitutes the  appropriate tactics is not always the same.

Getting practical

At the practical level, in terms of what we actually do with the tactics we choose, we also have to make appropriate choices. At the hackathon there were 3 platforms: Bolt , CoLoop, and Discover.ai, each containing a variety of tools, not all of which are appropriate for all tasks. The mindset of strategies driving tactics allows tools to be chosen consciously and appropriately. Researchers most often struggle with tool use when they don’t have a clear plan, when they somehow – whether consciously or not – expect the tools to help them work out what to do. 

It’s not necessary to be a slave to a particular tool, rather combining tools, mixing and matching their use throughout projects is often most appropriate. That might mean using different digital tools – bespoke and general-purpose - for different tasks. And there’s still a place for the good old fashioned highlighter pen. The range of options these days means we have an incredible amount of flexibility to design an appropriate toolbox, to combine tools in dynamic, creative and powerful ways.  

We then had a chat about pain points in using manual methods and general-purpose tools. In other words, teasing out what researchers are looking to AI to help with. Top of the list, unsurprisingly, amongst the hackathon audience was time. Most want to save time. Also mentioned were being able to access and combine lots of different types of data, mitigating personal biases, organising and managing large amounts of data, and reducing costs. And these different motivations affect what appropriate use of tools looks like. 

Prompting with purpose

The final part of my talk focused on prompting as one of the key practical differences in how researchers can go about doing qualitative analysis using generative-AI. Although all analysis is about asking questions about data, the way we can go about doing so, is expanding. When coding qualitative data we question what is interesting and meaningful within it, and capture our answers in the form of codes. The subsequent exploration of patterns are relationships between codes is also a form of questioning data.

But although we are very used to coming to data with questions, the form those questions – and the responses to them – take, can be very different when using Generative-AI tools. Most of the hackathon delegates had used chatbots like ChatGPT etc. and so know well that how we ask questions when using such tools matters.  And so we finished up having a bit of fun thinking about how we might ask better questions of our data using those tools.

We began thinking about general advice : be specific, provide context, give examples, and layered that up to think more deeply about what that actually means in practice.

This gave me the opportunity to use another of my favourite analogies. Originally borrowed from Professor Karen Andes who uses it to teach about the properties and dimensions of themes in qualitative analysis, I have adapted it to bring to life the importance of understanding different types of prompt, and how to construct Chain of Thought prompts for qualitative analysis.

If you tell a teenager to “do the washing” (an example of a zero shot prompt), you’re not likely to achieve the desired result, because everything might get shoved in together, all the darks and lights, the cottons and woollens, and be washed on an inappropriate cycle.

If you instructed said teenager to “sort the washing, then wash each pile separately” (a few shot prompt) you might get better results, but probably examples of what the piles should consist of would be more productive.

Much more detailed instructions, with clarity around the specific steps and their sequence (Chain of Thought prompt) is likely to be even more successful. And perhaps intervening along the way with additional instructions, might be appropriate.

Fellow parents of teenagers in the room seemed to resonate with the laundry example, and we had some fun suggesting alternative ways to get our teenagers to do the laundry effectively.

But the point of course, was to reflect on what this means for doing qualitative analysis. So I finished up showing a few examples of the different types of prompts, illustrating the nature and extent of responses in relation to different prompts.

All in all it was a great day. After some great questions and discussions arising from my talk, we progressed onto the ‘hackathon’ part of the day, getting to experiment with the three platforms in groups, and discuss the possible implications for our work.

A thoroughly enjoyable afternoon, where I got to share some of my thoughts, hear about what others are doing in this space, and to meet some old friends, and make a bunch of new connections.     

If you're interested in learning more, check out our forthcoming events, or get in touch for us to design a bespoke course.