Using LLMs for psychological research

The idea was formed

At the time GPT-3.5 became available I was studying fullstack development. I was already somewhat proficient due to self-studies, but I was on parental leave, had the time, thought it could be fun, and some formal merits are good, right? The courses were all remote, and we students came to primarily communicate via discord. Messages per week in the channel ranged from 560-2200 during the first months, so it was quite active.

Early days in the LLM hype era, the capabilities of the models was not evident, and definitely not proven. I was quite fascinated about the possibilities they could enable.

In psychology research, qualitative research is sometimes used, interviews, transcriptions, observations etc. Sometimes you might do labeling and analysis of the content, to be used in a more quantitative way. The idea occurred to me: use LLMs to label/analyze chat messages, with regards to more nuanced psychological measures than sentiment for example. We might with the use of analyzed chat messages get real actionable information about student well-being and optimize learning environment.

Stress, Learning and Support

This section will be a compressed summary on how stress, learning and what psychological factors that relates. For those already knowledgeable about these relationships, skip ahead.

In short, we’ll use two established theories to help us understand. First, being Vygotsky’s theory of zone of proximal development. It states that learning is proceeding at an optimal pace when the demands are just above the comfort level. The second theory is Karasek’s demand-control-support model. Stress and well-being depend on the balance of demand, amount of percieved control of ones situation, and experienced support. High demand with ample control is fine, but high demands with low control leads to stress. Social support can act as a buffer.

So, in this specific context, if the ambition of the educational program is to have students learn and integrate, the demands should be set at an appropriate level, promoting the sense of control and providing support when needed. Sounds easy right?

Method

So the idea was: can we use zero shot LLMs (ie no fine-tuning, no specifically trained models, but rather commercially available ones) to analyze discord messages to determine experienced demand, control and support, as well as measures we’d expect to be dependent of those variables, like sentiment?

It would be possible, and one could argue interesting, to look at individual persons in the chat, and follow them longitudinally. But ethics is worth considering, so I opted to remove user identity from each message, and simply send the message content. I also chose to have the analysis done per week, so all messages for each week gets aggregated under one analysis.

After extracting the messages from Discord, the data was cleaned, PII removed, clustered by week, and sent to a LLM for analysis (initially GPT-3.5-turbo, later GPT-4o). The prompt was fairly simple:

Analyze the text and assign scores (0-100) for 'Demand', 'Control', 'Support', 'Sentiment', and 'Engagement' based on the Demand Control Support Model and sentiment analysis. Do your best attempt at interpreting, no excuses! Output the scores in JSON format. No other output is allowed.

Result

The rest was a matter of statistical analysis. Please read the paper for more details, but in short:

• There was significant correlation in the expected direction.
• Multiple regression analysis showed that the model (Karasek’s) was doing a good job at explaining differences in sentiment (R-squared=0.788).

So, what to take from it?

Several things:

• Convergent validity of the findings with the demand-control-support model leads us to believe that the LLM is doing a decent enough job of analyzing large amounts of chat data, most of it in swedish.
• Monitoring chat like this can give an aggregate measure of information pertaining to psychological variables of interest for educational settings.
• Use of LLMs in research have a way to go in regards of being validated, but in comparison to manual labeling and rating, which has its own issues with validity, its seemingly a valid tool which opens up new research possibilities.
• Monitoring via chat holds the potential for some frightening situations, the possibility existed beforehand of course, but with the enormous cost decreases in comparison to doing it manually, its significantly more likely to happen in the future.

Looking back

It was a fun project, at a certain point in time where using LLMs like this was really quite new. For me personally it offered a glimpse into what could possibly be done at a reasonable cost. It would have been totally possible to do the same study without LLMs, but enormously costly considering the amount of messages. I think I haven’t, and possibly, many others too, fully comprehended the amount of insight that could be gotten on a persons inner functioning by observation and monitoring, at least not in a fully automated way.

Related posts

• Refactoring a retrieval pipeline through evaluation - Another dive into using machine learning for practical applications, this time optimizing retrieval in my knowledge management system.