Survey insights on Steroids with ChatGPT

My biggest challenge

Most of the user research I do will, at some stage, incorporate some version of a very specific question. Works across more or less any industry, any audience. It goes:

1. What’s your biggest challenge as a social media creator?

2. What’s your biggest marketing challenge right now?

3. What’s your biggest challenge right now as a talent manager?

You get the point. Make sure you ask the respondent to be as detailed or specific as possible.

The point? To pinpoint, with mind-reading-like accuracy, what kind of challenges your audience is dealing with. And perhaps most importantly – what exact words, tone of voice, and vocabulary they use to voice those frustrations.

Knowing the answer to this question is half the battle in sales and marketing.

I’d like to gather at least 300-400 qualified responses, but the amount you’ll need will ultimately come down to who your audience is and how much time or money you’ve got (it’s not always practical or necessary to collect data from a very large sample set.).

This calculator should be able to point you in the right direction for now.

Now,  I’ve been asking this question for years across various businesses and projects, and the actual data can be somewhat… overwhelming.

I mean, look at this:

What would typically follow was hours, or rather days of work, labeling each response in a separate column based on common emerging themes.

Sure– “I never have time to grow my business,” “manage my time,” and “finding time” could easily be labeled “Time management”.

But what do you make of “Steady work and ongoing partnerships instead of one-off projects to have more predictable income” or “Pushing past burnout and overwhelm! The landscape has changed drastically, and I feel like I can't keep up.”

That’s complicated. Takes more time and certainly more brainpower. Also, the longer the responses, the more valuable the insights, so you can’t afford to mess up.

It’s always been a “No pain, no gain” kind of situation for me (a decent coping mechanism, to be fair.).

No matter how painful or time-consuming, I'd do the work and would typically end up in a situation where I’d have an actionable insight or two. That’s great leverage. Plus, it’ll teach me how to talk like my audience. But at what cost?

The work is tricky to outsource, too. You need a baseline understanding of the target audience and context to categorize and label responses accurately. (Perhaps why there are an estimated 48,924 market research businesses in the U.S., employing 133,913 people. ).

So here we are… and the answer to the question I've been asking others is now my biggest challenge. How ironic.

Enter OpenAI

… and ChatGPT, specifically.

Rather than those screenshots you’ve seen on Twitter and LinkedIn, I’ve gone through the trouble of including the actual prompt in plain text, step by step. Yeah, you’re welcome!

Let’s go step-by-step and start from scratch.

Prompt: I recently ran a survey and asked my audience what their single biggest challenge is. I have hundreds of open-ended text-based answers. There's one answer per line, and I have them in a spreadsheet. Can you analyze the answers for me, look for common patterns, themes that emerge and give me an executive summary of what you find? Confirm and I'll send the data set.Answer: Yes, I can certainly help with that. Before I begin, do you have any specific questions or areas of focus that you would like me to address in the analysis?

I paste the answers, a total of 419 lines of information, and am immediately met with the following error message:

Inconvenient but easy to fix. Through trial and error I’ve discovered, at least as of Jan 4, 2023, that ChatGPT is limited to approximately 15,000 characters per prompt (likely 4,000 tokens.).

So I turn to my copilot for a different kind of question.

Prompt: I know, from experience, that your character limit is around 15,000 characters per prompt. How do I count characters in google sheets?Answer: To count the number of characters in a range of cells, you can use the SUMPRODUCT function in combination with the LEN function. For example, to count the number of characters in a range of cells from A2 to A, you can use the formula =SUMPRODUCT(LEN(A2:A)). This will add up the number of characters in each cell in the range.

Turns out that I’m looking at approximately 17,000 characters across my 419 lines of data. Split my prompt in two and we should be good.

Should the data set be bigger I’d probably save the responses in a text file and use the split command in the Terminal on my Mac.

> split -C 15000 file.txt

This will create a series of smaller files with names like xaa, xab, xac, etc. Each of these files will contain at most 15,000 characters, and the original file will be split into as many of these smaller files as needed to ensure that no single line is cut off in the middle. (Disclosure: I didn’t write this paragraph. Or did I?)

At this stage, I opt for overcommunication to ensure that my copilot and I are on the same page.

Prompt: Okay. Because of your limit, I'll need to give you the survey answers across two separate prompts. I'll label them "Part 1" and "Part 2" etc. After each prompt, write "Got it" to confirm that you have received the answers, and await my instructions before you do any analysis or commentary. Got it?Answer: Got it. I will wait for your instructions before doing any analysis or commentary. Please go ahead and send the first part of the survey answers.

I proceed to paste my two text files as per my prior instructions.

My wingman seems to dig my style of (over)communication and play along. A bit annoying, and I’m certainly to blame.

Answer: Got it. To start the analysis, I will first need to clean and preprocess the text data. This will involve removing any duplicates, converting all the text to lowercase, and removing any punctuation or special characters.Next, I will identify the most common words and phrases in the survey answers. This will give us a good idea of the main challenges and issues that are being mentioned most frequently by the respondents.To get a more detailed understanding of the themes and patterns in the data, I will use techniques such as word frequency analysis and natural language processing. This will involve applying techniques such as tokenization, stemming, and lemmatization to the text data, in order to identify the main themes and topics that are being mentioned.Once I have identified the main themes and patterns in the data, I will provide you with an executive summary of my findings, including a list of the most common challenges and issues mentioned by the respondents, and any other insights that emerge from the data.Please let me know if you have any specific questions or areas of focus that you would like me to address in the analysis.

I’d prefer a more concise response, but this is helpful if someone was ever to question the process and technique. Plus, two-stage prompting and chain of thought (CoT) prompting (“let’s think step-by-step”) has been proven to increase accuracy.

Now for the exciting part (and saving days of painful work.):

Prompt: Thank you. Let's start with a top 10 list of the biggest problems.Answer: Here is a list of the ten most common challenges and issues mentioned in the survey answers:Time management: Many respondents mentioned that they struggle with finding enough time to do everything that they need to do, whether it's creating content, finding clients, or managing their business.Finding clients or brand partnerships: A number of respondents mentioned that they struggle with finding new clients or brand partnerships, or with getting enough paid collaborations.Consistent income: Many respondents mentioned that their income is inconsistent, and that they struggle with finding ways to make a stable living as a content creator or influencer.Engagement on social media: Several respondents mentioned that they struggle with getting enough engagement on their social media accounts, whether it's likes, comments, or followers.Staying motivated: A number of respondents mentioned that they struggle with staying motivated and finding the energy to continue creating content and growing their business.Productivity: Some respondents mentioned that they struggle with being productive and getting things done, whether it's due to time management issues or other challenges.Staying consistent: Several respondents mentioned that they struggle with staying consistent in their content creation and business efforts.Finding paid collaborations: Some respondents mentioned that they struggle with finding paid collaborations with brands, and that they often have to rely on gifted product or commission-based deals.Balancing life and work: A number of respondents mentioned that they struggle with balancing their personal life and work as a content creator or influencer.Mental health: Several respondents mentioned that they struggle with mental health issues, whether it's due to the demands of the job or other factors.I hope this helps! Let me know if you have any other questions or would like more detailed information on any of these challenges.

Look at that. From 3 days of work to 30 seconds. Let’s go further.

Prompt: Could we group or combine any of these categories and narrow down the list to the five most important areas? Answer: Sure, here is a revised list of the 5 most common challenges and issues mentioned in the survey answers, along with a brief description of each and the percentage of respondents who mentioned each issue:Time management and productivity: Many respondents mentioned that they struggle with finding enough time to do everything that they need to do, whether it's creating content, finding clients, or managing their business. Some also mentioned that they struggle with being productive and getting things done.Finding clients or brand partnerships: A number of respondents mentioned that they struggle with finding new clients or brand partnerships, or with getting enough paid collaborations.Consistent income: Many respondents mentioned that their income is inconsistent, and that they struggle with finding ways to make a stable living as a content creator or influencer.Engagement and growth on social media: Several respondents mentioned that they struggle with getting enough engagement on their social media accounts, whether it's likes, comments, or followers. They also mentioned that they struggle with growing their audience and reaching a wider audience on social media.Mental health and well-being: A number of respondents mentioned that they struggle with mental health issues, whether it's due to the demands of the job or other factors. Some also mentioned that they struggle with finding time for themselves and taking care of their overall well-being.I hope this revised list provides a clearer picture of the most common challenges and issues mentioned by the respondents, along with the percentage of respondents who mentioned each issue. Let me know if you have any other questions or would like more information on any of these topics.

That’s incredible. From 419 rows of unstructured data to insights in… seconds.

Are these insights 100.00% accurate? No, but neither are average-intelligent humans decoding survey answers or marketing research manually.

From here, we could either go into each area and explore them further or expand on some of the qualitative aspects of your marketing research and ask questions such as:

Example Prompt: I'd like to get a better sense of how our users talk. Can you provide additional insights across these areas along with a few quotes or full responses that can help capture the tone of voice and vocabulary they use to describe each challenge?

Example Prompt: Can you provide additional commentary, analysis and insights on on the topic of mental health and well-being. Please include quotes and the most notable highlights/responses to provide additional context. I might decide to do further follow-up questions with these individual respondents, so you should also include a few questions I should ask them to uncover even more relevant insights.

Example Prompt: Can you write an executive summary of the survey for our leadership team?

You get the idea. Here’s another fun example prompt, along with the actual answer.

Prompt: Can you help write a persona based on these insights? You can give the persona a name, place they live and some additional fictional traits that help the reader relate to their situation.

Answer:Name: Rachel Location: Los Angeles, CA Rachel is a 25-year-old content creator and influencer based in Los Angeles. She has been working in the industry for a few years and has built up a decent following on social media, but she struggles with finding enough time to do everything that she needs to do. Rachel is always on the lookout for new clients and brand partnerships, but it can be tough to stand out in a crowded market. She also struggles with inconsistent income and finding ways to make a stable living as a content creator. Rachel is constantly trying to grow her audience and get more engagement on social media, but it can be frustrating when her posts don't get as much attention as she thinks they deserve.In addition to these professional challenges, Rachel also struggles with mental health issues and finding time for self-care in the midst of a busy schedule. She tries to stay positive and motivated, but it's not always easy.

Poor Rachel. So many struggles.

We’re barely scratching the surface.

Admittedly, the persona in our prior example ain’t great. It’s one-sided, focused only on the challenging aspects of our target audience. It doesn’t at all paint a full picture of who the user actually is.

To solve this, we’d have to ask our audience additional questions (common in marketing research, of course), and repeat the steps we’ve taken so far.

Here are a few sample questions:

1. What are you most excited about for next year?

2. What are your goals as a social media creator?

3. If you hired a personal assistant, what is the first thing you’d ask them to do?

4. Ask a smaller subset of your users to describe what a day in their life typically looks like. 10-20 answers will get you quite far.

Proceed, and you might end up with something like this:

Not a fan of personas? No problem. Here’s the data structured in a table:

What’s next

It’s common in market research to conduct qualitative interviews with a subset of survey respondents to uncover additional insights. Here’s another example:

Prompt: Can you create a list with follow-up survey questions specifically for time management and productivity? These should be structured to allow for a recorded interview that we will conduct over zoom with a smaller subset of our users. The interview should be about 30 minutes long.

Answer: Sure! Here is a list with follow-up survey questions specifically for time management and productivity:How do you currently manage your time as a content creator?What are your biggest time management challenges?How do you prioritize tasks and manage your workload?Have you tried any time management techniques or tools (e.g. calendars, to-do lists, time blocking)? If so, which ones have you found helpful?How do you balance the need to be productive with the need to take breaks and rest?How do you stay motivated and focused when working on content creation tasks?What strategies do you use to stay on top of deadlines and avoid procrastination?How do you manage your energy levels and avoid burnout?Have you faced any specific time management challenges during the pandemic (e.g. working from home, balancing personal and professional responsibilities)? If so, how have you addressed them?How do you think your time management skills could be improved, and what steps are you taking to make that happen?I hope these questions are helpful! Let me know if you have any other questions or would like more information.

These questions are not perfect, but they make for a pretty good starting point.

And as far as the qualitative interview process goes, should you be heading down that path,  I’d probably record the interview, transcribe the text, and head right back to where we started to classify, analyze and extract relevant insights from each question.

– Aron

p.s. if you enjoyed this post, consider subscribing and sharing with others.