Last Updated: 27 May 2020
Recently, numerous influencers (including the likes of Brian Dean and Larry Kim) have espoused the importance of researching your audience in order to understand their needs, wants and desires. This has increasingly become the case in the age of Rankbrain and semantic search, With Google focusing far more on answering users’ questions.
So, how do we find out what questions people in our vertical are asking?
It turns out that the answer is both relatively easy and surprisingly scalable, by using nothing more than a crawling tool and some basic googling.
Ready to get started?
Tools you will need:
- Google!
- Beamusup/Screaming Frog, or any equivalent site crawling software
- Excel
What we want to do first is find the most popular, authoritative forums in our niche. For the purposes of this tutorial we’ll be working in the gardening niche, but this can be applied to any niche market or vertical.
The reasons we’re looking for forums specifically are as follows:
1. Forums are full of your vertical’s consumers and enthusiasts.
2. Forums tend to be where lots of questions get asked, and these often reveal what your audience desires, has problems with, and fears.
3. They tend to be well-organised into subsections, which helps us perform research more easily, especially using a web crawler.
4. Few marketers use them for research, particularly in terms of finding keywords, so untapped keywords, content gaps and and content opportunities may present themselves more easily.
First, we need to find the forums and forum subsections that we’re interested in. As we’re working on the premise of doing research for a gardening
supplies website we’d go through the below steps (remember, we’re looking for high authority, popular and high traffic forums, so basic keyword+forum searches will do.)
In this instance the problem or tools sections of this particular forum would work for conducting market research. The forum has lots of posts and threads, so we can assume it’s got plenty of users.
Step 1:
First, either start up your crawler of choice, or install one if you haven’t already (we’ll be using Beamusup)
The process we’re detailing is a little different for BUU than it is for Screaming Frog, but the principles are essentially the same. Xenu link Sleuth is another crawler option here.
[showmodule id=”3238″]
Step 2:
Let’s copy the URL of the ‘problem solving’ section with a view to researching further into what manner of content marketing we could base around the
forum’s discussions.
We want to go to Configuration first and change the number of threads downwards, so we aren’t putting in too many server requests at once.
Step 3:
Now we just have to wait a little while for the crawl to find some pages for us. We won’t be going through the whole forum, due to the number of pages on it-a few thousand pages should be enough for now.
Step 4:
Once we’ve completed a large enough crawl, we’ll be exporting it into Excel.
Typically, forum URLs and page titles are auto-populated with the title of the post, which is a happy coincidence as far as filtering is concerned.
Once we have the Page URL and title in Excel, we simply need to play around a little bit. Delete everything in the sheet except for the title, as we won’t need anything else.
Step 5:
We have a few options here – either filter on A-Z and browse through the sheet looking for the typical who, what,when, where, why, how, etc, OR create a named range containing key question-related terms and run a formula through Excel. Exceljet has a more detailed step-by-step process here.
https://exceljet.net/formula/cell-contains-one-of-many-things
By using this technique we can find a big list of top 10 ideas that users have put on the site:
There are also numerous threads beginning with how, why, where, what etc. that give us plenty of ideas:
Hey presto! We now have a substantial list of topics which we can look into researching for content marketing purposes. The main advantage of this
technique over using Google or manually going onto the likes of Quora, Reddit etc. is that one is able to directly research a large, targeted section of
the desired target audience at scale, easily amassing a substantial database of topics in the process.
