Competitive Content Intelligence: Learning from the Competition

Spying on Your Competitor's Content


Content marketing failure is common. And heartbreaking.

First comes hours of evangelizing the need for great content to your colleagues, clients, and boss. Then you finally get permission to make something. Next, you spend hours painstakingly researching, writing, and creating images. And, with great hope and trepidation, you hit publish…

And nothing happens.

I’ve been there.  We all have. But don’t worry – there’s hope for better content marketing, with just a little analysis.

One way you can improve your content marketing success rate is by conducting a competitive content analysis before starting to create content.  That way, you can learn what works with your audience before you ever put pen to paper.

Here’s a process we’ve found helpful in our own content marketing efforts:

Define Your Goals

Why are you creating content? Are you trying to obtain inbound links, social shares, or become a thought leader? Are you trying to move prospects along in the buying cycle, or convert existing customers to evangelists?

Once you define your goals, you can begin analyzing your competition for how they meet those goals.

In this post, you’ll learn how to do competitive analysis based on inbound links and social shares for SEO, social marketing, and audience development goals.

Example: Let’s say I’m starting a new site that sells Coffee of the Month Club subscriptions.  My goal is to rank highly for queries like “coffee of the month club” and “coffee of the month club reviews”.

Accordingly, I’ll be looking to create content that attracts links and social shares to improve my domain authority.

Gather a List of Competitors

Remember, your SEO competitors aren’t necessarily your business competitors.  While your direct competitors may be other companies like yours, your SEO and social competitors are any site getting attention in your SERPs and social feeds.  This means affiliates, informational sites, news sites, and even Wikipedia can be your competitor.

Example:  For my coffee site, I’ll be looking at other coffee blogs.  While none of these sites focus on coffee subscriptions, there are many popular coffee blogs with lots of search and social visibility.  Three of these are:

(If I was working on this project, I would include more sites. Keep in mind this is just an example.)

Collect Content URLs

Grab your competitor’s content and assess its effectiveness at gathering links and shares.

The easiest way to do this is by URL.  (This makes some assumptions that your competitors have implemented some basic SEO best practices around 1 permanent URL per piece of content.  If they haven’t, this still works, but you’ll need to aggregate the multiple URLs together in the data analysis stage.)  Start by grabbing a list of content-bearing URLs from your set of competitors.

Some of the tools that can do this are:

  • Screaming Frog (My favorite search spider, not free but definitely worth it.)
  • Xenu (Free. Only runs on Windows.)
  • Your competitor’s .XML sitemaps (If they’ve configured them correctly.)

Analyze Social & Link Popularity of Competitor URLs

Now that you have a list of your competitor’s content URLs, you can evaluate them based on their social and link popularity.

To do that, you’ll want to put all of your competitor’s URLs into a spreadsheet or database, and then find their associated link popularity and social metrics.

You can grab site-wide URL link popularity data from your favorite source of link data (I like Open Site Explorer and Majestic SEO), then use a database merge or a VLOOKUP in Excel to get it all into a table.

Many services gather social popularity data for URLs.  You can use the API for individual services (Tom Critchlow put together a great post about how to do this in Google Docs), or you can use a service that aggregates them like SharedCount.

There are several tools that automatically gather URLs and take social metrics in one step:


I’ve used Social Crawlytics to crawl each of the sites and assess their social popularity.  Next I gathered link popularity data from MajesticSEO and used VLOOKUP to get them all into one table.  My end result looks like this:

ineedcoffee shares and links by URL


Scale the Data to Enable Multi-Site Comparisons

Now that we have data on the popularity of individual competitors, we want to be able to look across the competitive sites to analyze trends.  (If one type of article is incredibly popular across a variety of our competitors’ sites, it represents niche demand for that kind of content.)

To do this, we’ll make a new metric in our spreadsheet that can be compared across sites.  Because it’s not fair to compare the number of social shares from older sites with newer sites and sites with really large social followings with smaller social followings, we’ll scale the popularity proportionally and then compare it across sites.

To do this, we’ll create a ‘Relative Popularity Metric’ for each site.We’ll make this metric by finding the most shared piece of content in each channel, and then scaling each piece of content as % shared of that content.

If the most popular article on a blog got 250 tweets, we would assign that a ‘1’ for Twitter popularity.  If an article only got 100 Tweets, that would get a ‘.4’ rating.

(Note: There are some weaknesses to this approach.  If a site has very few social shares at all (ie the average post gets between 1 and 5), you won’t have enough data to separate signal from noise.  You could also do this statistically, which could result in a different set of insights.)

Example: I’ve added scaled metrics to my data from





Put It All Together

Now that you have URL, link popularity, social popularity, and an aggregate metric for each competitor, you can combine these into one master spreadsheet.

Conducting Competitive Analysis

You are now in a position to make apples-to-apples comparisons across your competitor’s content.  Start by sorting by relative popularity on whatever social networks matter the most to your audience. ( If you’re in a B2B market, it’s probably LinkedIn and Twitter. If you’re working in B2C markets, it could be Facebook and Pinterest.)

Now you can see, across all of your competitors, which pieces of content are the most popular with your audience.

When I do this for my example sites, it looks something like this:

aggregate social shares across 3 sites

When I do this exercise, I dig down into the most popular URLs and look at a few different factors:

  • What forms of content are the most popular? Are the most popular posts numbered lists?  Conventional blog posts? Videos? How-To Articles? Photos? Graphs? Memes?
  • What tone is the most popular? Authoritative or conversational? Do people share long, detailed resources, or short, breezy posts?
  • What content topics are the most popular? Are there any topics that are consistent winners across different sites?
  • What sort of headlines are the most successful posts?  Are they more in the classical, direct response style (aka “1 Weird Tip to…”), or more thoughtful and metaphorical?
  • What kind of emotional and persuasive triggers are used within the content? How do they create believability – through data, social proof, or citing others?
  • Who is sharing this content? You can AJ Kohn’s Google+ Ripple Tool or Topsy to identify the influencers sharing competitor’s content.

Now that you understand what kind of content types, topics, and emotional appeals are popular, you can create a more effective content strategy.

Be Different, But Use Elements of What’s Popular

We have always been shameless about stealing great ideas. – Steve Jobs

Now here’s the hard part: Use what you’ve learned in the previous exercise to build a content strategy that’s differentiated yet informed by what your competitors are doing.

When I do this exercise, I like to make three columns: motivation for sharing/linking, form, and topic.  Then I mix and match.


  •  I notice a couple of articles about iced coffee and cold-brewed coffee in my top 100 URLs.  I’ll definitely make sure I create content on that topic.
  • Lists, like numbered lists of coffee makers and holiday gift lists, are popular.  I’ll definitely include some list-based content in my plan.  Perhaps a list post about how to make cold-brewed coffee.
  • I see a couple of articles about coffee filters and their efficacy in my list.  I’ll make sure I hit these up as well.
  • Starbucks seems to draw emotional reactions on both sides.  There’s some potential to linkbait around controversy here, so I’ll make sure to play to this hook.
  • Great coffee at low prices seems to be a perennially popular topic.  I’ll make sure I hit this as well.

Good Luck, and thanks for reading.

(Photo Credit)

Do you have any awesome competitive content techniques? Please share them in the comments…



  • I really like the idea of sorting social popularity and reach of content by business type, i.e. B2B vs. B2C.

    Also appreciate your quick wrap up of the opportunities that stick out to you.


  • Great comprehensive way to come up with content. We are looking for ways to scale our content production for our clients so we can do content ideation every few months rather than for each post or set of posts we are doing. This process will fit in nicely to what we are putting together. Awesome resources!

  • Thanks Jon! Glad you enjoyed it.
    It’s worth noting that you could add comments to this method, if you found comments to be a worthwhile goal. You could write an XPath script and grab number of comments in Google Docs.

2 pingbacks