Ending the Guesswork Behind Ranking Difficulty & Proving what Factors REALLY Matter

At SpyFu, we’re good at quantifying the value and opportunity of SEO work. We tell you how many clicks you will get by moving into a certain position and how much those clicks worth. It’s all pretty sophisticated. But knowing which keywords to pursue is only half of the equation; you also need to know how hard it will be to rank on those keywords. And as it stands, figuring that part out is up to you.

Until now.

It’s a ritual every SEO regularly performs: looking through the actual SERP’s to get a sense of how hard it will be to rank on a given term. If you’ve been in the business long enough, getting a read on difficultly has become second nature, amounting to instinct or even simple common sense.

A quick scan of the top 10 or 20 results might communicate that familiar giants like Wikipedia or Amazon gobble up the first five to seven spots. A little more digging reveals that the rest of the ranking domains are already well-established in the niche. “Hmm, this doesn’t look good…”

So you abandon your initial list and start searching for other keywords in the same tier of traffic that the competing domains might have missed. Or maybe you go straight to deeper, long-tail terms.

But how can you know for sure? If determining difficultly is half the battle, you could be leaving a lot of opportunity on the table by relying on a subjective “best guess.” To me, this type of fuzzy logic cries out for the making of a great algorithm. An opportunity to convert common sense into code.

So, I set out on a mission: figure out how to quantify ranking difficulty. That way, our users can stop relying on instinct to prioritize their efforts. Instead, they can use hard data to easily create a real-time, low-hanging-fruit report.

SpyFu’s ‘Keyword Ranking Difficulty’

What we came up with is pretty amazing.

As announced in a recent blog post and expanded upon in a separate webcast, SpyFu has integrated a powerful new metric into our term pages, ‘Keyword Ranking Difficulty.’ By comparing a keyword’s difficulty score to others, you get a solid measure by which to gauge difficulty. No interpretation or guess-work required.

I’m putting together a separate post that breaks out all the nitty-gritty details on what goes into deriving ‘Ranking Difficulty’ – everything from the methodology I employed in my research to the specific data sets. But for those of you who are data-driven, here’s a rough overview of the process.

It made sense to me that a good place to start was to determine what contributed to ranking in the first place. More specifically, what things contribute that I can already measure? My ideas eventually fell into three broad categories: on-page/on-SERP metrics, domain-level metrics, and URL/page-level metrics.

I then gathered all of the data points for these categories for every URL and domain in about 65,000 search results (Top 50 of 1389 randomly selected keywords) to determine what metrics can predict which results get to the top.

What I discovered is as important as – if not more so – than the difficulty score itself.

A Quick Note about what the Numbers Mean

Before we get into the meat of the data, here’s what you should know going in. The numbers you see in the charts represent correlation coefficients. These numbers tell us whether there is a signal in the noise, and if so, how strong the signal is. The values range from -1 to 1. 0 means there is no signal found, just noise. 1 is perfect correlation and -1 signifies a perfect inverse correlation.

On-Page / On-SERP Metrics

onpage ranking factors sheet SERP ranking factors sheet

On-page/On-SERP metrics have a wide range of correlations to Google rank; even some sub-metrics have significant variations within themselves. Take for example the # of Keyword Hits in URL. For the top 10 results, the average correlation came in at a weak 0.025. But within the URL, the correlation between ranking and the root domain proved to be a much stronger 0.094.

Here are the most and least important ranking factors we discovered within this category for the top 10 results:

Most Important Ranking Factors

  1. result is homepage
  2. # of keyword hits in URL – root domain
  3. exact match domain

Least Important Ranking Factors

  1. # of keyword hits in URL – subdomain
  2. # of keyword hits in URL – query string
  3. # of keyword hits in URL – first level of path

Domain-Level Metrics

Domain ranking factors sheet

The most interesting thing we discovered about domain-level metrics by far was their correlation to ranking in the top 1 to 50 segment. The on-page factors we looked at in the previous section were mostly important to domains that ranked in the top 10 spots only.

Most of those metrics had very little growth as we moved through the ranking segments, which is significant to note because while getting into the number 1 position is the goal, you much first get into the top 1000, top 100, and top 10. Conversely, factors that decrease in correlation from the top 10 to the top 50 are less important.

Most Important Ranking Factors

  1. domain strength (SpyFu)
  2. domain trust flow (Majestic SEO)
  3. domain age

Least Important Ranking Factors

  1. domain pagerank
  2. domain backlinks / referring domains
  3. domain backlinks

URL / Page-Level Metrics

Page ranking factors sheet

In order to get the summary numbers I calculate for domain-level metrics, I pulled # of Backlinks, Citation Flow, etc. from MajesticSEO for every unique domain in the search results for the 1389 keywords. There may have been a few randomly obscure domain names they didn’t have data on, but nothing significant.

But, when I did the same thing for each unique URL in the search results, I only got data back for about 40% of the URL’s.That leaves a lot of holes. I’ve included charts for both of the scenarios we ran since the choice makes a pretty big difference in the growth rates and the top 30 to top 50 results brackets.

Most Important Ranking Factors

  1. class c blocks linking to URL (Majestic)
  2. referring domains (Majestic)
  3. URL backlinks

Least Important Ranking Factors

  1. URL trust flow
  2. URL backlinks/referring domains
  3. URL citation flow (Majestic)

More to Come

Deriving ‘Ranking Difficultly’ for our users was a large undertaking and gave us new insights into the factors that affect organic ranking. We’ve only just begun to explore how we can use what we’ve learned to make your life as an SEO easier and more effective by providing you with the best data and metrics available. Keep your eye on the site over the next couple of months as we roll out a ton of powerful new features.