How Processing Fluency Impacts Web Marketing - Whiteboard Friday

Posted by randfish

As marketers, we're trained to think that our audiences consider the rational inputs we display, and through them, come to rational conclusions. But what about cognitive biases that might influence processing and decision making?

In today's Whiteboard Friday, Rand dives into how processing fluency impacts web marketing, and explains why things which are easier for us to digest are more likely to induce action.

For reference, here is a still of Rand's whiteboard.

Video Transcription

"Howdy Moz fans, and welcome to another edition of Whiteboard Friday. This week I want to start with a conundrum. In fact, it's a conundrum from a research project that is based on a fluency bias. Fluency bias being one of the many cognitive biases in the field of psychology.

Let me start by asking you a question. Do you believe the statement, "What alcohol conceals, sobriety unmasks"? So a large number of participants in a research study were asked whether they believed this, and a second group, another group of participants in the same study were asked separately whether they believed the statement, "What alcohol conceals, sobriety reveals." What do you think were the results? Take a minute to guess.

People believed this one massively more by a shocking margin. And you would think to yourself, "Well, I am not nearly so foolish a person as to think that my belief in a statement like this would be biased by rhyme, conceals/reveals," and yet that is exactly what happened time and time again. This study can be reproduced with success. Far more people believe "What alcohol conceals, sobriety reveals," rather than the alternate use of the word "unmasks."

This is called, one of my favorite cognitive biases in the world, the "rhyme as reason" bias. Rhyme as reason. Let me give you another famous example that some of you have probably already jumped to. Do you remember Johnny Cochran in the famous O.J. Simpson trial, declaring to the jury, "If the gloves does not fit, you must acquit. If the glove does not fit, you must acquit."

So human beings, especially in the marketing and technology world, are trained, we are trained to think that people are logical, that people consider the rational outcomes, the rational inputs, and they come to a rational decision based on those inputs. And yet a cognitive bias, like rhyme as reason, would suggest that's not really the case at all. We are biased by all sorts of things.

Rhyme as reason is one of many fluency biases. The fluency bias or the fluency processing bias essentially suggests that things which are more easy for us to comprehend, which are more simple for us to digest, lots of good examples here. Attractive people on magazine covers are more likely to draw our eyes. Concepts that are simple for us to understand, phrases that we've heard many times, things that relate to things in our memory, all of these are simpler for us to understand and therefore more credible, more believable, and more likely, in the marketing world, to induce action.

Let's take this over to web marketing for a second and think about things where this happens. Page speed load time. When something loads more quickly, not only are we more likely to stay on the page, we're more likely to trust the brand more. We're more likely to recommend it to others. We're more likely to use it ourselves.

In fact, when Microsoft did a famous research study where they increased the amount of time before search results were returned by a mere 250 milliseconds, which is undetectable to the human eye, right? If you were shown a film strip and then there was a 250 millisecond cut, your eye could not detect it. Your brain would not know that you had been shown that image, and yet what they found was that abandonment rates went up. People searched less, and they searched less often, and they were less likely to return to the site.

This is fluency bias at work. The aesthetic attractiveness of a website's or a web page's layout is likely to drive us to take more action or to take less action, to recommend something, to tweet it, to share it, to link to it. No wonder, right?

The pronounceability of a brand name. One of my favorite, favorite examples is that a study looked at the pronounceability of stock market ticker symbols during their IPO, at a public market offering. And you would think to yourself, "Now, wait a minute. These are some of the smartest human beings in the world, who are working at hedge funds, who are working at large investment portfolios. There is no way that they are going to be taken in by the pronounceability of a stock ticker symbol." Why does it even matter whether a stock ticker symbol is pronounceable or not? And yet pronounceability has a high correlation with more successful IPOs in their first two weeks after offering.

Insanity. Insanity. We are all subject to this. No matter how smart you believe yourself or you audience to be, fluency biases, processing fluency, and cognitive biases as a whole are undoubtedly having an effect on your audience.

The familiarity of user experience. Some of you have seen some of the screen shots from Moz Analytics and probably maybe a few of you have gotten access to the private beta, and over the next couple of months more people will. Inside that product you'll notice that it looks very similar to another product. Right? There's sort of a, "Oh, look at that. There's the navigation on the left-hand side. There's a little graph up here, and the time frame is over here, and then there's a chart of data down here." That reminds me a lot of Google Analytics, which many people who are watching this Whiteboard Friday and might be using Moz Analytics are almost certainly familiar with. And that is no error. That familiarity of user experience, that, "Oh, yes. I have been here before. Oh, yes. I am familiar with how to use a web analytics product or a search engine or an e-commerce site."

There's a reason that these follow into patterns and why these patterns are successful when they are repeated and deviation from those patterns can actually be dangerous. The legibility of font and text in a blog post, in a piece of content can influence whether it's shared more or less.

The ease of discovery and shareability of something. If something is very easy to copy and paste inside my browser so that I can easily tweet it, or if I am sent a link by somebody in an email that just says, "Hey, if you would retweet this that would be great," and it goes directly to their tweet, wow, this is very easy. It's very easy for me to share it, and therefore I am more likely to do so. Processing fluency dictates it is thus.

I would urge you, whenever you're thinking about your marketing campaigns, whether those be in the SEO world with things like your domain name, your title, your URL. Your URL, in a study by Bing, domain name and URL, the little part in the search results that's green, actually had a significant biasing effect on where clicks went. Almost as significant, in fact, a little more significant than whether there was a rel=author profile picture, according to Google. These are separate studies, but the data should match up.

The readability of that content. Social, the sharing time. When was it shared? Was it shared at a time when I'm going to see it? Was it shared at a time when I'm likely to be on a device where I'm more likely to share? Maybe that's mobile if it's a retweet. Maybe that's desktop if it's something where I actually want someone to take action, or a laptop, or a tablet.

The length of the content. Length is very much a part of processing fluency because very long articles, depends on the subject matter, but we have a tendency not to read or to comprehend and process all of that information.

In advertising, your copy, your layout, your design, this is classic ad agency world stuff that people have been doing for decades. And in content, the style, the UX, the complexity of that content.

Again, another really good example, Moz's own search ranking factors, which are produced every two years, and this summer we're coming out with a new version. It will be first presented at MozCon and then appear on the web. But the complexity of the new UI, that we launched in 2011, made it such that engagement on those pages was far less because you had to click over to different tabs to actually see the numbers, as opposed to seeing it all on one page. It reduced the shareability, the number of links it got, as compared to when it was done in 2005, 2007, and 2009. Fascinating, fascinating stuff.

If you were investing in web marketing channels, in content marketing and SEO, in social, and advertising of any kind, I would urge you to think about the fluency of the work that you're producing and whether people can really consume it as effectively as you're hoping they can. This can have a big impact on the effectiveness of the work that you do.

All right everyone. I hope you've enjoyed this edition of Whiteboard Friday, and we'll see you again next week. Take care."

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

via Moz Blog http://moz.com/blog/processing-fluency-impacts-web-marketing-whiteboard-friday

SEO Tactics Die, But SEO Never Will

Posted by Dr-Pete

This is a post that has been gnawing at the edges of my brain for years, and I think the time has finally come to write it. Our recent Moz re-brand launched the inevitable 4,789th wave (and that’s just this year) of "SEO Is Dead" posts. This isn't a post about our reasons for broadening our brand (Rand has talked extensively about that) – it’s a post about why I think every declaration of SEO's demise misses something fundamental about our future. This is going to get philosophical, so if you’d rather go make a sandwich, I won’t stop you.

The Essence of Search

Let’s start with a deceptively simple question – How big is the internet? I’ll attempt to answer that by creating a graph that borders on being silly:

The internet is so big that even Google got tired of counting, and it's growing exponentially. Five years have passed since they announced the trillion mark, and the article suggests that URL variations now make the potential indexed page count theoretically infinite.

We can't just print out the internet and read it at our leisure. We need a filter – a way to sift and sort our collected content – and that's essentially all that search is. However search evolves or whatever happens to Google, the expansion of human knowledge is accelerating. Unless we suffer a technological cataclysm, we will need search, in some form, for the rest of human history.

Searchers and Searchees

As long as search exists, it also stands to reason that there will be two groups of people: (1) People who want to find things, and (2) People who want to be found. On any given day, we may each be both (1) and (2), and the "people" who want to be found could be businesses, governments, etc., but for every search there will be some entity who wants to have a prominent position in that search result.

The desire to be found isn't new or unique to online search – just ask Melvil Dewey or call up "AAA Aardvark Plumbing" in the Yellow Pages. What's unique to online search is that the system has become so complex that automated technology governs who gets found, and as the scope of information grows, that's not about to change. Ultimately, whenever a system controls who will be found, then there will be a need for people who understand that system well enough to help entities end up on the short list.

This goes beyond manipulative, "black hat" practices – data needs to be structured, rules complied with, and many pieces put into place to make sure that the information we put out there is generally friendly with the systems that catalog and filter it. Over time, these systems will get more sophisticated, but they will never be perfect. As long as search exists, there will be a need for experts who can optimize information so that it can be easily found.

SEO Is Not One Tactic

When we say "SEO Is Dead!", we’re usually reacting to the latest tactical fad or announcement from Google. Ultimately, though, SEO is not one tactic and even though Google currently dominates the market, SEO doesn't live and die with Google. I'm 42 years old, and the public internet as we know it now hasn't existed for even half of my life. Google is a teenager, and I strongly suspect I'll outlive them (or at least their dominance).

There's no doubt that search is changing, and our industry is barely out of its infancy. In the broad sense, though, the need for people who can help construct findable information and attract people to that information will outlive any single tactic, any individual SEO expert, and even any search engine.

The Construct: Search in 2063

Sergei had spent his entire adult life learning how to manipulate The Construct. Fifteen years earlier, the unthinkable had happened – the collected knowledge of humanity had grown so quickly that there was no longer enough space in the accessible universe to store it in. The internet became The Construct, and it now spanned both space and time.

Since no human could adequately comprehend 4-dimensional data (early attempts at neural interfaces drove a few pioneers to insanity), The Construct had to be projected onto a 3-dimensional orb suspended in a vacuum, affectionately known as the “space egg.” With more than a decade of practice, Sergei manipulated the egg like an omelette chef at a 5-star brunch, and what his clients paid him made their $37 mimosas look reasonable.

This morning was worse than most. The Construct’s AI had detected an unacceptable level of manipulation and was adjusting the Core Algo. Sergei could already see the surface of the egg being rewritten, and the change was costing his clients millions with every passing minute. Luckily, his defensive bots were already at work, rewriting semantic data to conform to the ripples in the Algo. One thing was certain: the life of a Space Egg Optimizer was never dull.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

via Moz Blog http://moz.com/blog/seo-tactics-die-but-seo-never-will

How Processing Fluency Impacts Web Marketing - Whiteboard Friday

Posted by randfish

As marketers, we're trained to think that our audiences consider the rational inputs we display, and through them, come to rational conclusions. But what about cognitive biases that might influence processing and decision making?

In today's Whiteboard Friday, Rand dives into how processing fluency impacts web marketing, and explains why things which are easier for us to digest are more likely to induce action.

For reference, here is a still of Rand's whiteboard.

Video Transcription

"Howdy Moz fans, and welcome to another edition of Whiteboard Friday. This week I want to start with a conundrum. In fact, it's a conundrum from a research project that is based on a fluency bias. Fluency bias being one of the many cognitive biases in the field of psychology.

Let me start by asking you a question. Do you believe the statement, "What alcohol conceals, sobriety unmasks"? So a large number of participants in a research study were asked whether they believed this, and a second group, another group of participants in the same study were asked separately whether they believed the statement, "What alcohol conceals, sobriety reveals." What do you think were the results? Take a minute to guess.

People believed this one massively more by a shocking margin. And you would think to yourself, "Well, I am not nearly so foolish a person as to think that my belief in a statement like this would be biased by rhyme, conceals/reveals," and yet that is exactly what happened time and time again. This study can be reproduced with success. Far more people believe "What alcohol conceals, sobriety reveals," rather than the alternate use of the word "unmasks."

This is called, one of my favorite cognitive biases in the world, the "rhyme as reason" bias. Rhyme as reason. Let me give you another famous example that some of you have probably already jumped to. Do you remember Johnny Cochran in the famous O.J. Simpson trial, declaring to the jury, "If the gloves does not fit, you must acquit. If the glove does not fit, you must acquit."

So human beings, especially in the marketing and technology world, are trained, we are trained to think that people are logical, that people consider the rational outcomes, the rational inputs, and they come to a rational decision based on those inputs. And yet a cognitive bias, like rhyme as reason, would suggest that's not really the case at all. We are biased by all sorts of things.

Rhyme as reason is one of many fluency biases. The fluency bias or the fluency processing bias essentially suggests that things which are more easy for us to comprehend, which are more simple for us to digest, lots of good examples here. Attractive people on magazine covers are more likely to draw our eyes. Concepts that are simple for us to understand, phrases that we've heard many times, things that relate to things in our memory, all of these are simpler for us to understand and therefore more credible, more believable, and more likely, in the marketing world, to induce action.

Let's take this over to web marketing for a second and think about things where this happens. Page speed load time. When something loads more quickly, not only are we more likely to stay on the page, we're more likely to trust the brand more. We're more likely to recommend it to others. We're more likely to use it ourselves.

In fact, when Microsoft did a famous research study where they increased the amount of time before search results were returned by a mere 250 milliseconds, which is undetectable to the human eye, right? If you were shown a film strip and then there was a 250 millisecond cut, your eye could not detect it. Your brain would not know that you had been shown that image, and yet what they found was that abandonment rates went up. People searched less, and they searched less often, and they were less likely to return to the site.

This is fluency bias at work. The aesthetic attractiveness of a website's or a web page's layout is likely to drive us to take more action or to take less action, to recommend something, to tweet it, to share it, to link to it. No wonder, right?

The pronounceability of a brand name. One of my favorite, favorite examples is that a study looked at the pronounceability of stock market ticker symbols during their IPO, at a public market offering. And you would think to yourself, "Now, wait a minute. These are some of the smartest human beings in the world, who are working at hedge funds, who are working at large investment portfolios. There is no way that they are going to be taken in by the pronounceability of a stock ticker symbol." Why does it even matter whether a stock ticker symbol is pronounceable or not? And yet pronounceability has a high correlation with more successful IPOs in their first two weeks after offering.

Insanity. Insanity. We are all subject to this. No matter how smart you believe yourself or you audience to be, fluency biases, processing fluency, and cognitive biases as a whole are undoubtedly having an effect on your audience.

The familiarity of user experience. Some of you have seen some of the screen shots from Moz Analytics and probably maybe a few of you have gotten access to the private beta, and over the next couple of months more people will. Inside that product you'll notice that it looks very similar to another product. Right? There's sort of a, "Oh, look at that. There's the navigation on the left-hand side. There's a little graph up here, and the time frame is over here, and then there's a chart of data down here." That reminds me a lot of Google Analytics, which many people who are watching this Whiteboard Friday and might be using Moz Analytics are almost certainly familiar with. And that is no error. That familiarity of user experience, that, "Oh, yes. I have been here before. Oh, yes. I am familiar with how to use a web analytics product or a search engine or an e-commerce site."

There's a reason that these follow into patterns and why these patterns are successful when they are repeated and deviation from those patterns can actually be dangerous. The legibility of font and text in a blog post, in a piece of content can influence whether it's shared more or less.

The ease of discovery and shareability of something. If something is very easy to copy and paste inside my browser so that I can easily tweet it, or if I am sent a link by somebody in an email that just says, "Hey, if you would retweet this that would be great," and it goes directly to their tweet, wow, this is very easy. It's very easy for me to share it, and therefore I am more likely to do so. Processing fluency dictates it is thus.

I would urge you, whenever you're thinking about your marketing campaigns, whether those be in the SEO world with things like your domain name, your title, your URL. Your URL, in a study by Bing, domain name and URL, the little part in the search results that's green, actually had a significant biasing effect on where clicks went. Almost as significant, in fact, a little more significant than whether there was a rel=author profile picture, according to Google. These are separate studies, but the data should match up.

The readability of that content. Social, the sharing time. When was it shared? Was it shared at a time when I'm going to see it? Was it shared at a time when I'm likely to be on a device where I'm more likely to share? Maybe that's mobile if it's a retweet. Maybe that's desktop if it's something where I actually want someone to take action, or a laptop, or a tablet.

The length of the content. Length is very much a part of processing fluency because very long articles, depends on the subject matter, but we have a tendency not to read or to comprehend and process all of that information.

In advertising, your copy, your layout, your design, this is classic ad agency world stuff that people have been doing for decades. And in content, the style, the UX, the complexity of that content.

Again, another really good example, Moz's own search ranking factors, which are produced every two years, and this summer we're coming out with a new version. It will be first presented at MozCon and then appear on the web. But the complexity of the new UI, that we launched in 2011, made it such that engagement on those pages was far less because you had to click over to different tabs to actually see the numbers, as opposed to seeing it all on one page. It reduced the shareability, the number of links it got, as compared to when it was done in 2005, 2007, and 2009. Fascinating, fascinating stuff.

If you were investing in web marketing channels, in content marketing and SEO, in social, and advertising of any kind, I would urge you to think about the fluency of the work that you're producing and whether people can really consume it as effectively as you're hoping they can. This can have a big impact on the effectiveness of the work that you do.

All right everyone. I hope you've enjoyed this edition of Whiteboard Friday, and we'll see you again next week. Take care."

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

via Moz Blog http://feedproxy.google.com/~r/seomoz/~3/4pW_lBHGeAg/processing-fluency-impacts-web-marketing-whiteboard-friday

SEO Tactics Die, But SEO Never Will

Posted by Dr-Pete

This is a post that has been gnawing at the edges of my brain for years, and I think the time has finally come to write it. Our recent Moz re-brand launched the inevitable 4,789th wave (and that’s just this year) of "SEO Is Dead" posts. This isn't a post about our reasons for broadening our brand (Rand has talked extensively about that) – it’s a post about why I think every declaration of SEO's demise misses something fundamental about our future. This is going to get philosophical, so if you’d rather go make a sandwich, I won’t stop you.

The Essence of Search

Let’s start with a deceptively simple question – How big is the internet? I’ll attempt to answer that by creating a graph that borders on being silly:

The internet is so big that even Google got tired of counting, and it's growing exponentially. Five years have passed since they announced the trillion mark, and the article suggests that URL variations now make the potential indexed page count theoretically infinite.

We can't just print out the internet and read it at our leisure. We need a filter – a way to sift and sort our collected content – and that's essentially all that search is. However search evolves or whatever happens to Google, the expansion of human knowledge is accelerating. Unless we suffer a technological cataclysm, we will need search, in some form, for the rest of human history.

Searchers and Searchees

As long as search exists, it also stands to reason that there will be two groups of people: (1) People who want to find things, and (2) People who want to be found. On any given day, we may each be both (1) and (2), and the "people" who want to be found could be businesses, governments, etc., but for every search there will be some entity who wants to have a prominent position in that search result.

The desire to be found isn't new or unique to online search – just ask Melvil Dewey or call up "AAA Aardvark Plumbing" in the Yellow Pages. What's unique to online search is that the system has become so complex that automated technology governs who gets found, and as the scope of information grows, that's not about to change. Ultimately, whenever a system controls who will be found, then there will be a need for people who understand that system well enough to help entities end up on the short list.

This goes beyond manipulative, "black hat" practices – data needs to be structured, rules complied with, and many pieces put into place to make sure that the information we put out there is generally friendly with the systems that catalog and filter it. Over time, these systems will get more sophisticated, but they will never be perfect. As long as search exists, there will be a need for experts who can optimize information so that it can be easily found.

SEO Is Not One Tactic

When we say "SEO Is Dead!", we’re usually reacting to the latest tactical fad or announcement from Google. Ultimately, though, SEO is not one tactic and even though Google currently dominates the market, SEO doesn't live and die with Google. I'm 42 years old, and the public internet as we know it now hasn't existed for even half of my life. Google is a teenager, and I strongly suspect I'll outlive them (or at least their dominance).

There's no doubt that search is changing, and our industry is barely out of its infancy. In the broad sense, though, the need for people who can help construct findable information and attract people to that information will outlive any single tactic, any individual SEO expert, and even any search engine.

The Construct: Search in 2063

Sergei had spent his entire adult life learning how to manipulate The Construct. Fifteen years earlier, the unthinkable had happened – the collected knowledge of humanity had grown so quickly that there was no longer enough space in the accessible universe to store it in. The internet became The Construct, and it now spanned both space and time.

Since no human could adequately comprehend 4-dimensional data (early attempts at neural interfaces drove a few pioneers to insanity), The Construct had to be projected onto a 3-dimensional orb suspended in a vacuum, affectionately known as the “space egg.” With more than a decade of practice, Sergei manipulated the egg like an omelette chef at a 5-star brunch, and what his clients paid him made their $37 mimosas look reasonable.

This morning was worse than most. The Construct’s AI had detected an unacceptable level of manipulation and was adjusting the Core Algo. Sergei could already see the surface of the egg being rewritten, and the change was costing his clients millions with every passing minute. Luckily, his defensive bots were already at work, rewriting semantic data to conform to the ripples in the Algo. One thing was certain: the life of a Space Egg Optimizer was never dull.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

via Moz Blog http://feedproxy.google.com/~r/seomoz/~3/56M5JYdNyW8/seo-tactics-die-but-seo-never-will

Determining Relevance: How Similarity Is Scored

Posted by Matt Peters

Today's web search engines have sophisticated ways of measuring whether a web page is related to a given query, based on decades of research in Information Retrieval. Come join me as I explore the inner workings of a search engine's relevance engine and explain what it means for SEOs.

Determining Relevance

When a user submits a query to a search engine, the first thing it must do is determine which pages in the index are related to the query and which are not. Throughout this post, I will refer this as the "relevance" problem. More formally, we can state it as follows:

Given a search query and a document, compute a relevance score that measures the similarity between the query and document.

The "document" in this context can also refer to things like the title tag, the meta description, incoming anchor text, or anything else that we think might help determine whether the query is related to the page. Practically, a search engine computes a number of relevance scores using different page elements and weights them all to arrive at one final score.

The relevance problem has been extremely well studied in the research community. The first papers go back several decades, and it is still an active area of research. In this post, I focus on the most influential approaches that have stood the test of time.

Relevance vs Ranking

Conceptually, we can separate relevance determination from ranking the relevant documents, even if they are implemented as a single step inside a search engine. In this mental framework, the relevance step first makes a binary (True/False) decision for each page, then the ranking step orders the documents to return to the user.

I'll present some data later in this post that vividly illustrates this split and how it relates to different ranking signals.

Query and Document Models

Translating the query and document from raw strings into something we can do computation with is the first hurdle in computing a similarity score. To do so, we make use of "query models" and "document models." The "models" here are just a fancy way of saying that the strings are represented in some other way that makes computation possible.

The above image illustrates this process for the query "philadelphia phillies" and the Wikipedia page about the Phillies. The final step in computing the similarity score runs the query and document representations through a scoring function.

Query Models

The following image illustrates some different types of query models:

The building blocks at the bottom include things like tokenization (splitting the string into words), word normalization (such as stemming where common word endings are removed), and spelling correction (if a query contains a misspelled word, the search engine corrects it and returns results for the corrected word).

Built on top of these building blocks are things like query classification and intent. If the search engine determines that a particular query is time sensitive it will return news results, or if it thinks the query intent is transactional it will display shopping results.

Finally, at the top of the pyramid are more abstract representations of the query such as entity extraction or latent topic representations (LDA). Indeed, Google knows that the "philadelphia phillies" are a major league baseball team and since it is baseball season returns last night's score at the top of the search results (in addition to the knowledge graph on the right).

Document Models

Like query models, there are several different types of document models commonly used in search.

TF-IDF is one of the oldest and most well known approaches that represents each query and document as a vector and uses some variant of the cosine similarity as the scoring function. A language model encodes some information about the statistics of a language and includes knowledge such as the phrase "search engine optimization" is much more common then "search engine walking." Language models are used heavily in machine translation and speech recognition, among other applications. They are also extremely useful in information retrieval. Yet another class of models uses the probability ranking principle, which directly models the probability of relevance given the query and document. Of these, Okapi BM25 has been shown to be particularly effective.

Correlation study

By now, you are probably wondering if search engines actually use any of these things, and if so, which ones are the most important. To explore this, we designed a correlation study similar to ones we have run in the past (see this for some background on the general approach). In this case, we collected the top 50 results from Google-US for about 14,000 keywords. This resulted in about 600,000 pages that we then crawled and used to compute a number of different similarity scores.

As you can see, the language model approach performed the best with a mean Spearman correlation of 0.10, consistent with results published in the research literature.

If we do some stemming of both the query and document first and recompute, the correlations increase slightly across the board:

This suggests that Google is indeed doing some type of word normalization or stemming in their relevance calculation.

Relevance vs Ranking revisited

Comparing these correlations vs Page Authority (an aggregate in-link metric in our Mozscape index) on the same data set, we see a substantial difference:

This begs the question: if these sophisticated similarity scores are so useful, why aren't the correlations higher? The answer lies in the conceptual relevance vs ranking split I discussed earlier.

To convince myself, I constructed an experiment as illustrated below:

To run the experiment, I first took 450 random pages from our dataset stratified across the top 50 results (so that they include nine #1 ranked pages, nine #2 ranked pages, etc.). Then I added the 450 random pages to the top 50 pages in each search result to make one group of 500 pages for each keyword. Since 50 of these pages are in the search result, and 450 are not, 10% of them are relevant to the keyword and 90% are not (the assumption here is that if the page appears in a Google search then it is relevant). Then for each keyword, I collected the Page Authority and Language Model similarity score and sorted by each (the tables in the middle).

Finally, I computed the Precision at 50, which is the percentage of the top 50 results sorted by PA/Language Model score that are actually in the search result. This directly measures the extent to which PA or the Language Model can separate relevant from irrelevant pages. Since 10% of the 500 documents are in the search result, we can achieve a 10% precision by randomly sorting them. This 10% precision is our baseline (bottom gray bars in the image).

The results are striking. The PA precision is very close to the baseline, which says that is does no better then a random number at determining relevance even though it does do a good job at ranking the top 50 once they are known to be relevant. On the other hand, the Language Model precision is close to 100%. Put another way, the Language Model is nearly perfect in determining which of the 500 pages are in the search result, but does a poor job at actually ranking those relevant documents.

Takeaways

This type of query-document similarity scoring is well established in the research literature and underlies every modern information retrieval system. As such, it is fundamental to search and is immune to algorithm change.

Since search engines use sophisticated query and document models, there is no need to optimize separately for similar keywords. For example, any page targeting "movie reviews" will also target "movie review."

Finally, you can use the conceptual split between relevance and ranking in your workflow. When creating or modifying existing content, first concentrate on making the page relevant to a broad set related keywords. Then concentrate on increasing the search position.

More Ranking Factors results coming soon

These are the first results we've released from the 2013 Ranking Factors project. As in years past, the project includes both an industry survey and large correlation study. I'll be presenting the results at MozCon this year (so get your tickets if you haven't already!), and we'll be following it up with a full report sometime later this summer.

To dig deeper

Here are all the slides from my SMX Advanced talk:

I highly recommend the book Introduction to Information Retrieval by Manning et al. It is available for free online reading from their site and provides a comprehensive description of everything discussed in this post (and much, much more). In particular, see Chapters 2, 6, 11 and 12.

Thanks for reading. I look forward to continuing the discussion in the comments below!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

via Moz Blog http://feedproxy.google.com/~r/seomoz/~3/gf7OAmqyb5g/determining-relevance-how-similarity-is-scored