General Guidelines

Published on December 2016 | Categories: Documents | Downloads: 35 | Comments: 0 | Views: 361
of 137
Download PDF   Embed   Report

Comments

Content

General Guidelines Version 4.2

June 7, 2013

General Guidelines Overview ................................................................................. 4 Part 1: Utility Rating Guidelines ............................................................................ 5
1.0 Introduction to Utility Rating ............................................................................................................................. 5 1.1 1.2 1.3 1.4 1.5 1.6 2.0 Important Rating Definitions and Ideas .................................................................................................. 5 The Purpose of Search Quality Rating .................................................................................................... 6 Raters Must Represent the User .............................................................................................................. 6 Browser Requirements ............................................................................................................................. 6 Ad Blocking Extensions ........................................................................................................................... 6 Internet Safety Information ....................................................................................................................... 6

Understanding the Query .................................................................................................................................. 7 2.1 Understanding User Intent ........................................................................................................................ 7 2.2 Task Language and Task Location (Locale) ............................................................................................ 8 2.3 Queries with Multiple Meanings................................................................................................................ 8 2.4 Query Meanings Can Change Over Time ............................................................................................... 10 2.5 Classification of User Intent: Action, Information, and Navigation: Do-Know-Go ........................... 11

3.0

Understanding Websites ................................................................................................................................. 12 3.1 3.2 3.3 Important Definitions and Ideas ............................................................................................................. 12 Identifying the Purpose of the Page ...................................................................................................... 13 Understanding Web Page Content ........................................................................................................ 13

4.0

Utility Rating Scale for Result Blocks ............................................................................................................ 15 4.1 4.2 4.3 4.4 4.5 Vital (V) ..................................................................................................................................................... 16 Useful (U) .................................................................................................................................................. 25 Relevant (R).............................................................................................................................................. 30 Slightly Relevant (SR) ............................................................................................................................. 32 Off-Topic or Useless (OT) ....................................................................................................................... 35

5.0

Rating Porn (P), Foreign Language (F), and Didn’t Load (D) Results ........................................................ 39 5.1 5.2 5.3 5.4 5.5 Porn (P) Flag ........................................................................................................................................... 39 Porn Result Utility Rating ...................................................................................................................... 39 Reporting Illegal Images ........................................................................................................................ 41 Foreign Language Results .................................................................................................................... 42 Rating Results with Content that Doesn’t Load .................................................................................. 45

6.0 Location and Utility Rating ............................................................................................................................... 47 6.1 6.2 Locale is Important.................................................................................................................................. 47 Queries that “Ask” for Nearby Results or Nearby Information .......................................................... 50

Proprietary and Confidential – Copyright 2013

1

6.3 6.4 6.5 6.6 6.7 6.8

User Location and Understanding the Query ....................................................................................... 53 Queries with an Explicit Location .......................................................................................................... 55 Vital Ratings for Queries with a User Location .................................................................................... 56 Rating Queries with User Location and Explicit Location .................................................................. 57 Rating Local Intent Queries .................................................................................................................... 57 Rating Examples with User Location, Explicit Location, and Local Intent ....................................... 57

Part 2: Page Quality Rating Guidelines .............................................................. 61
7.0 Overview of Page Quality Evaluation............................................................................................................. 61 7.1 7.2 Introduction to Page Quality .................................................................................................................. 62 Important Information about the Page Quality Guidelines ................................................................. 62

7.3 Your Money or Your Life (YMYL) ............................................................................................................ 63 8.0 Landing Page Considerations ........................................................................................................................ 64 8.1 8.2 8.3 8.4 8.5 8.6 Identifying the Purpose of the Page ...................................................................................................... 64 Identifying the Main Content, Supplementary Content, and Advertisements................................... 65 Rating the Quality of the Main Content ................................................................................................. 66 Rating the Quantity of Helpful Main Content ........................................................................................ 69 Rating the Helpfulness of the Supplementary Content ....................................................................... 70 Rating the Layout of the Page/Use of Space on the Page .................................................................. 71

9.0 Answering Homepage and Website Questions ............................................................................................. 74 9.1 9.2 9.3 9.4 9.5 9.6 Finding the Homepage of the Website .................................................................................................. 74 Is the Purpose of the Page Consistent with the Website? .................................................................. 76 Who is Responsible for the Content of the Website and the Content of the Page? ........................ 77 Does the Website Have an Appropriate Amount of Contact Information? ....................................... 78 What Kind of Reputation Does the Website Have? ............................................................................. 79 Is the Homepage of the Website Updated/Maintained? ...................................................................... 83

10.0 Additional Page Quality Rating Guidance .................................................................................................... 83 10.1 Assigning a Page Quality Rating to “Your Money or Your Life” (YMYL) Pages .............................. 84 10.2 Assigning a Page Quality Rating to Encyclopedia Pages.................................................................. 84 10.3 Assigning a Page Quality Rating to Pages with no Main Content/Error Messages ........................ 85 10.4 Balancing Page Level and Website Level Questions to Assign an Overall Page Quality Rating .. 86 10.5 How to Check for Copied Content ........................................................................................................ 86 11.0 Assigning an Overall Page Quality Rating ................................................................................................... 88 11.1 Highest Quality Pages ........................................................................................................................... 89 11.2 High Quality Pages ................................................................................................................................. 90 11.3 Medium Quality Pages ........................................................................................................................... 90 11.4 Low Quality Pages.................................................................................................................................. 91 11.5 Lowest Quality Pages ............................................................................................................................ 92

Proprietary and Confidential – Copyright 2013

2

12.0

Webspam: a Special Type of Lowest Page Quality .................................................................................... 93 12.1 What is Webspam?................................................................................................................................. 93 12.2 When to Check for Spam ....................................................................................................................... 93 12.3 Looking for Technical Signals .............................................................................................................. 93 12.4 Ads and Spam Pages ............................................................................................................................. 98 12.5 Doorway Pages ..................................................................................................................................... 101 12.6 Thin Affiliates ........................................................................................................................................ 102

13.0

Page Quality Rating FAQs .......................................................................................................................... 103

Part 3: Block Utility Rating ................................................................................ 105
14.0 15.0 16.0 17.0 18.0 19.0 Overview of Block Utility (BU) Rating ........................................................................................................ 105 The Relationship between PQ and Utility .................................................................................................. 105 Queries with Multiple Interpretations and Intents .................................................................................... 107 Specificity of Queries and Landing Pages ................................................................................................ 109 Utility and Freshness ................................................................................................................................... 114 Misspelled and Mistyped Queries and Results ......................................................................................... 119 19.1 Misspelled and Mistyped Queries....................................................................................................... 119 19.2 Name Queries ....................................................................................................................................... 120 19.3 Spelling Suggestion Result Blocks .................................................................................................... 122 20.0 21.0 22.0 Non-Vital Results for URL Queries............................................................................................................. 125 Product Queries: Action (Do) vs. Information (Know) Intent .................................................................. 126 Mobile Landing Pages ................................................................................................................................. 128

Part 4: Using the Evaluation Platform .............................................................. 130
23.0 Introduction ................................................................................................................................................... 130 24.0 Accessing the Evaluation Platform (EP)..................................................................................................... 130 25.0 Evaluation Platform Screenshot .................................................................................................................. 130 26.0 Block Utility Task Page Screenshot ............................................................................................................ 131 27.0 Notes about Using the Block Utility Rating Interface................................................................................ 133 28.0 Using the “Report a Problem / Release this Task” Button ....................................................................... 133 29.0 Reporting Results with Duplicate Landing Pages ..................................................................................... 134 30.0 Using the Page Quality Rating Scale in Block Utility Rating Tasks ........................................................ 137

Proprietary and Confidential – Copyright 2013

3

General Guidelines Overview
Welcome to the Search Quality Rating Program!
As a Search Quality Evaluator, you will work on many different types of rating projects. The General Guidelines primarily cover block utility (BU) rating; however, the concepts are also important for many other types of rating tasks. Please note:
• •

For brevity, we refer to “Search Quality Evaluators” as “raters” in these guidelines. The General Guidelines contain many rating examples. Because webpage content changes so frequently, we use screenshots created when the rating examples were written. Some screenshots may seem outdated when you read through the guidelines.

Proprietary and Confidential – Copyright 2013

4

Part 1: Utility Rating Guidelines
1.0 Introduction to Utility Rating
1.1 Important Rating Definitions and Ideas

Query: This refers to the word(s), number(s), and/or symbol(s) that a user types in the search box of a search engine. In these guidelines, queries will have square brackets around them. If a user types the word iphone in the search box, we will display: [iphone]. User: A user is a person who types queries into a search engine when looking for information. These guidelines assume that the user is typing queries on a desktop or laptop computer. User Intent: When a user types a query, he is trying to accomplish something. We refer to this goal as the user intent. Locale: All queries have a locale which is the language and location for the task. Task locations are represented by a two-letter country code. For a current list of country codes, click here. User Location: Some queries also have a User Location, which provides more specific information about where the user is located. The User Location is usually a city. Search Engine Results Page (SERP): The page a search engine shows after a user enters a query in the search box. The SERP is made up of result blocks. Here is a screenshot of one result block from the [iphone] SERP with the parts labeled:

The Landing Page (LP) is the page you see after you click the title link in the result block. We will use the word result to refer to the result block and the landing page.

Proprietary and Confidential – Copyright 2013

5

1.2

The Purpose of Search Quality Rating

Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that are helpful for users in their specific language and locale.

1.3

Raters Must Represent the User

It is very important for you to represent users in the locale you evaluate. You must be very familiar with the task language and location in order to represent the experience of users in your locale. If you do not have the knowledge to do this, please inform your employer.

1.4

Browser Requirements

Check with your employer for browser requirements. You may use helpful browser add-ons or extensions, but please do not use add-ons or extensions which interfere with the user experience of the page.

1.5

Ad Blocking Extensions

Do not use add-ons or extensions that block ads for Block Utility rating or Page Quality rating. These add-ons or extensions may cause you to give incorrect ratings. As a rater, only use an ad blocking extension or add-on if specifically instructed to do so in the project-specific instructions.

1.6

Internet Safety Information

In the course of your work, you will visit many different webpages. Some of them may harm your computer unless you are careful. Please do not download any executables, applications, or other potentially dangerous files, or click on any links that you are uncomfortable with. We strongly recommend that you have antivirus and anti-spyware protection on your computer. This software must be updated frequently or your computer will not be protected. There are many free and for-purchase antivirus and anti-spyware products available on the web. Here are links to Wikipedia articles with information about antivirus software and spyware: • • Wikipedia page about antivirus software: http://en.wikipedia.org/wiki/Antivirus_software Wikipedia page about spyware: http://en.wikipedia.org/wiki/Spyware

We suggest that you only open files with which you are comfortable. The file formats listed below are generally considered safe if antivirus software is in place. • • • • • .txt (text file) .ppt or .pptx (Microsoft PowerPoint) .doc or .docx (Microsoft Word) .xls or .xlsx (Microsoft Excel) .pdf (PDF) files

Proprietary and Confidential – Copyright 2013

6

If you encounter a page with a warning message, such as “Warning-visiting this web site may harm your computer,” or if your antivirus software warns you about a page, you should not try to visit the page to assign a rating. You may also come across pages that require RealPlayer or the Adobe Flash Player plug-in. These are safe to download.

1.7 Releasing Tasks
Most raters have difficulty rating tasks now and then. Some queries are about highly technical topics (e.g., computer science or physics) or involve very specialized areas of interest (e.g., gaming or torrents). Please release the task if, after research, you don’t understand the query or user intent for the task or the content of the LP. Release tasks when: • • • • • Lack expertise: You feel that you personally can’t rate the query. Suspicious files: The task contains unknown or suspicious file formats. Offensive content: You believe that the LP will be offensive to you or you feel uncomfortable visiting the LP. Technical problem: The query/instructions/results have obvious rendering or formatting issues. Wrong language: The query/task is in the wrong language, i.e., it cannot be understood by users in the locale.

2.0

Understanding the Query

Understanding the query is the first step in evaluating the task. A “query description” will sometimes be provided to help you understand the query. You must use the information in query descriptions when assigning ratings. If you don’t understand the query or user intent, do web research using the Google search engine or an online dictionary or encyclopedia. If you still don’t understand the query or user intent, please release the task. Important: If you research the query on Google, please do not rely on the top results on the SERP. A query may have other meanings not represented on Google’s search results pages. Do not assign a high rating to a webpage just because it appears at the top of a list of search results on Google.

2.1 Understanding User Intent
You must understand user intent to evaluate the landing page. Think about users in your task location typing the query right now. What are users trying to accomplish?
Query [population of paris], English (US) [Fedex], English (US) [weather], English (US) Likely User Intent Find the current population of Paris, France. Navigate to the Fedex website, track a package, or find a FedEx location. Find weather information in the User Location right now.

Proprietary and Confidential – Copyright 2013

7

2.2 Task Language and Task Location (Locale)
All queries have a task language and task location (locale). The locale is important for understanding the query and user intent. Users in different parts of the world may have different expectations for the same query.

2.3 Queries with Multiple Meanings
Many queries have more than one meaning. For example, the query [apple], English (US) might refer to the computer brand or the fruit. We will call these possible meanings query interpretations. Dominant Interpretation: The dominant interpretation of a query is what most users mean when they type the query. Not all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after doing a little web research. Common Interpretation: A common interpretation of a query is what many or some users mean when they type a query. A query can have multiple common interpretations. Minor Interpretations: Sometimes you will find less common interpretations. These are interpretations that few users have in mind. We will call these minor interpretations.

Proprietary and Confidential – Copyright 2013

8

Query: [apple] Locale: English (US)

Query: [mercury] Locale: English (US)

Proprietary and Confidential – Copyright 2013

9

2.4 Query Meanings Can Change Over Time
Remember to think about the query and its current meaning as you are rating. We will assume users are looking for current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc., unless otherwise specified by the query.

The interpretation of the query [iphone], English (US) has changed over time as new iPhone models have been introduced. The first iPhone was introduced in 2007. Users searching for [iphone], English (US) at that time were looking for the newly introduced first iPhone model. Most users now are looking for the most recent or upcoming iPhone model. In the future, new models will come out and the dominant interpretation will change again.

Proprietary and Confidential – Copyright 2013

10

2.5 Classification of User Intent: Action, Information, and Navigation: Do-Know-Go
Sometimes it is helpful to classify user intent for a query in one or more of these three categories: • • • Action intent – Users want to accomplish a goal or engage in an activity, such as download software, play a game online, send flowers, find entertaining videos, etc. These are Do queries: users want to do something. Information intent – Users want to find information. These are Know queries: users want to know something. Navigation intent – Users want to navigate to a website or webpage. These are Go queries: users want to go to a specific page.

An easy way to remember this is Do-Know-Go. Classifying queries this way can help you figure out how to rate a webpage. Important: Many queries fit into more than one type of user intent.

2.5.1

Action Queries: Do

The intent of an action query is to accomplish a goal or engage in an activity on the web. The goal or activity may be to download, to buy, to obtain, to be entertained by, or to interact with a resource that is available on the web. Users want to Do something.
Query [download adobe reader], English (US) [online personality test], English (US) [what is my bmi?], English (US) [buy Citizen Kane DVD], English (US) [flowers], English (US) [bubble spinner 2], English (US) Likely User Intent Download software. Take an online personality test. Calculate the BMI (body mass index). Purchase this DVD. Order flowers online or look at pictures of flowers. Play Bubble Spinner 2 online or download the game.

2.5.2

Information Queries: Know

An information query seeks information on a topic. Users want to Know something; the goal is to find information.
Query [Switzerland], English (US) [cryptology use in WWII], English (US) [how to remove candle wax from carpet], English (US) [definition of paradox], English (US) [allergies], English (US) [big dog breeds], English (US) Likely User Intent Find travel and tourism information for planning a vacation or holiday, or find information about the Swiss geography, languages, economy, etc. Find information about how cryptology was used in World War II. Find information on how to remove candle wax from carpet. Find a definition of “paradox.” Find information about allergies. Find information about big dog breeds.

2.5.3

Navigation Queries: Go

The intent of a navigation query is to locate a specific webpage. Users have a single webpage or website in mind. This single webpage is called the target of the query. Users want to Go to the target page. The most helpful page for a navigation query is the navigational target page.

Proprietary and Confidential – Copyright 2013

11

Query [kayak], English (US) [youtube], English (US) [ebay], Italian (IT) [best buy store locator], English (US) [sony customer support], English (US) [canon.com camcorders], English (US)

Likely User Intent Go to the Kayak homepage. Go to the YouTube homepage. Go to the Italian eBay homepage. Go to the store locator page on the Best Buy website. Go to the customer support page on the Sony website. Go to the camcorders page on the Canon website.

2.5.4

Queries with Multiple User Intents (Do-Know-Go)

Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more likely than another intent.
Query [firefox], English (US) [bmi], English (US) [iphone], English (US) Likely User Intent Do and Go. Many users may want to download the Firefox browser (Do) from the official Firefox website (Go). Do and Know. Some users may want to calculate (Do) their BMI (body mass index). Some users may want to learn what BMI is or how to interpret their BMI number (Know). Do, Know, and Go. Users may be looking to purchase an iPhone (Do), find information (Know), or go to the iPhone page on the Apple website (Go).

3.0

Understanding Websites

In this section, you will learn about websites and webpages. You need to understand websites in order to give utility and page quality ratings.

3.1

Important Definitions and Ideas

Website or site: A website or site is a collection of webpages. Webmaster: This refers to the person who designs, develops and/or maintains a website. Webpage or web page: This refers to any page on a website. It may be the homepage or a subpage on a website. URL: The URL is the web address of a search result block, such as http://www.microsoft.com or http://windows.microsoft.com/en-US/windows/products. Homepage of a website: This refers to the main page of a site. It is usually the first page that users see when the site loads. For example, http://www.apple.com is the homepage of the Apple computer company site, and http://www.yahoo.com is the homepage of the Yahoo! company site. The websites of some businesses and other entities have “major sections” which may also be considered websites. Yahoo! has Yahoo! Finance, Yahoo! Sports, Yahoo! Mail, etc. For example, http://www.finance.yahoo.com is the homepage of the Yahoo! Finance site. Subpage on a website: This refers to a page on a site that is not the homepage. For example, http://www.apple.com/iphone is a subpage on the Apple site, and http://finance.yahoo.com/options is a subpage on the Yahoo! Finance site.

Proprietary and Confidential – Copyright 2013

12

3.2

Identifying the Purpose of the Page

The first step in understanding a webpage is figuring out why it was created. We will call this the purpose of the page. Every page on the Internet is created for a purpose (or for multiple purposes). Most pages are created to be helpful for users. 3.2.1 Pages with a Helpful Purpose

Common helpful page purposes include (but are not limited to): • • • • • • • • To share objective information about a topic. To share personal or social information. To express an opinion or point of view. To entertain. To share pictures, videos, or other forms of media. To sell products or services. To allow users to post questions so that other users can answer. To allow users to share files or to download software.

Here are a few examples where it is easy to understand the purpose of the page:
Type of Page News website homepage Shopping page Video page Currency converter page Purpose of the Page To display news. To sell or give information about the product. To allow users to watch a video. To calculate equivalent amounts in different currencies.

Why do we care about the purpose of the page? In order to understand and evaluate a page, you must understand what the page is trying to achieve.

3.3

Understanding Webpage Content

All of the content on a webpage can be classified as one of the following: Main Content (MC), Supplementary Content (SC), or Advertisements (Ads). In order to evaluate webpages, you will need to be able to distinguish between these different parts of the page.

3.3.1

Identifying the Main Content (MC)

Main Content is any part of the page that directly helps the page achieve its purpose. MC can be text, images, videos, page features, etc. Webmasters control the MC of the page.
Type of Page and Purpose News website homepage: the purpose is to display news. News article page: the purpose is to display a news article. Store product page: the purpose is to sell or give information about the product. Video page: the purpose is to allow users to view a video. Currency converter page: the purpose is to calculate equivalent amounts in different currencies. Blog post page: the purpose is to display a blog post. Search engine homepage: the purpose is to allow users to enter a query and search the Internet. Bank login page: the purpose is to allow users to log in to bank online. MC Highlighted in Yellow MC - News Homepage MC - News Article MC - Shopping Page MC - Video Page MC - Currency Converter MC - Blog Post Page MC - Search Engine HP MC - Bank Login Page

Proprietary and Confidential – Copyright 2013

13

3.3.2

Identifying the Supplementary Content (SC)

Supplementary Content is content which contributes to a good user experience on the page, but does not directly help the page achieve its purpose. SC is created by webmasters and is an important part of the user experience. For example, one common type of SC is navigation links which allow users to visit other parts of the website. Sometimes the easiest way to identify SC is to look for the parts of the page which are not MC or Ads.
Type of Page and Purpose News website homepage: the purpose is to display news. News article page: the purpose is to display a news article. Store product page: the purpose is to sell or give information about the product. Video page: the purpose is to allow users to view a video. Currency converter page: the purpose is to calculate equivalent amounts in different currencies. Blog post page: the purpose is to display a blog post. Search engine homepage: the purpose is to allow users to enter a query and search the Internet. Bank login page: the purpose is to allow users to log in to bank online. SC Highlighted in Blue SC- News Homepage SC - News Article SC - Shopping Page SC - Video Page SC - Currency Converter SC - Blog Post Page SC - Search Engine HP SC - Bank Login Page

3.3.3

Identifying the Advertisements (Ads)

Advertisements are content and links that are displayed for the purpose of monetizing or making money from the page. Ads are sometimes labeled as "ads," "sponsored links," “sponsored listings,” “sponsored results,” etc. Usually, you can mouse over the content or click on the links to determine whether they are Ads. Ads may change when you reload the page, and different users may see different Ads on the same page. Webmasters can choose to display Ads on their page, but they don’t directly control the content of the Ads. However, we will consider a website responsible for the overall quality of the Ads displayed.
Type of Page and Purpose News website homepage: the purpose is to display news. News article page: the purpose is to display a news article. Store product page: the purpose is to sell or give information about the product. Video page: the purpose is to allow users to view a video. Currency converter page: the purpose is to calculate equivalent amounts in different currencies. Blog post page: the purpose is to display a blog post. Search engine homepage: the purpose is to allow users to enter a query and search the Internet. Bank login page: the purpose is to allow users to log in to bank online. Ads Highlighted in Red Ads - News Homepage Ads - News Article No ads Ads - Video Page Ads - Currency Converter Ads - Blog Post Page No ads No ads

3.3.4

Summary of the Parts of the Page

Let's put it all together. Here are the examples again, with all parts of the page labeled:
Type of Page and Purpose News website homepage: the purpose is to display news. News article page: the purpose is to display a news article. Store product page: the purpose is to sell or give information about the product. Video page: the purpose is to allow users to view a video. Currency converter page: the purpose is to calculate equivalent amounts in different currencies. Blog post page: the purpose is to display a blog post. Search engine homepage: the purpose is to allow users to enter a query and search the Internet. Bank login page: the purpose is to allow users to log in to bank online. MC, SC, and Ads Highlighted Summary - News Homepage Summary - News Article Summary - Shopping Page Summary - Video Page Summary - Currency Converter Summary - Blog Post Page Summary - Search Engine HP Summary - Bank Login Page

Proprietary and Confidential – Copyright 2013

14

MC and SC are important parts of the page. It is easy to understand the need for MC: MC is the reason the page exists. SC is also important. Almost every web page needs navigation links, and most pages can be improved by features and content designed to help users get the most out of the page and website. Many pages have Ads. Without advertising and monetization, some webpages could not exist. Do not worry too much about identifying every little part of the page. Carefully think about which parts of the page are the MC. Next, look for the Ads. Anything left over can be considered SC.

4.0

Utility Rating Scale for Result Blocks

The Utility rating scale offers five rating options that apply to all types of result blocks in BU rating.
Rating Vital (V) Useful (U) Relevant (R) Slightly Relevant (SR) Off-Topic or Useless (OT) Description A special rating category. Very helpful for many or most users. Helpful for many users OR very helpful for some users. Helpful for fewer users. Helpful for no or very few users.

On BU rating tasks, you will see the rating scale on the Utility Slider. The slider uses abbreviations for the five rating options as follows:

The slider is labeled query-dependent because the utility rating is based on how helpful the result (the result block and landing page) is for the query. Please note that you may assign in-between ratings. For example, notice that there are 3 small tick marks in between R and U on the slider. Use in-between ratings if you think the utility of a result falls between two labels. You can either drag the slider or click on the point that you want the slider to land on. Most result blocks have a title link and landing page. Some result blocks have no title link.

Proprietary and Confidential – Copyright 2013

15

For result blocks with a title link and landing page, the utility rating is based primarily on the MC of the LP. We call these blocks Title Link Result Blocks (TLRB).

For result blocks with no title link, the utility rating is based primarily on the content and prominent links inside the result block. We call these blocks No Title Link Result Blocks (NTRB).

4.1

Vital (V)

The Vital (V) rating is used for these very special situations: • • There is a dominant interpretation and clear user intent to navigate to a specific website or webpage, and the LP is exactly what the user is looking for. The dominant interpretation of the query is an entity (such as a person, place, business, restaurant, product, company, organization, etc.), and the LP is the official webpage associated with that entity.

In both cases, the query must have a dominant interpretation. If there is no dominant interpretation, it is not possible to assign a V rating. The V rating is based on the LP of the title link. Result blocks with no title link (NTRBs) cannot receive the V rating. Most V pages are very helpful; however, this is not a requirement for a V rating. Some V pages are “official,” but not very helpful. To give a V rating, the landing page and website must “work” – the LP must load, function, and be maintained. Finally, when in doubt, don’t use the V rating.

4.1.1 Go Queries with Vital (V) Pages For Go queries, the V page is the page requested by the user. If the query “asks” for the homepage of a website, only the homepage gets the V rating. If the query “asks” for a subpage, only that particular subpage gets the V rating.

Proprietary and Confidential – Copyright 2013

16

Query and User Intent [nytimes], English (US) Go to the New York Times online newspaper.

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Description of the LP

The New York Times homepage and target of the query.

Landing page link: LP [nytimes sports], English (US) Go to the sports section of the New York Times online newspaper. Landing page link: LP [walmart.com], English (US) Go to the official homepage of the Walmart online retail site. The Walmart homepage and target of the query. The New York Times sports section page and target of the query.

Landing page link: LP [walmart storefinder], English (US) Go to the storefinder page on the Walmart website. The Walmart storefinder page and target of the query.

Landing page link: LP [lady gaga twitter], English (US) Go to Lady Gaga’s official Twitter page. Lady Gaga’s Twitter page and target of the query.

Landing page link: LP [google maps], English (US) Go to Google Maps.

The Google Maps site and target of the query. Note that even though the title link just says “welcome,” the rating is Vital.

Proprietary and Confidential – Copyright 2013

17

4.1.2

Entity Queries with Vital Pages

For entity queries, the official homepage of the entity is Vital. Some entities show a “splash page” to announce something to visitors to their website. A splash page should be rated V if the page behind it is V.
Query and User Intent [Madonna], English (US) Go to the homepage of Madonna’s official website or find information about her. [Gary Danko], English (US) Go to the homepage of the Restaurant Gary Danko website or find information about the restaurant. Landing page link: LP [Bourne Legacy], English (US) Go to the movie’s official homepage or find information about it. Homepage of the movie (on the movie studio’s website). Landing page link: LP LP of the Result Block, Result Block, Utility Rating Landing page link: LP Description of the LP

Homepage of Madonna’s official website.

Homepage of Restaurant Gary Danko.

[Maytag], English (US) Go to the homepage of this appliance manufacturer or find information about the brand. [ipod nano], English (US) Go to the iPod nano page on the Apple website or find information about it.

Landing page link: LP

Homepage of the Maytag website.

Landing page link: LP

iPod nano page on the Apple website.

Proprietary and Confidential – Copyright 2013

18

Query and User Intent [Baseball hall of fame], English (US) Go to the Baseball Hall of Fame website or find information about it [Masters Golf Tournament], English (US) Go to the Masters website or find information about it.

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Description of the LP

Homepage of the Baseball Hall of Fame website.

Landing page link: LP Homepage of the Masters Golf Tournament website.

Landing page link: LP [Freakonomics blog], English (US) Go to the Freakonomics Blog. Freakonomics Blog page on the Freakonomics website.

Landing page link: LP [Harvard], English (US) Go to the Harvard University website or find information about it. Homepage of the Harvard University website.

4.1.3

Queries with No Vital Result

Most queries do not have Vital results. Here are situations for which there is no V result. • • • • The query does not have a dominant interpretation. The query is not an entity or is not a navigation query. No official website or webpage exists for the entity. No person or entity can “own” the topic of the query.

Proprietary and Confidential – Copyright 2013

19

Query

Vital Result?

Description There is no dominant interpretation. The following entities are all common interpretations. Each interpretation has an official homepage, but none is V since there is no dominant interpretation. • Americans with Disabilities Act • American Dental Association • American Diabetes Association This is an information query. Knitting is an activity anyone can do and that anyone can create a website for. There is no one official source for knitting information. No one can own this topic. This is an information query. No person or entity can claim ownership of the query [diabetes]. The query [ipod] English (US) has a Vital result, but [ipod reviews] does not. [ipod reviews] is an Know query. Users are looking for information that many sites can provide. The query [Britney Spears] is an entity query, but [how old is britney spears] is not. This is an Know query. Users are looking for information that many sites can provide.

[ADA], English (US)

No V result is possible.

[knitting], English (US)

No V result is possible.

[diabetes], English (US) [ipod reviews], English (US) [how old is britney spears?], English (US)

No V result is possible. No V result is possible. No V result is possible.

Be careful with queries such as [diabetes], [knitting], [furniture], [cars], [toys], etc. There is no Vital result possible because no one can “own” the topic of these queries. TLRBs with URLs that match the query may not even be helpful. Sites claiming to be official may not actually be official sites. The V rating should NOT be assigned on the basis of the result block or appearance of the URL alone. You must visit the LP and examine the website. Here are some examples of result blocks that may look Vital, but are not:
Query LP of the Result Block, Result Block Vital? No: This LP is not an official homepage for Ashley Tisdale; it is a fan site. This is the “real” official V result: Official Ashley Tisdale Website. When you are unsure if a celebrity site is official, please explore the site. You will usually be able to find information telling you that the site is a fan site.

[Ashley Tisdale] , English (US)

Landing page link: LP

[Branson, Missouri] , English (US)

Landing page link: LP

No: The title of the TLRB and the LP display the words “Branson.com Official Website,” but it is the homepage of the Branson.com website, not the homepage of the official city of Branson, Missouri website: Official Vital Result for the City of Branson, Missouri.

Proprietary and Confidential – Copyright 2013

20

Query [Diabetes], English (US)

LP of the Result Block, Result Block
Landing page link: LP

Vital? No: No V result is possible for this query because it is an information query and no one can claim ownership of it.

Landing page link: LP

[fabric], English (US)

No: No V result is possible for this query because it is an information query and no one can claim ownership of it.

[chairs], English (US)

Landing page link: LP

No: No V result is possible for this query because it is an information query and no one can claim ownership of it. This is an example of a spam page (see Section 14.4.2, Parked Domains).

4.1.4

Vital Pages for People Queries

This section describes the use of the Vital rating for queries which are names of people, such as [oprah], [barack obama], and [lady gaga]. This section does not apply to queries which include both a name and other words, such as [lady gaga twitter]. For a query which is the name of a real (non-fictional) living person, the V rating should be used when: • • The query has a clear dominant interpretation, i.e., most people are looking for information about one particular person. For example [madonna], [barack obama], [shaquille o'neal]. The result is the homepage of the person's official website, if such a website exists.

Many or most people queries do not have a dominant interpretation. Queries like [ben smith], [mary jones], [elizabeth tucker], [susan greene], [paul richards], and [chad hancock] can have no V result because different users may be looking for different people. Even unusual sounding name queries may not have a dominant interpretation. For example, the queries [sam wen], [tran nguyen], and [david mease] can have no V result because there are multiple people with each of these names and it is not clear that most users are looking for any one particular individual. We will consider a person’s website official if it is created by the person or an authorized agent of that person. The website must be maintained and have information or content which establishes that the website officially represents the person. This is a very high standard. When in doubt, do not use the V rating. Important: Websites that are under construction or obviously unmaintained should not be rated V, even if they were at one point created by or authorized by the person in the query. Please consider a personal website to be unmaintained if there is old or stale information displayed prominently. For personal websites which are generally very frequently updated, please look for updates within the last 4 months. If the personal website feels unmaintained, do not use the V rating.

Proprietary and Confidential – Copyright 2013

21

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Description of the LP

Homepage of Tiger Woods’ official website.

[tiger woods], English (US) Landing page link: LP This is Tiger Woods’ Twitter page. It is not his official website. This result is not Vital. Landing page link: LP

Homepage of Oprah’s official website.

[oprah], English (US) Landing page link: LP This is not Oprah’s official website, even though the URL (oprahwinfrey.com) matches her name. This result is not Vital. Landing page link: LP Homepage of Emma Watson’s official website.

[emma watson], English (US) Landing page link: LP This is a well maintained Facebook page but it is not Emma Watson’s official website. This result is not Vital.

Proprietary and Confidential – Copyright 2013

22

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Description of the LP There are multiple people with this name. It is not clear which Sam Wen users may be looking for. There is no dominant interpretation, and therefore no V result is possible for this query. Amanda Bynes is an actress. The LP displays an “Under Construction” message. The copyright date shown is 2000. There is no content on the page that establishes this as her official website.

[sam wen], English (US) This result is not Vital. Landing page link: LP

[amanda bynes], English (US)

This result is not Vital.

4.1.5 Vital pages for URL queries Some queries look like URLs. We will call these queries URL Queries. Here are some types of URL queries: • • Exact, perfectly-formed, working URLs, such as [http://www.ibm.com] or [ www.ibm.com] or [ibm.com]. Imperfect URL queries: Queries that look like URL queries, but are not “working URL” queries. These URLs do not load if you type or paste them into your browser address bar. Even so, we believe users have a specific page in mind.

Some queries, such as [yahoo] or [yahoo mail] are website names. These queries do not contain “.com” or “www” or other standard components of a URL. These are Go queries, but we will not consider them URL queries. Assign a V rating to results for URL queries based on user intent. Assign a V rating only if there is little or no doubt that the page matches user intent.
Query and User Intent [www.myspace.com] , English (US) This is a perfect working URL query. The user intent is clear: go to the MySpace website. [www.yahoo.c0m], English (US) [yahoo.xcom], English (US) Even though these queries are imperfect URL queries, it is clear the user wants to go to the Yahoo! website. LP of the Result Block, Result Block, Utility Rating Landing page link: LP Description of the LP

Homepage of the MySpace website.

Landing page link: LP

Homepage of the Yahoo! website.

Proprietary and Confidential – Copyright 2013

23

Query and User Intent

LP of the Result Block, Result Block, Utility Rating

Description of the LP

Landing page link: LP

Homepage of the huffintonpost.com website. This is spam. It is a fake search page. See Section 12.4.3 for more information about this result and about fake search pages.

[huffintonpost.com], English (US) This query is a working URL, but it is clearly mistyped. The clear user intent is to navigate to huffingtonpost.com.

This result is not Vital. It is not the result the user is looking for.

Landing page link: LP

Homepage of the Huffington Post website.

The clear user intent is to navigate to huffingtonpost.com. This result is Vital. Landing page link: LP

[wwww.ibm.com], English (US) Even though the URL doesn’t load, it is clear that the user wants to go to the IBM homepage.

Homepage of the IBM website.

[tax form 1040 irs.gov], English (US) Even though the query contains spaces, it is clear that the user wants to go to the page on the official IRS government website for the current 1040 tax form. [i hire chemists.com], English (US) Even though the query contains spaces, it is clear that the user wants to go to the iHireChemists website.

Landing page link: LP

Page with the 1040 tax form on the official IRS website.

Landing page link: LP

Homepage of the iHireChemists website.

Proprietary and Confidential – Copyright 2013

24

4.1.6

Vital Pages and Geographic Location

Some international entities have multiple versions of their homepages for different languages or countries. A homepage should be rated Vital when: • • There is clear user intent to navigate to a specific website or webpage, and the landing page is exactly what the user is looking for. There are multiple versions of the official page, and the page seems right for users in the locale.

Otherwise, please assign a rating based on utility for the user.
Query and User Intent [United Nations], French (FR) Go to the United Nations website. French users expect to go to the version of the website that is appropriate for French speaking users. [United Nations], Russian (RU) Go to the United Nations website. Russian users expect to go to the version of the website that is appropriate for Russian-speaking users. LP of the Result Block, Result Block, Utility Rating Landing page link: LP Description of the LP Homepage of the French version of the United Nations website. The French version of the website is V for all French task locations. It is the right official page for French speaking users. The United Nations website has six versions of its website: Arabic, Japanese, English, French, Russian, and Spanish. This page is a “choose your language” page. The Russian version of the website is V for this query. The “choose your language” page is not. “Choose your language” page on the United Nations website. Since the LP is the page “asked for” in the query, it is V, even though a Russian version of the United Nations website exists.

Landing page link: LP

This result is not Vital.

Landing page link: LP [un.org], Russian (RU) Go to un.org.

4.2

Useful (U)

A rating of Useful is assigned to title link result blocks (TLRBs) and non-title link result blocks (NTRBs) that are very helpful for many or most users. U results are highly satisfying and a good “fit” for the query. In addition, they often have some or all of the following characteristics: high quality, authoritative, entertaining, and/or recent (such as breaking news on a topic). A query can have many Useful results. Have high standards for using the U rating. This is especially important for queries with many, many on-topic results.

Proprietary and Confidential – Copyright 2013

25

4.2.1

Examples of Useful (U) Result Blocks
LP of the Result Block, Result Block, Utility Rating Landing page Link: LP (Pleasant Grove High School in Utah) Explanation

Query and User Intent

Landing page link: LP (Pleasant Grove High School in California)

[pleasant grove high school], English (US) Navigate to the homepage of a high school named “Pleasant Grove High School” or find information about it. This is a Go or Know query.

This query does not have a dominant interpretation. Each of these LPs is the homepage of a high school in the US with this name. The high schools are located in the states of Utah, California, Alabama, and Texas. Since there appear to be only four high schools in the country with this name, the homepage of each school is a good match of the query and merits a U rating.

Landing page link: LP (Pleasant Grove High School in Alabama)

Landing page link: LP (Pleasant Grove High School in Texas)

Landing page link: LP [poison ivy], English (US) Find information about poison ivy. This is a Know query.

Page on an authoritative website that answers this question very well. This result would be very helpful for many or most users. There are many, many ontopic results for this query. This result merits a U rating.

Proprietary and Confidential – Copyright 2013

26

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

[Manresa reviews], English (US) Read reviews for this restaurant. This is a Know query.

Page with over 75 reviews for this restaurant. This page would be very helpful for many or most users.

Landing page link: LP [broadway tickets], English (US) Purchase tickets to a Broadway show. This is a Do query.

Page on a reputable ticket site on which to complete this transaction. This page would be very helpful for many or most users. There are many, many ontopic results for this query. This result merits a U rating.

Landing page link: LP [csco], English (US) Find stock quote information for Cisco. This is a Know query.

CSCO is the stock symbol for the Cisco Corporation. This page would be very helpful for many or most users. There are many, many ontopic results for this query. This result merits a U rating. LL Bean is well-known for selling backpacks for kids of different ages. There is a lot of product information and many user reviews for each backpack. This page would be very helpful for many or most users. There are many, many ontopic results for this query. This result merits a U rating. The MC of the LP allows users to select their location. Users in the US would click the “United States” link to get to the appropriate homepage. Because this store exists in many different countries, a “choose your location” page is very helpful for most users.

[kids backpacks], English (US) Find information about kids backpacks or purchase a kids backpack.

Landing page link: LP

Landing page link: LP [ikea], English (US) Go to the official Ikea homepage for US users.

Proprietary and Confidential – Copyright 2013

27

Query and User Intent

LP of the Result Block, Result Block, Utility Rating

Explanation Page on the official Sting website with the requested lyrics. There are many lowquality lyrics results, but we can have confidence in the accuracy of the lyrics on this site. There are many, many ontopic results for this query. This result merits a U rating.

[every breath you take lyrics], English (US) Find the lyrics to this song written by Sting. This is a Know query.

Landing page link: LP

[dance video], English (US) Find a dance video to watch. There are many good, entertaining, and popular dance videos on video websites. Users are looking for good or entertaining dance videos.

Landing page: http://www.youtube.com/watch?v=dMH0bHeiRNg Please visit the LP to watch the video

This is a very popular video of a comedian demonstrating dance styles from previous decades. There are many, many ontopic results for this query. This result merits a U rating.

Landing page link: LP

[purple coneflowers], English (US) Find images of or information about purple coneflowers.

This is a title link result block (TLRB), even though it has special content. There are helpful images in the result block, as well as many, many additional helpful images on the LP.

Proprietary and Confidential – Copyright 2013

28

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

[1600 amphitheatre parkway, Mountain View, CA], English (US) Users may be looking for directions, a map, information about what business resides at this address, etc.

This is a non-title link result block (NTRB). There is no title link but the block itself is clickable. Clicking the block takes users to the LP.

This is a non-title link result block (NTRB). [barack obama], English (US) Users may be looking for news, biographical information, images, etc. There is a lot of content in the block itself (images and information). Even though there is no title link, there are links in the result block that go to various landing pages. This result (result block and landing pages) would be helpful for many users.

Proprietary and Confidential – Copyright 2013

29

4.3

Relevant (R)

A rating of Relevant is assigned to results that are helpful for many users or very helpful for some users. R results have fewer valuable attributes than U results. R results should still “fit” the query, but they might be less comprehensive, less up-to-date, come from a less authoritative source, or cover only one important aspect of the query. R results should not be low quality, out-of-date, or inaccurate. R results are average to good. The R rating can apply to non-title link result blocks (NTRBs) as well. A NTRB may not be highly satisfying in some way, or it may serve a minor interpretation.

4.3.1

Examples of Relevant (R) Pages
LP of the Result Block, Result Block, Utility Rating Explanation Page with a map of the city of Seoul. This page would be helpful for some users. There are many, many maps of Seoul with more detailed information covering a wider area of the city. This page is not helpful enough for a rating of U.

Query and User Intent

Landing page link: LP [map of seoul, korea], English (US) Travel to Seoul; find current information about the city.

[Tom Cruise], English (US) Find biographical information or current news/celebrity gossip about Tom Cruise.

Landing page link: LP

A page of information about Tom Cruise. This page would be helpful for many users or very helpful for some users. There are many, many pages about Tom Cruise. This page is not helpful enough for a rating of U.

Landing page link: LP [abe lincoln’s birthday], English (US) Find this specific piece of information. Wikipedia page that displays the birthdays of all US presidents, including the birthday of Abraham Lincoln. However, Lincoln’s birthday is not prominently displayed.

Proprietary and Confidential – Copyright 2013

30

Query and User Intent [every breath you take lyrics], English (US) Find the lyrics to the song “Every Breath You Take,” which was written by Sting. This is a Know query. [latin dance video], English (US) Find a Latin dance video to watch. There are many good, entertaining, and popular Latin dance videos on video websites. Users are looking for good or entertaining Latin dance videos.

LP of the Result Block, Result Block, Utility Rating

Explanation

Landing page link: LP

Page on a lyrics website with the requested lyrics. There are many lyrics websites on the web. Often, pages with lyrics are not 100% accurate. R is an appropriate rating for average pages with the requested lyrics.

Landing page: http://www.youtube.com/watch?v=F5zCGgZMaWQ Please visit the LP to watch the video There are many, many Latin dance videos. This video is entertaining and the dancers are good, but there are many non-professional videos of talented amateur dancers.

[Nicaraguan money], English (US) Find out what Nicaraguan currency is called, what it looks like, the exchange rate, etc.

This is a NTRB. While it gives some basic information and is helpful, it is not highly satisfying. For example, it doesn’t show what the currency looks like.

[mercury], English (US) Find information about the Mercury car brand, the planet, or the chemical element.

This is a NTRB. It helps users understand some different meanings of the word “Mercury,” and the links lead to these SERPs: Element LP and Planet LP. However, it is missing at least one common interpretation: the car brand.

Proprietary and Confidential – Copyright 2013

31

Query and User Intent

LP of the Result Block, Result Block, Utility Rating

Explanation A page on crunchbase.com with a profile of the Shutterfly company . The MC of the LP has a lot of helpful information the company. It also has a link to shutterfly.com in the “General Information” portion of the page, which is part of the MC. This is a high quality page on a member site on the TechCrunch network. The LP has some helpful information about Shutterfly. (The page has a link to the target of the query in a logical spot on the page, but this link does not play a large role in rating.) This page merits a rating of R.

Landing page link: LP [shutterfly], English (US) Go to the homepage of the Shutterfly website

4.4

Slightly Relevant (SR)

A rating of Slightly Relevant is assigned to results that are helpful for some or few users. SR results may serve a minor interpretation, be low quality, have stale (outdated) information, be too specific, too broad, etc. to receive a higher rating. The SR rating can apply to non-title link result blocks (NTRBs) as well. A NTRB may be too specific, have too little content, or serve a minor interpretation.

4.4.1

Examples of Slightly Relevant (SR) Pages
LP of the Result Block, Result Block, Utility Rating Landing page link: LP Explanation This is a low quality article. The writing quality is poor. The article is written by a person without expertise in marriage or relationship counseling. Even though the article is about the query, the page is low quality and untrustworthy. It would be helpful for few users.

Query and User Intent

[lack of sex and problems with my marriage], English (US) Find help for marital issues.

Landing page link: LP [hot dog], English (US) Find information about hot dogs, such as recipes or nutrition information. The movie “Hot Dog,” which came out in 1984, is a minor interpretation. This IMDb page would be helpful for few users.

Proprietary and Confidential – Copyright 2013

32

Query and User Intent [best golfers], English (US) Professional golfers are ranked based on money earned in tournaments. We will assume users are interested in the most current annual Money List.

LP of the Result Block, Result Block, Utility Rating

Explanation

Landing page link: LP The landing page shows the 2009 PGA Money List. It is outdated and stale for the query today. It would be helpful for few users.

Landing page link: LP [britney spears], English (US) Find current news or pictures related to Britney Spears. 2004 article about the annulment of Britney’s first marriage. This is very old stale news that would be helpful for few users.

[honda odyssey], English (US) We will assume that users are interested in the current Honda Odyssey model, unless specified otherwise.

Landing page link: LP The information on this Kelley Blue Book page about the 2010 Honda Odyssey would be considered stale for the query today. It would be helpful for some or few users.

Landing page link: LP [abe lincoln’s birthday], English (US) Find this specific piece of information.

This landing page mentions the month and day, but not the year, of Lincoln’s birth. Most users would be interested in also knowing the year. There is not enough information about the topic of the query. This page would be helpful for few users.

[map of texas in the late 1800s], English (US) View a map that shows what Texas looked like in the late 1800s.

Landing page link: LP This landing page describes various maps of Texas in the 1800s, but does not display any maps. The page is related to the query but does not fit the user intent and would be helpful for few users.

Proprietary and Confidential – Copyright 2013

33

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

[ebay], English (US) The dominant interpretation is to go to www.ebay.com. This landing page has information about web traffic to the ebay.com website. It would be helpful for few users.

[every breath you take lyrics], English (US) Find the lyrics to the song “Every Breath You Take,” which was written by Sting. This is an information query.

Landing page link: LP

Even though this page does display the lyrics, it is a lower quality page than most lyrics pages. Notice the pop-up displayed on top of the lyrics and how the lyrics are split across two columns, as well as all the other distracting content on the page. It would be helpful for few users.

Landing page for the Vatican City link: LP

[the vatican], English (US) Find information about the Vatican.

This is a NTRB. This result block does not have enough content to be satisfying. The LP for the Vatican City link in the result block is a SERP for [Vatican City]. This result is not very helpful overall.

[another word for allowed], English (US) Find synonyms for the word “allowed.”

This is a NTRB. While this result block may help a few users think of another word for “allow,” it is not very helpful overall.

Proprietary and Confidential – Copyright 2013

34

4.5

Off-Topic or Useless (OT)

A rating of Off-Topic or Useless should be assigned to results that are helpful for no or very few users. OT results are unrelated to the query and/or have no utility. A rating of OT also applies when there is lack of attention to an aspect of the query that is important for satisfying user intent. The OT rating can also apply to NTRBs. An OT NTRB may be unrelated to the query, inaccurate, very unhelpful, or useless.

4.5.1

Examples of Off-Topic or Useless (OT) Pages
LP of the Result Block, Result Block, Utility Rating Landing page link: LP Explanation Does not fit the user intent: This LP is about the 2004 Australian Open, not the 2008 Australian Open. It is OT because it does not pay attention to an aspect of the query that is important for satisfying user intent: “2008.” It would be helpful for no or very few users.

Query and User Intent [Australian Open mens singles result 2008], English (US) Find a page that displays the 2008 men’s singles result for this tennis tournament.

[german cars], English (US) Find information about German cars or go to official homepage of a German automaker.

Landing page link: LP Does not fit the user intent: This LP is the homepage of Subaru, a Japanese car company, not a German car company. It would be helpful for no or very few users.

[anderson high school, austin] Go to the homepage of Anderson High School in Austin, Texas or get information about the school.

Landing page link: LP Does not fit the user intent: This LP is the homepage of Anderson High School in Cincinnati, Ohio, not Anderson High School in Austin, Texas. It would be helpful for no users.

Landing page link: LP [opi blue nail polish], English (US) Find pages about the Opi brand’s blue nail polish. Does not fit the user intent: This LP has the Essie brand of blue nail polish, not the Opi brand. It would be helpful for no or very few users.

Proprietary and Confidential – Copyright 2013

35

Query and User Intent [company to get rid of the possum in my attic], English (US) Find a company to trap and remove a possum from the attic.

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation Does not fit the user intent: This LP is the homepage of a pest control company in Australia. US users would need a US company to take care of this problem. There is a mismatch between the page and the locale that makes the landing page OT.

Landing page link: LP [hot dog], English (US) Find information about hot dogs, such as recipes. Keyword matches only: This LP has information about doghouses and happens to display the word hot. It is OT.

Landing page link: LP [tooth loss five years old], English (US) Find information about tooth loss in a five-year-old child. Keyword matches only: This LP has information about tooth loss in pike fish and displays the words five years old. This page is OT because it has keyword matches only and would be helpful for no users.

Landing page link: LP [mountain bikes], English (US) Find information about or purchase a mountain bike. [how to quit smoking], English (US) Find information on ways to quit smoking.

Links and ads only: Even though this LP has tabs and links that appear related to the query, neither the landing page nor the links provide any information about mountain bikes. The page is useless and should be rated OT. Gibberish: This LP has gibberish text. Read this sentence: “The tropical ubiquity was being singularized burring.” The quality of the landing page is so low that the page is OT.

Landing page link: LP

[american express], English (US) Go to the American Express card or get information about the company and its products and services.

Landing page link: LP

Insufficiently related to the query: The LP is a humorous blog post about a wife helping her husband buy a suit. The page mentions “American Express” on the page, but is insufficiently related to the query to be helpful for users and is OT for the user intent.

Proprietary and Confidential – Copyright 2013

36

Query and User Intent [every breath you take lyrics], English (US) Find the lyrics to the song “Every Breath You Take,” which was written by Sting.

LP of the Result Block, Result Block, Utility Rating

Explanation

Landing page link: LP

Insufficiently related to the query: This LP displays the wrong lyrics. Even though the title on the page is “Every Breath You Take,” the lyrics displayed are for the song “Every Little Thing She Does is Magic.”

[drive safely texas], English (US) There are several drivers education driving course websites with “drive safely” in the URL. Users may want to go to one of these websites. Insufficiently related to the query: This is a TLRB with special content. This result has no connection to the query and is off-topic.

[what is wedding in spanish], English (US)

This is a NTRB. The translation in the result block is completely wrong and inaccurate. Mañana means “tomorrow,” not “wedding.”

[starting order for today’s nascar race], English (US) Find information about the starting order of the Nascar race that day.

This is a NTRB. The content is off-topic. The content is not what the user is looking for and is OT. It does not show the starting order of a Nascar event.

Proprietary and Confidential – Copyright 2013

37

Query and User Intent

LP of the Result Block, Result Block, Utility Rating

Explanation

[united tech logo], English (US) Find images of the United Technologies Corporation logo.

This is a NTRB. The content is off-topic. The result block shows stock information, not the company logo.

[rachel mcadams & ryan gosling couple image], English (US) Find an image of Rachel McAdams and Ryan Gosling.

Landing page link: LP

The LP has an image of Amanda Seyfried and Adam Brody, not Rachel McAdams and Ryan Gosling. The page has a link in the SC on the lower part of the page titled “rachel mcadams & ryan gosling.” The MC of the page is not about Rachel McAdams and Ryan Gosling. The link to an image of them is in the SC and many users would never notice this link. This inconspicuous link does not provide enough utility to give the page a rating above OT.

Landing page link: LP [zoo atlanta], English (US) Go to the Zoo Atlanta website.

The LP on about.com is about the Louisville Zoo. The page has a link in the SC titled “Visit Zoo Atlanta On The Cheap – Save Money Without Coupons at Zoo Atlanta.” The MC of this page is not about Zoo Atlanta. The link to an article about Zoo Atlanta is in the SC and many users would never notice this link. This result should not be rated above OT.

Proprietary and Confidential – Copyright 2013

38

5.0

Rating Porn (P), Foreign Language (F), and Didn’t Load (D) Results

You will assign P, F, and D flags to result blocks when appropriate. All flags are query-independent, meaning that they do not depend on the query. Here is a screenshot of the flags you will use when the result block has certain characteristics:

And here is a screenshot of a result block that shows when the P flag would be used.
Query and User Intent Result Block, Utility Rating, Flag Explanation

[freeones], English (US) Go to the Freeones website.

The LP is the homepage of the Freeones website, a porn site. This result merits a Vital utility rating and the P flag.

5.1

Porn (P) Flag

Please assign the Porn flag to all porn pages, whether the query is porn-seeking or not. The P flag does not depend on the query or user intent. A page will be considered porn if it has pornographic content, including porn images, links, text, pop-ups, and/or prominent porn ads. An image may be considered porn in one culture or country, but not another. Please use your judgment and knowledge of the locale. Remember: The P flag is used to indicate that the result is porn. It doesn’t matter whether the query “asks for porn” or not.

5.2

Porn Result Utility Rating

5.2.1 Utility Rating for Clear Non-Porn Intent Queries If the user intent behind a query is clearly not porn-seeking, a landing page that has porn for its MC should be rated OT. • • When the user intent is clearly not porn, a porn result should be considered unhelpful or useless. Uninvited porn is a very bad experience for many users.

Remember to assign the P flag for all porn pages.

Proprietary and Confidential – Copyright 2013

39

Query [toys], English (US) [how tall is a camel], English (US) [car pictures], English (US)

User Intent Find toys to buy. Find the answer to this question about camels. Find pictures of cars.

Landing Page Example Warning – this page is porn. Example Warning – this page is porn. Example Warning – this page is porn.

Utility Rating OT OT OT

Porn Flag? Yes Yes Yes

Sometimes, the MC of a landing page is helpful for the query, but the page happens to display porn ads or porn links outside the MC. The query and the helpfulness of the MC have to be balanced with the user experience of the page. Use your judgment and represent users in your locale.

5.2.2

Utility Rating for Possible Porn Intent Queries

Some queries have both non-porn and porn interpretations. For example, all of the following English (US) queries are possible porn intent queries, but they also have a non-porn intent: [girls], [gay], [thong], [breast], [sex], [spanking]. We will call these queries “possible porn intent” queries. For these queries, please rate as if the non-porn interpretation is dominant, even if you think users are looking for porn. For example, please rate as if a likely intent of [bikini], English (US) is shopping. Rate the porn interpretation as a minor interpretation, even if you think most users are looking for porn.
Query [breasts], English (US) [breasts], English (US) [pictures of girls], English (US) [pictures of girls], English (US) User Intent Find anatomy or health information about breasts. Find anatomy or health information about breasts. Find pictures of girls. Find pictures of girls. Landing Page Example Example Warning – this page is porn. Example Example Warning – this page is porn. Utility Rating U SR U SR Porn flag? No Yes No Yes

5.2.3

Utility Rating for Clear Porn Intent Queries

For very clear porn queries where no other intent is possible, assign a rating to the porn landing page based on how helpful it is for the user. Even though there is porn intent, the page should still be assigned a P flag. Do not simply rate all porn pages for porn queries as R or U. Even though the query is porn and the result is porn, the page must fit the query and be helpful to get a high utility rating. Pages that provide a poor user experience - such as pages that try to download malicious software - should also receive low ratings, even if they have some images appropriate to the query. Porn stars, porn movies, names of specific porn websites, etc., can have V pages. Assign a P flag to all porn pages. Even Vital porn pages must be assigned a P flag.

Proprietary and Confidential – Copyright 2013

40

Query [freeones], English (US) [freeones], English (US) [jenna jameson], English (US) [jenna jameson], English (US) [anime sex pictures], English (US) [cheerleader porn], English (US)

User Intent Navigate to the Freeones homepage. Navigate to the Freeones homepage. Find porn pictures of Jenna Jameson or go to her official website. Find porn pictures of Jenna Jameson or go to her official website. Find anime sex pictures. Find porn pictures of cheerleaders.

Landing Page Example: http://www.freeones.com Warning – this page is porn. Example: http://www.baberoad.com Warning – this page is porn. Example Warning – this page is porn. Example Warning – this page is porn. Example Warning – this page is porn. Example Warning – this page is porn.

Utility Rating V OT V OT U to R range U to R range

Porn Flag? Yes Yes Yes Yes Yes Yes

5.3

Reporting Illegal Images

Child Pornography and Bestiality When working on rating projects in any task location or locale, you must follow United States federal law, which considers child pornography and bestiality to be illegal. Definition of Child Pornography An image is child pornography if it is a visual depiction of someone who appears to be a minor (i.e., under 18 years old) engaged in sexually explicit conduct (e.g., vaginal or anal intercourse, oral sex, bestiality or masturbation as well as lascivious depictions of the genitals), or sadistic or masochistic abuse. The image of sexually explicit conduct can involve a real child; a computer-generated, morphed, composite or otherwise altered image that appears to be a child (think of images that have been altered using “Photoshop”); or an adult who appears to be a child; and the image can be nonphotographic -- e.g., drawings, cartoons, anime, paintings or sculptures – so long as the subject is engaging in sexually explicit conduct and which is obscene. If it is indistinguishable from child pornography, it is child pornography. Even if the image has literary (think of the famous book “Lolita”), artistic, political (think of political cartoons), or scientific (think of images for a medical text book) value, please send the link to your employer (as instructed below). Depiction of the genitals does not require the genitals to be uncovered. Thus, for example, a video of underage teenage girls dancing erotically, with multiple close-up shots of their covered genitals, or images of children with opaque underwear that focus on the genitalia could be considered child pornography. An image of a naked child (e.g., in the bathtub or at a nudist colony) is not considered child pornography as long as the child is not engaging in sexually explicit conduct, or the focus is not on the child’s genitalia. Visual depictions of adults who look like adults (e.g., a 35 year old man play-acting in diapers, or an obvious woman dressed as a school girl) are not child pornography. (If you don't think it's a minor, it probably isn’t child pornography.) However, if you cannot tell that the person in the image is over 18 (e.g., an under-developed 18 year old whose body hair has been waxed), that is child pornography. Definition of Bestiality Bestiality or zoophilia is defined as human-animal sexual interaction. Reporting Instructions Please report illegal and offensive images as instructed by your employer.

Proprietary and Confidential – Copyright 2013

41

5.4

Foreign Language Results

5.4.1 Using the Foreign Language (F) Flag Please assign a Foreign Language (F) flag when the language on the landing page is not one of the following: • • • The task language English A language which is commonly used by a significant percentage of the population in the task location

For example, most users in Ukraine speak Russian. Therefore, landing pages in the Russian language should not be assigned the F flag for rating tasks in Ukrainian (UA). Here is a screenshot of a result block that shows when the F flag would be used.
Query and User Intent Result Block, Utility Rating, Flag Explanation The LP is the homepage of the Baidu website. The result is the target of the query, but is in a foreign language (Chinese). This result merits a Vital utility rating and the F flag.

[baidu], English (US) Go to the Baidu website.

Important: • • • • Please assign the F flag even if you personally understand the language but most users in your locale do not. Please remember to flag all foreign pages with the F flag, even if the query “asks” for a foreign language page. Assign the F flag based on the language of landing page, not the appearance of the result block. Sometimes it is difficult to say what language the landing page is in. The LP may have multiple languages or no words at all. In these cases, try to represent users in your locale. Does it feel like a foreign language page? You may look at MC, SC, ads and even the website the page is on. When in doubt, don’t use the F flag.

5.4.2 Utility Ratings for Foreign Language Results You must assign a utility rating for all result blocks in your task, even if the result blocks have a foreign language landing page. In most cases, pages you flag as F should be rated OT, because they cannot be understood by most users in your locale and are therefore useless. (Recall that if users in your locale can read the language, then you shouldn’t be using the F flag.) Occasionally, you will encounter helpful F pages. If the query is clearly “asking” for a foreign language result, then the utility rating of the foreign language page should not be OT. For example, please assign the V rating and F flag for baidu.com if the query is [baidu.com], English (US). Videos are often an example where foreign language pages are helpful and desired. Think about user intent and what pages are good for users. If the query “asks” for a foreign language song, band, film, sporting event, etc., then a video of the song, band, film, sporting event, etc. is helpful since it can probably be understood or enjoyed even though it is in a foreign language. For these types of queries, foreign language results are often expected. If the video is someone talking *about* the song, band, film, or event, the result probably cannot be understood or enjoyed and should be assigned an OT rating and the F flag.

Proprietary and Confidential – Copyright 2013

42

Query and User Intent

LP of the Result Block, Link to Video, Result Block, Utility Rating Link to LP Screenshot Link to view the video:

Explanation

[gangnam style], English (US) Watch a video of this song.

This is a very popular video in the US, even though the song is in Korean. This result is extremely helpful. The language of the LP is mostly English (even though the video is in Korean), so the F flag is not needed.

Link to LP Screenshot Link to view the video

[video of celine dion singing s’il suffisait d’aimer], English (US) Watch a video of Celine Dion singing this song.

This video is just what English (US) users are looking for, even though the video is not in English. The language of the LP is mostly English (even though the video is in French), so the F flag is not needed.

[alex c], English (US) Watch a song by this German compuser or find information about him.

Link to LP Screenshot Link to view the video

The query is for the German composer, Alex C. The landing page has a video with one of his songs sung by Y-Ass in German. It would be helpful for many English (US) users who type the query, even though it is not in English. The language of the landing page isn’t clear. Use your judgment when deciding whether to assign the F flag. The query is for “Kasal, Kasali, Kasalo,” a Filipino film. The LP is a clip from the movie in Filipino (Tagalog), the language spoken in the Philippines. It would be helpful for many English (US) users who type the query, even though it is not in English. Use your judgment when deciding whether to assign the F flag.

[Kasal, Kasali, Kasalo], English (US) Watch a trailer of this Filipino film or find information about it.

Link to LP Screenshot Link to view the video

Link to LP Screenshot Link to view the video

[the beatles video], English (US) Watch a video of the Beatles.

The video on the LP documents a visit by the Beatles to Tokyo. Most of the spoken language on the video is in Japanese. This video would be helpful for no or very few English (US) users. Use your judgment when deciding whether to assign the F flag.

Proprietary and Confidential – Copyright 2013

43

5.4.3 English Language Results The following rating guidance is for raters in non-English locales. You may stop reading this section if the language spoken in your locale is English! Your utility ratings need to reflect how helpful the result is for users in your locale. We know that you can read English (you are reading this document!), but you should only give high utility ratings to English results if users in your locale would expect or desire them. Unless requested by the query, English results should be considered useless if most users in the locale can't read them. Here are some examples using Korean (KR) as the task language. In Korea, most users don't read English. Unless the query "asks for" English results, we will consider them unhelpful or even useless (OT):
Query and User LP of the Result Block, Result Block, Utility Rating Landing page link: LP Explanation

This Korean language Wikipedia page on Barack Obama is very helpful for Korean (KR) users. [Barack Obama], Korean (KR) Find information about Barack Obama. Landing page link: LP

This English Wikipedia article about Barack Obama has similar content to the Korean Wikipedia page. However, very few users would be able to read this page. (Note: this query does not “ask for” English language results.)

Landing page link: LP

[Titanic 1997], Korean (KR) Purchase a DVD or find information about the movie “Titanic,” released in 1997.

Although the query was typed in English, most Korean users would expect to see Korean language reviews or Korean stores for purchasing the movie. This Korean result on a Korean website is very helpful.

Landing page link: LP

This English language landing page has helpful content. However, very few users would be able to read this page. (Note: this query does not “ask for” English language results.)

In some locales, English is one of the official languages or a commonly spoken language. In these locales, English websites are easy to use and could be helpful (depending on the query). For example, the Singapore government recognizes four official languages: English, Malay, Chinese, and Tamil, but English is the first and most dominant language in Singapore.

Proprietary and Confidential – Copyright 2013

44

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

This Wikipedia page in English about Obama would be very helpful to users in Singapore. [Barack Obama], Chinese_Simplified (SG) Find information about Barack Obama. Landing page link: LP This Wikipedia page in Chinese about Obama would also be very helpful to users in Singapore.

5.5

Rating Results with Content that Doesn’t Load

5.5.1 Using the Didn’t Load (D) Flag Didn’t Load (D) is used to indicate technical problems that prevent you from viewing any LP content. Use the D flag when: • • The MC of the landing page is a web server or web application error message and there is no other content on the page (no navigation links, no home link, no SC, no Ads). See this Wikipedia article for descriptions of different types of error messages. The landing page is completely blank (no MC, no SC, no Ads).

Assign the D flag based on the landing page, not the result block. Here is an example of a D landing page. You cannot tell that the landing page doesn’t load by looking at the result block.
Query and User Intent LP of the Result Block, Result Block, Utility Rating Landing page link: LP [douglas instruments], English (US) Navigate to the homepage of this website. This page displays a generic 404 message. There is no MC, SC, or Ads on the page. This result merits an OT utility rating and the D flag. Explanation

Here are screenshots of other types of landing pages that should be assigned the D flag: D1, D2, D3.

Proprietary and Confidential – Copyright 2013

45

Additionally, use the D flag for • • Malware warnings, such as “Warning – visiting this web site may harm your computer!” Pages that solicit certificate acceptance requests

Here is an example of a malware warning:

Here is an example of a certificate acceptance request:

5.5.2 Rating Results that Do Not Load All result blocks must be given a utility rating. If the landing page truly doesn’t load, assign the D flag and rate the page OT. True Didn’t Load (D) pages are useless. Sometimes the page partially loads or has an error message. Give utility ratings based on how helpful the result is for the query. Error messages can be customized by the webmaster and are part of a well functioning website. Sometimes these pages are helpful for the query.

Proprietary and Confidential – Copyright 2013

46

Query and User Intent [boys snow shoes], English (US) Find information about or purchase boys snow shoes.

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation The MC has an error message, but the LP has a lot of SC. However, the page has no information about boys’ snow shoes and is unhelpful for the query. Do not assign the D flag.

Landing page link: LP [bible passages], English (US) Find specific passages in the bible.

In spite of the customized “No results found” message on the LP, it has links to all passages in the bible, organized by book. It would be very helpful for most users. Do not assign the D flag.

6.0 Location and Utility Rating

6.1

Locale is Important

In Understanding the Query (Section 2.0), we gave an example of a query that means different things to users in different locations. It is important to think about the locale when thinking about the query.

You must also think about the location of the results you are rating. For some queries, results from other countries may be helpful or desired.

Proprietary and Confidential – Copyright 2013

47

For other queries, results from outside the rating locale may be unhelpful or useless, even if the result is in the right language. For example most users want to make purchases from stores in their own country (unless the item is very unusual).

Do not hesitate to lower the rating to OT if there is a mismatch between the locale and the result which makes the result useless for a user in the locale.
Query and User Intent LP of the Result Block, Result Block, Utility Rating Landing page link: LP This result is helpful for users in the US. The ingredients and measurements are familiar to US users. [white chocolate berry cheesecake recipe], English (US) Find a specific type of cheesecake recipe. Landing page link: LP Most US users would have difficulty making this recipe. The measurements are in metrics and some of the ingredients and terminology are British. This result is not helpful for US users. Explanation

Proprietary and Confidential – Copyright 2013

48

Query and User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

Landing page link: LP [human rights violations], English (US) Find examples or information about human rights violations.

Most people in the US would be interested in international human rights violations. For this query, results about countries other than the US are just fine. International news sources might be some of the best resources. All these results are helpful for the query. Use your common sense to decide what a user in your location would be interested in.

Landing page link: LP

[washing machines to buy], English (US) Buy a washing machine; compare prices on washing machines.

Landing page link: LP

Most users want to purchase common items from stores in their locale. It would be too expensive to ship a washing machine from the UK to the US. This result is useless for US users.

Landing page link: LP [house painters], English (US) Find a company to do house painting. A painting contractor in the UK is useless for a US user looking for a house painter.

[car insurance], English (US) Research or purchase car insurance; compare car insurance rates.

Landing page link: LP An insurance company that operates in Ireland and sells insurance to users in Ireland is useless for US users researching car insurance.

Landing page link: LP [purchase boys bedding online], English (US) Purchase bedding for a boys room online. Cottonbox is a children’s linen store in Australia. They ship to Australia and New Zealand, so this result is useless for US users looking to buy linens.

Proprietary and Confidential – Copyright 2013

49

6.2

Queries that “Ask” for Nearby Results or Nearby Information

Some queries “ask” for nearby information or nearby results. We call these "local intent" queries.

Here are more examples of local intent queries: • [pizza] • [time] • [weather] • [yoga class] • [coffee shops] • [movie showtimes] • [train station] • [car repair] • [plumber] • [swimming pool] • [dentists] • [farmers market] • [bank of america atm locations]

Proprietary and Confidential – Copyright 2013

50

Many queries are clearly not asking for nearby information or nearby results. We call these “non-local intent” queries.

Here are more examples of non-local intent queries: • [hotmail] • [easter] • [scrabble cheat] • [translate] • [definition of sedentary] • [aapl] • [beyonce] • [angry birds] • [small dog breeds] • [dance videos] • [oscars 2012] • [pick up lines] • [boyfriend lyrics] • [bank of america login]

Proprietary and Confidential – Copyright 2013

51

And some queries could go either way. Some users may want nearby results and others may not.

Here are some examples of queries with both local and non-local intent • [hotels] • [post office] • [apple store] • [Citibank] • [bmw] • [best buy] • [office depot] • [target] • [library] • [bank of america] • [the gap] • [barnes and noble] Use your common sense when thinking about queries and whether they have possible local intent.

Proprietary and Confidential – Copyright 2013

52

6.3

User Location and Understanding the Query

Every query has a locale. Some queries also have information about where the user is located: the user location. The user location is always inside the locale and is usually a city. Sometimes the user location can change our interpretation of the query or our understanding of user intent. For users in Sunnyvale, California, the query [turmeric] could have two different interpretations: a popular restaurant named Turmeric or the spice turmeric.

Proprietary and Confidential – Copyright 2013

53

In most other user locations, there is no restaurant (or anything else) named Turmeric. In most user locations, there is just one interpretation of the query [turmeric]: the spice. Even with no user location, we will assume that the dominant interpretation of [turmeric] is the spice. The Sunnyvale restaurant is not well known outside of Sunnyvale, California.

For many or most queries, the user location does not change our understanding of the query. Here are some examples: • [facebook] • [pictures of kittens] • [how many miles between the earth and the moon] • [reviews of hd tv] • [yahoo mail] • [Chinese restaurants] • [pizza] • [Citibank]

When is the user location important in understanding query interpretation and user intent? Sometimes or rarely-- it depends on the query. Please use both web research and your personal judgment to answer this question. Ask yourself, "Would users in one city be looking for something different than users in another?" For many queries, the answer to that question is “no.”

Proprietary and Confidential – Copyright 2013

54

6.4

Queries with an Explicit Location

Sometimes users tell search engines exactly what kinds of results they are looking for by adding the desired location in the query. We'll call this location inside the query the "explicit location." The explicit location makes queries much easier to understand and interpret.

Sometimes the explicit location matches the user location or locale, and sometimes it doesn't. When there is an explicit location in the query, pay attention to it! Users have told the search engine something very important by including it - exactly what they are looking for.

Proprietary and Confidential – Copyright 2013

55

6.5

Vital Ratings for Queries with a User Location

In most cases, the user location does not affect the Vital rating.

Sometimes, user location allows us to better understand the intent of the query and give a result a Vital rating. Here is an example of this relatively rare situation. There are multiple libraries named Belmont Library in the US. Without a user location, we do not know which library the user is looking for and no V rating is possible. With a user location, a V rating is possible.

Proprietary and Confidential – Copyright 2013

56

6.6

Rating Queries with User Location and Explicit Location

The user location may play a large role in assigning utility ratings when: • • The user location changes our understanding of the query or user intent. The query is a local intent query, i.e., the user is looking for local information or results nearby.

For many queries, the user location is not very important when assigning utility ratings. The explicit location always plays a large role in understanding the query and assigning utility ratings.

6.7

Rating Local Intent Queries

When there is a user location for a local intent query, such as [chinese restaurants] with a user location of Boston, MA, results in or near the user location are the most helpful. How close is "near"? Most people are not willing to travel very far for a gas station, coffee shop, supermarket, etc. Those are types of businesses that most users expect to find very nearby. Users might be willing to travel a little farther for certain kinds of local results: doctors’ offices, libraries, specific types of restaurants, public facilities like swimming pools, hiking trails in open spaces, etc. Sometimes users may accept results that are even farther away, such as a very specialized medical clinic. In other words, when we say users are looking for results "nearby," the word "nearby" can mean different distances for different queries. So, as always, please use your judgment.

6.8

Rating Examples with User Location, Explicit Location, and Local Intent
LP of the Result Block, Result Block, Utility Rating Landing page link: LP Explanation This LP is a fairly comprehensive list of Chinese restaurants in Mountain View. Many restaurants on the list have reviews. There is a map and features to narrow down the restaurants to certain categories.

Query, Explicit and User Locations, User Intent

Query: [chinese restaurants], English (US) Explicit Location: None User Location: Mountain View, California User Intent: This query has local intent. The user wants to find Chinese restaurants in the city of Mountain View, California.

Landing page link: LP This landing page is the website of Hunan Chili, a Chinese restaurant in the user location.

Landing page link: LP This landing page is for a Chinese restaurant in Chicago. It is unlikely users in Mountain View would find this helpful.

Proprietary and Confidential – Copyright 2013

57

Query, Explicit and User Locations, User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

This is a popular pizza shop in Mountain View, CA.

Landing page link: LP

Query: [pizza], English (US) Explicit Location: None User Location: Mountain View, California User Intent: Most users want to find a pizza place nearby. This query has clear local intent, whether or not the task has a user location.

This is one of many pizza shops in Mountain View, CA.

Landing page link: LP

This is a pizza shop in Los Angeles, which is about 350 miles from Mountain View. This result is unhelpful for this query.

Landing page link: LP Most users are looking for a pizza shop nearby. Finding pizza information is a less likely user intent.

Proprietary and Confidential – Copyright 2013

58

Query, Explicit and User Locations, User Intent Query: [target], English (US) Explicit Location: None User Location: None User Intent: This query has both local and nonlocal intent. Users may want to shop online or visit a store nearby.

LP of the Result Block, Result Block, Utility Rating

Explanation

Landing page link: LP The dominant interpretation of this query is the Target store and this is the official page of Target.

Landing page link: LP

The dominant interpretation of this query is the Target store and this is the official page of Target. The LP of target.com is V for this query even when the rating task has a user location.

Query: [target], English (US) Explicit Location: None User Location: Keizer, Oregon User Intent: This query has both local and nonlocal intent. Users may want to shop online or visit a store nearby. Landing page link: LP

This LP has the address, phone number and hours for the Target store in Keizer, Oregon. This result is very helpful for users who want to visit the store.

Query: [target keizer], English (US) Explicit Location: keizer User Location: None User Intent: This query has local intent. The explicit location (keizer) tells us that users are looking for the Keizer location of Target.

Landing page link: LP The explicit location (keizer) tells us that users are looking for the Keizer location of Target. This LP is the official page for the Target store in Keizer, Oregon.

Proprietary and Confidential – Copyright 2013

59

Query, Explicit and User Locations, User Intent

LP of the Result Block, Result Block, Utility Rating Landing page link: LP

Explanation

Query: [minimum wage rate], English (US) Explicit Location: None User Location: Wichita, Kansas User Intent: Find out what the minimum wage is. Different states in the US have different minimum wage rates. Users might be interested in the minimum wage in their state and how it compares to other places.

This is the official Kansas state website page on minimum wage for Kansas.

Landing page link: LP This LP has the minimum wage rates for all states. It is on the official US Department of Labor website.

Landing page link: LP This is the official California state website page on minimum wage for California.

Landing page link: LP

Query: [pictures of kittens] English (US) Explicit Location: None User Location: Pittsburgh, Pennsylvania User Intent: Users are looking for pictures of kittens. This is a non-local intent query. There is no obvious user intent to find pictures of kittens in Pittsburgh. The user location plays no role in the utility rating.

The LP is a page full of kitten pictures. The result block has a nice assortment of pictures as well.

Landing page link: LP This is a local listing of pets needing homes in the Pittsburgh area. There are no pictures of any pets directly on this page and few pictures on the individual listings.

Proprietary and Confidential – Copyright 2013

60

Part 2: Page Quality Rating Guidelines
7.0 Overview of Page Quality Evaluation

This part of the General Guidelines is about Page Quality (PQ) rating. Since page quality is an important consideration when assigning a utility rating, you should apply the concepts in this section to block utility (BU) rating tasks, as well as any other tasks that ask for a utility rating. Here is a screenshot of the Page Quality rating task page:

Proprietary and Confidential – Copyright 2013

61

In each page quality rating task, you will be given a URL. You will click on the URL and evaluate the landing page. There are three questions at the top of the PQ rating task page: 1) Does the landing page load? If you answer no to this question, you will merely submit the task. You may write a comment if you think it will be helpful. 2) Is the page in the task language, an acceptable language or English? (An acceptable language is one commonly spoken in your locale.) If you answer no to this question, you will merely submit the task. 3) Is the page porn? If you answer yes to this question, you will merely submit the task. For all other landing pages, you will answer a series of questions. Some of these questions focus specifically on the landing page while other questions are about the website that hosts the landing page. After carefully considering what you have learned from answering the page and website quality questions, you will use the sliding scale (slider) at the bottom of the task to assign an overall PQ rating of Highest, High, Medium, Low, or Lowest (or a rating in between two of these ratings). This is what the slider looks like on a page quality task:

7.1

Introduction to Page Quality

You have probably noticed that webpages vary in quality. There are high quality pages: pages that are well-written, trustworthy, organized, entertaining, enjoyable, beautiful, compelling, etc. You have probably also found pages that seem poorly-written, unreliable, poorly-organized, unhelpful, shallow, or even deceptive or malicious. We would like to capture these observations in PQ rating. Unfortunately, if we ask you to rate the quality of a page without giving any guidance, the result is dissent among raters. One rater will rate a page High quality, while another rates the same page Low quality. Why do we disagree? • • We may focus on different parts of the page or different aspects of the page. One rater might rate based on the content of the page and another based on the layout of the page. We may even have different ideas of what High quality means for a landing page. What makes an encyclopedia article High quality? What makes a product page High quality?

These guidelines are important because they explain what to consider when rating the quality of a landing page. When you read these guidelines, think carefully about the examples, and thoughtfully answer the questions about the landing page and website. As with all of your rating and feedback, this data is used for evaluation purposes only.

7.2

Important Information about the Page Quality Guidelines

The goal of these guidelines is to standardize our approach to page quality rating. An important part of this document is the examples, which include images and descriptions of webpages. Please make sure you look at each example. Clicking on the webpage images will enlarge them so that you can read the text on the page. At first, PQ rating may seem difficult. There are several aspects of the page and the website to look at and think about. As you gain PQ rating experience, you will be able to rate efficiently and with confidence. This type of rating takes practice. Rereading sections of these guidelines and thinking about the examples may help when you encounter difficult rating tasks. Please send feedback if you have a question about a particular rating task. Many examples and explanations have been added on the basis of rater questions.

Proprietary and Confidential – Copyright 2013

62

These guidelines are specific to “regular” webpages. Occasionally you may be asked to rate a landing page which is not a webpage. For example, you may be asked to rate a PDF file, a Microsoft Word document, a PNG or JPEG image file, etc. When the landing page of the URL is not a webpage, some of the questions in the rating task or considerations in these guidelines may not apply. In this case, please use your judgment. Do not consider the country or location of the page or website for page quality rating. For example, English (US) raters should use the same PQ standards when rating pages from other English language websites (UK websites, Canadian websites, etc.) as they use when rating pages from US websites. In other words, you should not lower your PQ rating because the page location (UK, Canada) does not match the task location (US). When you are rating PQ rating tasks, try not to think about how helpful the landing page could be for a particular query. Page Quality rating is query-independent, meaning that the rating you assign does not depend on the page’s relevance or utility to a query. Please do not struggle with each PQ rating. Please give your best rating and move on. If you are having trouble deciding between two ratings, please use the lower rating. Finally, the questions in the rating template and the quality considerations in the Page Quality Rating Guidelines do not cover absolutely every aspect of page quality. If you find pages which you truly believe to be high or low quality, please rate them as such, even if the reason is based on something not covered in this document. Please explain your reasoning and include any additional criteria you considered in the comment section. As always, we ask you to use your judgment.

7.3 Your Money or Your Life (YMYL)
In the following sections of these guidelines, you will learn about characteristics of landing pages and websites that are important considerations when evaluating PQ. There are some pages for which PQ is particularly important. We call these pages Your Money or Your Life (YMYL) pages. They are pages that can have an impact on your current or future well being (physical, financial, safety, etc.). YMYL pages should come from reputable websites and the content should be created with a high level of expertise and authority. Here are some examples of YMYL pages: • • • • • Pages that solicit personal information, such as personal identification numbers, bank account numbers, drivers license numbers, etc., which could be used for identify theft. Pages used for monetary transactions, on which users might give their credit account or bank account information; for example any page that allows you to buy something. Pages that offer medical or health information that could impact your physical well being. Pages offering advice on major life decisions, such as pages on parenting, purchasing a home, a vehicle, etc. Pages offering advice on major life issues that could impact your future happiness and finances, such as pages giving legal or financial advice.

Before evaluating a page for page quality, you should ask yourself whether the landing page is a YMYL page? If the answer is Yes, please have especially high standards when answering all questions for PQ rating. In particular, please be very careful evaluating the reputation of the page and whether the content has been created with the necessary authority and expertise. Here are some examples of pages where authority and expertise would be less important from the user’s point of view: • • • Pages on a gossip website. Pages on a humor website. Pages on an entertainment website.

Your standards may be somewhat lower when evaluating these types of pages.

Proprietary and Confidential – Copyright 2013

63

8.0

Landing Page Considerations

In the following sections, we discuss the aspects of page quality specific to the landing page of the title link, as well as what you need to examine on the website that the landing page belongs to.

8.1

Identifying the Purpose of the Page

In Section 3.2, you learned about identifying the purpose of the page. As a reminder, every page on the Internet is created for a purpose (or for multiple purposes). Most pages are created to be helpful for users. Some pages are created merely to make money with little effort to help users. Some pages are even created to cause harm to users. The examples you saw in Section 3.2 were fairly straightforward. With the explanations provided, you should have been able to understand which pages were and were not created to be helpful for users. Here are two examples of helpful pages where the purpose of the page is not as obvious: 1) Christopher Columbus page: This page looks quite nice, but it starts off with "Christopher Columbus was born in 1951 in Sydney, Australia." This is obviously inaccurate! What is the purpose of this page? In this case, exploring the website that hosts the page can help us understand its purpose. This website was built by educators to teach about interpreting information found on the Internet. After reading about the website here: About This Site, it should be clear that the purpose of this page is to serve as an educational tool. The information on the page is deliberately inaccurate so that it can be used as an example of misinformation on the Internet. Note: This Christopher Columbus page is actually a High quality page. Once you understand its purpose (to serve as an educational tool), it becomes clear that this page is carefully thought out, well executed, and achieves its purpose well. 2) Here is another example of a page that at first glance may seem pointless or strange: OmNomNomNom page. This page is from a humorous site that encourages users to post photos with mouths drawn on them. The purpose of the page is humor or artistic expression. Even though the "About" page on this website is not very helpful, the website explains itself on its "FAQ" page: OmNomNomNom FAQ page. Why do we care about the purpose of the page? The purpose of the page will help you answer all of the landing page and website questions. All questions should be answered in the context of the purpose of the page. Ultimately, your PQ rating will depend on how well the page achieves that purpose, given everything you have learned from answering the landing page and website questions. Keep in mind that for almost any helpful purpose, it is possible to find examples of high and low quality pages. The purpose of the page alone does not determine the PQ rating. For example, we will not consider informational pages to be higher quality than entertainment pages, even though they often have more serious content. There are high and low quality informational pages, and there are high and low quality entertainment pages.

8.1.2 Lack of Purpose, Harmful Purpose, and Deceptive Pages As a rater, you may encounter the following types of pages.


Lack of purpose: Sometimes it will be impossible to figure out the purpose of the page despite your best effort to do so. Many "lack of purpose" pages are gibberish (nonsensical) or auto-generated. These pages serve no real purpose. Harmful purpose: Some pages are clearly harmful or malicious. For example, pages designed to "phish" for the user’s government-issued identification number (such as a Social Security number), bank account information, or credit card information are harmful because the purpose is to steal private information. Malicious download pages are another type of harmful page.



Proprietary and Confidential – Copyright 2013

64



Deceptive pages: These pages are designed to look as though they have a helpful purpose but actually exist for some other reason. Deceptive pages are usually created to make money using ads or affiliate links rather than to help users. For example, some deceptive pages are designed to look as though they have helpful information, but in reality they are created to get users to click on ads.

Here are some examples of pages that were not created to be helpful for users and should always be considered overall Lowest quality.
Type of Lowest Quality Page Example Lack of Purpose 1 Lack of Purpose 2 Lack of Purpose 3 Lack of Purpose 4 Lack of Purpose 5 Lack of Purpose 6 Deceptive 1 Deceptive 2 Explanation

Lack of Purpose

These pages do not have a purpose.

Deceptive Deceptive

Not only does this page have many ads, but all the helpful-looking links lead to pages full of ads and very little text. The title of this page is “Washing Machine Reviews,” but all the content is product information copied from another source and all links are affiliate links. The purpose of this page is clearly to profit from an affiliate program, rather than publish reviews. The title of this page is “Rachael Ray Diet Blog,” but the page has nothing to do with Rachael Ray or her diet or her products. In fact, there is a brown-text-on-brown-background section at the bottom of the page (which we consider to be hidden text) that says “Disclaimer: Rachael Ray is not affiliated with nor does she sponsor or endorse this blog.” Important: This page is deceptive in spite of the disclaimer! This page is malicious. It is a shopping page that “phishes” for private information, including the user’s Social Security number, driver’s license number, and bank account information.

Deceptive

Deceptive 3

Malicious

Malicious Example

Note: Lack of Purpose pages, Harmful pages, and Deceptive pages all violate the Quality Guidelines section of Google’s Webmaster Guidelines. In general, any page or website which violates the Quality Guidelines section of Google’s Webmaster Guidelines should be considered Lowest quality.

8.2

Identifying the Main Content, Supplementary Content, and Advertisements

In Section 3.3, you learned about the parts of the web page. As a reminder, all of the content on a webpage can be classified in the following way: Main Content (MC), Supplementary Content (SC), and Advertisements (Ads). The landing page questions in PQ tasks require you to identify all parts of the landing page. Here is the summary table we showed you in Section 3.3.4. Please take another look at the examples. They will help you understand what parts of the page are MC, SC, and Ads.
Type of Page and Purpose News website homepage: the purpose is to display news. News article page: the purpose is to display a news article. Store product page: the purpose is to sell or give information about the product. Video page: the purpose is to allow users to view a video. Currency converter page: the purpose is to calculate equivalent amounts in different currencies. Blog post page: the purpose is to display a blog post. Search engine homepage: the purpose is to allow users to enter a query and search the Internet. Bank login page: the purpose is to allow users to log in to bank online. MC, SC, and Ads Highlighted Summary - News Homepage Summary - News Article Summary - Shopping Page Summary - Video Page Summary - Currency Converter Summary - Blog Post Page Summary - Search Engine HP Summary - Bank Login Page

Proprietary and Confidential – Copyright 2013

65

8.3

Rating the Quality of the Main Content

The Main Content is the reason the page exists, and rating the quality of the MC is the most important step in PQ rating. You must think about the purpose of the page in order to evaluate the MC. High or highest quality MC allows the page to achieve its purpose in a highly satisfying way. Understanding the purpose of the page is extremely important for rating the quality of the MC. (Remember that all your rating must be done in the context of the purpose of the page.) For each page, spend a few minutes examining the MC. Read the article, watch the video, examine the pictures, play with the calculator or online game, etc. Remember - MC also includes page features and functionality, so test the page out. For example, if the page is a product page on a store website, put at least one product in the cart to make sure the page and the shopping cart are functioning. If there is a lot of content, give yourself a few minutes to browse through the MC on the page. Then assign a rating to the quality of the MC: • • • • • • no main content lowest low medium high highest

The purpose of the page will help you determine what high quality content means. For example, high or highest quality encyclopedia articles should be accurate, clearly written, and comprehensive. High or highest quality shopping content should help you find the products you want, research the products thoroughly, and make purchasing the products easy. High or highest quality humor content should be entertaining. For all pages and all purposes, creating high quality MC takes a significant amount of at least one of the following: time, effort, expertise, and/or talent. High or Highest Quality Main Content Let's consider various types of pages and purposes and what is needed to create high or highest quality Main Content. • Medical Pages. Many medical pages exist to inform people on diseases and conditions. Highest quality medical content is written by people or organizations with medical accreditation (i.e., with professional expertise). The text and other content is written or produced in a professional style and is edited, reviewed, and updated on a regular basis, i.e., the content involves a high degree of time and effort to create and maintain. Medical pages are YMYL pages. Hobby Pages. Many people create pages to share information about their hobbies. Here, time and effort as well as expertise and possibly talent are important. The highest quality content is produced by those with a lot of knowledge and experience who then spend time and effort creating content to share with others who have similar interests. For example, fish aquarium enthusiasts have created some of the highest quality content on the web about how to set up and take care of a fish tank. Social Networking Pages. Social networking pages for individuals exist to allow people to connect socially and express their personality. Most pages are created by the person they are about, and so the creator of the page is an “expert”-- we are all experts on our own lives. High quality content on social networking sites is often the end result of a lot of time and effort. The content is frequently updated with lots of posts, social connections, comments by friends, links to cool stuff, etc. Social networking content with few or no updates and little engagement or little effort should be considered low quality. Video Pages. Many people post videos on video sharing sites. The content of these videos varies, from home videos to documentary footage of events. Videos vary tremendously in quality as well. Time, effort, expertise, and often talent are needed to create a high or highest quality video.







Proprietary and Confidential – Copyright 2013

66

Low or Lowest Quality Main Content Main Content quality may be rated low or lowest for many different reasons. Often, the content is created without adequate time, effort, expertise, or talent. Consider this. Most students have to write papers for high school or college. Many students take shortcuts to save time and effort by doing one or more of the following: • • • • • • • Buying papers online or getting someone else to write for them. Making things up. Writing quickly with no drafts or editing. Filling the report with large distracting pictures. Copying the entire report from an encyclopedia, or paraphrasing content by changing words or sentence structure here and there. Filling up pages with completely obvious sentences that repeat the topic of the paper. ("Argentina is a country. People live in Argentina. Argentina has borders. Some people like Argentina.") Using a lot of words to communicate only basic ideas or facts (“Pandas eat bamboo. Pandas eat a lot of bamboo. It’s the best food for a Panda bear.”)

Unfortunately, the content of many webpages is similarly created. When it is clear that the MC is created with deceptive intent and without putting in enough effort, time, expertise, or talent, please assign a low or lowest MC quality rating. Please note that copied or “scraped” content which is created with deceptive intent and very little time, effort, expertise, or talent violates the Quality Guidelines section of Google’s Webmaster Guidelines. Sometimes, time and effort were clearly involved when the page was created, but the content of the page does not allow the page to achieve its purpose. For example, expertise may be lacking on a topic for which expertise is really important. This content should be rated low or lowest.

Low or Lowest Quality Main Content Examples Here are some examples of low or lowest quality MC. Many of the examples are articles. You may find it helpful to read the MC text out loud. Think about the quality of the MC as you read.
Type of Low or Lowest Quality MC Very poorly written content A lot of text but little actual information Commonly known information only Commonly available content on nonauthoritative source Non-authoritative/ untrustworthy author or website MC Quality Example Low/Lowest MC 1 Explanation This content has many problems: poor spelling and grammar, complete lack of editing, inaccurate information, and no citing of sources of facts. Many of the sentences in this example are either meaningless or state something obvious. For example: "Popping pimples could be or could be not the new trend of getting rid of them." Most people already understand how to put on a hat. This content is commonly known information: “Turn the hat upside down. Pick up the hat you want to put on with both hands. Grasp the hat from the bottom.” While the text in this example is not copied word-for-word from a source we would consider original, many sources list these exact same techniques. Very little effort and no obvious expertise have gone into this particular example. And this is a topic where expertise/authoritative sources are important. This is a YMYL page. The level of expertise of the author of this content is not clearly communicated. Providing this background information is particularly important for medical, financial, or other subjects for which expertise is needed. This is a YMYL page.

Low/Lowest MC 2

Low/Lowest MC 3

Low/Lowest MC 4

Low/Lowest MC 5

Proprietary and Confidential – Copyright 2013

67

Type of Low or Lowest Quality MC

MC Quality Example

Explanation For auto-generated content, a basic template is designed from which hundreds or thousands of pages are created, sometimes by using an RSS feed or API. This particular page displays text (in the red box) explaining that the page is autogenerated. Auto-generated content should be rated lowest quality.

Auto-generated MC

Low/Lowest MC 6

Machine-generated text that has no meaning at all

Low/Lowest MC 7

This example is a page with random copied or gibberish text.

No MC at all, by deliberate design

Low/Lowest MC 8

Some pages have errors causing there to be no MC. Other pages intentionally have no MC. If the page has no MC, use the "no main content" rating. This is an example of a page that deliberately has no Main Content (the part that looks like MC is actually Ads). Note that this page is deceptive as well, since the title of the page is “chickenrecipes.com,” but the only content is Ads designed to look like a list of recipes.

Lowest Quality Main Content Examples- Copied, Scraped, or Paraphrased Content Reminder: Some lowest quality MC pages contain only copied or “scraped” content and were created with deceptive intent. This is a violation of the Quality Guidelines section of Google’s Webmaster Guidelines. Copied content should be considered lowest quality MC. Copied content may be: • • Copied exactly from an identifiable source. Sometimes a complete article is copied. Sometimes just parts of the article are copied. Text that has been copied exactly is usually the easiest type of copied content to identify. Paraphrased slightly, making it difficult to find the exact matching original source. Sometimes just a few words are changed. Sometimes whole sentences are changed. Because of the changes, this type of copied content is harder to identify.

See Section 10.5 for information on how to check for copied content. Some copied content pages get all of their content by making a copy of results from a search engine or news source. Because these are copies of “dynamic” pages that change frequently, you often will not be able to find an exact matching original source.
Type of Copied MC Description All of the MC is copied. The MC consists primarily of content copied from another source. Most of the MC is copied. Sometimes, we feel very sure that the content is copied, but we are unable to find the original source because it no longer exists. No effort has gone into creating this page. All content comes directly from “feeds” from other sites. Copied Content Example MC Quality - Copied 1 MC Quality - Copied 2 Original Source of the Content MC Original Source 1 MC Original Source 2

MC Quality - Copied 3

The original appears to no longer exist.

The MC consists of just RSS feeds, news feeds, or a copy of results from a search engine.

MC Quality - Copied 4

RSS feeds, news feeds, search engine results.

Proprietary and Confidential – Copyright 2013

68

8.4

Rating the Quantity of Helpful Main Content

Overall High or Highest quality pages have enough helpful Main Content to accomplish their purpose and be very satisfying to users. The quantity of helpful MC on overall Low or Lowest quality pages is often insufficient for their purpose. Some Low quality pages are unsatisfying and do not achieve their purpose well because they have a bare minimum of helpful MC. Some Lowest quality pages have so little helpful MC (or no MC) that they do not achieve their purpose at all. Sometimes there is a lot of MC (for example, many words or pictures), but it is not helpful for the purpose of the page. In all these cases, there is not enough helpful MC to achieve the purpose of the page. Use the following ratings to indicate the quantity of helpful MC on the page:
Quantity of Helpful Main Content Rating no main content unsatisfying so-so satisfying very satisfying Description No MC at all on the page. Not enough helpful MC to achieve the purpose of the page and be satisfying to users. Just enough helpful MC to achieve the purpose of the page and be somewhat satisfying to users. Enough helpful MC to achieve the purpose of the page and be satisfying to users. Enough helpful MC to achieve the purpose of the page very well and be very satisfying to users.

The amount of helpful content needed depends on the purpose of the page. Here are some MC rating examples:
Purpose of the Page MC Quantity Example Very Satisfying – a page about the French Revolution. Informational: the amount of content needed to be satisfying depends on the topic and the purpose of the page. Unsatisfying – another page about the French Revolution. Satisfying – a page about baroque pearls. Very Satisfying – a page for users interested in the TomTom XXL 550 GPS navigator. Shopping/ Informational: A very satisfying shopping page might include all or many of the following: photos, specifications, manufacturer information, professional and user reviews, and convenient shopping features such as a menu to choose size and color. Unsatisfying – another page for users interested in theTomTom XXL 550 GPS navigator. Explanation An informational page on the French Revolution should have a lot of information because this is a very large topic. This page has a satisfying amount of helpful MC for its purpose. The title of this page indicates that it is about the French Revolution. However, it’s a random collection of specific content that makes it unsatisfying for the general topic of the French Revolution. This page is about baroque pearls, which is a much narrower topic than the French Revolution. This page does not have a lot of content, but it is still satisfying for its purpose. Note that the tabs on the page lead to even more information and have many customer reviews. Please consider the content under or behind tabs to be part of the content of the page. Such content should even be considered part of the MC of the page. This page for the same product has very little helpful MC. Note that the image of this webpage has been annotated with some additional information to help you recognize that this page is a “thin affiliate” and is therefore a violation of the Quality Guidelines section of Google’s Webmaster Guidelines. The amount of MC in this example is unsatisfying. This page has a very satisfying amount of MC for users interested in short wedding dresses. An abundance of pictures plus options to view by price range, style, etc. are part of what makes this page so satisfying. This page achieves its purpose very well. This page does not have enough MC to be satisfying for its purpose.

Very Satisfying – a page for users interested in short wedding dresses. Unsatisfying

Proprietary and Confidential – Copyright 2013

69

Purpose of the Page Login: Login pages and some other types of pages need very little content to be satisfying or achieve their purpose.

MC Quantity Example

Explanation This page has login functionality, as well as clear information about what the user is logging into. In this example, the MC is boxed in red. Please read the MC in this example, including the completely unhelpful "answer" to the question in the red box. Some websites rely on users to create virtually all of their MC. If there is no user participation, the amount of content on the page is unsatisfying. This Q&A page has no answer to the question. (Note that it is possible that the page may someday have participation and more content.) This gibberish page has a lot of text, but none of it is helpful or satisfying for any purpose.

Satisfying

Unsatisfying Q&A: For a Q&A page, the MC includes the question and the answers.

Unsatisfying

Lack of Purpose: It is possible to have an unsatisfying amount of helpful MC on a page even though there is lot of text.

Unsatisfying

8.5

Rating the Helpfulness of the Supplementary Content

Supplementary Content can be a large part of what makes a High or Highest quality page very satisfying for its purpose. Features designed to help shoppers find other products they might also like can sometimes be as helpful as the MC of a shopping page. Ways to find other cool stuff on entertainment websites can keep users happily browsing. Sometimes, the comments on a blog post are the most interesting part. Take a look at the SC and rate it on this scale: • • • • • no supplementary content distracting/not helpful so-so helpful very helpful

So-so or Unhelpful Supplementary Content Examples Low or Lowest quality pages frequently have SC which is unhelpful or distracting for the purpose of the page.
SC Rating distracting/unhelpful SC Example SC - Distracting/Unhelpful Explanation This page has a lot of text at the bottom and the sides which is not actually helpful to users. See the content in green towards the bottom. (Note: This page also has no MC and is overall Lowest quality.) Some pages have way, way too many links, obscuring the page and distracting from the MC. This example is cropped to make it easier to view. The full webpage has even more links than are shown in this image. The only links on this page are the buttons saying “Click Here.” These are affiliate links to merchant sites, which allow the website owner to monetize the page. On the surface, this page seems to be a product review, but closer inspection shows that the page is designed to get users to click on the “click here” links: there is no other way off this page. This page is deliberately created to make money by trying to get users to click on the monetized links. This page has no SC by design, in order to increase the chances of a click on the monetized links. (This is an example of a deceptive page and is considered overall Lowest quality.)

distracting/unhelpful

SC - Distracting/Unhelpful

no supplementary content

No SC

Proprietary and Confidential – Copyright 2013

70

Helpful or Very Helpful Supplementary Content Examples
SC Rating very helpful helpful SC Example SC Video - Very Helpful SC Shopping - Helpful Explanation The MC of this video page is a "Saturday Night Live" episode. Below the main video, there are many other videos that users may be interested in. This SC is very helpful. This shopping page has helpful navigation features at the left to move around to different categories of shopping items. This SC is helpful.

Here are some recipe pages which range from very helpful to so-so in SC:
SC Rating very helpful Recipe SC Example Very Helpful Explanation This recipe page has helpful features, such as the ability to print out a list of ingredients. Behind the tabs, there are reviews, nutritional information, a video on how to knead bread dough, etc. The SC is very helpful for a recipe page. This page has reviews, a video, preparation time information, a “recipe box” feature, etc. This SC is very helpful for a recipe page. While there is a lot of helpful Main Content on this page (including both text and images), there is little helpful recipe-specific Supplementary Content. There are no reviews, there is no shopping list, there is no recipe printing feature, etc. This SC is just so-so for a recipe page.

very helpful

Very Helpful

so-so

So-So

You must consider the purpose of the page to decide whether SC is helpful or distracting. Helpful SC for recipe pages (e.g., features to print lists of ingredients or videos on cooking techniques mentioned in the recipe) may be very different than helpful SC for encyclopedia pages (e.g., a list of other resources for the topic). Reminder: These Page Quality guidelines are specific to “regular” webpages. Occasionally you may be asked to rate a landing page that is not a webpage. For example, you may be asked to rate a PDF file, a Microsoft Word Document, a PNG or JPEG image file, etc. PDF files and other non-webpages may not have any SC. For these types of pages, the absence of SC is expected and is not a sign of low quality. Please rate these types of pages using your best judgment. Here is an example of a high quality PDF file, which we display here as an image (just like all other examples in these guidelines). There is no SC, but none would be expected for this type of file: High Quality PDF. Of course, not all PDF files are high quality. Here is an example of a gibberish PDF which should be rated overall Lowest quality: Lowest Quality PDF.

8.6

Rating the Layout of the Page/Use of Space on the Page

Use of space refers to the position of and the amount of space on the page dedicated to MC, SC, and Ads. You will rate on this scale: • • • • • misleading or deceptive poor so-so good excellent

The page should be designed and organized to achieve its purpose. While every page is different, pages that use space effectively should have these characteristics: • • • • • The MC should be prominently displayed and "front and center.” The MC should be immediately visible when a user opens the page. It should be very clear what the MC actually is. The page layout, organization and use of space, as well as the choice of font, font size, background, etc., of the page should make this clear. The MC and SC together should take up most of the space on the page. Ads and SC should be arranged so as not to distract from the MC. It should be clear what parts of the page are Ads, either by explicit labeling or simply by page layout.

Proprietary and Confidential – Copyright 2013

71

Some pages are "prettier" or more professional looking than others, but you should not rate based on how “nice” the page looks. Pages can have lovely images, a pretty background, or a professional "look," but fail to achieve their purpose. Low Quality Page with Poor Use of Space has a nice image at the top, but otherwise the use of space is poor: the MC is placed below many Ads (and note that the MC is also low quality). Here is an example of a very low quality page that looks professional: Low Quality Page with Good Use of Space. On the other hand, a page can be very functional, use space well, and achieve its purpose without being “pretty”: High Quality Page with Good Use of Space, but Not Pretty. Like everything else, good layout and use of space depends on the purpose of the page. What constitutes good page layout for a shopping page may be very different from what constitutes good page layout for an information page. Consider these two pages. The shopping page on the left has good layout and use of space for its purpose, which is to sell a child’s easel. The page on the right also has good layout and use of space for its purpose, which is to display the current time in different time zones. Both pages have excellent page layout and use of space for their purpose even though they look very different and the page on the right has little SC.
Learning Express Shopping Page US Naval Observatory Master Clock Time Page

Notice that the page layout in the example on this side is very clean and the information is displayed in a highly functional way. The only SC is a link, which gives users additional information about the source of this displayed content. This page achieves its purpose well and is high quality.

You must think about the purpose of the page and whether the page layout allows the page to achieve its purpose. Let's look at a few other examples. Please note that we are only discussing the use of space in these examples. The first three examples are Low or Lowest quality pages.

Proprietary and Confidential – Copyright 2013

72

Use of Space Rating misleading or deceptive poor

Use of Space Example Misleading or Deceptive Poor

Explanation Advertising should never disguise itself as the MC of the page. Pages with Ads that are designed to look like MC should be considered deceptive. Advertising should never make it very difficult or impossible to find or use the MC. In this example, some users might not even notice the MC because it is under a long list of Ads. A lot of scrolling is required to see the MC. This example has mildly deceptive layout. This is a download page, so the download text and links are the MC. But the large green “Download now” button at the top is an Ad. Users could easily click on that button without realizing that it is an Ad and not part of the MC. At first glance, it appears that this page has little MC and that the page was created with little effort. However, close inspection of the page and consideration of the purpose of the page reveal that this page has good layout and use of space for its purpose, which is to display the official US Naval Observatory Master Clock time in 7 different time zones. No additional content is needed, and the page displays the clock information in a clear, easy-to-read format. Users interested in this information would be happy with the page layout.

poor

Poor

Good

Good

Use of Space Examples for Q&A Pages Here are some examples for Q&A (question and answer) pages. Q&A pages with good use of space should devote much or most of the space to the questions and answers, and the layout/use of space should make the question and the answers clear. Good layout might also help users understand at a glance how many answers there are and which are the best answers, as well as clearly highlight the author and the dates of the answers. The SC should play a supporting role on the sides of the page, and Ads should be clearly distinguished from the SC and the answers.
Use of Space Rating Use of Space Q&A Example Explanation The use of space in this example is poor or even deceptive. The Ads and SC take up most of the space. The page also uses the same "look" (color, font, and layout of the text), both for answers to the question and for the Ads/links, making it very difficult to distinguish which are answers to the question and which are Ads. (There are other aspects of this page which make it overall Lowest quality: there are no features for users to rate the answers, and the page is overwhelmed by Ads and links.) This is another example of a Q&A page that has poor use of space. This question does not have an answer, but it might take you a while to realize this. The page has a lot of SC and Ads, much of which is shown where you would expect answers to be displayed. There is a section called “Relevant answers,” but in fact these are pages from the same website with different questions, many of which are actually unrelated to this question. The “Relevant answers” section should be considered SC. Compare the page to this example used earlier: MC Q&A Unsatisfying Quantity. The use of space in the earlier example makes it immediately obvious that the question is unanswered. Of course, with no answer, we will consider MC Q&A Unsatisfying Quantity overall Low quality, even though the use of space and layout is good. This is an example of good layout and good use of space on a Q&A page, It's very clear what each part of the page is. The question is immediately visible, the authors are next to the questions and answers, all responses are dated, and the answers are sorted according to user feedback so that the best answer is at the top, immediately following the question. Note: The overall PQ rating for this page is Medium.

misleading or deceptive

Misleading or Deceptive

poor

Poor

good

Good

Note: Invasive pop-ups or large flashing/animated/distracting Ads that cannot easily be closed are an example of poor use of space because they take attention away from the MC. Here is an example of a page with a pop-up that cannot be closed: Annoying Pop-Up. This page is considered “misleading or deceptive” and should get an overall PQ rating of Lowest.

Proprietary and Confidential – Copyright 2013

73

9.0 Answering Homepage and Website Questions
For page quality rating, we ask you to explore the website and answer some questions about the website. Website checks are very important for distinguishing between overall High and Highest quality, as well as between overall Low and Lowest quality. These website checks are especially important for YMYL pages, such as • • Stores or other websites which users entrust with credit card or other personal or financial information. Pages with medical or financial information.

9.1

Finding the Homepage of the Website

To answer the website questions, you must visit the homepage associated with the URL in the task. How do you find the homepage of the task URL? Click on the URL and examine the landing page. If the landing page is not the homepage of a website, you will usually see either a link labeled "home" or a logo to click on. The “home” link is usually at the top of the landing page, but sometimes you will see it at the bottom of the page. In the following examples, we have included the URL of the task page, as well as the URL of its associated homepage. We have also included an image that shows where to click on the landing page to navigate to the homepage. In the image, you will see a red box around the link or the logo you would click to navigate to the homepage.
URL on the Task Page http://www.williamssonoma.com/products/s hun-premier-7-pieceknife-block-set Homepage of the Website Image that shows where to click Williams-Sonoma Homepage

http://www.williams-sonoma.com

http://www.mattcutts.com/blog http://www.mattcutts.co m/blog/spam-reports-infive-languages If you are curious as to why http://www.mattcutts.com is not considered the homepage, go to http://www.mattcutts.com and see what that page looks like. It's pretty clear that the blog homepage is actually http://www.mattcutts.com/blog, not http://www.mattcutts.com. http://answers.yahoo.com In this case, we will consider http://answers.yahoo.com the homepage, rather than http://yahoo.com. Why? Because clicking on the logo takes the user to http://answers.yahoo.com. In addition, http://answers.yahoo.com has information about the Yahoo! Answers website. It is very difficult to find specific information about answers.yahoo.com on the yahoo.com homepage. http://hms.harvard.edu http://hms.harvard.edu/ about-hms/facts-figures In this case, we will consider the Harvard Medical School page at http://hms.harvard.edu to be the homepage, rather than http://www.harvard.edu (which is the homepage of Harvard University). Clicking the logo at the top of http://hms.harvard.edu/about-hms/facts-figures takes users to http://hms.harvard.edu, not to http://www.harvard.edu. Harvard Medical School Facts and Figures Page Specific Yahoo! Answers Page Matt Cutts Blog Page

http://answers.yahoo.co m/question/index;_ylt=A nAYEU1fED6ncg1jRCF y30kk5XNG;_ylv=3?qid =20091214193523AAQ qHQS

Proprietary and Confidential – Copyright 2013

74

URL on the Task Page

Homepage of the Website http://www.facebook.com

Image that shows where to click

http://www.facebook.co m/sarahpalin

In this case, we will consider http://www.facebook.com the homepage. Clicking the “facebook” logo at the top of http://www.facebook.com/sarahpalin takes users to http://www.facebook.com http://twitter.com

Sarah Palin's Facebook Page

https://twitter.com/Barac kObama

In this case, we will consider http://twitter.com the homepage. Clicking the “twitter” logo at the top of https://twitter.com/BarackObama takes users to http://twitter.com

Barack Obama's Twitter Page

Whenever possible, look for a “home” link on the landing page, or try clicking on the logo at the top of the page. Sometimes, the navigation links and structure also have a clear hierarchy, with the homepage featured prominently. Webmasters usually make it easy to get to the homepage of the website, and homepages usually have links to a lot of helpful information you need to evaluate PQ. If there is no clear home link or logo or navigation structure on the page you are evaluating, try looking at some other pages on the site. If finding the homepage is hard, usually either the website design is amateurish or the website is low quality. Use the questions in the next section to decide which. Important: If all else fails, you can modify the URL by removing everything to the right of “.com” or “.org” in the URL. Example 1: http://www.kernel.org/pub/linux/utils/boot/syslinux Screenshot of the LP Although there is no “home” link or logo on the landing page for this URL, we can get to the homepage (http://www.kernel.org/) by removing everything in the URL to the right of “.org”.

The landing page for kernel.org is titled “The Linux Kernel Archives,” and the page tells us that this “is the primary site for the Linux kernel source.” To find information about kernel.org, please visit the kernel.org Wikipedia page. You will learn that kernel.org is “a main repository of source code for the Linux kernel, the base of the popular Linux operating system.” You can read about Linux here: Linux Wikipedia page. Even though the landing page for http://www.kernel.org/pub/linux/utils/boot/syslinux has an atypical layout and is not a “pretty” page, the website is the highest authority on Linux, and the page achieves its purpose very well. Software developers would be happy with the directory landing page. Directory pages can be hard to rate. This particular directory page is a very high quality page. Other directory pages may be lower quality. Researching the reputation of the website will help you to assign an appropriate PQ rating.

Proprietary and Confidential – Copyright 2013

75

Example 2: http://www.library.cornell.edu/colldev/mideast/arbwomn.htm Screenshot of the LP Although there is no “home” link or logo on the landing page and the page does not have nice formatting, we can modify the URL by removing everything to the right of “.edu.” By doing so, we learn that the page comes from the Cornell University Library website:

This Cornell University Library Wikipedia article tells us that library.cornell.edu is the library system of Cornell University, a highly-respected university in the United States. In spite of the appearance of the page, we will take the reputation of Cornell University into consideration and assume that many users would be happy with the landing page. Occasionally, your rating task will include a URL for which there are two or more justifiable “homepage” candidates. For example, you may find a landing page that has a “home” link, as well as a website logo with a link. Usually these links go to the same page, but sometimes they have different landing pages. Or you may find a complicated relationship between subdomains and the top level domain associated with a URL. For example, you may not be sure whether the homepage of the URL http://finance.yahoo.com/news/category-stocks is http://finance.yahoo.com or http://www.yahoo.com). Many websites have a complicated directory structure, and you may not be sure which page is reasonably the “homepage” associated with the landing page of a given URL. As always, please use your judgment. You may consider information from any reasonable homepage candidate. In general, please prefer the homepage candidate which has the most information relevant to the landing page. Please note that we include finding the homepage as part of the Page Quality guidelines because we are trying to find information and answer questions about a specific landing page and the website it is associated with. Frequently, some of the information we need to know about the landing page (such as authorship, contact information, etc.) is not on the landing page itself because it is the same for every page on the site or subdomain, or it is contained within a certain branch of a directory on the site. Finding the homepage (or relevant subdomain or appropriate page in a directory) is a good technique for locating this kind of information. Please use any reasonable “homepage” candidate that allows you to find the information you need to rate effectively. Note: Amateurish (i.e., non-professional) website design may not be a PQ issue, at least for certain types of pages (for example, websites of family photos). Amateurish website design is less acceptable for YMYL websites, such as legal, financial, and medical websites or shopping websites that ask for your credit card information.

9.2

Is the Purpose of the Page Consistent with the Website?

Now that we have looked at the page and figured out its purpose, we will do the same thing for the website. Sometimes websites explain their purpose right on the homepage. Some websites put this information on a subpage. Examine the homepage for this information. Also look for "about" or "about us" links or "contact us" links or even "FAQ" links. You want to make sure the purpose of the page you are evaluating and the purpose of the website are consistent. Here are the website purpose questions in the rating task: • • Describe the purpose of the website. Is the page consistent with the purpose of the website?

Proprietary and Confidential – Copyright 2013

76

Consistency Check consistent

Example Page with a recipe for pumpkinpecan pie Page with videos unrelated to recipes

Information about the Website Help page which says that the site is a recipe site

Purpose of the Page Share a recipe.

Purpose of the Website Provide recipes and other cooking resources.

Explanation The stated purpose of the website and the page are consistent. The stated purpose of the website (to provide recipes) is inconsistent with the presumed purpose of the page (to display videos, all of which happen to be unrelated to recipes).

inconsistent

About page which says that the site is a recipe site

Share a large collection of clearly nonrecipe videos.

Provide recipes.

Note: Page with videos unrelated to recipes above has a collection of random and unrelated content and videos. Some people might justifiably say the page has no clear purpose. It also appears that the content may be copied or scraped from some other page on the web. These are some of the many indications that this page is overall Lowest quality. In many cases, landing page and website questions give a consistent picture. We ask you to do page and website checks because sometimes a Lowest quality page only clearly fails on one question. Page with videos unrelated to recipes fails on many. If the purpose of the page is very clearly inconsistent with the purpose of the website, please examine it closely. Usually, Lowest will be the most appropriate overall PQ rating. If the purpose of the website is deceptive or harmful, or if the website has no purpose (for example, the website is made up of gibberish pages), all pages on the website should be considered overall Lowest quality.

9.3

Who is Responsible for the Content of the Website and the Content of the Page?

Every page belongs to a website, and it should be clear: • • Who (what company, business, foundation, individual, etc.) is responsible for the content on the website. Who is responsible for the content on the page you are evaluating.

The website check questions you will answer in the rating task are: • • Is it clear who is responsible for the content on the website? Is it clear who is responsible for the content on the page?

Websites are usually very clear about who is responsible for the content on the website and the content on the page. There are many reasons for this. Commercial websites have copyright material they want to protect. Businesses want users to know who they are. Artists, authors, musicians, and other original content creators usually want to be known and appreciated. Foundations often want support and even volunteers. High quality stores want users to feel comfortable buying online. The homepage of the website should indicate who is responsible for the content of the website. Most websites have "contact us" or "about us" or "about" pages that give you information about who owns the site. Many companies have an entire website or blog devoted to who they are and what they are doing, what jobs are available, etc. Please look carefully and explore the website. Blogs or even social networking pages may be the way companies communicate who they are. Here is an example of a website where it is clear who is responsible for the content of the website: "WHO IS GuS?" page. This website has a nice page about who they are. Note that this page is titled “WHO IS GuS?” The page is not titled “about” or “about us.” You will need to look at the information provided on the website, not necessarily the title of the page on which the information appears.

Proprietary and Confidential – Copyright 2013

77

Important: The question “Who is responsible for the content of the website and the content of the page?” is not necessarily looking for an individual who is responsible. Often the website is responsible for the content, not an individual person. The IBM Corporation is responsible for the content on ibm.com. The Cleveland Clinic is responsible for the content on clevelandclinic.org. An individual is not responsible for the content on these websites, even though many individuals probably contribute to creating and maintaining the content. The business or organization is responsible for the content. Website Owner and Content Creator are Different In some cases, the website is owned by one company, but the content on the page is provided by a different company or individual. For example, social networking sites allow individuals to create pages. The social networking website is responsible for many aspects of the site, but individuals are responsible for what is on their pages (subject to the terms and restrictions of the social networking site).
Who is Responsible? clear Example Oprah's Facebook Page Page with no "about us" link or information About Us page with no true company information Explanation This is Oprah’s Facebook page. She or her media company is responsible for the content on this page. Facebook (the company) is responsible for the website. There is no “about us” information on who is responsible for the page content or for the website. The “About Us” page does not give the name of a company or physical address. No other page on this site has information either. Clicking the “contact us” link will open the default email program on your computer with an email addressed to the website. Clicking the link will not provide contact information for the website.

unclear

unclear

9.4

Does the Website Have an Appropriate Amount of Contact Information?

Most websites are interested in communicating with their users. Usually, this means that websites offer paths of communication that at the very least include phone numbers and email addresses. High or highest quality websites will usually offer many ways by which users can get in touch, such as email addresses, phone numbers, and physical addresses. Sometimes, this contact information is even organized by department and provides the names of individuals to contact. The types and amount of contact information needed depend on the type of page and the type of website. Contact information is extremely important for YMYL websites. Be extra critical of shopping websites. Most stores have contact information or contact processes prominently featured. Sometimes this information is listed as "customer service." Users often need to contact stores for questions and returns. Stores usually work very hard to win users' trust by offering many ways to contact them. Some retailers have special sections of their website devoted to customer service. Some kinds of websites need less detailed and a smaller amount of contact information for their purpose. For example, humor websites may not need the kind of detailed contact information we would expect an online banking website to have. Websites that require a lower level of trust should still have the name of the company, a phone number and/or an email address, at a minimum. Find the contact information associated with the website. Usually, homepages have a "contact us" link. There should be clear contact information somewhere on the website. Sometimes, the contact information is located on the "about us" page and sometimes it is displayed directly on the homepage. Please explore the website if you cannot find a "contact us" page. Sometimes you will find the contact information on a "corporate site" link. Be a detective. The website check question you will answer in the rating task is: Does the website have sufficient contact information for its purpose?

Proprietary and Confidential – Copyright 2013

78

Sufficient Contact Information? sufficient

Example

Explanation This page provides the street address and phone number of the company, as well as email addresses and phone numbers (and even names) of various people/departments in the company. These three examples are customer service pages on popular and well-regarded stores/shopping sites that help customers contact the store. Notice that these pages use a variety of approaches, from giving users a list of phone numbers to providing interactive features that navigate users through the customer service process. In this example, there is a "Company Site" link in the upper right hand corner (we added the blue box to help you find it). Clicking on that link gives the corporate site homepage: Morningstar corporate page. From the corporate homepage, there is a “Global Contacts” page under the “Company” tab: "Global Contacts" tab. Clicking on this link gives the “Global Contacts” page: Morningstar global contacts page. This rating task requires raters to be detectives. (Note: this is an example of a highly reputable financial site—we definitely do not want this type of page to get low quality ratings!) In this example, there is a blank email form with no indication where it would go. Although there is no other contact information anywhere on the website, this is a humor website and contact information is not very important. We will consider the contact information to be sufficient for this type of page. In this example, the landing page does not have a link to contact information, or even a “home” or logo link to get to the homepage. However, as you saw in Section 9.1, modifying the URL shows us that this page is on the Cornell University Library site: library.cornell.edu . This page has many tabs which serve to provide various types of contact information: About Us, Libraries, Hours and Maps, and Help. This is a blank email form with no indication where it would go. There is absolutely no other contact information anywhere on the website. Since this is an online store where users would enter credit card information, contact information is extremely important. This is a YMYL page. There is no contact information at all anywhere on this website. Since this site offers medical information, contact information is extremely important. This is a YMYL page. There is no contact information at all anywhere on this website. Since this site offers medical information, contact information is extremely important. (Note: there is also absolutely no information about who is responsible for the content of this website.) This is a YMYL page.

Sufficient Contact Info 1

sufficient

Sufficient Contact Info 2 Sufficient Contact Info 3 Sufficient Contact Info 4

sufficient

Sufficient Contact Info 5

sufficient

Sufficient Contact Info 6

sufficient

Sufficient Contact Info 7

insufficient

Insufficient Contact Info 1

insufficient

Insufficient Contact Info 2

insufficient

Insufficient Contact Info 3

9.5

What Kind of Reputation Does the Website Have?

A website's reputation is based on the experience of real users, as well as the opinion of people who are experts in the topic of the website. Stores frequently have user ratings, which can help you understand a store’s reputation based on the reports of people who actually shop there. Many other kinds of websites have reputations as well. You might find that a newspaper website has won journalistic awards. You might find that a medical information site is endorsed by physician groups. The reputation of a website is especially important for YMYL pages. The reputation of a website is also very important when the information on the website demands a high level of authoritativeness or expertise, such as medical information websites. When a high level of authoritativeness or expertise is needed, the reputation of a website should be judged by what expert opinions have to say. For example, high quality medical websites are endorsed by prominent physician groups.

Proprietary and Confidential – Copyright 2013

79

Reputation research in PQ rating is very important. A positive reputation from a consensus of experts is often what distinguishes an overall Highest quality page from a High quality page. A negative reputation should not be ignored and is a reason to give an overall PQ rating of Low or Lowest. The website check question you will answer in the rating task is: What kind of reputation does the website have? • • • • negative or malicious reputation mixed reputation positive or OK reputation little or no information found

Reviews and ratings of websites can be very helpful for determining reputation. Any store or website can get a few negative reviews, and that is normal. Large stores and companies have thousands of reviews and most receive a few negative ones. However, it is not normal for businesses or websites to have all or predominantly negative reviews. Use the “negative or malicious reputation” rating if there are many believable, detailed negative reviews (and very few positive reviews), or if there are informative/believable complaints from multiple sources detailing malicious behavior. Note: Frequently, you will find little or no information about the reputation of a small website. This is not indicative of positive or negative reputation. Sometimes, you will find information about a website which is not related to its reputation. For example, pages with information about Internet traffic to the website are not about the reputation of the website. Please select “little or no information found” if the information you find is unrelated to the reputation of the website. How to Search for Reputation Information: The usual way to find out about reputation is by searching for comments, reviews, and articles and references about the website. Usually, one quick search is all you need. Here is how to research the reputation of the website: 1. Identify the "homepage" of the website. 2. Try one or more of the following searches on Google: Format of Reputation Search Queries [homepage] [homepage.com] [homepage reviews] [homepage –site:homepage.com] [“homepage.com” –site:homepage.com] [“homepage.com” –site:homepage.com reviews” [link:homepage.com] Reputation Search Queries (using IBM as an example) [ibm] [ibm.com] [ibm reviews] [ibm –site:ibm.com] [“ibm.com” –site:ibm.com] [“ibm.com” –site:ibm.com reviews] [link:ibm.com]

3. Browse through the results to see what others have to say about the website. 4. For businesses, there are many sources of reputation information and reviews. Here are some examples: Yelp, SiteJabber, Better Business Bureau, Amazon, and Google Product Search. 5. Look for other websites that reference the website you are researching. On Google search, you can use the “link:” operator. 6. See if there is a Wikipedia article or news article from a well known news site. Wikipedia can be a good source of information about companies and organizations.

Proprietary and Confidential – Copyright 2013

80

Website

Wikipedia Page About the Website Wikipedia article about annualcreditreport.com

Description Users in the US can obtain free credit reports on this website by providing their Social Security Number. Notice that this Wikipedia article tells us that “AnnualCreditReport.com is the only federally mandated and authorized source for obtaining a free credit report.” Etsy.com is a popular shopping website for handmade and vintage and other unique items. Users provide credit account information when making purchases on the website. According to Wikipedia, the Cleveland Clinic “is currently regarded as one of the top 4 hospitals in the United States as rated by U.S. News & World Report.” Users can trust medical information on this website.

annualcreditreport.com Wall Street Journal article about annualcreditreport.com Wikipedia article about etsy.com etsy.com CNN article about etsy.com Wikipedia article about clevelandclinic.org clevelandclinic.org US News & World Report article about the best hospitals in the US

Here is an example that uses some of the website research techniques listed above. URL of the landing page: http://www.decormyeyes.com/pd.asp?prod_id=1940 (Decor My Eyes). 1. Identify the homepage of the website, which is "decormyeyes.com." 2. Issue the query [decormyeyes reviews] to see what others have to say about the website. Here is what this search looks like: Search results for [decormyeyes reviews]. You will notice many extremely negative reviews and complaints about this malicious site. The New York Times has an article extensively detailing the malicious behavior of this website. 3. Also try this query: [decormyeyes -site:decormyeyes.com] 4. Try the search [décor my eyes site:bbb.org]. You will see that the Better Business Bureau (BBB) gives this business a very low rating: BBB page about Decor My Eyes. 5. Look for articles on Wikipedia and well known news sites about DecorMyEyes. These articles on Wikipedia and the New York Times describe the deceptive techniques used by DecorMyEyes and provide other negative information about the website and its owner. Please note the following • The Better Business Bureau does not have ratings for all businesses. You will sometimes find high ratings on BBB because there is very little data on the business, not because the business has a positive reputation. However, low ratings on BBB are usually the result of multiple unresolved complaints. Please consider very low ratings on the BBB site to be evidence for a negative reputation. Including “site:homepage.com” in your query restricts the search results to include only pages from the site “http://www.homepage.com.” Another helpful trick is to try adding "-site:homepage.com" to the query, which restricts your search results to pages about homepage.com, but not on the website homepage.com.



Sometimes it is helpful just to search using the name of the homepage (i.e., "homepage"), and sometimes it is helpful to search for the URL (i.e., "homepage.com"). If you are having trouble finding any information, try both. Here are some other examples:
Website Reputation Research Search results for [atgstores.com complaints] Search results for [atgstores.com reviews] Reputation Explanation

atgstores.com

Negative

You will notice many detailed extremely negative reviews about this site. Please see these pages: ATQ Stores negative review 1, ATQ negative review 2, and ATQ negative review 3.

Proprietary and Confidential – Copyright 2013

81

Website

Reputation Research Search results for [help hospitalized veterans scam]

Reputation

Explanation You will notice many detailed negative articles about this organization on news sites and charity watchdog sites: HHV negative review 1, HHV negative review 2, HHV negative review 3, and HHV negative review 4. You will find two negative reviews. This one has lots of detail and is believable: Denison negative review . The review at the top of this page is questionable: Denison questionable review. It offers no details and feels spammy. Notice that there is a credible response by the owner further down the page. Although we could not find pages with positive reviews about this business, we can infer a positive reputation by looking at the following: Denison positive article, Denison positive article, and Denison positive article. In this case, we should consider the very credible sounding negative review, but it should not outweigh the overall generally positive reputation of the business. We would consider denisonyachtsales.com to have an OK or positive reputation.

hhv.org/

Negative

denisonyachtsale s.com

Search results for [denisonyachtsales.com reviews – site:denisonyachts.com]

Positive to Mixed

csmonitor.com (Christian Science Monitor)

Search results for [csmonitor.com site:csmonitor.com] Search results for [llbean.com]

Positive

Notice the highlighted section in this Wikipedia article about the Christian Science Monitor newspaper, which tells us that the newspaper has won seven Pulitzer prizes. From this information, we can infer that the csmonitor.com website has a positive reputation. From the numerous positive user reviews on these sites, we can infer that llbean.com has a positive reputation: LL Bean positive reviews 1, LL Bean positive reviews 2, and LL Bean positive reviews 3.

llbean.com

Search results for [llbean.com reviews – site:llbean.com]

Positive

Search results for [kernel.org] kernel.org Search results for [kernel.org – site:kernel.org] Positive

From this Wikipedia article, we learn that “Kernel.org is a main repository of source code for the Linux kernel, the base of the popular Linux operating system. It makes all versions of the source code available to all users. It also hosts various other projects, like Google Android. The main purpose of the site is to host a repository for Linux kernel developers and maintainers of Linux distributions.” Kernel.org is a highly reputable resource. It happens to have an atypical layout, but is considered to be very high quality.

Final notes: • Reputation is different than popularity. Popular websites are used by many people and are frequently (but not always) high quality. Reputation is based on both experiences of users as well as the opinion of experts in the topic of the website. Some medical websites with positive reputations are those recommended by prominent medical groups. Stores with good reputations are ones that serve their customers well (not just the ones that sell the most products). Reputation research is necessary. Do not just assume websites you personally use have a good reputation. Please do research! You might be surprised at what you find. You are not evaluating whether the website works or doesn’t work. If a website clearly has a bad reputation, the site is considered lowest quality regardless of whether the website works or not. Many “bad” websites readily take a user’s money or personal information. The important question is what the site does with the money or information.



Proprietary and Confidential – Copyright 2013

82

9.6

Is the Homepage of the Website Updated/Maintained?

High or highest quality websites stay current. Most continue to add high quality content on a frequent basis. For example, most news websites add content continuously throughout the day and archive content that is relevant. Users can trust high quality websites because they maintain their content. For example, medical advice changes all the time. A High or Highest quality medical information site will remove information that has become outdated and will quickly correct errors. The frequency of maintenance should depend on the purpose of the website. We would expect the homepage of a news website to be updated many times a day. Other websites might have a much slower cycle. Most homepages should be updated at least every few years. The website check question you will answer in the rating task is this: Is the homepage of the website maintained or cared for? Homepage Checks - You cannot check every page on a website to make sure the content is maintained, but please do inspect the homepage. Make sure that at least the homepage looks reasonably maintained and updated. Do the following: • Check copyright dates, if available. These should be recent, or at least not more than 2 years old. Note: Some amateur websites may not be vigilant in updating the site’s copyright date. If you see an old date but strongly feel the page is OK, you may consider the site maintained. For example, it’s probably OK and even expected if a website of family photos does not have a copyright date. It’s also OK if a small volunteer organization website has not updated its copyright date if the group is obviously still updating other parts of the homepage and website. But we would expect business and professional websites to have recent copyright dates. Look for a "last updated" date or other indication that the website is maintained. Most homepages should have evidence of recent updates. Check to see if links on the homepage work and that basic functionality of the homepage works. (One broken link or so is OK- every website has those now and then.) Do a few random clicks to check. The images and formatting should also look OK. The page should feel cared for.

• •

Medical Information Examples - Here are two examples (both from medical information sites) that do not appear to be updated/maintained. Note that medical websites require up to date information and a high degree of user trust. 1. Unmaintained website example: The information at the bottom of the page indicates it was last updated in 2005. It does not look like anyone is taking care of this website/maintaining the accuracy of this information. 2. Site for sale example: There is a note at the top that the site is for sale. There are many other indications that this page is overall Low quality, but the "for sale" notice should make you suspicious about updates/maintenance of the information.

10.0 Additional Page Quality Rating Guidance
This section is included to give additional rating guidance, answer rater questions, and address common rating mistakes.

Proprietary and Confidential – Copyright 2013

83

10.1 Assigning a Page Quality Rating to “Your Money or Your Life” (YMYL) Pages
Page quality is especially important for pages that can affect “your money or your life.” By this, we are referring to the following kinds of pages:

• • • •

Pages that offer medical/health information or other information that can impact your physical well being (your life). Pages offering advice on major life issues that may impact your future happiness. Pages in this category include legal advice pages, financial advice pages, etc., and even pages that offer advice on parenting, home buying, purchasing a vehicle, etc. (your life). Monetary transaction pages, where you give your credit card information (your money). Pages that ask for personal information like Social Security or other government issued identification numbers, bank account numbers, insurance account numbers, or anything else which could be used for identify theft (your money or your life).

We ask raters to have particularly high standards when considering the following aspects of PQ rating for YMYL pages:



• •



Main content quality, in particular who creates the content and the expertise of the author. Pages that exist to inform people on diseases and health related conditions should be written by people or organizations with professional expertise. The content should be written or produced in a professional style and be edited, reviewed, and updated on a regular basis. Contact information. Websites that require a high level of user trust, such as medical websites, banks, and even shopping sites that accept credit cards, require very helpful contact information. Reputation. The reputation of a website is also very important when the information on the website demands a high level of authoritativeness or expertise, such as medical information websites. When a high level of authoritativeness or expertise is needed, the reputation of a website should be judged by what expert opinions have to say. For example, high quality medical websites are endorsed by prominent physician groups. Maintained/updated website. Medical, legal, financial, and other YMYL websites require up-to-date information.

10.2 Assigning a Page Quality Rating to Encyclopedia Pages
There are many encyclopedia-type websites. Some are highly respected publications appearing in all major libraries and are standard references. Some are websites edited by anonymous users with no editorial oversight or fact checking. Reputation research is extremely important. High and Highest page quality ratings can only be used for encyclopedias with very good reputations. As a rater, you will frequently encounter Wikipedia pages. In general, Wikipedia does have a good reputation. Wikipedia is a very popular resource and generally valued for accuracy. However, there is no single author or organization that vouches for the accuracy of Wikipedia articles. Individual Wikipedia articles should be evaluated on the basis of page-level checks because the quality of pages on Wikipedia varies. Wikipedia articles with a lot of detailed, information-rich MC and external references can usually be rated in the High range. Some Wikipedia articles may even be rated as high as Highest, although this rating is usually considered too high for a Wikipedia article on a medical, legal, or financial topic which requires a high level of expertise. These types of articles can get PQ ratings up to High if they have a lot of high quality MC, are well-researched, and do a good job of citing their resources. Otherwise, a PQ rating in the Medium range is appropriate. Naturally, articles with very little MC should get lower PQ ratings.

Proprietary and Confidential – Copyright 2013

84

10.3 Assigning a Page Quality Rating to Pages with no Main Content/Error Messages
Pages may lack Main Content for various reasons. Sometimes, the page is “broken” and the content does not load properly or does not load at all. Sometimes, the content is no longer available and the page displays an error message with this information. Sometimes the page is deliberately designed without MC. Every website probably has some “broken” non-functioning pages. This is normal, and those individual nonfunctioning or broken pages on an otherwise maintained site should be considered overall Low quality. This is true even if other pages on the website are overall High or Highest quality. Sometimes website checks for a broken or error message page reveal that the individual page is not an isolated example, but rather a symptom of an unmaintained site (or possibly a deceptive or malicious site). If that is the case, the page should be rated Lowest quality. If the page has no MC by design, the rating should be Lowest. Not all pages with error messages are Low or Lowest quality pages, however. If the purpose of the page is to communicate that content has been moved or is no longer available, and the page does a good job of communicating this message, the overall PQ rating may be higher; it may be Medium or even High. The page quality rating will depend on the website level checks and the content of the page. Reminder: Page quality ratings are query-independent. The rating will depend on how well the page achieves its purpose and how well it passes page and website checks. Here are some examples of “broken” or error message pages:
Example Rating Discussion This is an example of a “broken” looking page which appears to be missing Main Content. You might think that this page is just “missing” the MC due to a problem with this particular page. In fact, this website has hundreds of pages that look the same way—no MC, just Ads. This website is either unmaintained or deceptive. Either way, this page should be rated Lowest quality. This is another example of a “broken” looking page which appears to be missing MC. Even though the website is a legitimate news website and passes all other website checks, we will consider this an overall Low quality page. This is an example of a “custom 404 page.” “Custom 404 message” pages are designed to alert users that the URL they are trying to visit no longer exists. Some websites do a nice job of not only alerting users about a problem, but also giving them help. This particular page displays the bare minimum of content needed to explain the problem to users, and the only help offered is a link to the homepage. The website is high quality, passes all website checks, and has a good reputation. But there obviously is not a lot of time and effort put into the particular page. The page accomplishes its purpose and there is nothing “wrong” with the page, but it is not a high quality page. The overall PQ rating here should probably be Medium. This is an example of a very high quality “custom 404” page. The MC of this page is the cartoon, the caption, and the search functionality (which is specific to the content of the website). Clearly, time, effort, and talent were involved in the creation of the MC. This page achieves its purpose well, and the website hosting the page passes all website checks and has a good reputation. An overall PQ rating here should probably be on the high end of the range.

No MC 1

Lowest

No MC 2

Low

Medium quality custom 404

Medium

High quality custom 404

High

Proprietary and Confidential – Copyright 2013

85

10.4 Balancing Page Level and Website Level Questions to Assign an Overall Page Quality Rating
For some page quality rating tasks, the page level questions (i.e., the questions about the landing page) will be the most important questions to consider. For other tasks, the website level questions (i.e., the questions about the website) will play an equally large or even a larger role. As always, use your best judgment. Important: If either page level or website level questions reveal a significant quality concern, the page should get an overall Low or Lowest rating. Please give a high rating only if both the page level and the website level questions can justify a high rating. Website level questions play the biggest role (and you should use an overall Low or Lowest page quality rating) when: • The website is harmful/deceptive/malicious. In this case, all pages on the website should be rated overall Lowest quality. • The website has an extremely negative or malicious reputation. If so, all pages on the website should be rated overall Lowest quality. • The website obviously or significantly violates the Quality Guidelines of Google’s Webmaster Guidelines. If so, all pages on the website should be rated overall Lowest quality. • The website clearly completely fails one or more of the website level checks in this document. If so, all pages on the website should be rated Low or Lowest quality. The overall rating depends on the purpose of the site and the check which revealed the quality concern. For example, all pages should be considered Lowest if the website is a store and there is absolutely no contact information, customer service information, or ownership information. For a store, this is an extremely important quality check! Website level questions play a medium to large role in determining the overall page quality rating when: • The content on the site is very uniform, i.e., when all content is produced by the same organization and there is little variation in the content or quality of pages. An example is a medical website produced by a reputable physician group where each page on various diseases has a similar amount of high quality content. • The content of the website is produced by different authors or organizations, but the website has very active editorial standards and guarantees a level of accuracy/quality. An example is a science journal with very high standards for publication. • The website has an extremely positive reputation from experts in the topic of the website, i.e., the website is acknowledged to be one of the most authoritative sources on the topic. Page level questions are most important when: • The website clearly “passes” all website level checks. If the website checks are “cleared,” the overall page quality rating will usually depend almost entirely on the content of the page. • The website has many different authors/contributors, or there is a large variation in the quality of the pages. Even if a website is popular and has a good reputation, you should rely on page level checks when the content is produced by different people. Page level questions are particularly important when the website has little or no active editorial standards, i.e., anyone can publish any content on the website.

10.5 How to Check for Copied Content
So how do you determine whether the content is copied or copied with minimal alteration? How do you identify the original source of the content? This can be difficult, but please follow these steps. • Copy a sentence or a series of several words in the text. It may be necessary to try a few sentences or phrases just to be sure. When deciding what sentence or phrase to copy, try to find a sentence or series of several words without punctuation, unusual characters, or suspicious words that may have replaced the original text. Search using Google by putting the entire sentence or phrase within quotation marks inside the search box. For example, try searching for the sentence [“Many details are omitted or altered while many of the perils that Dorothy encountered in the novel are not at all mentioned in the feature film”] or the phrase [“timid Munchkins come out of hiding to celebrate the demise”]. Sometimes, it is helpful to try the same search without the quotes, e.g. [timid Munchkins come out of hiding to celebrate the demise].



Proprietary and Confidential – Copyright 2013

86



Compare the pages you find that match the sentence or phrase. Is most of their MC the same? If so, does one clearly come from a highly authoritative source which is known for original content creation (newspaper, magazine, medical foundation, etc)? Does one source appear to have the earliest publication date? Does one source seem to reasonably be the original?

Use your best judgment. Sometimes it is clear that the content is copied from somewhere, but you cannot tell what the original source is. Or sometimes the content found on the original source has changed enough that searches for sentences or phrases may no longer match the original source. For example, Wikipedia articles can change dramatically over time. Old copies may not match the current content. If you strongly suspect the page you are evaluating is not the original source, go ahead and use the Low quality MC rating. Sometimes content is intentionally revised to make it difficult to determine that the content has been copied. Content may even be put through a translator to revise it. For example, if the original content is in English, it may be put through a translator twice: first to change it to a foreign language and second to translate it back to English. Text that has been changed in this way will often sound nonsensical. Any time you find copied content or suspect the page has copied content, please explain in the comment box. Please include the original source (URL or description) if you are able to find it. We will now walk you through two examples to determine if the content is copied. Example 1 – No clear original source • Copied content - suspicious. There is a paragraph at the top, followed by a line. Then there is an article below. Notice something a bit funny? Instead of the word "flower" or "flowers," this article uses "f" or "fs" in the text. This page looks suspicious. Let's try to figure out if the content is copied from elsewhere. Let's use the sentence: "Flowers which last only one day, like day lilies, do not dry well." It does not have the odd "fs" abbreviation. Do a search on Google with that sentence in quotes: ["Flowers which last only one day, like day lilies, do not dry well."].



You will see that there are many webpages with this sentence, though most use the word "flowers" rather than "fs" in the text. In fact, if you go to the last page of Google results, you'll find this: "In order to show you the most relevant results, we have omitted some entries very similar to the 25 already displayed. If you like, you can repeat the search with the omitted results included." Clicking on the blue link will give you over a hundred results for this sentence, many of which contain this article text or links to pages containing this article text. But none of these results seems to be very authoritative, and no single web result looks like it is the original source of the article. In some cases, even though the original source of content may be difficult or impossible to identify, you can still be fairly certain you have a copy. In the example above, it's highly unlikely that this is the original source of the content. It has the odd "fs" and "f" abbreviations (probably done so that search engines will not detect that this is copied content). There are also copying errors and other alterations on this page. Look at the bottom and you'll see "For Ber Colors: Rapid drying in a very warm, dry and bly-lit place will produce b blossoms; s drying in a more humid spot will produce more muted colors." While we cannot be sure of the original source, this is clearly a copy. It's actually less helpful than an unaltered copy. The abbreviations make it difficult to understand the text in places. If you see similar issues when rating, please make a note in your comments.

Proprietary and Confidential – Copyright 2013

87

Example 2 – Clear original source • • • Clear original source of content found on other sites. This example is from an actual page quality rating task. Some raters indicated that this page has copied content. Does it? Let's continue through the steps to see. Search for this phrase: “A master of creating the illusion of three-dimensional forms and figures on flag walls.” You will find many results. We should try to determine if there is one plausible original source for this content. Let's start with our URL. On the landing page, click the "about us" link to see who is responsible for the content of the website: About us page of clear original source. You'll find this information which shows that this is a very authoritative source for the content: Since the laying of the Capitol cornerstone by George Washington in 1793, the Architect of the Capitol (AOC) has served the United States as builder and steward of many of the nation's most iconic and indelible landmark buildings. These include the U.S. Capitol, Capitol Visitor Center, Senate Office Buildings, House Office Buildings, Supreme Court, Library of Congress, U.S. Botanic Garden and Capitol Grounds. Now let's look at this result which came up when we searched for the phrase on Google: Page with copied content. This is a far less authoritative source. In addition, the text at the bottom of the article cites "the Architect of the Capitol" at the bottom in the Credits section: “Images and descriptions online, courtesy Architect of the Capitol.” There are other copies of this article on the web, but we can see that the original URL is a page on a highly authoritative website for this content and that other sources cite this page. At this point, we can conclude that Clear original source is the original source of the article, though there are many copies on other websites.

11.0 Assigning an Overall Page Quality Rating
Each page quality task begins with a click on the task URL. Page quality rating is a series of steps, including examining both the landing page and the website it belongs to. Which questions are most important? How do you combine your responses into an overall PQ rating? As with the rest of these guidelines, the answer depends on the purpose of the page and type of website. You will have to use your judgment to combine all that you learned about the page and the website into one overall page quality rating. In this section, we’ll give you some guidance and some examples. Remember: In order to give an overall rating above Medium, every aspect of page and website quality you look at must be medium or high. If any aspect of quality seems low, whether the aspect pertains to the page or the website, choose a lower overall rating.

Proprietary and Confidential – Copyright 2013

88

11.1 Highest Quality Pages
Highest quality pages are highly satisfying to users. Highest quality pages have a large amount of very high quality MC, very helpful SC, and use very good webpage layout. The author(s) of the content on Highest quality pages should have a very high level of expertise in the subject. (Reminder: On high quality websites, we do not need to know the name of the author who wrote the content on the page. When the website is a very high quality website, we will assume that pages are written with a high level of expertise.) In addition, Highest quality pages are often found on websites that have a very good reputation from experts in the topic (even if average users or raters are unaware of the site or its reputation). Reputation checks are an important part of identifying Highest quality pages. • • • • • • • • Highest quality pages have an obvious purpose and they achieve that purpose very well. The MC of Highest quality pages is created by people with a high level of expertise in the topic. Highest quality pages have a very satisfying amount of MC. The page layout on Highest quality pages makes the MC immediately visible ("front and center"). The space on Highest quality pages is used well. The SC on Highest quality pages is helpful and contributes to a very satisfying user experience. Highest quality pages usually have near professional quality content, even though ordinary individuals may create the content. Highest quality pages frequently appear on high quality websites with very positive reputations for their purpose or topic, such as: Highest PQ Highest 1

Award winning newspaper sites for news. Authoritative sites for medical information. Well-known “go-to” recipe sites for recipes. Highly regarded and trusted shopping sites.
Explanation This is a complete paper published in a highly reputable medical journal (Annals of Oncology). All of the on-page and website-level PQ checks look good. This is the kind of content that academic and scientific journals are known for and valued for. The National Institutes of Health has an extremely good reputation. They maintain archives of content for the US public. A PQ rating of Highest is appropriate for most pages on a highly reputable and authoritative site. This article, which gives a relatively short summary of a report about studies that looked at fitness and longevity, is perhaps not the highest quality page on the site, so a PQ rating of High to Highest is appropriate for this page. This is a pancreatic cancer reference page on a highly reputable and popular website (Mayo Clinic). All of the onpage and website-level PQ checks look good. This website has the reputation of being one of the best web resources for medical information of this type. A PQ rating of High to Highest is appropriate for this page. The purpose of this page is to answer frequently asked questions about the Smithsonian Institution. The page provides a highly satisfying quantity of information about the Institution as well as helpful SC, and its layout is clear and accessible. The website the page belongs to is well-maintained, authoritative, and has an excellent reputation. Contact and other information about the Smithsonian Institution is appropriately detailed and easy to find. This page merits a Highest PQ rating. The purpose of this page is to provide an online BMI (Body Mass Index) calculator. The National Heart Lung and Blood Institute (which is part of the National Institutes of Health) has an extremely good reputation and this page achieves its purpose very well. The purpose of this page on the Microsoft website is to provide tips for using Excel. The page is authoritative and provides a highly satisfying quantity of helpful information about using Excel. A PQ rating of Highest is appropriate for this page. The purpose of this page on the official National Parks Service website is to provide information about Yosemite National Park. The page is authoritative and provides a highly satisfying quantity of helpful information about the park. The page merits a Highest PQ rating. The purpose of this page is to provide information about and allow users to buy the LLBean Deluxe Book Pack, a specific type of school backpack. The product is manufactured by LL Bean, so the page on the LLBean website is authoritative. The page provides a lot of helpful product information, as well as 600 user reviews and a lot of helpful SC. A PQ rating of Highest is appropriate for this page.

Highest 2

Highest 3

Highest 4

Highest 5

Highest 6

Highest 7

Highest 8

Proprietary and Confidential – Copyright 2013

89

11.2 High Quality Pages
High quality pages are satisfying to users. High quality pages have a large amount of high quality content, good use of space, and are trustworthy and authoritative. • • • • • • • • High quality pages have an obvious purpose and they achieve that purpose well. High quality content is created by people with appropriate expertise. High quality pages have high or highest quality MC. High quality pages have a satisfying amount of MC. The page layout of High quality pages makes the MC clearly visible. The space on High quality pages is used reasonably well. The SC on High quality pages is helpful. High quality pages appear on all sorts of websites, large and small, but the website of the page you are evaluating should "pass" all website checks, including the reputation check. High quality pages may be found on websites with a positive reputation or no reputation. They may even be found on mixed reputation websites, if they have enough positive evidence to support an overall High quality rating. High quality pages will not be found on websites with a negative reputation.
Explanation This is a health news article on a popular and reputable website (WebMD). All of the on-page and website-level PQ checks look good. The purpose of this page is clear, and there is a very large quantity of MC. The article does an extremely good job of citing resources (which are themselves reputable), so even though we don’t know who is responsible for the content, our concerns about medical accuracy/expertise are somewhat allayed. The references themselves are also valuable. The page has a satisfactory amount of helpful SC. The layout is good. A rating of Highest is too high for a Wikipedia article on a medical (or other YMYL) topic. However, the quantity of information and the references allow us to rate higher than Medium. Although this page is not authoritative, the resources cited confer some evidence of authority/accuracy. This page achieves its purpose well and merits a PQ rating in the High range. This is a blog post in the Style section of an award winning news website (New York Times). All of the on-page and website level PQ checks look good. Although this is not a “serious” news story, most blog posts by regular contributors on award winning news websites are considered High quality. As you can see here, the author of the post is a regular contributor to parenting and other blogs on the New York Times website.

High PQ High 1

High 2

High 3

11.3 Medium Quality Pages
Many pages are overall Medium quality. There is nothing “wrong” with Medium pages. However, it is easy to identify dimensions along which the page could be improved: they could have higher quality content, more content, more helpful and specific supplementary content, better layout, a better reputation, etc. • • • • • • • • Medium quality pages have a purpose and they achieve that purpose. Medium quality pages may have medium quality content, or even high quality content. The amount of MC on Medium quality pages is OK, though it may not be extensive. The page layout on Medium quality pages makes the MC visible. The space on Medium quality pages is used reasonably well. The SC on Medium quality pages is helpful or OK. Medium quality pages appear on all sorts of websites (and, in fact, many pages on the web are Medium quality). The website of the page you are evaluating should still "pass" all website checks. Medium pages may appear on websites with positive, mixed reputation, or no reputation. Having many negative reviews is a possible reason to give a Medium rating, even to pages on a popular website. You will need to use your judgment, taking into consideration the mix of positive and negative reviews, the reasons for the negative reviews, and the overall reputation and popularity of the website.

Proprietary and Confidential – Copyright 2013

90

Medium PQ

Description The PQ of this Wikipedia page is in the Medium range. Although some Wikipedia pages can be High or even Highest quality, this one does not have a lot of MC or include external references. Many Wikipedia articles have an extensive amount of information on the subject of the article, but this one does not. For example, it doesn’t include information about the range of sizes of baroque pearls, prices, etc. This directory page is designed to provide information about a particular business. The quantity of MC on this particular page is lacking as there are no reviews, no information about the business other than contact information, no website link, etc. The page meets the minimum standards required for a Medium quality page, but there are many aspects that could be improved upon to make it more satisfying.

Medium 1

Medium 2

11.4 Low Quality Pages
Low quality pages come in many flavors. Usually, Low quality pages are lacking in one or a few of the aspects we consider for the page or the website. Low quality pages may only be acceptable to users if there are no other higher quality pages. Many Low quality pages feel unsatisfying, especially to more discriminating users. Some Low quality pages may feel OK, but the website itself lacks enough information to feel credible or trustworthy. Please note: If a page has low quality MC, please rate the page Low or Lowest quality. If the website that the page is on “fails” one of the website checks, please rate the page Low or Lowest quality. In other words, if any of your checks find an area of concern, do not hesitate to use Low or Lowest ratings. • • • • • • • • • • Low quality pages usually have a purpose, though the purpose may be somewhat unclear or the page may not achieve that purpose well. Low quality pages may have low quality MC. The MC may be higher quality but copied from another source (perhaps with minimal alteration). The amount of MC on Low quality pages may be lacking. The page layout on Low quality pages may be poor. The space on Low quality pages may not be used well. The SC on Low quality pages may be unhelpful or distracting or lacking. Low quality pages may have an obvious problem with functionality or may have errors in displaying content. For example, the MC of the page may fail to load. (Please note that if many pages on the website have problems, you should consider the website unmaintained and use the Lowest rating). Low quality pages exist on all sorts of websites. In fact, many high quality websites have a few low quality pages. For example, most websites have a few pages with non-loading content or very little MC. Negative reputation alone can be the reason for a Low rating, but to assign this rating there must be evidence of an overwhelmingly negative reputation found on multiple sources. There are many “flavors” of Low quality pages. If a page does not live up to the standards established in these guidelines for any reason, please use the Low (or Lowest) rating.
Description While the conversation on this Q&A page may be enjoyable or important to the individual participants, we must evaluate this page for the average user who is not a participant. There is no medical expertise involved in this discussion, which is enough to lower the PQ rating to the Low range for this YMYL page. In addition, the content is very superficial and obviously untrustworthy: "it depends, do u feel any pain? if so take an aspirin and go to the hospital.. do u feel any pain down your left arm or your jaw go to the hospital.." "your stupid !!! are you trying to kill your self !!! there i smore than 200 poisons in cigarettes ....... do you want to die ????" Even though the title of the article is “Cholesterol Warning Signs,” it is created without medical expertise (the author is an editor, novelist, and screenplay writer), which is enough to lower the PQ rating for this YMYL page to the Low range. In addition, most of the MC is general and available on many websites. Most people think of their pets as family members. We should have the same high standards for pet medical/health information as we do for our own medical/health information. The author does not have the expertise to be providing medical advice for a malnourished cat, which is enough to lower the PQ rating to the Low range for this YMYL page.

Low PQ

Low 1

Low 2

Low 3

Proprietary and Confidential – Copyright 2013

91

11.5 Lowest Quality Pages
Lowest quality pages are severely lacking in one or more of the aspects we consider for the page or the website. The lowest overall PQ rating should be used for pages on websites that obviously fail one or more website checks, even if all page level ratings are OK. Lowest quality pages contribute little to the Internet. Many or most users would find Lowest pages unsatisfying. In most cases, users would be better off without Lowest quality pages. • • • • • • • • Lowest quality pages may not have a purpose, or the page may have an unclear/deceptive or malicious purpose. Lowest quality pages may have a purpose but fail to achieve that purpose Lowest quality pages may have low or lowest quality MC. The MC may be medium or higher quality but completely copied from another source. The amount of MC on Lowest quality pages may be lacking. There may not be any MC. The page layout on Lowest quality pages may be poor. The SC on Lowest quality pages may be unhelpful or distracting or nonexistent. The page may appear on a website which significantly “fails” the website level checks; for example, a website with absolutely no information about who created the website or how to contact the owner of the site. Extremely negative reputation can be a justification for the Lowest rating in cases of deceptive or malicious behavior.

The page may appear on a website that obviously or significantly violates the Quality Guidelines section of Google’s Webmaster Guidelines.
Lowest PQ Lowest 1 Description The author of this article has no medical expertise. The MC quality is extremely low, including grammatical errors and unprofessional writing. The writing quality is uneven, and the more professional sounding sentences appear to be copied content (search for "But on the basis of some observations experts have come up with a few probable causes"). A rating of Lowest is appropriate for this YMYL page. There is no evidence of medical expertise or authority of the information on this website. This is normally enough to lower the PQ rating to Low. On this page, the content on the page appears to be copied. Search for "A rash is a change in skin which affects its color, appearance, or texture." The content on this website is highly variable/disorganized (see the main links at the bottom "Naturally Slim | Syphilis | Birth Defects | Salmonellosis"). The combination of copied content, little effort, and the odd collection of content is enough to lower the PQ rating for this page to Lowest for this YMYL page. There is no evidence of medical expertise, and the text contains obvious grammatical errors as well as severe medical inaccuracies. For example, "The measles are a disease of the respiratory system. Its like a rarely deadly version of chicken pox and most triggers will not consequence in dying." This YMYL page merits a PQ rating of Lowest. This page has no MC, only links to what look like fitness blog results. However, all of the links go to other pages with similar links. Here is an example. These pages have no purpose and are deceptive. This page merits a rating of Lowest on the PQ rating scale. This is a fake search page with Ads and no MC. The page lacks any purpose and merits a PQ rating of Lowest. Although the title of the page is “AMF Bowling Coupons,” it does not offer any bowling coupons or provide links to bowling coupons. The MC quality is extremely low, including grammatical errors. For example, “When you return to our main website of the AMF, which have led eClub. Here you can in your e-mail and send you special offers. Even if you have an e-mail account, you can get a killer to.” This page merits a PQ rating of Lowest. The purpose of this page is to provide an online BMI (Body Mass Index) calculator. This page does not pass pagelevel or website-level checks, and the text is copied from other websites. It is Lowest quality even if the BMI calculator works.

Lowest 2

Lowest 3

Lowest 4

Lowest 5

Lowest 6

Lowest 7

Proprietary and Confidential – Copyright 2013

92

12.0

Webspam: a Special Type of Lowest Page Quality

12.1 What is Webspam? Webspam is the term for webpages that are designed by webmasters to trick search engines and draw users to their websites. In these guidelines, we sometimes refer to webspam as “spam,” and webmasters who use deceptive techniques as “spammers.” Pages with webspam violate the Quality Guidelines section of Google’s Webmaster Guidelines. In the coming pages, you will learn how to identify some of the deceptive techniques used to create webspam. Identifying spam is an important rater skill and will allow you to use the Lowest overall quality rating with confidence. Important: Pages that are clearly spam should receive the Lowest overall PQ rating. However, you can also use the Lowest PQ rating when the page isn’t technically spam.

12.2 When to Check for Spam
You should use the spam detection techniques described in these guidelines when you feel the page needs further investigation. Because spam pages change so quickly, you will be given links to screenshot examples to help you understand spam and learn how to check for it. You may notice that some examples fall into more than one spam category.

12.3 Looking for Technical Signals
When evaluating a page for spam, you should start by looking for the following “technical signals.” • • • • Hidden text and hidden links. Keyword stuffing. Sneaky redirects. Cloaking with JavaScript redirects and 100% frames.

This section describes these technical signals and provides tips and tools on how to identify them. 12.3.1 Hidden Text and Hidden Links Webmasters add hidden text and/or hidden links to lure search engines and users to their pages. Hidden text is visible to the search engine, but not to the user who might find it distracting or annoying. Here are some things you should know about hidden text: • It may be completely invisible to the human eye. • It may be in the same color as the background color on the page, or in a color that is so close to the background color that it almost invisible and will not be noticed. • It may be formatted in a very, very small font size (e.g., 1-point) so that it will not be noticed. • It may be placed outside the normal viewing area. For example, there may be a large blank space between the normal viewing area and a “hidden” area of text all the way at the bottom of the page or far to the right. • Sometimes there is just a line or two of hidden text, but you may even see a whole page of it. • Most hidden text is there to trick the search engine, but occasionally you will find hidden text that is not spam. For example, if the webmaster merely hides the date of an update, it is not spam.

Proprietary and Confidential – Copyright 2013

93

Hidden text may be revealed by: • Applying Ctrl-A (or "⌘" and "A" for Apple computer users). • Looking outside the normal viewing area. • Using advanced techniques: disabling CSS, disabling JavaScript, viewing the source code.

12.3.2 Apply Ctrl-A to the Landing Page After you have clicked on the URL, simultaneously press the “Ctrl” and “A” keys (the keyboard shortcut for “Select All” for PC users), or "⌘" and "A" or "Command" and "A" (the keyboard shortcuts for Apple computer users) and then scroll down the whole page. This technique sometimes reveals text that has been hidden. Example of hidden text revealed by applying Ctrl-A: LP before applying Ctrl-A LP after applying Ctrl-A Tiny text is not always exposed using Ctrl-A. You should be suspicious of horizontal lines or bars on the LP because sometimes they contain hidden text. A simple technique for revealing this type of hidden text is to select and copy the suspicious line or bar, paste it in your word processor, and increase the font size. You may also try using the techniques described below.

12.3.3 Look Outside the Normal Viewing Area Be suspicious of large blank areas on the bottom and far right portions of the page. Use the vertical and horizontal scroll bars to see if it appears there is text on the portion(s) of the page outside the main viewing area. 12.3.4 Advanced Techniques Use these techniques when the page is suspect and you want to dig deeper. Disabling CSS: Disabling CSS sometimes reveals hidden text. Here are instructions for disabling CSS using the Web Developer toolbar (a Firefox add-on that can be helpful detecting spam). • • • Click on “CSS.” On the dropdown menu, click on “Disable Styles.” Click on “All Styles.”

You do not need to check every page for hidden text in CSS, but please do check if the page is suspect. Example of hidden text revealed by disabling CSS: LP before disabling CSS Disabling CSS LP after disabling CSS Disabling JavaScript: Spammers sometimes use JavaScript to hide text. JavaScript using the Web Developer toolbar: • • • • Click on “Disable.” On the dropdown menu, click on “Disable JavaScript.” Click on “All JavaScript.” Refresh the page. Here are instructions for disabling

You can also disable JavaScript using your browser menu. For example, this is how you would do it in Firefox.

Proprietary and Confidential – Copyright 2013

94

Disabling JavaScript using your browser window in Firefox: • Go to “Tools.” • Click on “Options.” • Click on “Content” or “Web Features.” • To disable JavaScript, uncheck the “Enable JavaScript” box . • Click “OK.”

Example of hidden text revealed by disabling JavaScript: LP before disabling JavaScript Disabling JavaScript LP after disabling JavaScript Important: When you are done looking for spam on a particular page, please remember to go back and enable JavaScript. If you do not do this, certain features on pages you open will not work.

Viewing Source Code: Viewing the source code sometimes reveals hidden text. This is how you would do it in Firefox.
Viewing Source Code in Firefox: • Go to “View.” • Click on “Page Source.” or • Right click on the page. • Click on “View Page Source.”

Look for large areas of keyword stuffing in the source code. Keyword stuffing is discussed in Section 12.3.5. Here is an example of hidden text revealed by viewing the source code: Landing page Viewing source code Source code of the LP Please note that a page should not be considered spam when the keyword stuffing appears in the meta tags only. Meta tags are easy to identify because they start with the words "meta name.” Here is an example of keyword stuffing that appears in the meta tags only and is not considered to be spam: Landing page Source code with keyword stuffing in the meta tags only 12.3.5 Keyword Stuffing Keyword Stuffing: Webmasters sometimes load pages with an excessive amount of keywords. Here are descriptions of what you might see: • • • • • Keywords repeated many times on the page Words that are related to keywords repeated many times on the page Multiple misspellings of keywords on the page Pages with a large amount of what look like gibberish or random keywords. Pages that appear to be programmatically or automatically generated text that doesn’t really make sense.

Webmasters also sometimes load pages with irrelevant keywords on topics that are unrelated to the query, such as mortgages, cell phones, ringtones, gambling, weather, etc. Whether the keywords are related or unrelated to the query, the intent is to draw search engines and users to the page.

Proprietary and Confidential – Copyright 2013

95

It is sometimes difficult to decide when the keywords on a page should be considered keyword stuffing. We ask you to consider a page to be spam if you think the number of keywords on the page is excessive and would be annoying and distracting to the real user. Please note: Hidden text and keyword stuffing often go together. Hidden text frequently contains keyword stuffing. Recognizing keyword stuffing Some keyword stuffing is visible to the human eye and you will not have to use any special techniques to see it. In other cases, it is hidden. You will discover hidden keyword stuffing by using the techniques already described. Important: Hidden keyword stuffing will always be considered spam (unless it is only in the source code meta tags). Here are some examples that most users would consider excessive and annoying, even though in some cases the keywords are in the portion of the page “below the fold,” which users would have to scroll down to see:
Fake Feed with Keyword Stuffing Examples: The keyword stuffing on this page is just a collection of links with nonsensical link names. Notice how many times the words “las vegas” and “casino” appear on the page. Fake Blog with Keyword Stuffing Example: Notice how many times the word “Imodium” appears in this fake blog entry Computer-Generated Page with Keyword Stuffing Example

– Notice all the spelling variations of “Nissan” and the computer-

generated gibberish text.

12.3.6 Keyword Stuffing in the URL URLs may also contain keyword stuffing. These URLs are computer-generated based on the words in the query and are often formatted with many hyphens (dashes) in them. They are a strong spam signal. Keyword Stuffing in the URL Example Here are some additional examples of URLs with keyword stuffing. We have removed the hyperlinks from these examples because they have stopped working and/or become malicious. You do not need to click through to the landing page in order to see that there is keyword stuffing in the URL and that they are spam. • • http://frat-boy-blog-gay.grandbrooklynlodge.cn/boy-brief-frat-in-their-wet.html http://brazilian-model-alexandra.wantloweryour.cn/brazilian-model-adriana-lima.html u=hpmonmpi

12.3.7 Sneaky Redirects Sneaky Redirects: We call it a sneaky redirect when a page redirects the user from a URL on one domain to a different URL on a different domain, with spam intent. Search engines “see” the first page, while the user is sent to a different page and sees different content. Here are some other things you should know about sneaky redirects: • • • While being redirected, you may notice that the page redirects through several URLs before ending up on the landing page. Sneaky redirects may take the user to one of several rotating domains; so clicking on the same URL several times may send you to different landing pages each time. Some sneaky redirects take users to well-known merchant websites, such as Amazon, eBay, Zappos, etc.

Proprietary and Confidential – Copyright 2013

96

Recognizing sneaky redirects • Compare the two URLS: Compare the URL in the rating task to the URL of the landing page to see if it makes sense that one would redirect to the other. A redirect from a company’s old homepage to its new homepage on a different domain is not sneaky. Redirects from one page on a domain to another page on the same domain are also not sneaky. Look at the domain registrants: If you suspect that a sneaky redirect has taken place, you should check to see “whois” the registrant (or owner) of the two domains. If the registrant is the same, the redirect is less likely to be sneaky.



12.3.8 Using “Who Is” Here are instructions for checking “whois” the domain registrant: • • • • • Go to the site of a “whois” provider. Here are two you can use: http://www.domaintools.com and http://whois.mtgsy.net/default.php. Some computers also allow you to run a command like whois example.com from a terminal window. Enter the URL of one domain in the search box on the “whois” page. Sometimes, you will need to delete some leading or following characters. For example, if the URL is http://supportapj.dell.com/support, you will enter just “dell.com” in the search box of the whois provider. Open another “whois” page. Enter the URL of the other domain in the search box on the second “whois” page. Compare the domain registrants for the two URLs. If you find that they have the same domain registrant, you will *typically* conclude that the page is not spam. If they are different and do not seem related, it is probably spam.

Sneaky Redirect Example URL before clicking (task URL) URL after clicking (URL of the landing page) Using a “whois” provider to learn about the domain registrants Entering the domain of the task URL Domain registrant information of the task URL Entering the domain of the LP URL Domain registrant of the LP URL The domain registrants in this example are different. Since these two domain registrants have no relationship, this is a sneaky redirect.

Non-Sneaky Redirect Example Trans World Airlines was acquired in by American Airlines in 2001. Click here to read about this acquisition. The domain registrant of twa.com and the domain registrant of aa.com are the same, so this is not a sneaky redirect. Please be aware that domain names with the same domain registrant can look very different. For example, Barnes and Noble, the bookseller, owns the following domains: www.barnesandnoble.com, www.bn.com, and www.books.com.

Proprietary and Confidential – Copyright 2013

97

12.3.9 Cloaking It is called “cloaking” when the webmaster shows different pages to the search engine and the user. True cloaking is somewhat rare, but spammers do use other methods to show different pages to search engines than to users. Two such techniques used by spammers are:   JavaScript redirects 100% frame

Spammers use JavaScript redirects to show one page to search engines while sending users to a different page. Looking at the page first with JavaScript enabled and then with JavaScript disabled reveals the differences. Webmasters sometimes cloak what users see by using frames. Two frames (pages) exist, but one frame takes up 100% of the screen. The user sees one frame (page), but the search engine sees both frames. Here are instructions for looking at the different frames in Firefox:
Viewing Frame Information in Firefox • Right-click on the page. • Click “This Frame.” • Click “View Frame Info.” • Compare the URL of the frame with the URL of the page. If they are different, the page is probably 100% framed, and should be considered spam.

100% Frame Example URL of the landing page Viewing frame information Comparing the URLs (of the landing page and the frame) Domain registrant of the landing page (neoobe.com in Westchester, California) Domain registrant of the frame (animaturk.com in Istanbul, Turkey) Since the URLs and the domain registrants are different, the page is probably 100% framed.

12.4 Ads and Spam Pages
Many webpages have Ads. Pages with satisfying and helpful MC and Ads are not considered spam. However, we will consider pages to be spam if they have Ads and: • No MC, deceptive MC, or unhelpful MC • Copied MC • Auto-generated MC All of these pages exist primarily to get clicks on Ads.

12.4.1 No Main Content Some pages literally have no MC. The only thing on the page is Ads and possibly links to other similar pages. We will consider these pages to be spam. Here is an example: No MC Example

12.4.2 Parked (Expired) Domains Here is a type of website which frequently has no MC: Parked Domains.

Proprietary and Confidential – Copyright 2013

98

When companies go out of business and are acquired by another company, change their name, or fail to pay their domain registration fee, the website name “expires” and may be purchased by someone else. Spammers sometimes buy these websites and cover the pages with Ads. Such sites are referred to as “parked domains” or “expired domains.” Spammers also purchase the following kinds of domains, which are similar in appearance to parked domains: • • Domains which are close in spelling to real domains, hoping that users will mistype the domain name or URL and land on their websites, which contain Ads. Domains that users might type when looking for a website to use.

A typical parked/expired domain contains some or all of the following: • • • Ads Links to other similar pages on the website A fake search box

Here are some examples of queries that spammers have purchased domains for, hoping that users would land on the websites and click Ads.
Query LP of the Result Block, Result Block, PQ Rating, Utility Rating Link to landing page: LP This result is spam. It is a fake search page. [chairs], English (US) No Vital page is possible for this query because it is an information query and no one can claim ownership of it. Explanation

Link to landing page: LP This result is spam. It is a fake search page. [knitting], English (US) No Vital page is possible for this query because it is an information query and no one can claim ownership of it.

[huffintonpost.com], English (US) This query is a working URL, but it is clearly mistyped. The clear user intent is to navigate to huffingtonpost.com: LP

Link to landing page: LP This result is spam. It is a fake search page. This result is not Vital. It is not the result the user is looking for.

Proprietary and Confidential – Copyright 2013

99

Recognizing Parked/Expired Domains • • • Look at the links. Typically, all of the links on a parked domain are Ads. There is little to no original content on the page. Look at the domain name (URL). On a parked domain, the domain name (URL) often has little or nothing to do with the content on the webpage. You may see the keywords, but the links are usually generic and the linked pages are not really associated with the query. Look at the page on the Internet Archive. Go to http://www.archive.org/index.php to enter the URL and view the page as it appeared previously, when its original owner maintained it. If the original site was different, it is probably a parked domain.

Parked Domain Examples: Parked Domain Example 1 Parked Domain Example 2 Parked Domain Example 3

12.4.3 Deceptive MC: Fake Search Pages with Ads A fake search page is a page with a list of links that looks like a page of search results. You will see a “search box” on the page, but if you submit a new query in the search box, you just get a different page of links. If you click on a few of the links, you will see that the page is just a collection of Ads disguised as search engine results. Fake Search Page Examples Fake Search Page Example 1 Fake Search Page Example 2 Fake Search Page Example 3 Fake Search Page Example 4 Fake Search Page Example 5

12.4.4 Deceptive MC: Fake Blogs and Forums with Ads A fake blog or fake forum contains fake posts that are either nonsensical, copied from another source, or are written purely to get clicks on Ads or affiliate links. Ads and affiliate links may appear within the text of the fake blog or forum entry, or on other parts of the page. Fake blogs and forums are deceptive and are spam. Spammed blogs and spammed forums are different from fake blogs and fake forums. A spammed blog or forum is a real working site with real posts, but which has been spammed with posts that contain Ads and/or porn links. We do not want to penalize a real blog or forum because someone else has put spam on it. If you believe that the blog or forum is a legitimate site that has been spammed by someone else, please do not consider the result spam. Fake Forum/Message Board Posts Fake Post 1 Fake Post 2

Proprietary and Confidential – Copyright 2013

100

12.4.5 Copied MC and Ads Copying content from some sources is not necessarily illegal, nor is it plagiarism. Webmasters who copy content usually do not claim to be original content creators and may, in fact, assign credit to the originator of the content. However, even if they do give credit to others, it is considered to be copied content. We will call a page spam if the entire MC is copied content and the page is created to make money from ads. Copied Content Example Original Content Copied Content

12.4.6 Auto-generated Main Content and Ads Automatically generated (auto-generated) content is content that has been generated programmatically (not by a human). Often it will consist of paragraphs of random text that make no sense to the reader but which may contain words in the query. Some examples of auto-generated content include: • Text translated by an automated tool without human review or curation before publishing. • Text generated using automated synonymizing or obfuscation techniques. • Stitching or combining content from different webpages without adding sufficient value. • Text generated from scraping Atom/RSS feeds or search results. Note: Web publishers (such as the BBC, CNN, Usenet, CNet, NY Times, Atom, and others) publish information online that is readily available to users through RSS (Really Simple Syndication). RSS feeds appear on many pages. However, we will call a page spam if the entire MC is auto-generated and was created just to make money. Computer-Generated Text Example Computer-Generated Page Example

12.4.7 Unhelpful MC and Ads Some MC is created just for the purpose of putting ads on the page. If the MC clearly has no other purpose than to get users to click on Ads, the page should be considered spam. Unhelpful MC Examples Unhelpful MC Example 1 Unhelpful MC Example 2 Unhelpful MC Example 3 In other cases, the MC quality may be poor and unhelpful, but you may not feel the page is spam. These pages can still be rated Lowest PQ even if the page is not technically spam.

12.5 Doorway Pages
Doorway pages are typically large sets of poor-quality pages where each page is optimized for a specific keyword or phrase. In many cases, doorway pages are written to rank for a particular phrase and then funnel users to a single destination. Whether deployed across many domains or established within one domain, doorway pages tend to frustrate users.

Proprietary and Confidential – Copyright 2013

101

Doorway pages are sets of pages that all look very much the same and do not provide meaningful content for users. For example, consider the website http://www.hair-removal-hair-laser.com Hair Removal Homepage Screenshot, It contains links for all of the states in the US. Clicking on a link makes you think that you are getting a customized page for that state, but you will find that there is no actual information on any of the pages. There is no information specific to cites, but there are many ads. These pages are spam.
Doorway Pages Example California URL and landing page Florida URL and landing page San Francisco URL and landing page Miami URL and landing page http://www.hair-removal-hair-laser.com/ca.html: Hair Removal California Page Screenshot http://www.hair-removal-hair-laser.com/fl.html: Hair Removal Florida Page Screenshot http://www.hair-removal-hair-laser.com/City/California/Hair-removal-SanFrancisco.html: Hair Removal San Francisco Page Screenshot http://www.hair-removal-hair-laser.com/City/Florida/Hair_Removal_Miami_FL.html: Hair Removal Miami Page Screenshot

12.6 Thin Affiliates
An affiliate earns money from affiliate commissions. An affiliate website exists primarily to make money. Affiliates often show content from other "real" merchant sites, such as Amazon or eBay, or a good hotel or travel website. When users click on links to buy products or make reservations, they are oftentimes redirected to the real merchant page. A thin affiliate is a site that offers little additional information and does not offer substantial value to users compared to many other sources on the web. For example, an affiliate that has only copied content from the merchant site is considered a thin affiliate. This is a moneymaking spam technique.

12.6.1 Recognizing Thin Affiliates To help determine if a page is a thin affiliate, you can do the following: • Look for original content on the page. The quality of an affiliate page or site depends on how much added value, usefulness, or original/additional information is available on the page that is not easily available elsewhere on the web. If the page has the same “cookie-cutter” text or functionality found on dozens or hundreds of other sites, it is more likely to be spam. Look at the domain registrants. If clicking a button takes you to another page, check to see “whois” the registrant (or owner) of the two domains. If the registrant is the same, the page is typically not a thin affiliate. Please follow the instructions for checking “whois” in Section 12.3.8.



12.6.2 Recognizing True Merchants Features that will help you determine if a website is a true merchant include: • • • • • • • • • a “view your shopping cart” link that stays on the same site. a shopping cart that updates when you add items to it. a return policy with a physical address. a shipping charge calculator that works. a “wish list” link, or a link to postpone the purchase of an item until later. a way to track FedEx orders. a user forum that works. the ability to register or log in. a gift registry that works.

Proprietary and Confidential – Copyright 2013

102

Please note the following: • • • A page does not need to have all of these features to be considered a true merchant. Yahoo! Stores are true merchants – they are typically not thin affiliates. Some true smaller merchants take users to another site to complete the transaction because they use a third party to process the transaction. These merchants are not thin affiliates.

Many large web retailers offer affiliate programs. Some of the most common examples are Amazon.com, eBay.com, Zappos.com, Allposters.com, Hotels.com, Orbitz.com, and Overstock.com. There are two methods you can use to learn if a result is a thin affiliate: • • Check the properties of an image on the page to find the source of the image. Attempt to make a purchase to see if a transaction can be completed on the website.

Shoe Store Thin Affiliate Example – We will focus on one pair shoes on the landing page. Method 1 – Checking properties of an image on the page Landing Page Checking the properties of the image. On the LP, we right-click the shoe image and then scroll down in the pop-up window and click on Properties. Checking the source of the image. Element Properties box tells us that the image comes from shoemall.com Properties of the image. Method 2 – Attempting to make a purchase Clicking on the shoe on the LP to make a purchase to find information about the shoe takes users to shoemall.com to complete the transaction or find information. Users cannot make a purchase on the LP.

13.0

Page Quality Rating FAQs
Answer With practice, the amount of time needed for accurate PQ ratings will decrease. The steps are important and are designed to help you assess many different aspects of PQ. You may be surprised by what you find. Pages which initially look low quality may turn out to be medium or high quality with careful inspection. The reverse may happen as well. We want your most informed, thoughtful opinion. No! The goal is to do the exact opposite. These steps are designed to help you analyze the page without using a superficial "does-it-look-good?" approach. Remember - we are not just talking about expertise. High quality pages involve time, effort, expertise and/or talent. But since you ask, pretty much any topic has some form of expert, though there are some topics or types of pages where expertise is less important than other aspects for MC quality rating. For most page purposes and most topics, you can find experts, even when the field itself is niche or non-mainstream. For example, there are expert alternative medicine websites with leading practitioners of acupuncture, herbal therapies, etc. There are also pages about alternative medicine written by people with no expertise or experience. The MC quality ratings should distinguish between these two scenarios.

Question Why do we have to do all these steps? This takes forever! Are we just giving high quality ratings to pages that "look" good?

You talked about expertise when rating MC. Does expertise matter for all topics? Aren't there some topics for which there are no experts?

Proprietary and Confidential – Copyright 2013

103

Question

Answer For almost any type of page, there is a range of content quality. Remember - high quality content is defined as content that is very satisfying, useful, or helpful for its purpose. For example, there are high quality celebrity gossip pages and low quality celebrity gossip pages. Often, the purpose of a gossip page is to share scandalous, but plausibly true personal information about well-known people, so the MC of a gossip page is high quality when it is juicy and from a somewhat plausible source. Gossip pages are not judged by their accuracy. And, yes, there are high quality gossip pages!

Aren't there some types of pages that always have low quality content?

I've never seen a high quality page of type X. If there are no high quality pages of this type, why are we giving existing pages a low quality rating? Some of these criteria seem unfair. For example, some art pages do not have a purpose. Are these pages low quality?

For some topics or page purposes, there may not be many (or any!) high quality pages now, but in the future there may be. We need a uniform set of standards that apply to all pages, even for pages that have not yet been created.

Art pages have a purpose: artistic expression. Certainly, pages created for artistic expression do not deserve the low quality rating simply because they have no other purpose. Artistic expression, humor, and entertainment are valid page purposes.

Are forum pages always low quality?

No. Forum pages vary. We need to evaluate forum pages using the same criteria as all other pages. There are some forum pages with detailed information on specific issues written by people who are experts in the topic being discussed. There are also shallow discussion threads with very little content. No type of page (news, forum, shopping, encyclopedia, etc.) is automatically high or low quality. No. Q&A pages vary. We need to evaluate these pages with the same criteria as all other pages. Sometimes, it can be difficult to assess the accuracy of the information or the expertise/knowledge of the person answering the question. In these cases, you may need to do a bit of research. If the page is asking for medical advice, be skeptical about the expertise of the participants in the discussion. If the question is about daily life, then it is far more likely that the participants in the discussion have the necessary experience/expertise. Some Q&A pages are detailed and have accurate and reliable information. Many others have little participation or inaccurate/incomplete information. On some Q&A pages, the question itself is incomplete or fragmented. We must evaluate these from the perspective of web users, not the participants in the discussion on the page. Remember - no type of page (such as news, Q&A, store, etc.) is automatically high or low quality.

Are Q&A pages necessarily low quality?

Proprietary and Confidential – Copyright 2013

104

Part 3: Block Utility Rating
14.0 Overview of Block Utility (BU) Rating

For every BU rating task, you must • • • • • Understand the query (release the task if you can't!). Assign a utility rating to ALL results. Assign a PQ rating to the landing page of the result block if a PQ slider is present. Identify duplicates (See Section 29.0). Flag Foreign Language (F), Porn (P), and Didn't Load (D) results.

You will develop your own workflow as you master BU rating. You may decide to identify duplicates first or assign PQ ratings before utility ratings. Remember: If you can’t understand the query and user intent, please release the task. For this reason, understanding the query is a good first step. All blocks in BU rating have a utility slider. Please assign a utility rating for all results. Please assign an OT rating for: • • • • Unsolicited Foreign Language (F) results Unsolicited Porn (P) results Didn’t Load (D) results Any other kind of useless result

Some results don’t have a PQ slider. If a result block has no PQ rating slider, you do not have to give a PQ rating. If there is a PQ slider, please assign a PQ rating based on the landing page. If you are unable to evaluate the PQ of a Foreign Language or Didn’t Load result, you should assign a PQ rating of Medium.

15.0

The Relationship between PQ and Utility

The utility rating is based on both the query and the result. For this reason, the utility rating slider is labeled "querydependent." You must carefully think about the query and user intent when assigning a utility rating.

The PQ rating slider is labeled "query-independent," meaning that it does not depend on the query. Do not think about the query when assigning a PQ rating to the LP.

Proprietary and Confidential – Copyright 2013

105

Here is some guidance about assigning utility and PQ ratings: • • Useless results should always be rated OT, even if the landing page is high quality. Useless is useless. On-topic, helpful, but low quality results should get lower utility ratings than on-topic, helpful, high quality results. The utility scale encompasses all aspects of "helpfulness," and many users find low quality results less helpful than high quality results. Your ratings must reflect this. The U rating should be given to helpful, high quality pages which are a good fit for the query. The U rating may also be used for results which are very helpful, medium quality, and have other very desirable characteristics, such as very recent information. Do not use the U rating if a page is low quality or has any other undesirable characteristic, such as outdated information, or if it is a poor fit for the query. We have very high standards for the U rating. SR is often a good rating for low quality but on-topic pages. However, a page can be so low quality that it is useless for any query. Gibberish pages are good examples of this and should be rated OT. The V rating is special and does not depend on the quality of the landing page. If the user intent of the query requests a specific page, then that page should get the V rating, even if it is low quality.
LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing Page: LP Explanation PQ: Read this content out loud. The article on the LP is shallow and unintentionally humorous. There are four paragraphs of text, but it basically says that electric cars use electricity and have to be charged. Utility: This page would be helpful for few users. Landing Page: LP PQ: There is a lot of helpful MC on the landing page. Utility: The query asks how electric cars work, but the article on the LP is about mobility scooters.



• • •

Query and User Intent

[how do electric cars work], English (US) Find information about how electric cars work.

Proprietary and Confidential – Copyright 2013

106

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating

Explanation PQ: This is a YMYL topic. The page has many characteristics of low quality: no contact information, no indication of who wrote the contact, no evidence of medical expertise/authority and heavy monetization from Ads which distract from the MC. This page is not trustworthy. Utility: Even though the article is topical, the information is not reliable and potentially misleading or dangerous.

Landing Page: LP

[symptoms of dehydration], English (US) Find information about the symptoms of dehydration. Landing Page: LP

PQ: This is a YMYL topic. This page is on a highly authoritative medical website and has a lot of reliable and accurate MC. This page is very trustworthy. Utility: This result would be very helpful for most users.

16.0

Queries with Multiple Interpretations and Intents

Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user intents for this query (Do-Know-Go) and there are different models of iPhones, but all users are basically referring to the same thing: the phone made by Apple, Inc. Some queries truly have different possible meanings. We refer to these different meanings as query interpretations. When giving utility ratings for results involving different query interpretations, think about how likely the query interpretation is and how helpful the result is. For example, a very helpful result for a dominant interpretation should be rated U, because it is very helpful for many or most users. (Some queries with a dominant interpretation have a V result.) A very helpful result for a common interpretation may be U or R, depending on how likely the interpretation is. A very helpful result for a very minor interpretation may be SR or lower because few users may be interested in that interpretation. There are some interpretations which are so unlikely that results should be rated OT. We call these “no chance” interpretations. Reminder: PQ rating is query-independent. Do not think about query interpretation when doing PQ ratings.

Proprietary and Confidential – Copyright 2013

107

Query Interpretation

Example Query and Interpretation

Highest Possible Utility Rating For a Very Helpful Result

[apple], English (US): Apple computers. Most users who type this query want results on Apple computers. [windows], English, (US): the Microsoft operating system. Most users who type this query want results on the Microsoft Windows operating system. [median], English (US): the mathematical formula. Most users who type this query want results about the mathematical formula. There is no V result possible for this query. [guinea pig], English (US): the small furry animal often kept as a pet. Most users who type this query want results about the animal. There is no V result possible for this query. V or U

Dominant Interpretation: Of all the users who type the query, most users would want this interpretation.

[apple], English (US): The fruit. Some users who type this query could want results about the fruit. [windows], English (US): The glass paned windows for a home. Some users who type this query could want results about glass windows for a house. Common Interpretation: Of all the users who type the query, some users would want this interpretation. [ada], English (US): The American Dental Association, the American Diabetes Association, or the American with Disabilities Act. Some users could want information about any of these interpretations. [mercury], English (US): The car brand, the planet, or the chemical element. Some users could want information about any of these interpretations. [sandals], English (US): The open type of shoe or the chain of resorts in the Caribbean Sea. Many or some users could want information about either of these interpretations. U or R, depending on how likely the common interpretation is.

[ada], English (US): The Atlanta Development Authority or the American Darters Association. Few users would want information about either of these interpretations. Minor Interpretation: Of all the users who type the query, few users would want this interpretation. [mercury], English (US): The Mercury Magazine (published by the Astronomical Society of the Pacific) or Mercury Records (a record label in the UK). Few users would want information about these interpretations. [hot dog], English (US): “Hot Dog,” a movie that was in movie theaters in 1984. Few users would want information about this interpretation.

R or SR Results for less likely interpretations should get lower ratings because they are helpful for fewer users.

“No chance” Interpretation: An interpretation so minor that almost no one would ever want it.

[guinea pig], English (US): A pig from New Guinea, which is an island country located near Australia. (There probably are pigs in New Guinea, but it is extremely unlikely that the user typing the query would have that interpretation in mind.)

OT No users are interested in “no chance” interpretations.

Proprietary and Confidential – Copyright 2013

108

17.0

Specificity of Queries and Landing Pages

Some queries are very general and some queries are specific. Here are some examples that compare levels of specificity of English (US) queries:
Query [chair] [cameras] [Toyota] [library] [interview questions] [discount stores in houston] More Specific Query [dining room chair] [Nikon cameras] [Toyota hybrid] [Harvard library] [interview questions for teachers] [walmart stores in houston] Even More Specific Query [ikea “henriksdal” highback upholstered chair] [Nikon d5000 slr] [Toyota Prius 2010] [Harvard Anthropology library] [practice interview questions used for Teach For America] [walmart 9555 South Post Oak Road houston]

Results for specific queries are easier to rate on the utility scale because we know more about what the user is looking for. Giving a utility rating for results for general queries can be difficult. As always, your rating is based on how helpful the result is for the query, not the specificity fit. Of course, PQ rating is not based on the query, so specificity is not a concern for PQ rating.
Query and User Intent LP of the Result Block, Result Block, PQ Rating, Utility Rating Explanation This is a popular travel aggregator website, and the hotel page on the site can help users find hotels in the US. Users can read reviews, compare hotels, and make a reservation. Utility: This LP fits the query. It would be very helpful for most users. [hotels], English (US) PQ: Orbitz is a popular website with a good reputation. This is a very popular hotel chain with hotels available in most of the US at many different price levels. Landing page link: LP Even though the list of possible hotel chains is long, the homepage of an individual chain that offers different prices, features, and location options could be very helpful. Utility: This LP is more specific than the query, but it would still be helpful for many or most users. PQ: The Marriott website gives information on Marriott hotels, a popular chain of hotels. Marriott is an expert on Marriott hotels and the information is highly authoritative.

Landing page link: LP

Users are probably planning a trip, but this query is very general and vague.

Proprietary and Confidential – Copyright 2013

109

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating

Explanation This is the webpage of the Marriott Courtyard hotel in Emeryville, California.

[hotels], English (US)

Landing page link: LP

Users are probably planning a trip, but this query is very general and vague.

Utility: The LP is too specific for the query, but this is a wellknown brand and users can navigate to other Marriott hotels from this page. Few users would find this page helpful. PQ: The Marriott website gives information on Marriott hotels, a popular chain of hotels. This page is professional, well designed, functions well and Marriott has a good reputation. Marriott is an expert on Marriott hotels and the information is highly authoritative.

Landing page link: LP

The LP is the official Target homepage. Utility: The landing page fits the query. PQ: Target is a popular store which sells many different products. Target has a good reputation but is not an expert or authority on much of what it sells.

Landing page link: LP

The LP is the “store locator” page on the Target website. Utility: The LP is more specific than the query, but many or most users would be interested in this page. PQ: Target is the expert about where its stores are located. This page is the most authoritative page on this topic.

[Target], English (US) Go to target.com or find a local Target store.

Landing page link: LP

The LP is the “baby” page on the Target website. Utility: The LP is more specific than the query, but many or most users would be interested in this page. PQ: Target has a good reputation, but is not an expert or authority on much of what it sells.

Proprietary and Confidential – Copyright 2013

110

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP

Explanation The LP is the “electronics” page on the Target website. Utility: The LP is more specific than the query, but many or most users would be interested in this page. PQ: Target has a good reputation, but is not an expert or authority on much of what it sells.

[Target], English (US) Go to target.com or find a local Target store. Landing page link: LP

The LP on the Target website is for a specific type of junior colored jeans. Utility: The LP is much more specific than the query and would be helpful for very few users. PQ: Target has a good reputation, but is not an expert or authority on much of what it sells. The LP is the official Honda Accord page.

Landing page link: LP [Honda Accord], English (US) Users probably want to buy a car and are interested in finding information about the Honda Accord. There are currently two models of the Accord today (the Accord Sedan and the Accord Coupe), and a Plug-In model will be coming out in 2014.

Utility: The LP fits the query. The query asks about the Accord and the landing page is about the Accord. PQ: The Honda website gives information on Honda cars, a very popular car maker. This page is professional, well designed, functions well, and Honda has a good reputation. Honda is an expert on Honda cars and the information is highly authoritative. The LP is the official Honda Automobiles webpage. There is a prominent “Accord” link on the page, as well as a lot of helpful features for users interested in Accords. Utility: The LP is more general than the query; however, the information on the page would be very helpful for many or most users. PQ: Honda is an expert on Honda cars and the information is highly authoritative.

Landing page link: LP

Proprietary and Confidential – Copyright 2013

111

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP

Explanation The LP is the official page of the Accord Coupe. Utility: The LP is more specific than the query, but since there are three Accord models, official pages for any of the three models are U.

[Honda Accord], English (US) Users probably want to buy a car and are interested in finding information about the Honda Accord. There are currently two models of the Accord today (the Accord Sedan and the Accord Coupe), and a Plug-In model will be coming out in 2014.

PQ: Honda is an expert on Honda cars and the information is highly authoritative. Landing page link: LP The LP is the official “Accord Sedan Warranty” page. Utility: The LP is much more specific than the query. It would be helpful for few users. PQ: Honda is an expert on Honda cars and the information is highly authoritative.

Landing page link: LP

The LP is the official Power Equipment page on the Honda website. It has nothing to do with the Honda Accord. Utility: The LP is does not fit the query. Most users would not find this result helpful. PQ: Honda is an expert on Honda power equipment and the information is highly authoritative.

[chicken recipes], English (US) Users probably want to prepare a chicken dish and are looking for some recipes to choose from. Users probably expect and want a list of recipes.

Landing page link: LP

Users can find many chicken recipes (with reviews) on this page on a popular recipe website. Utility: This LP fits the query. It would be very helpful for most users. PQ: Food Network is a popular website with a good reputation for providing high quality recipes and cooking information.

Proprietary and Confidential – Copyright 2013

112

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating

Explanation This LP has a single recipe for chicken parmesan. Utility: This is a popular type of chicken recipe on a popular website, but the LP is more specific than the query. Some or few users would find this page helpful. PQ: Food Network is a popular website with a good reputation for providing high quality recipes and cooking information. The LP has 20 recipes for fried chicken, a popular chicken dish. Utility: Even though there are 20 different recipes on the LP, they are all for the same basic dish. Therefore, this LP is also more specific than the query. Some or few users would find this page helpful. PQ: Allrecipes.com is a popular website with a good reputation for providing high quality recipes with lots of reviews.

Landing page link: LP

[chicken recipes], English (US) Users probably want to prepare a chicken dish and are looking for some recipes to choose from. Users probably expect and want a list of recipes.

Landing page link: LP

Landing page link: LP This is a low quality page with distracting pop-ups that appear when you hover your mouse over hyperlinked words in the list of recipes. These pop-ups would actually prevent users from reading the titles of some of the recipes. Very few users would find this page helpful.

Proprietary and Confidential – Copyright 2013

113

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating

Explanation Many users research purchases carefully. The LP is a cnet.com “Digital cameras” review page, with information about many different digital cameras organized by price, manufacturer, and camera features. Utility: This LP fits the query. The query asks for digital cameras and the LP is about digital cameras.

Landing page link: LP

[digital cameras], English (US) Find information about digital camera brands and options before buying a camera. Landing page link: LP

PQ: Cnet.com is a popular website known for providing helpful reviews of consumer electronics products. This article has a lot of very helpful MC and SC. This LP is a cnet.com review page about the Nikon Coolpix P520. Utility: It is a comprehensive review of a popular type of camera made by a respected brand, but it is more specific than the query. PQ: Cnet.com is a popular website known for providing helpful reviews of consumer electronics products.

18.0

Utility and Freshness

Some queries demand very recent or "fresh" information. Users may be looking for “breaking news,” such as an important event or natural disaster happening right now. Imagine a user in Kansas typing the query [tornado], English (US) because a big storm is coming. Information about last year's tornado season is not helpful. Users need information about tornadoes headed their way right now! Queries for recurring events, such as elections, sports events, TV shows, conferences, etc., also demand current information. Assume users are looking for the most recent or current information about the event, unless the query indicates otherwise. For example, if the Olympics are happening right now, users typing the query [olympics] want information about the current Olympics, not results from years ago. If the next Olympics are a few months away, users are probably interested in the next upcoming Olympic Games. Some queries demand current information, such as [population of paris], [next federal holiday], etc. Requests for prices such as [airfare from ny to sfo], should be interpreted as asking for current price. Unless otherwise stated, users usually want information about the most recent model/version for product queries, such as [iphone], [windows operating system], [toyota camry], etc.

Proprietary and Confidential – Copyright 2013

114

Here are some different types of queries demanding current/recent results: • • • • “Breaking news” queries such as [tornado] Recurring event queries such as [olympics], [american idol], [redsox schedule], [tax forms], [elections] Current information queries such as [population of paris], [amount of US debt], [airfare from ny to sfo], Product queries such as [iphone], [Toyota camry], [windows operating system]

When a query demands current or recent content, only pages with current, recent or updated content should get high utility ratings. (Remember- we care about the content, not the date the page was created.) For these queries, pages about past events, old product models, old prices, outdated information, etc. are not helpful and should be considered "stale." When the query demands recent information, stale results should be given low Utility ratings because they are not helpful for users. In some cases, stale results are useless and should be rated OT. Freshness is generally less of a concern for PQ rating. “Stale” pages can have high PQ ratings. For example, some highly reputable news websites maintain “archival” content. However, unmaintained/abandoned “old” websites or unmaintained and inaccurate/misleading content is a reason for a low quality rating.
Query and User Intent LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP Explanation

[nasa mars landing], English (US) On the date this example was written, NASA had just landed the “Curiosity” rover on Mars. Most users are probably looking for breaking news and pictures about the event.

This LP has an article about the “Curiosity” rover landing, published on the same day as the landing.

Landing page link: LP.

This LP has an article about the launching of the “Curiosity” rover, published nine months before its landing on Mars. The content on this page would be considered stale for the query:

Proprietary and Confidential – Copyright 2013

115

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP.

Explanation

[los angeles traffic], English (US) Users are looking for current information about traffic conditions in Los Angeles. Even pages about traffic conditions the day before would be considered stale for the query.

This LP shows traffic conditions in Los Angeles on the date this example was written (August 7, 2012).

Landing page link: LP This LP shows traffic conditions in Los Angeles on August 6, 2012. It is stale and useless for the query even though it is only one day old.

Landing page link: LP.

[Nobel Peace Prize Winner], English (US) Find the name of the most recent winner of this prize. This is an information query. At the time this example was written, 2011 was the most recent year that the Noble Peace Prize was awarded.

This LP on the New York Times website has information about the most recent winner of this prize.

Landing page link: LP.

This LP on the BBC website has information about the 2006 winner of this prize. This page is stale for the query.

Proprietary and Confidential – Copyright 2013

116

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP.

Explanation

[60 minutes], English (US) 60 Minutes is a popular weekly TV show. Users probably want to watch or find information about the most recent episode. At the time this example was written, the show that aired on September 23, 2012 was the most recent episode.

This LP is an article about the most recent episode of 60 Minutes which aired on September 23, 2012

Landing page link: LP. This LP is about an episode of 60 Minutes that aired on September 16, 2012. Users might be looking for the episode from the previous week.

Landing page link: LP.

This LP has the 2012 Olympics gymnastics results.

[olympics gymnastics results], English (US) Users probably want to find a page that displays gymnastics results from the most recent Olympics.

Landing page link: LP.

This LP has the 2008 Olympics gymnastics results. It is stale for the query.

Proprietary and Confidential – Copyright 2013

117

Sometimes, queries don't demand the most recent information. For example, some queries, such as [birthdate of ronald reagan] and [atomic weight of gold] request basic facts that are well understood and established and unlikely to change. Finally, for some queries, there may be a "newsy" or recent information user intent as well as a more "timeless" information user intent. Users issuing queries for celebrities or politicians may be interested in biographical information or may be looking for the latest news or gossip. Please note: Information about almost any subject can become stale or outdated over time. Please use your judgment. Clearly outdated or stale pages should receive low utility ratings, unless the query is "asking" for the outdated content.
Query and User Intent LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP. Explanation

[arthritis], English (US) Users are looking for information about this disease. Because there is ongoing research done on treatments and cures for most diseases, users are probably looking for authoritative, up-todate information on this subject.

Medline Plus is an authoritative website that regularly updates its content. Even though there is no date on the page indicating when it was updated, we can assume the content is up-to-date.

Landing page link: LP.

This 2006 article would be considered stale for this medical query.

Landing page link: LP. This landing page has the information users are looking for. Notice that the requested information is even displayed in the snippet.

[when was Franklin Delano Roosevelt born], English (US) Users want to find this well understood fact.

We couldn’t find a stale page for this query.

Proprietary and Confidential – Copyright 2013

118

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP.

Explanation

[barack obama], English (US) Barack Obama is the President of the United States. We will assume users are looking for news, current or biographical information about him, images, etc.

This article contains up-to-date news about Obama (at the time this example was written).

Landing page link: LP. This landing page has an article from 2008 about Barack Obama choosing Joe Biden as his running mate.

Note: The date the page was created may be different from when the content was last updated or modified. When content is updated, the page will sometimes show the date of the update, not the date the page was created. If you are curious about the content of a page, try the "Wayback Machine" on the Internet Archive: http://archive.org/web/web.php. Not all pages are available, but this tool may help you understand how some pages were created and how their content has changed over time.

19.0

Misspelled and Mistyped Queries and Results

19.1 Misspelled and Mistyped Queries
You will notice that some queries are misspelled or mistyped. Here are some examples of queries that are obviously misspelled.
Misspelled Query [federal expres], English (US) [my sapce], English (US) [the ecomonist], English (US) [expdeia], English (US) [New England Patroits], English US) [byonce knowles], English (US) Query Interpretation The only reasonable query interpretation is the company named Federal Express. The only reasonable query interpretation is the website MySpace. The only reasonable query interpretation is the news and economics publication. The only reasonable query interpretation is the travel website. The only reasonable interpretation is the NFL football team. The only reasonable interpretation is the famous singer/actress, Beyonce Knowles.

Some misspelled or mistyped queries are more difficult to interpret. Use your judgment and do query research. For obviously misspelled or mistyped queries, you should base your rating on user intent, not necessarily on exactly how the query has been spelled or typed by the user. For queries that are not obviously misspelled or mistyped, you should respect the query as written and assume users are looking for results for the query as it is spelled.

Proprietary and Confidential – Copyright 2013

119

19.2 Name Queries
Consider the query [John Stuart], English (US). There is a very famous Jon Stewart, the comedian and host of a popular US television show. But we should not assume that the query [John Stuart] has been misspelled. There are many people named John Stuart. We will respect the query as written and assume the user is looking for someone named “John Stuart”.
Query and Query Interpretations LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP Explanation

Official homepage of Jamie Fox, the guitarist.

[Jamie Fox], English (US) There are many reasonable interpretations for this query including the guitarist named Jamie Fox, Jamie Fox the politician, Jamie Fox Photography, Jamie Fox the actress, and regular people named Jamie Fox. Helpful results for more prominent people named Jamie Fox should receive higher ratings than helpful results for less prominent people named Jamie Fox. Because Jamie Foxx is such a famous actor, we will consider Jamie Foxx to be a minor interpretation, not offtopic. Landing page link: LP

Official homepage of Jamie Fox Photography.

Landing page link: LP

Wikipedia article about Jamie Fox, the New Jersey politician.

Landing page link: LP

IMDb page about Jamie Fox, the actress.

Proprietary and Confidential – Copyright 2013

120

Query and Query Interpretations [Jamie Fox], English (US) There are many reasonable interpretations for this query. Because Jamie Foxx is such a famous actor, we will consider Jamie Foxx to be a minor interpretation, not offtopic.

LP of the Result Block, Result Block, PQ Rating, Utility Rating

Explanation

Landing page link: LP

Official homepage of Jamie Foxx, the actor.

Landing page link: LP

[Micheal Jordan], English (US) There are many people named “Micheal Jordan”. Helpful results for more prominent people named Micheal Jordan should receive higher ratings than helpful results for less prominent people named Micheal Jordan. Landing page link: LP

LinkedIn page for the name “Micheal Jordan.” The page lists all users of the LinkedIn site named “Micheal Jordan.” LinkedIn is a very popular t social networking website for professionals in the US. This result would be very helpful for many or most users trying to find information about someone named “Micheal Jordan.”

LinkedIn page for Micheal Jordan, a facilities services professional in Atlanta, Georgia.

Because Michael Jordan is such a famous athlete/celebrity, we will consider Michael Jordan to be a minor interpretation.

Landing page link: LP

Michael Jordan’s player profile page on the NBA basketball website.

Proprietary and Confidential – Copyright 2013

121

19.3 Spelling Suggestion Result Blocks
When a user misspells or mistypes a query, search engines may display spelling suggestions. Users rely on "Did you mean" or “Do you mean” suggestions to tell them when they might be spelling something incorrectly and as a way to get better search result pages. “Did you mean” type result blocks are NTRBs, but they do have a prominent link to a page of search results for the spelling suggestion. Your utility rating should reflect both the helpfulness of the suggestion itself and the helpfulness of the LP of the suggestion. For every spelling suggestion, please be sure to look at the suggestion and click through to the landing page of the suggestion before rating. Sometimes, you will find it helpful to compare the landing page of the suggestion to the landing page of the original query. Here is some specific guidance on assigning utility ratings to spelling suggestion result blocks:
Rating Vital (V) Use this rating when: Do not use the V rating for spelling suggestion result blocks. The result is very helpful for many or most users because: • The query is clearly and severely misspelled. • The spelling suggestion is very likely or exactly what users are looking for. • The landing page of the suggestion is very helpful. The result is very helpful for some users or helpful for many users because: • The query is likely or very likely misspelled. • The spelling suggestion is the correct spelling for a likely or somewhat likely intent of the original query. • The landing page of the suggestion is helpful. The result is helpful for few users. One or more of the following apply: • It is likely that the query is not misspelled. • The spelling suggestion represents an unlikely interpretation. • The spelling suggestion is trivial, such as adding or deleting a space which makes little difference to the query and/or the results on the landing page of the suggestion. • The landing page of the suggestion is less helpful. The result is unhelpful or useless. One or more of the following apply: • The query is clearly not misspelled. • The spelling suggestion itself is misspelled. • The spelling suggestion changes the meaning of the query. • The spelling suggestion is clearly not what the user is looking for. • The spelling suggestion is offensive (such as uninvited porn, racial slur, etc.) • The landing page of the suggestion is unhelpful, i.e., the results are useless for the original query.

Useful (U)

Relevant (R)

Slightly Relevant (SR)

Off-Topic/Useless (O)

Query and Query Interpretations

LP of the Query, LP of the Spelling Suggestion, Utility Rating Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

Explanation

[irratated], English (US) This query has one reasonable interpretation: the word “irritated.”

The query is clearly misspelled. The suggestion is the correct spelling of the query, and the LP of the suggestion is very helpful.

Proprietary and Confidential – Copyright 2013

122

Query and Query Interpretations ["jack angle" scrap metal] , English (US) This query has one interpretation: Jack Engle & Co., a scrap metal company in Los Angeles.

LP of the Query, LP of the Spelling Suggestion, Utility Rating Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

Explanation

The query is clearly misspelled. The suggestion is the correct spelling of the query, and the LP of the suggestion is very helpful.

[imetrex], English (US) This query has one interpretation: the drug for treating migraine headaches.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP The query is clearly misspelled. The suggestion is the correct spelling of the query, and the LP of the suggestion is very helpful.

[utube], English (US) This query has one reasonable interpretation: the youtube.com website.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

The query is clearly misspelled. The suggestion is the correct spelling of the query, and the LP of the suggestion is very helpful.

[culcable], English (US) The query is misspelled or mistyped. The most likely interpretation is the word “culpable,” though perhaps this could be a misspelling of a type of cable. [vanilla cupcake mix make 1 cup cake], English (US) The meaning of the query is clear. There is a misspelling or typo: cupcake is the correct spelling.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

The query is misspelled. The suggestion is the correct spelling of a likely or very likely interpretation, and the LP of the suggestion is very helpful.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

The query has a mild misspelling and the spelling suggestion is technically correct. The results on the LP of the suggestion are similar to the results for the original query.

[one persons wealth raises the value of money] The meaning of the query is clear. There is a misspelling or typo: “persons” should be written “person’s”.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

The query has a misspelling and the spell suggestion is technically correct. The results on the LP of the suggestion are similar to the results for the original query.

Proprietary and Confidential – Copyright 2013

123

Query and Query Interpretations [twilight new moon sound track] The query intent is clear. There is a possible mild misspelling. “Soundtrack” is the more common and possibly preferred spelling,

LP of the Query, LP of the Spelling Suggestion, Utility Rating

Explanation “Soundtrack” may be the more common and preferred spelling, but “sound track” is not technically incorrect. The LP of the suggestion is perhaps a bit more helpful than the LP for the original query. This suggestion is helpful for some or few users.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

[cynderella], English (US) There are several interpretations for the query as spelled, including a bridal shop, entertainment agency, a cake company, etc.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

It is likely that the query is not misspelled. The results on the landing page of the suggestion probably do not match the intent of the original query. However, this suggestion could be helpful for a few users.

[he likes me a lot but doesn't want to get hurt], English (US) This query is not misspelled.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP The suggestion is misspelled even though the original query is not.

[girl in pink dress plays outside], English (US) This query is not misspelled.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP The original query is clearly spelled correctly. This suggestion is useless.

[how shades are cut], English (US) This query is not misspelled.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

The query is clearly not misspelled. The suggestion changes the meaning of the query.

[people can take a wall and turn it into a tv], English (US) This query is not misspelled.

Landing page link of the original query: LP Landing page link of the spelling suggestion: LP

The query is clearly not misspelled. The suggestion changes the meaning of the query.

Proprietary and Confidential – Copyright 2013

124

20.0

Non-Vital Results for URL Queries

Raters sometimes ask the question, “For a well-formed working URL query, are the only acceptable utility ratings for a result either V or OT?” The answer is No - there can be other helpful results for URL queries. Some users issue URL queries to find information about a website, such as reviews or recent news. We recommended this to you as one method of reputation research in the PQ guidelines. Real users do this too. Results which give reviews and reputation information can be very helpful for a URL query. However, websites that offer usage statistics about a website are not usually helpful results for URL queries. Most users aren’t interested in this kind of information.

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP

Explanation

This LP is the target of the query.

[coldwatercreek.com], English (US) Go to http://www.coldwaterc reek.com , the website of Coldwater Creek, a well-known US company that sells women’s clothing and accessories online and in retail stores.

Landing page link: LP

This LP displays coupon codes that users can use when making online purchases on the Coldwater Creek website. Online coupon codes are very popular in the US. This is a well-known coupon website where users can often find working coupon codes. This result displays active coupon codes that would be very helpful for most users shopping online at coldwatercreek.com.

Landing page link: LP

This LP has 3 reviews of the coldwatercreek.com website. This is a well-known review website. This result would be helpful for many users considering whether to make purchases at coldwatercreek.com. If there were more reviews and the reviews were more recent, the utility rating would be Useful only.

Proprietary and Confidential – Copyright 2013

125

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP

Explanation

This LP is the target of the query.

[ratemyprofessors. com], English (US) Go to http://www.ratemy professors.com, a website where students can rate their college professors.

Landing page link: LP The LP is a Huffington Post article dated August 31, 2012 about the ratemyprofessors.com website. Some users might be interested in this article.

Landing page link: LP The landing page has an article dated April 17, 2006 about the ratemyprofessors.com website. Few or no users would be interested in this outdated information.

21.0

Product Queries: Action (Do) vs. Information (Know) Intent

Some product queries, such as [ipad reviews], have a clear information-seeking (Know) intent. Other product queries, such as [buy ipad], have a clear purchase (Do) intent. And some product queries, such as [ipad store.apple.com], have a clear navigation (Go) intent. However, most product queries don’t obviously specify one type of intent. Even though the ultimate goal may be to purchase a product, many other activities may take place first: researching the product (reviews, technical specifications), understanding the options that are available (brands, models, pricing), viewing and considering various options (browsing), etc. Give high utility ratings to results that allow users to research, browse, and decide what to purchase. PQ ratings for product results need extra care and attention. Often, the results for product queries are YMYL pages. Users need high quality information from authoritative sources when researching products, especially when products are expensive or represent a major investment/important life event (for example cars, washing machines, computers, wedding gifts, baby products, etc). When buying products, users need websites they can trust: good reputation, extensive customer service support, etc. Results for product queries may be important for both your money and your life (YMYL)!

Proprietary and Confidential – Copyright 2013

126

Query and User Intent [high definition tv], English (US) This query doesn’t specify a size or brand of TV. Users are probably doing research before making a purchase.

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: LP

Explanation

The LP has comprehensive information about different types of high def TVs (plasma, LCD, LED), as well as reviews about many specific models. The website is known for having helpful reviews and information about electronics.

Landing page link: LP [Toyota Prius], English (US) Most users who type this query are probably considering whether to buy a Toyota Prius and want information. The LP is on a well-known car review website. It would be very helpful for many or most users.

Landing page link: LP [floral girl toddler jeans size 3T], English (US) The query is very specific. The user wants to browse and find an item to buy. This is a helpful page for browsing and shopping for toddler girls jeans. There are many options for floral jeans in size 3T. This is a high quality store known for selling jeans.

Landing page link: LP [cufflinks], English (US) Find different cufflink styles and options. Browse or purchase cufflinks. The LP has a large collection of cufflinks from a very reputable retailer. There are many options and styles. This is a good page for browsing. There are helpful features at the top to choose styles, price range, etc.

Proprietary and Confidential – Copyright 2013

127

22.0

Mobile Landing Pages

When rating BU tasks, you should assume that users are on a desktop computer or a laptop. (Mobile rating projects have separate instructions.) Most results in BU tasks are designed for desktop and laptop users. Please assign ratings to mobile-specific pages as follows: • • When the mobile page is high functioning and would be satisfying for desktop/laptop users, then rate normally. When the mobile page has limited functionality and a non-mobile version exists that would be much more helpful for users, please do not assign above SR.

Please use your judgment when considering how helpful mobile versions of landing pages would be for users. Of course, if the mobile landing page is unrelated to the query or useless for any reason, a rating of OT is appropriate. PQ rating assumes that users are on a desktop or laptop and have a keyboard, mouse or track pad, and at least a normal sized monitor. Some mobile pages have very limited functionality and therefore are unsatisfying for a desktop or laptop user. A low PQ rating can be justified if the page is unsatisfying/does not achieve its purpose for laptop or desktop users.
Query and User Intent LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: Stanford homepage Explanation

[Stanford], English (US) Navigate to the homepage of the Stanford University website.

Landing page link: Stanford mobile homepage

The mobile version has less content and limited functionality compared to the desktop version. It would be helpful for few users.

Proprietary and Confidential – Copyright 2013

128

Query and User Intent

LP of the Result Block, Result Block, PQ Rating, Utility Rating Landing page link: CBS Sports homepage

Explanation

[cbs sports], English (US) Navigate to the homepage of the CBS Sports site. Landing page link: CBS Sports mobile homepage

Even though the mobile version has a lot of links, the font size is small and hard to read on a desktop. It would be helpful for few users.

Landing page link: People homepage

[people], English (US) Navigate to the homepage of the People magazine website.

Landing page link: People mobile homepage

Although the versions are different, the mobile version has a lot of the same content. Some users might not even notice that it is a mobile version.

Proprietary and Confidential – Copyright 2013

129

Part 4: Using the Evaluation Platform
23.0 Introduction
Welcome to the Evaluation Platform! The Evaluation Platform is the system you will use to acquire and rate tasks.

24.0 Accessing the Evaluation Platform (EP)
Go to this link to access the Evaluation Platform. http://www.raterhub.com/evaluation/rater

25.0 Evaluation Platform Screenshot
Here is a screenshot of the task acquisition page on the Evaluation Platform. Please note that changes are sometimes made to the appearance of pages on the EP, so it may not look exactly as it appears here.

The red numbers represent the following: 1. Tasks This text shows that you are at the task acquisition page on the Evaluation Platform. 2. [email protected] Your Gmail account. 3. Recent tasks Click this link to revisit tasks completed in the last few minutes. 4. Sign out Click this link to end your rating session. 5. Acquire if available Click this link to acquire a new task. 6. Experimental This text indicates that there is an Experimental task available. Please note there are other types of rating tasks (e.g. Result Review, Side-by-Side, etc.). Sometimes there will be only one task type available, but other times you may see more than one type displayed. When there is more than one type displayed, you may choose the type you want to acquire.

Proprietary and Confidential – Copyright 2013

130

26.0 Block Utility Task Page Screenshot

The red numbers represent the following: 1. [email protected] Your Gmail account. 2. Recent tasks Click this link to revisit tasks completed in the last few minutes. 3. Sign out

Proprietary and Confidential – Copyright 2013

131

Click this link to end your rating session. 4. Web This text indicates that the current task belongs to the Web property type. Other properties you may rate include Video, Image, Mobile, etc. 5. Experimental This text indicates that the current task belongs to the Experimental project type. Other project types you may rate include Result Review, Side-by-Side, etc. 6. Average Estimated Time This text indicates the average estimated time for completion of the task. 7. Instructions Click this tab to view the project-specific instructions. Some tasks display project-specific instructions on the task page instead of behind the Instructions tab. 8. Query You should understand the query before rating the task. 9. Locale This refers to the task location and task language associated with the query. 10. Report a Problem / Release this Task Click this link to report a problem and/or release the task. 11. Utility Rating Slider Use this rating slider to assign a utility rating. 12. Page Quality Rating Slider Use this rating slider to assign a page quality rating. 13. P – F – D Flags Assign any of these flags (Porn, Foreign Language, Didn’t Load) that apply to the result. 14. Report Dupe Use this button to report duplicate results. Results are considered to be duplicates if they have the same landing page. For more information about reporting dupes, please see Section 29.0. 15. Comment Use the result block comment box to write comments that will be helpful to you when assigning utility and PQ ratings. Note: All comments must be written in English. 16. Dupe Confirmation Checkbox Check this box to confirm that you are done flagging duplicate results. 17. Submit Use this button to submit the task and continue rating. 18. Submit and Stop Rating Use this button to submit the task and stop rating. 19. Cancel Use this button to cancel any ratings you have assigned in the task.

Proprietary and Confidential – Copyright 2013

132

27.0 Notes about Using the Block Utility Rating Interface
Result blocks are sometimes slow to load: Before assigning any ratings, please scan the task page to make sure that all result blocks with a utility rating slider have loaded. Please note that result blocks may be slow to load and/or that the task page may need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a few times, please release the task. Sometimes a side will not have any result blocks: If one side displays the message “This side did not generate any results,” please do not release the task. Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will lose them and they will have to be entered again.

28.0 Using the “Report a Problem / Release this Task” Button
You will report problems and release tasks using the “Report a Problem / Release this Task” button on the rating interface. This is what the “Report a Problem / Release this Task” button looks like:

Clicking the red button displays the following reporting and release options:

Proprietary and Confidential – Copyright 2013

133

Here are some examples for the “lack expertise” and “technical problem” release categories.

Lack expertise: These queries require specialized knowledge and may be difficult for some raters. Query [cortisol secretion shows a blank pattern of secretion] [minecraft gun and vehicles 1.1 mod installer [python pandas index like matrix] General Query Topic Scientific query Gaming query Technical query

Technical problem: These result blocks have question marks and other characters where text should be.

Please note: Based on the number and/or type of tasks that you release, you may be asked to provide additional details about the reason for some of the releases.

29.0 Reporting Results with Duplicate Landing Pages
In addition to assigning utility ratings to each result block, we would like you to tell us whether any results are duplicates (dupes).

Pre-Identified Duplicates
Some duplicate results are automatically detected and will be pre-identified for you. They will be highlighted by a grey or green background, and annotated by either "Same as..." or "Dupe of..." text right below the slider and above the result block. This is what these pre-identified dupes look like:

Proprietary and Confidential – Copyright 2013

134

Same as… pre-identified duplicate result

Dupe of… pre-identified duplicate result

Please note: • • • Results annotated by "Dupe of..." may be given different ratings/flags/comments. Results annotated by “Same as…” may not be given different ratings/flags/comments. Their utility ratings, PQ ratings, flags, and comments will be automatically transferred to each other. You cannot un-check dupes that have been automatically detected and pre-identified.

Rater-Identified Duplicates
We would like your help identifying duplicate results that have not been automatically detected. Please mark two results as dupes if they have essentially the same content on the main landing page, AND you would not want a search engine to return both results for the query. Please note that in this project dupe identification is query-dependent. • Specific queries: For queries where the user is looking for a specific piece of content (such as queries looking for song lyrics, queries looking for a specific article, etc.), obtaining that piece of content from different sites could be helpful for users to verify the information, so they should not be rated as dupes. Broad queries: If the query is broad, then returning the same piece of content is not what the user is looking for, and hence those results should be flagged as dupes. Results may be considered dupes even if they have different minor content on the page (such as different ads, images, or related links).



Proprietary and Confidential – Copyright 2013

135

Please identify dupes both within the same side and across sides. Even for cross-side results, you should still ask yourself the question "Would you want to see both results if they were returned by the same search engine?"

Reporting Duplicate Results
When you notice that the results in two or more result blocks are duplicates, please click on the Report Dupe button of one of the results. The current result (the result you clicked on) will be highlighted by a thick solid red border and the button's name will change to Select Dupes.

You can then check results that are duplicates of the current result, and the checked results will be highlighted by a red dotted border.

The duplicate results that you checked will be annotated by "Dupe of..." text right below the sliding rating scale above the result block. After selecting all dupes, please click the red Select Dupes button to return to the normal rating mode. The button's name will change back to Report Dupe, and you will be able to report other sets of dupes (if there are any). If you change your mind, you can always un-check a result.

Proprietary and Confidential – Copyright 2013

136

Dupes QUERY: [choosing and Installing a motorcycle battery] URL 1: http://www.caimag.com/wordpress/2010/03/06/motorcycle-battery-how-to-choose-install URL 2: http://www.articlesbase.com/motorcycles-articles/choosing-and-installing-a-motorcycle-battery-47798.html Reason: Both of these results display the same article (which also appears on many other pages on the web). The only real difference between the landing pages is the ads displayed around the article. The query is broad enough that users would not benefit by search engines returning more than one of these results. Dupes QUERY: [jason castro] URL 1: http://www.myspace.com/jasoncastromusic URL 2: http://www.myspace.com/jasoncastromusic?MyToken=503599bf-01cf-4427-bdf4-d63920c107f9 Reason: These two results have the same landing page, even though the URLs are different. Users would not benefit by search engines returning both results. Not Dupes QUERY: [material girl lyrics] URL 1: http://www.lyricsfreak.com/m/madonna/material+girl_20086925.html URL 2: http://www.lyrics007.com/Madonna%20Lyrics/Material%20Girl%20Lyrics.html Reason: Even though both pages display the lyrics to the song "Material Girl", users would probably want to have the option to visit both pages so that they could verify the accuracy of the lyrics. Users could benefit by search engines returning more than one page with the lyrics to the song.

30.0 Using the Page Quality Rating Scale in Block Utility Rating Tasks
You will use the Page Quality slider to rate the PQ of the main landing page of the result block. Important: Due to the number of results in most block utility tasks, you will base your PQ rating on the landing page questions alone (e.g., identifying the purpose of the page, identifying the MC, SC, and Ads, rating the quantity of MC, etc.). You are NOT required to perform the website level checks required in PQ rating tasks. In other words, you are not required to learn about the reputation of the website, assess whether the website is maintained, etc. However, if you do feel that the landing page seems low quality and you want to explore the website for corroborating evidence, please do so. Reminder: Most results in block utility tasks have both utility and PQ rating sliders, but some results only have a utility slider.

Proprietary and Confidential – Copyright 2013

137

Sponsor Documents

Or use your account on DocShare.tips

Hide

Forgot your password?

Or register your new account on DocShare.tips

Hide

Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in

Close