Rate Answer Relevance - 1.3

Answer Relevance
Guideline Version 1.3, 7/7/2020
Instructions
This hit provides a Query, Answer and Url. We expect the judges to rate the
Answer to the given Query as Excellent, Bad, Fair or Detrimental.
The Answer is Excellent if it is accurate, reasonably complete, well

formatted and answers the question directly.
The Answer is Bad if it does not meet any of the above criteria.
If answer is inaccurate or has bad formatting they should be immediately flagged
as bad. If answer is incomplete and does not help the user get the direct answer
he is looking for then too it should be marked bad.
The Answer is Fair if it you think it is accurate, slightly incomplete but user can
infer the core information he is looking for from the answer. It can also be marked
Fair if the intent is ambiguous and the answer is something specific but broadly
common interpretation.
Detrimental: The query OR answer can sometimes be controversial,

untrustworthy, misleading, biased, or embarrassing to show to the users. Judge
must identify these hits if encountered and flag them.
Important Note:
 To view search results pane correctly please use Chrome Browser and
install W5P plugin.
 If you don’t have the subject matter knowledge to make a judgment, use
the provided search engine results to get a better understanding of the
context and intent of the query and use your best judgment to judge.
 If you do not know how to judge the answer or if it is in a language you do

not understand then SKIP the hit.
Examples
Query - barometric pressure at 9,000 feet
Answer -
Absolute Absolute Atmospheric

Altitude Above Sea Level
Barometer Pressure
feet metre inches Hg mm Hg psia kg/cm2 kPa
9000 2743 21.4 543 10.9 0.739 75.3
10000 3048 20.6 523 10.5 0.711 72.4
15000 4572 16.9 429 8.29 0.583 57.2
31 more rows. Altitude above Sea Level and Air Pressure - Engineering ToolBox
https://www.engineeringtoolbox.com/air-altitude-pressure-d_462.html
Guideline - Answer is accurate, complete, well-formatted and direct (exact row

with answer present). Rate as Excellent
Query - standard hx440vas
Answer - The HX400-bbas is factory programmed with all USA, Canadian and
international NOAA Weather channels. It also has the capability to be
programmed 40 Land Mobile Radio (137 -174MHz) channels with CICSS or DCS
signaling by a Standard Horizon dealer.
http://www.standardhorizion.com/indexVS.cfm?cmd-
DisplayProduct&ProdCatID85&encProdID=52+DD5587EFA2C80FDCF82B4A30DD1
22&DivisionID=3&isArchived=0
Guideline - Query has hx440vas, but Passage answers for Hx400-bbas. Rate as
Bad
Query - changing my profile pic on my laptop
Answer -
1. To add a profile picture or change your current profile picture: Go to your
profile and hover over your profile picture.
2. Click Update Profile Picture. You can: Upload a photo from your computer.
3. Take a new photo.
4. Crop your photo and click Crop and Save.
5. If you don't want to crop your photo, click Skip Cropping in the bottom left,
then click Save.
How do I add or change my Facebook profile picture?

https://www.facebook.com/help/163248423739693
Guideline - This is a broad query that could have several intents: the user could be
referring to Facebook picture, their PC profile picture, or their Mac profile picture.
Alternatively, their laptop could be Windows or Mac. The passage refers to
instructions on how to change a Facebook profile picture on a laptop computer.
The answer is specific for an ambiguous query and it is not the most common
interpretation, hence it should be rated as Bad
Query - bond prices
Answer - The cost of dental bonding ranges from about $100 to $400 per tooth.
There are a number of factors that can influence the cost of your particular
treatment. The number of teeth you wish to treat is, of course, a major
consideration.
How Dental Bonding Alters the Look of Your Teeth – Dental…
https://www.everydayhealth.com>dental-health>cosmetic-dentistry>bond.
Guideline - There is a most common identifiable intent for this query, which is to
find the prices of bonds in the financial sense. This should be marked Bad for
intent since answering for dental bond prices would not satisfy most users.
Query - grey granite slabs

Answer - Steel Grey granite from India is a low variation durable granite with
shades of grays and small flecks of lighter grays. It is available in slabs in a
polished finish and recommended for interior and exterior projects for both
commercial and residential projects including landscaping, flooring, countertops
and walls.
Steel Gray Granit [Low Maintenance, Stain-Free, Scratch-Free]

https://aquagranite.com?product?steel-gray-granite
Guideline - The intent of this query is probably to learn more about granite slabs.
However, this passage is about one specific kind of grey granite that is sold in
slabs by the company on the source page. It is very unlikely a user would be
searching with this specific intent and the answer will not benefit the user, so this
should be marked Bad.
Query - 2016 nba draft lottery chances
Answer - NBA Draft Lottery 2016: Odds, Chances & Must-See Details • The NBA
Draft Lottery is the system that… • Rather than just giving the worst team the first
pick… • NBA Lottery Changes and Odds. Here are the number of balls, followed by
the team name… NBA Draft Lottery 2016: Odds, Chances & Must-See Details
https://coed.com/2016/17/nba-draft-lottery-odds-chances-information-preview-
facts/
Guideline - The user wants to learn about the 2016 NBA Draft lottery chances. The
answer text provides random headers, and the last bullet is the correct header for
the odds, however the actual answer is omitted from the answer text. This is an
example of an answer that is useless as a standalone. Rate as Bad
Query - take screenshot in windows 10
Answer -
View all
Windows 10: How to take screenshot

To take a quick screenshot of the active window, use the keyboard shortcut Alt +
PrtScn. This will snap your currently active window and copy the screenshot to the
clipboard. You'll need to open the shot in an image editor to save it
Guideline - Rate “Bad”. Picture and hyperlink “View all” in Answer content which
is formatting issue since only expected content is text, list, table or list of snippets.
Query - how much is in a roll or quarters
Answer - blog.goldeneaglecoin.com Image: blog.goldeneaglecoin.com How Many

Coins In A Roll Denomination Count Face Value Penny – 1 Cent 50 .50 Nickel – 5
Cent 40 $2.00 Dime – 10 Cent 50 $5.00 Quarters- 25 Cent 40 $10.00 3 more rows
https://coinauctionshelp.com/how-many-coins-in-a-roll/
Guideline – The answer begins with some unrelated bad text. This is formatting
issues. Rate as Bad.

Rate Answer Relevance - 1.3

Uploaded by

Copyright:

Available Formats

Rate Answer Relevance - 1.3

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Rate Answer Relevance - 1.3

Uploaded by

Copyright:

Available Formats

Answer Relevance

Guideline Version 1.3, 7/7/2020

The Answer is Excellent if it is accurate, reasonably complete, well

Detrimental: The query OR answer can sometimes be controversial,

 If you do not know how to judge the answer or if it is in a language you do

Absolute Absolute Atmospheric

feet metre inches Hg mm Hg psia kg/cm2 kPa

9000 2743 21.4 543 10.9 0.739 75.3

10000 3048 20.6 523 10.5 0.711 72.4

15000 4572 16.9 429 8.29 0.583 57.2

Guideline - Answer is accurate, complete, well-formatted and direct (exact row

Query - standard hx440vas

How do I add or change my Facebook profile picture?

Query - bond prices

Query - grey granite slabs

Steel Gray Granit [Low Maintenance, Stain-Free, Scratch-Free]

Query - 2016 nba draft lottery chances

Windows 10: How to take screenshot

Query - how much is in a roll or quarters

Answer - blog.goldeneaglecoin.com Image: blog.goldeneaglecoin.com How Many

You might also like