Jump to content
N-Europe

Metacritic's score weighting system revealed (amazing BS rankings inside!)


Dcubed

Recommended Posts

Ever wondered why games on Nintendo platforms often get inexplicably low metacritic ratings? Here's why!

 

http://www.gamasutra.com/view/news/189448/Metacritics_weighting_system_revealed.php

 

March 27, 2013 | By Simon Parkin

3 comments

More: Console/PC, Smartphone/Tablet, Indie, Business/Marketing, Exclusive, GDC

Share on facebook Share on linkedin Share on print Share on gmail More Sharing Services

 

 

 

 

The website Metacritic weights the scores of different video game critics and publications when calculating its aggregate 'metascore' for any particular title.

 

Each critic/ publication is assigned one of six different weightings of 'importance', with some publications exerting considerably more influence over a game's final 'metascore' than others.

 

This was revealed by Adams Greenwood-Ericksen of Full Sail University at a talk titled 'A Scientific Assessment of the Validity and Value of Metacritic' delivered at the Game Developer's Conference in San Fransisco this afternoon.

 

Metacritic confirmed to Greenwood-Ericksen during the course of his research that the site applies different weightings to incoming critics and publications' reviews in order to calculate its 'averaged' numerical score for any particular title.

 

However, it would not reveal how the weightings were assigned to different publications or on what criteria one critic was given a greater weighing than another.

 

The course director and his students then set about modelling the weightings based on data pulled from the site. Finally, after six months of work, the researchers compared their modeled scores to the actual scores and discovered that across the 188 publications that feed into Metacritic's video game score work, their findings were almost entirely accurate.

 

Greenwood-Ericksen stated they wanted to carry out the research as Metacritic scores are "very important to a lot of people" and pointed out that, when publishers withhold financial bonuses when a game doesn't reach its Metacritic target, livelihoods are tied up in the site's work.

 

He also reminded attendees that often a publisher's Wall Street stock can change on the basis of a Metacritic score., and as such the site's workings are of practical interest.

 

The findings will also be of interest to consumers as, if accurate, they reveal that some official magazines and sites (which are sponsored by platform holders in some cases) are assigned a greater weighting than independent sites and critics.

 

Here is the full listing of score weightings used by Metacritic according to Greenwood-Ericksen's research:

 

Weighting -- Critic/ Publication

 

Highest (1.5) -- Dark Zero

Highest (1.5) -- Digital Chumps

Highest (1.5) -- Digital Entertainment News

Highest (1.5) -- Extreme Gamer

Highest (1.5) -- Firing Squad

Highest (1.5) -- Game Almighty

Highest (1.5) -- Game Informer

Highest (1.5) -- GamePro

Highest (1.5) -- Gamers Europe

Highest (1.5) -- GameTrailers

Highest (1.5) -- GotNext

Highest (1.5) -- IGN

Highest (1.5) -- IGN AU

Highest (1.5) -- IGN UK

Highest (1.5) -- Just Adventure

Highest (1.5) -- Machinima

Highest (1.5) -- Planet Xbox 360

Highest (1.5) -- PlayStation Official Magazine UK

Highest (1.5) -- PlayStation Official Magazine US

Highest (1.5) -- Telegraph

Highest (1.5) -- The New York Times

Highest (1.5) -- TheSixthAxis

Highest (1.5) -- TotalPlayStation

Highest (1.5) -- VGPub

Highest (1.5) -- Videogameszone.de

Highest (1.5) -- Wired

Highest (1.5) -- Xboxic

Highest (1.5) -- Yahoo Games

Highest (1.5) -- ZTGames Domain

 

High (1.25) -- Absolute Games

High (1.25) -- ActionTrip

High (1.25) -- Adventure Gamers

High (1.25) -- Computer & Video Games

High (1.25) -- Console Gameworld

High (1.25) -- Da GameBoyz

High (1.25) -- Darkstation

High (1.25) -- Edge Magazine

High (1.25) -- EGM

High (1.25) -- EuroGamer Italy

High (1.25) -- EuroGamer Spain

High (1.25) -- G4 TV

High (1.25) -- Game Chronicles

High (1.25) -- GameDaily

High (1.25) -- Gameplayer

High (1.25) -- Gamer 2.0

High (1.25) -- Gamervision

High (1.25) -- Games Master UK

High (1.25) -- Gamespot

High (1.25) -- GameSpy

High (1.25) -- Gaming Age

High (1.25) -- Gaming Nexus

High (1.25) -- Maxi Consoles (Portugal)

High (1.25) -- Pelit

High (1.25) -- Play.tm

High (1.25) -- PlayStation Universe

High (1.25) -- PlayStation Official AU

High (1.25) -- PSM3 Magazine UK

High (1.25) -- PS Extreme

High (1.25) -- RPG Fan

High (1.25) -- Strategy Informer

High (1.25) -- Team Xbox

High (1.25) -- The Onion (AV Club)

High (1.25) -- Totally 360

High (1.25) -- WonderwallWeb

High (1.25) -- XGN

 

Medium (1.0) -- 1Up

Medium (1.0) -- CPU Gamer

Medium (1.0) -- Cubed3

Medium (1.0) -- Cynamite

Medium (1.0) -- D+Pad Magazine

Medium (1.0) -- DailyGame

Medium (1.0) -- Destructoid

Medium (1.0) -- Eurogamer

Medium (1.0) -- Everyeye.it

Medium (1.0) -- Game Revolution

Medium (1.0) -- Game Shark

Medium (1.0) -- Gameblog.fr

Medium (1.0) -- GameKult

Medium (1.0) -- Gamereactor Denmark

Medium (1.0) -- Gamers' Temple

Medium (1.0) -- GameShark

Medium (1.0) -- Gameblog.fr

Medium (1.0) -- GamesNation

Medium (1.0) -- GameStar

Medium (1.0) -- GameTap

Medium (1.0) -- Gaming Target

Medium (1.0) -- Gamereactor Sweden

Medium (1.0) -- The Guardian

Medium (1.0) -- Hardcore Gamer Magazine

Medium (1.0) -- HellBored

Medium (1.0) -- NiceGamers

Medium (1.0) -- Joystiq

Medium (1.0) -- Just RPG

Medium (1.0) -- Level 7.nu

Medium (1.0) -- Modojo

Medium (1.0) -- MondoXbox

Medium (1.0) -- Multiplayer.it

Medium (1.0) -- N-Europe

Medium (1.0) -- Netjak

Medium (1.0) -- NGamer Magazine

Medium (1.0) -- Nintendo Life

Medium (1.0) -- Nintendo Power

Medium (1.0) -- Nintendojo

Medium (1.0) -- Nintendo World Report

Medium (1.0) -- NZGamer

Medium (1.0) -- Official Nintendo Magazine UK

Medium (1.0) -- Official Xbox 360 Magazine UK

Medium (1.0) -- Official Xbox Magazine

Medium (1.0) -- Official Xbox Magazine UK

Medium (1.0) -- PALGN

Medium (1.0) -- PC Format

Medium (1.0) -- PC Gamer (Germany)

Medium (1.0) -- PC Gamer UK

Medium (1.0) -- PC Gamer

Medium (1.0) -- PC Powerplay

Medium (1.0) -- PGNx Media

Medium (1.0) -- Play Magazine

Medium (1.0) -- PlayStation LifeStyle

Medium (1.0) -- Pocketgamer UK

Medium (1.0) -- PT Games

Medium (1.0) -- Real Gamer

Medium (1.0) -- SpazioGames

Medium (1.0) -- Talk Xbox

Medium (1.0) -- The Escapist

Medium (1.0) -- Thunderbolt

Medium (1.0) -- Total VideoGames

Medium (1.0) -- Worth Playing

Medium (1.0) -- X360 Magazine UK

Medium (1.0) -- Xbox World 360 Magazine UK

Medium (1.0) -- Xbox World Australia

Medium (1.0) -- Xbox360 Achievements

Medium (1.0) -- Xbox Addict

 

Low (0.75) -- 360 Gamer Magazine UK

Low (0.75) -- 3DJuegos

Low (0.75) -- Ace Gamez

Low (0.75) -- Atomic Gamer

Low (0.75) -- BigPond GameArena

Low (0.75) -- Console Monster

Low (0.75) -- Deeko

Low (0.75) -- Eurogamer Portugal

Low (0.75) -- Game Focus

Low (0.75) -- Gameplanet

Low (0.75) -- Gamer Limit

Low (0.75) -- Gamer.nl

Low (0.75) -- Games Radar (in-house)

Low (0.75) -- Games TM

Low (0.75) -- Gamestyle

Low (0.75) -- GameZone

Low (0.75) -- Gaming Excellence

Low (0.75) -- Gaming Trend

Low (0.75) -- Impulse gamer

Low (0.75) -- Kombo

Low (0.75) -- MEGamers

Low (0.75) -- Metro Game Central

Low (0.75) -- MS Xbox World

Low (0.75) -- NTSC-uk

Low (0.75) -- PS Focus

Low (0.75) -- PSW Magazine UK

Low (0.75) -- Video Game Talk

Low (0.75) -- VideoGamer

 

Lower (0.5) -- Armchair Empire

Lower (0.5) -- Cheat Code Central

Lower (0.5) -- Game Over Online

Lower (0.5) -- Game Positive

Lower (0.5) -- Gamer's Hell

Lower (0.5) -- Gamereactor Sweden

Lower (0.5) -- Gamers.at

Lower (0.5) -- Giant Bomb

Lower (0.5) -- PS3bloggen.se

Lower (0.5) -- RPGamer

Lower (0.5) -- Vandal Online

 

Lowest (0.25) -- 9Lives

Lowest (0.25) -- Boomtown

Lowest (0.25) -- Computer Games Online RO

Lowest (0.25) -- GamerNode

Lowest (0.25) -- GamingXP

Lowest (0.25) -- IC-Games

Lowest (0.25) -- Insidegamer.nl

Lowest (0.25) -- Jolt Online Gaming

Lowest (0.25) -- Kikizo

Lowest (0.25) -- LEVEL

Lowest (0.25) -- Meritstation

Lowest (0.25) -- My Gamer

Lowest (0.25) -- Official PlayStation 2 Magazine UK

Lowest (0.25) -- Play UK

Lowest (0.25) -- WHAM! Gaming

 

I wonder how much they charge for the upper tiers... :rolleyes: (and which publishers asked the lower publications to be blacklisted...)

 

This also means that IGN (in agregate total) is worth 4.5 times as much as Eurogamer and N-Europe in terms of review scores... Certainly, they are a bastion of quality; a trustworthy, unbiased and reliable source who would never cave in to publisher influence...

 

And this is the score agregator that publishers use to determine pay bonuses... :nono:

Edited by Dcubed
Link to comment
Share on other sites

 

Today, the website Gamasutra "revealed" the weights that we assign to each gaming publication (for the purpose of calculating our Metascores), based on a presentation given at the Game Developers Conference this morning. There's just one major problem with that: neither that site, nor the person giving the presentation, got those weights from us; rather, they are simply their best guesses based on research (the Gamasutra headline is misleading in this respect).

 

And here's the most important thing: their guesses are wildly, wholly inaccurate. Among other things:

 

* We use far fewer tiers than listed in the article.

 

* The disparity between tiers listed in the article is far more extreme than what we actually use on Metacritic. For example, they suggest that the highest-weighted publications have their scores counted six times as much as the lowest-weighted publications in our Metascore formula. That isn't anywhere close to reality; our publication weights are much closer together and have much less of an impact on the score calculation.

 

* Last but definitely not least: Our placement of publications in each tier differs from what is displayed in the article. The article overvalues some publications and undervalues others (while ignoring others altogether), sometimes comically so. (In addition, our weights are periodically adjusted as needed if, over time, a publication demonstrates an increase or decrease in overall quality.)

 

Still, I like the idea that my reviews are as important as NGamer's reviews.

Link to comment
Share on other sites

Hmm... So why not show the actual weighting scores then?

 

Clearly I'm just supposed to believe them blindly without questioning their methodology or their reluctance to showcase any transparency in the matter...

Edited by Dcubed
Link to comment
Share on other sites

So...What exactly makes Eurogamer Spain&Italy more trustworthy than regular Eurogamer? And what makes Eurogamer Portugal be worth less?

 

Anyway, that list is so bullshit. Giving the highest weight to publications that are naturally biased towards certain games? (Thinking of the Official PS magazines, here)

Also, IGN having that much influence. Yeah, right...

 

EDIT: Right, misunderstood Cube's post completely. I'm still not likely to trust Metacritic, though.

Link to comment
Share on other sites

Hmm... So why not show the actual weighting scores then?

 

Clearly I'm just supposed to believe them blindly without questioning their methodology or their reluctance to showcase any transparency in the matter...

 

If it is true that they weight the reviews (and i do think that it is necessary to do so*), then it would be silly to release who the most influential sites are, as those will suddenly become targetted by games companies trying to bribe a higher score.

 

 

 

 

*Simply because i think that certain smaller sites might post low review scores for popular games just to increase web traffic, and so those scores may need to be weighted out....

Link to comment
Share on other sites

Well i'm not sure there is a law against it, but it certainly happens. You hear about journalists getting free stuff all the time by companies trying to get a higher score. (I can't post references, but i definitely remember this happening).

Link to comment
Share on other sites

Welcome to the world, gentlemen. Driven by money and business, corrupt and ruthless as fuck.

 

 

Anyhoo, I'd be interested to see the original article/paper/publication. Surely they've put their working in it, and thus it can quite easily be worked out whether or not it's accurate, despite what Metacritic might claim otherwise?

Link to comment
Share on other sites

Welcome to the world, gentlemen. Driven by money and business, corrupt and ruthless as fuck.

 

 

Anyhoo, I'd be interested to see the original article/paper/publication. Surely they've put their working in it, and thus it can quite easily be worked out whether or not it's accurate, despite what Metacritic might claim otherwise?

 

I would imagine they just pulled the scores and the rankings and then set it up as a huge equation which was solved using brute force....

Link to comment
Share on other sites

its a completely disgusting practice that bribes and gifts can influence a metacritic rating, especially given the weight that a lot of industries and financial backing put upon them

 

It's called lobbying, it happens in American politics all the time, gifts for favours basically, and the Americans seem oblivious to the corruption that allowing this causes. IGN, for example, have on a number of articles and podcasts I've listened to, especially the latter, mentioned gifts from gaming companies and made fun of the 'crap' gift and then followed up by saying they'll take that into consideration when scoring their game. I'm paraphrasing, and they do say all that in a joking manner, but to think that a nice gift won't sway a final score is acting in ignorance in my mind.

Link to comment
Share on other sites

×
×
  • Create New...