Posted March 27, 2013 (edited) Ever wondered why games on Nintendo platforms often get inexplicably low metacritic ratings? Here's why! http://www.gamasutra.com/view/news/189448/Metacritics_weighting_system_revealed.php March 27, 2013 | By Simon Parkin 3 comments More: Console/PC, Smartphone/Tablet, Indie, Business/Marketing, Exclusive, GDC Share on facebook Share on linkedin Share on print Share on gmail More Sharing Services The website Metacritic weights the scores of different video game critics and publications when calculating its aggregate 'metascore' for any particular title. Each critic/ publication is assigned one of six different weightings of 'importance', with some publications exerting considerably more influence over a game's final 'metascore' than others. This was revealed by Adams Greenwood-Ericksen of Full Sail University at a talk titled 'A Scientific Assessment of the Validity and Value of Metacritic' delivered at the Game Developer's Conference in San Fransisco this afternoon. Metacritic confirmed to Greenwood-Ericksen during the course of his research that the site applies different weightings to incoming critics and publications' reviews in order to calculate its 'averaged' numerical score for any particular title. However, it would not reveal how the weightings were assigned to different publications or on what criteria one critic was given a greater weighing than another. The course director and his students then set about modelling the weightings based on data pulled from the site. Finally, after six months of work, the researchers compared their modeled scores to the actual scores and discovered that across the 188 publications that feed into Metacritic's video game score work, their findings were almost entirely accurate. Greenwood-Ericksen stated they wanted to carry out the research as Metacritic scores are "very important to a lot of people" and pointed out that, when publishers withhold financial bonuses when a game doesn't reach its Metacritic target, livelihoods are tied up in the site's work. He also reminded attendees that often a publisher's Wall Street stock can change on the basis of a Metacritic score., and as such the site's workings are of practical interest. The findings will also be of interest to consumers as, if accurate, they reveal that some official magazines and sites (which are sponsored by platform holders in some cases) are assigned a greater weighting than independent sites and critics. Here is the full listing of score weightings used by Metacritic according to Greenwood-Ericksen's research: Weighting -- Critic/ Publication Highest (1.5) -- Dark Zero Highest (1.5) -- Digital Chumps Highest (1.5) -- Digital Entertainment News Highest (1.5) -- Extreme Gamer Highest (1.5) -- Firing Squad Highest (1.5) -- Game Almighty Highest (1.5) -- Game Informer Highest (1.5) -- GamePro Highest (1.5) -- Gamers Europe Highest (1.5) -- GameTrailers Highest (1.5) -- GotNext Highest (1.5) -- IGN Highest (1.5) -- IGN AU Highest (1.5) -- IGN UK Highest (1.5) -- Just Adventure Highest (1.5) -- Machinima Highest (1.5) -- Planet Xbox 360 Highest (1.5) -- PlayStation Official Magazine UK Highest (1.5) -- PlayStation Official Magazine US Highest (1.5) -- Telegraph Highest (1.5) -- The New York Times Highest (1.5) -- TheSixthAxis Highest (1.5) -- TotalPlayStation Highest (1.5) -- VGPub Highest (1.5) -- Videogameszone.de Highest (1.5) -- Wired Highest (1.5) -- Xboxic Highest (1.5) -- Yahoo Games Highest (1.5) -- ZTGames Domain High (1.25) -- Absolute Games High (1.25) -- ActionTrip High (1.25) -- Adventure Gamers High (1.25) -- Computer & Video Games High (1.25) -- Console Gameworld High (1.25) -- Da GameBoyz High (1.25) -- Darkstation High (1.25) -- Edge Magazine High (1.25) -- EGM High (1.25) -- EuroGamer Italy High (1.25) -- EuroGamer Spain High (1.25) -- G4 TV High (1.25) -- Game Chronicles High (1.25) -- GameDaily High (1.25) -- Gameplayer High (1.25) -- Gamer 2.0 High (1.25) -- Gamervision High (1.25) -- Games Master UK High (1.25) -- Gamespot High (1.25) -- GameSpy High (1.25) -- Gaming Age High (1.25) -- Gaming Nexus High (1.25) -- Maxi Consoles (Portugal) High (1.25) -- Pelit High (1.25) -- Play.tm High (1.25) -- PlayStation Universe High (1.25) -- PlayStation Official AU High (1.25) -- PSM3 Magazine UK High (1.25) -- PS Extreme High (1.25) -- RPG Fan High (1.25) -- Strategy Informer High (1.25) -- Team Xbox High (1.25) -- The Onion (AV Club) High (1.25) -- Totally 360 High (1.25) -- WonderwallWeb High (1.25) -- XGN Medium (1.0) -- 1Up Medium (1.0) -- CPU Gamer Medium (1.0) -- Cubed3 Medium (1.0) -- Cynamite Medium (1.0) -- D+Pad Magazine Medium (1.0) -- DailyGame Medium (1.0) -- Destructoid Medium (1.0) -- Eurogamer Medium (1.0) -- Everyeye.it Medium (1.0) -- Game Revolution Medium (1.0) -- Game Shark Medium (1.0) -- Gameblog.fr Medium (1.0) -- GameKult Medium (1.0) -- Gamereactor Denmark Medium (1.0) -- Gamers' Temple Medium (1.0) -- GameShark Medium (1.0) -- Gameblog.fr Medium (1.0) -- GamesNation Medium (1.0) -- GameStar Medium (1.0) -- GameTap Medium (1.0) -- Gaming Target Medium (1.0) -- Gamereactor Sweden Medium (1.0) -- The Guardian Medium (1.0) -- Hardcore Gamer Magazine Medium (1.0) -- HellBored Medium (1.0) -- NiceGamers Medium (1.0) -- Joystiq Medium (1.0) -- Just RPG Medium (1.0) -- Level 7.nu Medium (1.0) -- Modojo Medium (1.0) -- MondoXbox Medium (1.0) -- Multiplayer.it Medium (1.0) -- N-Europe Medium (1.0) -- Netjak Medium (1.0) -- NGamer Magazine Medium (1.0) -- Nintendo Life Medium (1.0) -- Nintendo Power Medium (1.0) -- Nintendojo Medium (1.0) -- Nintendo World Report Medium (1.0) -- NZGamer Medium (1.0) -- Official Nintendo Magazine UK Medium (1.0) -- Official Xbox 360 Magazine UK Medium (1.0) -- Official Xbox Magazine Medium (1.0) -- Official Xbox Magazine UK Medium (1.0) -- PALGN Medium (1.0) -- PC Format Medium (1.0) -- PC Gamer (Germany) Medium (1.0) -- PC Gamer UK Medium (1.0) -- PC Gamer Medium (1.0) -- PC Powerplay Medium (1.0) -- PGNx Media Medium (1.0) -- Play Magazine Medium (1.0) -- PlayStation LifeStyle Medium (1.0) -- Pocketgamer UK Medium (1.0) -- PT Games Medium (1.0) -- Real Gamer Medium (1.0) -- SpazioGames Medium (1.0) -- Talk Xbox Medium (1.0) -- The Escapist Medium (1.0) -- Thunderbolt Medium (1.0) -- Total VideoGames Medium (1.0) -- Worth Playing Medium (1.0) -- X360 Magazine UK Medium (1.0) -- Xbox World 360 Magazine UK Medium (1.0) -- Xbox World Australia Medium (1.0) -- Xbox360 Achievements Medium (1.0) -- Xbox Addict Low (0.75) -- 360 Gamer Magazine UK Low (0.75) -- 3DJuegos Low (0.75) -- Ace Gamez Low (0.75) -- Atomic Gamer Low (0.75) -- BigPond GameArena Low (0.75) -- Console Monster Low (0.75) -- Deeko Low (0.75) -- Eurogamer Portugal Low (0.75) -- Game Focus Low (0.75) -- Gameplanet Low (0.75) -- Gamer Limit Low (0.75) -- Gamer.nl Low (0.75) -- Games Radar (in-house) Low (0.75) -- Games TM Low (0.75) -- Gamestyle Low (0.75) -- GameZone Low (0.75) -- Gaming Excellence Low (0.75) -- Gaming Trend Low (0.75) -- Impulse gamer Low (0.75) -- Kombo Low (0.75) -- MEGamers Low (0.75) -- Metro Game Central Low (0.75) -- MS Xbox World Low (0.75) -- NTSC-uk Low (0.75) -- PS Focus Low (0.75) -- PSW Magazine UK Low (0.75) -- Video Game Talk Low (0.75) -- VideoGamer Lower (0.5) -- Armchair Empire Lower (0.5) -- Cheat Code Central Lower (0.5) -- Game Over Online Lower (0.5) -- Game Positive Lower (0.5) -- Gamer's Hell Lower (0.5) -- Gamereactor Sweden Lower (0.5) -- Gamers.at Lower (0.5) -- Giant Bomb Lower (0.5) -- PS3bloggen.se Lower (0.5) -- RPGamer Lower (0.5) -- Vandal Online Lowest (0.25) -- 9Lives Lowest (0.25) -- Boomtown Lowest (0.25) -- Computer Games Online RO Lowest (0.25) -- GamerNode Lowest (0.25) -- GamingXP Lowest (0.25) -- IC-Games Lowest (0.25) -- Insidegamer.nl Lowest (0.25) -- Jolt Online Gaming Lowest (0.25) -- Kikizo Lowest (0.25) -- LEVEL Lowest (0.25) -- Meritstation Lowest (0.25) -- My Gamer Lowest (0.25) -- Official PlayStation 2 Magazine UK Lowest (0.25) -- Play UK Lowest (0.25) -- WHAM! Gaming I wonder how much they charge for the upper tiers... (and which publishers asked the lower publications to be blacklisted...) This also means that IGN (in agregate total) is worth 4.5 times as much as Eurogamer and N-Europe in terms of review scores... Certainly, they are a bastion of quality; a trustworthy, unbiased and reliable source who would never cave in to publisher influence... And this is the score agregator that publishers use to determine pay bonuses... Edited March 27, 2013 by Dcubed Share this post Link to post Share on other sites
Posted March 27, 2013 Today, the website Gamasutra "revealed" the weights that we assign to each gaming publication (for the purpose of calculating our Metascores), based on a presentation given at the Game Developers Conference this morning. There's just one major problem with that: neither that site, nor the person giving the presentation, got those weights from us; rather, they are simply their best guesses based on research (the Gamasutra headline is misleading in this respect). And here's the most important thing: their guesses are wildly, wholly inaccurate. Among other things: * We use far fewer tiers than listed in the article. * The disparity between tiers listed in the article is far more extreme than what we actually use on Metacritic. For example, they suggest that the highest-weighted publications have their scores counted six times as much as the lowest-weighted publications in our Metascore formula. That isn't anywhere close to reality; our publication weights are much closer together and have much less of an impact on the score calculation. * Last but definitely not least: Our placement of publications in each tier differs from what is displayed in the article. The article overvalues some publications and undervalues others (while ignoring others altogether), sometimes comically so. (In addition, our weights are periodically adjusted as needed if, over time, a publication demonstrates an increase or decrease in overall quality.) Still, I like the idea that my reviews are as important as NGamer's reviews. Share this post Link to post Share on other sites
Posted March 27, 2013 (edited) Hmm... So why not show the actual weighting scores then? Clearly I'm just supposed to believe them blindly without questioning their methodology or their reluctance to showcase any transparency in the matter... Edited March 27, 2013 by Dcubed Share this post Link to post Share on other sites
Posted March 28, 2013 So...What exactly makes Eurogamer Spain&Italy more trustworthy than regular Eurogamer? And what makes Eurogamer Portugal be worth less? Anyway, that list is so bullshit. Giving the highest weight to publications that are naturally biased towards certain games? (Thinking of the Official PS magazines, here) Also, IGN having that much influence. Yeah, right... EDIT: Right, misunderstood Cube's post completely. I'm still not likely to trust Metacritic, though. Share this post Link to post Share on other sites
Posted March 28, 2013 Hmm... So why not show the actual weighting scores then? Clearly I'm just supposed to believe them blindly without questioning their methodology or their reluctance to showcase any transparency in the matter... If it is true that they weight the reviews (and i do think that it is necessary to do so*), then it would be silly to release who the most influential sites are, as those will suddenly become targetted by games companies trying to bribe a higher score. *Simply because i think that certain smaller sites might post low review scores for popular games just to increase web traffic, and so those scores may need to be weighted out.... Share this post Link to post Share on other sites
Posted March 28, 2013 Why the fuck is it even allowed to bribe review sites? Share this post Link to post Share on other sites
Posted March 28, 2013 Well i'm not sure there is a law against it, but it certainly happens. You hear about journalists getting free stuff all the time by companies trying to get a higher score. (I can't post references, but i definitely remember this happening). Share this post Link to post Share on other sites
Posted March 28, 2013 its a completely disgusting practice that bribes and gifts can influence a metacritic rating, especially given the weight that a lot of industries and financial backing put upon them Share this post Link to post Share on other sites
Posted March 28, 2013 Welcome to the world, gentlemen. Driven by money and business, corrupt and ruthless as fuck. Anyhoo, I'd be interested to see the original article/paper/publication. Surely they've put their working in it, and thus it can quite easily be worked out whether or not it's accurate, despite what Metacritic might claim otherwise? Share this post Link to post Share on other sites
Posted March 28, 2013 Welcome to the world, gentlemen. Driven by money and business, corrupt and ruthless as fuck. Anyhoo, I'd be interested to see the original article/paper/publication. Surely they've put their working in it, and thus it can quite easily be worked out whether or not it's accurate, despite what Metacritic might claim otherwise? I would imagine they just pulled the scores and the rankings and then set it up as a huge equation which was solved using brute force.... Share this post Link to post Share on other sites
Posted March 31, 2013 its a completely disgusting practice that bribes and gifts can influence a metacritic rating, especially given the weight that a lot of industries and financial backing put upon them It's called lobbying, it happens in American politics all the time, gifts for favours basically, and the Americans seem oblivious to the corruption that allowing this causes. IGN, for example, have on a number of articles and podcasts I've listened to, especially the latter, mentioned gifts from gaming companies and made fun of the 'crap' gift and then followed up by saying they'll take that into consideration when scoring their game. I'm paraphrasing, and they do say all that in a joking manner, but to think that a nice gift won't sway a final score is acting in ignorance in my mind. Share this post Link to post Share on other sites
