Support our Sponsors:

Go Back   Touch Arcade > Other > Site Feedback and News

Reply
 
Thread Tools Display Modes
  #1  
Old 02-06-2012, 11:23 AM
GreyDawn GreyDawn is offline
Member
 
Join Date: Feb 2009
Posts: 42
Default TA star ratings sytem broken - scores all too high

This is constructive criticism, I'm not simply nagging.

But anyone else find that Touch Arcade's scoring system for game reviews is uninformative and of little meaning/value/distinction because of the base fact that TA ALWAYS gives ratings that are very high.

Almost everything is either 3.5, 4, or 4.5 stars. When you rate 90% of games in this small and high range, there is no distinction between games, and thus the ratings have no meaning/substance.

I mean, I don't think anything lower than 2 stars even exists in TA's rating system. And TA consistently rates games that aren't very good and that they criticize a fair amount in the text of a review as 3.5 or occasionally 3. These are games that deserve 1-2.5 stars.

Saying 'everything's at least pretty good' as the ratings indicate does nothing for a consumer looking to single out a particularly good games. TA seems to always give benefit of the doubt to games and in general seems scared to rate games lower than 3.5.

THE CURVE IS TOO HIGH.

Again, when you rate almost all games as 3.5-4.5, it takes meaning away from the ratings, and highly indicates that TA wants to give as many games as possible inflated ratings (why this reason is, I'm not sure, it could well be not wanting to upset developers or sponsors, being scared to be too critical, generally wanting to support the platform so giving most everything overly complementary ratings, etc.).
Reply With Quote
  #2  
Old 02-06-2012, 11:32 AM
Eli's Avatar
Eli Eli is offline
TouchArcade Janitor
iPhone (Original), iOS 1.x
 
Join Date: Feb 1983
Location: Barcelona
Posts: 1,267,509
Default

We only post about games that are good, or otherwise noteworthy. You don't see any 1 star reviews because assigning a writer a game that is clearly terrible is a waste of their time and a waste of the reader's time. People seem to come to TouchArcade for recommendations of what to download, not what to avoid. It's not difficult to find junk on the App Store, you don't need a web site for that.

The result of only reviewing good games is a top-heavy scoring system, you're totally correct.

Personally, I think review scores are completely pointless, and fought very hard to keep them off TouchArcade. Unfortunately, for as much as I tried, those who wanted review scores pushed back just as hard. I really don't understand what it is about people that requires some kind of quantitative metric to compare things, but, Metacritic wouldn't be the juggernaut that it is if it wasn't for a demand for what they offer.

My suggestion is to just read the review and see if it sounds like something you're interested in regardless of how many stars are attached to it. Hell, some of my favorite movies barely break the 6/10 mark on IMDB, but that doesn't stop me from loving them.

Edit: And remember, this is how we describe our star rating system: "Our final scores are not the product of any traditionally objective measures such as graphics or sound, but simply reflect the games we would most recommend to others." A three star game is still a "good" game, at least in my eyes. And really, I've had a ton of fun with even the most flawed games. I think Spell Tower was one of the best puzzle games of 2011, but its interface on the iPhone is clunky as hell. Regardless, I've dumped days into it.

Last edited by Eli; 02-06-2012 at 11:57 AM..
Reply With Quote
  #3  
Old 02-06-2012, 12:50 PM
Brad Nicholson Brad Nicholson is offline
💪 Eli Sucks 💪
 
Join Date: Dec 2010
Posts: 50
Default

Sure, we don't use the full scale. But, like President-In-Chief Eli Hoddap just said, that's because we don't review garbage games. Telling you that a game sucks on a platform clotted with bad games isn't doing anyone any favors.

Giving you a heads up on a good one on the other hand? That's what we're all about. I'd say sit back, read the review, and don't even look at the rating. If someone sounds good, just grab it.

EDIT: Also, I'm not a big fan of people accusing us of inflating stuff. Go read a magazine and get back to me on inflated scoring.

Last edited by Brad Nicholson; 02-06-2012 at 12:54 PM..
Reply With Quote
  #4  
Old 02-08-2012, 05:55 PM
Barcā Barcā is offline
Senior Member
iPhone 4S, iOS 5.x
 
Join Date: Sep 2010
Posts: 438
Default

Quote:
Originally Posted by Brad Nicholson View Post
Sure, we don't use the full scale. But, like President-In-Chief Eli Hoddap just said, that's because we don't review garbage games. Telling you that a game sucks on a platform clotted with bad games isn't doing anyone any favors.

Giving you a heads up on a good one on the other hand? That's what we're all about. I'd say sit back, read the review, and don't even look at the rating. If someone sounds good, just grab it.

EDIT: Also, I'm not a big fan of people accusing us of inflating stuff. Go read a magazine and get back to me on inflated scoring.

Agreed.

Go ahead and read Gameinformer in particular. Then you will see a "Paid Review" and inflated scoring.
Reply With Quote
  #5  
Old 02-10-2012, 08:50 AM
jar0d jar0d is offline
Senior Member
iPhone 5s, iOS 7.x
 
Join Date: Jan 2010
Location: Europe
Posts: 515
Default

Quote:
Originally Posted by Barcā View Post
Agreed.

Go ahead and read Gameinformer in particular. Then you will see a "Paid Review" and inflated scoring.
See. If you're really interested in a game or not sure to try/to buy you check other sites as well (yes, there are other iOS-gaming sites - but not for me ).
Reply With Quote
  #6  
Old 09-21-2012, 03:59 AM
Ares Ares is offline
Member
 
Join Date: Jun 2010
Posts: 39
Default TA Rating System

I tried posting this as a comment to the lastest "So You Just Got an iPhone 5... Let's Get Started" article, but either Disqus won't let me or I'm not allowed to comment on TA anymore (I hope not).

Either way, I think the current rating system needs to be adjusted. My original comment is below.

--

If you don't give out ratings less than 2 stars, you should move to a 4-star rating system or re-weight the rating distribution over 5 stars.

Ratings are only valuable to users when they can be used effectively to convey or compare the quality of games. As it is right now, most of us who frequent TA already know that:

5 stars = Definitely worth checking out
4.5 stars = Very good if you can overlook some flaws
4 stars = An "average" game

By "average" game, I mean average amongst the games on the app store with at least a decent amount of polish, not average in terms of every game on the app store. (There is a lot of unplayable stuff out there, and most games would be 4.8 ~5.0 stars if rated on an absolute scale.)

Anything under 4 stars we already know not to bother with, be it for poor IAP models or clunky controls.

What I'm saying is: positioning TA as a game discovery site is great (it's exactly what many of us use it for) but weighting review ratings higher actually hurts this purpose rather than helps it. We want to know how games compare to other decent games, and it's harder to get this kind of information from a rating system that is skewed by games that even the article admits is not worth bothering with.

When ratings are weighted the way they are, savvy TA readers can tell it's in effect a 3-point system with the appearance of a 10-point system.
Reply With Quote
  #7  
Old 09-21-2012, 04:12 AM
Royce Royce is offline
Senior Member
iPad mini
 
Join Date: Mar 2011
Posts: 3,413
Default

There is a relevant thread with posts from administrators here:
http://forums.toucharcade.com/showthread.php?t=121844

And there has been endless discussion of this topic in many random game threads. Basically it sounds like they only begrudgingly include ratings at all and wish they had never started using them.
Reply With Quote
  #8  
Old 09-21-2012, 07:51 AM
Eli's Avatar
Eli Eli is offline
TouchArcade Janitor
iPhone (Original), iOS 1.x
 
Join Date: Feb 1983
Location: Barcelona
Posts: 1,267,509
Default

Quote:
Originally Posted by Ares View Post
When ratings are weighted the way they are, savvy TA readers can tell it's in effect a 3-point system with the appearance of a 10-point system.
It's actually way better than the console world where acceptable score ranges are 8.5/10 to 10/10. But, yeah, I wish we never had to implement review scores at all. Sort of a necessity with the way the app works. I think what causes my intense review score fatigue is how commenters take review scores about 10,000 times more seriously than we do when we're assigning them.
Reply With Quote
  #9  
Old 09-21-2012, 11:51 AM
JBRUU JBRUU is offline
Senior Member
iPad (4th Gen), iOS 6.x
 
Join Date: May 2012
Location: USA
Posts: 6,565
Default

I think most of the time you're never going to please everybody. Give a 5 star review? A vocal minority will critisize the iap "on principle" or hate on the "super short story". Give a lower score? Everyone will rail on you for hating on a game or the devs (as in dead trigger or shadowgun) or proclaim their love for the game adamantly. Dammed if you do, damned if you don't.

Of course I've never reviewed anything, but that's the impression I usually get.
Reply With Quote
  #10  
Old 09-21-2012, 01:24 PM
Ares Ares is offline
Member
 
Join Date: Jun 2010
Posts: 39
Default

Quote:
Originally Posted by Hodapp View Post
It's actually way better than the console world where acceptable score ranges are 8.5/10 to 10/10. But, yeah, I wish we never had to implement review scores at all. Sort of a necessity with the way the app works. I think what causes my intense review score fatigue is how commenters take review scores about 10,000 times more seriously than we do when we're assigning them.
I'm not familiar with all the console game ratings systems, but I frequent GiantBomb.com and they have an excellent 5-point rating system with no half-points. But even 8.5 to 10 (effectively a 4-point scale) is more indicative than a scale of 4 to 5 stars (effectively a 3-point scale).

I actually would prefer a move back to the old reviews that had no scores. It places an emphasis on the review itself and could lead to better review writing and some interesting discussion.

Your review score is your endorsement for a game/product. I'm not sure you should be taking them 10,000 times less seriously than readers. Developers will quote your review scores in their app descriptions, just like restaurants will quote Zagat scores and ratings. The scores should always be something you stand by. If not, just don't implement them.
Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Copyright 2012, TouchArcade.com, LLC.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Copyright ©2008 - 2011, TouchArcade.com. Privacy Policy / DMCA Copyright Agent