TA star ratings sytem broken - scores all too high

Discussion in 'Site Feedback and News' started by GreyDawn, Feb 6, 2012.

  1. GreyDawn

    GreyDawn Active Member

    Feb 6, 2009
    42
    0
    6
    This is constructive criticism, I'm not simply nagging.

    But anyone else find that Touch Arcade's scoring system for game reviews is uninformative and of little meaning/value/distinction because of the base fact that TA ALWAYS gives ratings that are very high.

    Almost everything is either 3.5, 4, or 4.5 stars. When you rate 90% of games in this small and high range, there is no distinction between games, and thus the ratings have no meaning/substance.

    I mean, I don't think anything lower than 2 stars even exists in TA's rating system. And TA consistently rates games that aren't very good and that they criticize a fair amount in the text of a review as 3.5 or occasionally 3. These are games that deserve 1-2.5 stars.

    Saying 'everything's at least pretty good' as the ratings indicate does nothing for a consumer looking to single out a particularly good games. TA seems to always give benefit of the doubt to games and in general seems scared to rate games lower than 3.5.

    THE CURVE IS TOO HIGH.

    Again, when you rate almost all games as 3.5-4.5, it takes meaning away from the ratings, and highly indicates that TA wants to give as many games as possible inflated ratings (why this reason is, I'm not sure, it could well be not wanting to upset developers or sponsors, being scared to be too critical, generally wanting to support the platform so giving most everything overly complementary ratings, etc.).
     
  2. Eli

    Eli ᕕ┌◕ᗜ◕┐ᕗ
    Staff Member Patreon Silver Patreon Gold

    #2 Eli, Feb 6, 2012
    Last edited: Feb 6, 2012
    We only post about games that are good, or otherwise noteworthy. You don't see any 1 star reviews because assigning a writer a game that is clearly terrible is a waste of their time and a waste of the reader's time. People seem to come to TouchArcade for recommendations of what to download, not what to avoid. It's not difficult to find junk on the App Store, you don't need a web site for that.

    The result of only reviewing good games is a top-heavy scoring system, you're totally correct.

    Personally, I think review scores are completely pointless, and fought very hard to keep them off TouchArcade. Unfortunately, for as much as I tried, those who wanted review scores pushed back just as hard. I really don't understand what it is about people that requires some kind of quantitative metric to compare things, but, Metacritic wouldn't be the juggernaut that it is if it wasn't for a demand for what they offer.

    My suggestion is to just read the review and see if it sounds like something you're interested in regardless of how many stars are attached to it. Hell, some of my favorite movies barely break the 6/10 mark on IMDB, but that doesn't stop me from loving them. :)

    Edit: And remember, this is how we describe our star rating system: "Our final scores are not the product of any traditionally objective measures such as graphics or sound, but simply reflect the games we would most recommend to others." A three star game is still a "good" game, at least in my eyes. And really, I've had a ton of fun with even the most flawed games. I think Spell Tower was one of the best puzzle games of 2011, but its interface on the iPhone is clunky as hell. Regardless, I've dumped days into it.
     
  3. Brad Nicholson

    Brad Nicholson 💪 <strong>Eli Sucks</strong> 💪

    Dec 1, 2010
    27
    0
    0
    #3 Brad Nicholson, Feb 6, 2012
    Last edited: Feb 6, 2012
    Sure, we don't use the full scale. But, like President-In-Chief Eli Hoddap just said, that's because we don't review garbage games. Telling you that a game sucks on a platform clotted with bad games isn't doing anyone any favors.

    Giving you a heads up on a good one on the other hand? That's what we're all about. I'd say sit back, read the review, and don't even look at the rating. If someone sounds good, just grab it.

    EDIT: Also, I'm not a big fan of people accusing us of inflating stuff. Go read a magazine and get back to me on inflated scoring.
     
  4. Barcâ

    Barcâ Well-Known Member

    Sep 10, 2010
    438
    0
    0

    Agreed.

    Go ahead and read Gameinformer in particular. Then you will see a "Paid Review" and inflated scoring.
     
  5. jar0d

    jar0d Well-Known Member

    Jan 4, 2010
    530
    0
    16
    Europe
    See. If you're really interested in a game or not sure to try/to buy you check other sites as well (yes, there are other iOS-gaming sites - but not for me :D;) ).
     
  6. Ares

    Ares Active Member

    Jun 20, 2010
    39
    0
    0
    TA Rating System

    I tried posting this as a comment to the lastest "So You Just Got an iPhone 5... Let's Get Started" article, but either Disqus won't let me or I'm not allowed to comment on TA anymore (I hope not).

    Either way, I think the current rating system needs to be adjusted. My original comment is below.

    --

    If you don't give out ratings less than 2 stars, you should move to a 4-star rating system or re-weight the rating distribution over 5 stars.

    Ratings are only valuable to users when they can be used effectively to convey or compare the quality of games. As it is right now, most of us who frequent TA already know that:

    5 stars = Definitely worth checking out
    4.5 stars = Very good if you can overlook some flaws
    4 stars = An "average" game

    By "average" game, I mean average amongst the games on the app store with at least a decent amount of polish, not average in terms of every game on the app store. (There is a lot of unplayable stuff out there, and most games would be 4.8 ~5.0 stars if rated on an absolute scale.)

    Anything under 4 stars we already know not to bother with, be it for poor IAP models or clunky controls.

    What I'm saying is: positioning TA as a game discovery site is great (it's exactly what many of us use it for) but weighting review ratings higher actually hurts this purpose rather than helps it. We want to know how games compare to other decent games, and it's harder to get this kind of information from a rating system that is skewed by games that even the article admits is not worth bothering with.

    When ratings are weighted the way they are, savvy TA readers can tell it's in effect a 3-point system with the appearance of a 10-point system.
     
  7. Royce

    Royce Well-Known Member

    Mar 22, 2011
    4,239
    179
    63
    There is a relevant thread with posts from administrators here:
    http://forums.toucharcade.com/showthread.php?t=121844

    And there has been endless discussion of this topic in many random game threads. Basically it sounds like they only begrudgingly include ratings at all and wish they had never started using them.
     
  8. Eli

    Eli ᕕ┌◕ᗜ◕┐ᕗ
    Staff Member Patreon Silver Patreon Gold

    It's actually way better than the console world where acceptable score ranges are 8.5/10 to 10/10. But, yeah, I wish we never had to implement review scores at all. Sort of a necessity with the way the app works. I think what causes my intense review score fatigue is how commenters take review scores about 10,000 times more seriously than we do when we're assigning them.
     
  9. JBRUU

    JBRUU Well-Known Member

    May 9, 2012
    6,758
    0
    36
    USA
    I think most of the time you're never going to please everybody. Give a 5 star review? A vocal minority will critisize the iap "on principle" or hate on the "super short story". Give a lower score? Everyone will rail on you for hating on a game or the devs (as in dead trigger or shadowgun) or proclaim their love for the game adamantly. Dammed if you do, damned if you don't.

    Of course I've never reviewed anything, but that's the impression I usually get.
     
  10. Ares

    Ares Active Member

    Jun 20, 2010
    39
    0
    0
    I'm not familiar with all the console game ratings systems, but I frequent GiantBomb.com and they have an excellent 5-point rating system with no half-points. But even 8.5 to 10 (effectively a 4-point scale) is more indicative than a scale of 4 to 5 stars (effectively a 3-point scale).

    I actually would prefer a move back to the old reviews that had no scores. It places an emphasis on the review itself and could lead to better review writing and some interesting discussion.

    Your review score is your endorsement for a game/product. I'm not sure you should be taking them 10,000 times less seriously than readers. Developers will quote your review scores in their app descriptions, just like restaurants will quote Zagat scores and ratings. The scores should always be something you stand by. If not, just don't implement them.
     
  11. Eli

    Eli ᕕ┌◕ᗜ◕┐ᕗ
    Staff Member Patreon Silver Patreon Gold

    How we write our reviews hasn't changed at all.
     
  12. Boardumb

    Boardumb Administrator
    Staff Member Patreon Silver Patreon Gold

    Apr 14, 2009
    8,766
    799
    113
    THE BOSS
    Sacramento, CA
    You're missing out on a lot of good games if you only look at 4 star and up.
     
  13. Ares

    Ares Active Member

    Jun 20, 2010
    39
    0
    0
    I said "could". But that isn't even the main point. I won't reiterate as I think my previous posts were pretty clear.
     
  14. Ares

    Ares Active Member

    Jun 20, 2010
    39
    0
    0
    I don't have the time to play lots of games these days. I still have a queue of really great games sitting around that I haven't gotten to. But I do enjoy "keeping up" with gaming news, regardless.

    Generally if a game is getting 3.5 stars or lower on TA, I know I won't be missing much. Definitely not a GREAT game.
     
  15. Fastbridge

    Fastbridge Well-Known Member

    Aug 20, 2009
    2,196
    0
    0
    Service advisor for Honda
    NYC, New York. USA
    TA needs a new rating system

    I'm starting this thread out of concern for the current rating system for reviews. Where I don't think it's totally broken, I do sometimes scratch my head when they give a game a perfect score. I truly feel a 5 star rating should be giving maybe a couple times a year. If that. Max Payne mobile was by no means a perfect game and that got a perfect score. That's just one example. I think Eli and the team really need to come up with something a little better. One suggestion would be maybe a 1to10 rating system. This will give you more room to rate a game and you could save the 10 score for true gems that deserve it. Or maybe even go to half stars as well which will give you more room to work as well. I've been a long time supporter of this site, but truly feel that some of these ratings are not warranted, and with the amount of people you have coming to this site for a accurate rating I think we deserve better. I know I'm not the only person that feels this way and I hope people chime in to express there concerns if there are any. If you go to the TA APP and look at all time 5 star reviews, you'll some that really don't belong there. Please TA, save five star scores for truly amazing games. A five star review should not be that easy to get, especially when in the review there ate complaints about the game. Thanks for listening guys! Btw the TA app is awesome.
     
  16. LOLavi

    LOLavi Well-Known Member

    May 23, 2011
    8,364
    0
    0
    iPhone gamer
    Finland
    The home page reviews do have half stars, but agreed, I like 1 to 10 more, I find them easier to uderstand.
     
  17. TheFrost

    TheFrost Well-Known Member

    Nov 18, 2010
    5,772
    0
    0
    A 1-10 scoring system would give a wider spectrum for scores, making the more understandable and, well, more variety

    Actually, I never even bother checking games with less than 4 stars reviews
     
  18. MidianGTX

    MidianGTX Well-Known Member

    Jun 16, 2009
    3,738
    10
    38
    Plenty of websites use 1-10 and they're constantly moaned at for only ever awarding 7s and above.
     
  19. Eli

    Eli ᕕ┌◕ᗜ◕┐ᕗ
    Staff Member Patreon Silver Patreon Gold

    Let's adopt a rating system where everything is 9.X where X is the actual rating that way everyone is happy. :)
     
  20. Ares

    Ares Active Member

    Jun 20, 2010
    39
    0
    0
    Who are you responding to? The consensus in this thread is that, on average, scores should be lower, not higher.
     

Share This Page