How Many Beta Testers Do You Need? What's Ur Experience?

Discussion in 'Public Game Developers Forum' started by Hug_a_Panda_Ben, Oct 22, 2009.

  1. Hug_a_Panda_Ben

    Hug_a_Panda_Ben Active Member

    Aug 11, 2009
    28
    0
    0
    Janitor
    California
    Alright so my team and I are getting ready to launch a playable version of our first iPhone game, Q Racer (I'd call it a casual learning game for adults...I can't stand how most of the "educational" games out there seem to be made for children).

    Anyway, we have some people signed up to help us test the game, but I was wondering how you other developers handle the testing phase of an app/game.

    Specifically, I was wondering...
    • How many testers is enough to get good feedback?
    • What's your experience been with testing?
    • Do you guys find testing helpful?
    • Any other tips or advice on beta testing a new game?

    Would love to hear your experiences on this...thanks in advance!
     
  2. In terms of numbers, you'll get diminishing returns on adding more testers. Say each tester finds on average 10 bugs / issues. With one tester, you've found 10 bugs. With two testers it may be, say, 15 bugs due to overlaps in what they find. Three testers may find a total of 18 bugs and so on.

    In other words, small groups can be very effective. I'd recommend a small group of, say, 5 testers to start with. Get their feedback, make adjustments and then release an updated test version to a new set of five testers, plus the original group of testers. repeat until you're happy.

    This way you get continuous feedback from fresh users and those who can rate the value of the changes.
     
  3. Real Racing Fan

    Real Racing Fan Well-Known Member

    Sep 18, 2009
    611
    21
    0
    I wish
    THE MATRIX
    One,two,three, (Crunch), the world may never know!)
     
  4. EssentialParadox

    EssentialParadox Well-Known Member

    Sep 21, 2009
    602
    0
    0
    UK / Toronto
    I think that's really good advice actually.
     
  5. Hug_a_Panda_Ben

    Hug_a_Panda_Ben Active Member

    Aug 11, 2009
    28
    0
    0
    Janitor
    California
    Thanks for the advice. This is great. I know some people have told me to subscribe to the "more is better" line of testing philosophy and while tempting, your logic just makes much more sense.

    How many rounds of testing do most people go through? I'm just wondering when is the right time to pull the plug on testing and launch the game. It seems like theoretically you could keep going since I feel like there's always something that could be improved or tweaked.
     
  6. Glad to be of help. I'm just regurgitating other people's good advice. If you haven't yet come across Joel on Software, then it's worth having a look at some of his older articles (see the 'reading lists' in the sidebar).

    As for how many rounds of testing, I think this is where more is better. In an ideal world, iterate until people run out of reasonable complaints :) In reality, it's purely a judgement call.
     
  7. Warder

    Warder Active Member

    For my new game I got 5 beta testers. 2 of them never filed their report at the end. I attached an excel spreadsheet and told everyone to fill in what they wanted. That way they wouldn't be overwhelmed.

    The 3 that submitted feedback were very useful. They gave me some feedback confirming things I knew needed to be fixed, as well as suggested new ideas.
     
  8. Flickitty

    Flickitty Well-Known Member

    Oct 14, 2009
    761
    1
    0
    iPhone Dev
    We didn't have any official Beta Testers at all. Maybe it shows, maybe it doesn't, but our game has been regarded as 'very polished'.

    We had a 2nd Gen iPod, iPhone 3G and iPhone 3GS at our disposal.

    My girlfriend is probably the closest thing we had to a beta tester, and gave me real insight into the mind of a gameplayer. Her initial attempts to play the game were very short erratic strokes, which were ineffective for Flickitty. Her questions also led to the simple in-game instructions. They are so simple that the game has a big presence outside the US, despite not being localized (English only).

    The Engine Designer used his kids for beta testers. If you want REAL feedback, use a kid. They are brutal when it comes to telling the truth and trying to figure things out logically. Of course this won't work if your game is more advanced and complex. But kids sure do know the definition of 'fun'.

    Just be wary when it comes to Beta Testers. A lot of people just want a free game, and put no effort into actual testing nor providing feedback. The iPhone and iPods are similar enough in hardware that you don't need a large base of Beta Testers (for Pocket PC we needed hundreds of testers for the different devices).
     
  9. Hug_a_Panda_Ben

    Hug_a_Panda_Ben Active Member

    Aug 11, 2009
    28
    0
    0
    Janitor
    California
    Beta Testing Update

    Hey guys,

    Just wanted to post an update to my original post on asking how many beta testers and how you guys test your games. Got a bunch of great ideas from the forum members that I wanted to give back by posting my experience with testing.

    Ended up with 5 testers all of which I personally knew (one of which was my girlfriend...I know Flickitty mentioned how he used his gf as a tester...wasn't copying him...was just looking for people to test). At this early stage I wanted to make sure I got feedback back so I opted not to go with random strangers (for people who have done that I'm curious to hear what your experience has been like). Among the testers we had a good mix of iPod touches and iPhones (3G and 3GS).

    The game was in pretty rough shape (ie we didn't spend any time polishing the game) and calling it a beta was probably a little premature. Alpha was more like it. What I wanted to test was the basic game interaction and whether people thought it was fun or not. The whole thinking being why waste a bunch of time implementing a bunch of features when the core gameplay sucks.

    Rather than doing an excel spreadsheet I put together a google document/form (very handy and something I recommend) to put together a quick 20 question survey. All the testers responded so we got some good feedback and thanks to the google document system they were all consolidated onto one spreadsheet.

    How was the feedback? All of them did not like the game. Haha. It actually wasn't that surprising cause even before the test even I was not digging our own game as well...so like others have mentioned the testing confirmed a lot of things that my team and I had already suspected. The nice thing though was we uncovered some other things that on our own without testing we probably wouldn't have uncovered.

    So now we've delayed our game probably about another 2 months to work on a new interaction/gameplay system which we hope to test again in about a month...trying to keep to the iterate iterate iterate philosophy.

    I posted some more details about our 1st game beta testing experience.

    Anyway, thanks again for the responses. If you guys have any questions/feedback just give me a holler.
     
  10. YmirMobile

    YmirMobile Active Member

    We have also used the "Girlfriends and Kids as Testers-Method" which gave us quite good results. Then we advanced to the Beta-Phase were we asked a rather small group of friends and acquaintances for feedback.
    Regarding the comments on bugs and general gameplay this was a very good way.
    I wonder though if a bigger Beta-tester Group might help in creating some sort of small buzz for your game, as it is anyways so hard to get attention for your product. Has anybody used testing as a Marketing-Tool?
     
  11. GlennX

    GlennX Well-Known Member

    May 10, 2009
    761
    0
    0
    UK
    I asked all of my iPhone/iPod owning Facebook friends and 12 or so signed up (mostly old colleagues), only three of them gave feedback at all though to be fair, I sent them a version too early. I had a little testing from Testology (a company run by a friend who ran the test departments at Bullfrog and Lionhead) which was valuable for eliminating bugs but possibly a little too early.

    I also asked for Beta testers on TA at the end of the project. 30 people signed up, over half of them actually played the game and maybe 8 gave useful feedback with a few of them being real stars. Almost all 30 contacted me for promo codes when the game was released though, which maybe gives some insight to the real reason people sign up for beta testing.

    Some of the testing was done in a social group on TA which, with hindsight, was a bad idea. I'd also set up a Google group and I ended up having to post versions and take feedback in two places because the guy who set up the social group seemed to disappear.

    A smaller group of more interestd testers would have worked better. Maybe asking for testers in this forum rather than upcoming games would have been a good idea.

    Next time I will probably invite the best of those TA testers and a few others i've met on TA since Ground Effect was released and insist on communicating via an email group as that would let me include a few non-TA people.
     
  12. EssentialParadox

    EssentialParadox Well-Known Member

    Sep 21, 2009
    602
    0
    0
    UK / Toronto
    Thanks for posting about your experience.

    I got the jist of what your game is about, and your aims on how to improve it. Are any of you reading up about game design? You talk about implementing "incentives and rewards" to keep people interested in the game, but I'm afraid that might be the wrong direction. There are many, many articles out there about what makes an addictive game. If you're not reading them, I would very much recommend searching gamasutra.com and go from there.

    I think your encounter also points out an important characteristic of beta testers; If you choose seasoned gamers to play your game I believe they'll tend to have better suggestions and ideas about why the game fails in some areas. However with a more casual player, though they may be able to tell you that it isn't fun, they may not be experienced enough to understand why.
    I don't have much to base this assumption off, but after seeing a couple of your feedback excerpts and the comments made in a beta test group I was recently part of, I noticed the older and more hardcore players gave some very good ideas. So maybe it's something to consider when choosing beta testers.
     
  13. jak56

    jak56 Well-Known Member

    well a lot of devs get teams of 20 for beta testing
     
  14. Hug_a_Panda_Ben

    Hug_a_Panda_Ben Active Member

    Aug 11, 2009
    28
    0
    0
    Janitor
    California
    Yeah, I have read quite a few articles on Gamasutra with regard to game design. I really like reading the case studies on there as well as the marketing articles they sometimes have. It's a great resource.

    If I gave you the impression that I thought "incentives and rewards" was the key to making the game better I probably misspoke. If anything I didn't want to use those types of "incentives" (like achievements and the like) until after we had our basic gameplay (like moving your character, speed, general rules of the game, etc...). So the point of the testing was to see if 1) see if on a longshot some users actually enjoyed the game 2) identify any problems or angles we might not have anticipated.

    I've noticed it's difficult after working on a game to look at your own game objectively. This is where a set of fresh eyes (via a tester) can offer some perspective on your product. At least this helped for us.

    I think finding seasoned gamers isn't a bad idea...but I do think a mix of testers if always ideal. I think seasoned gamers (like myself) typically have higher expectations since they've played so many games. They may have good ideas for improving your game (that casual gamers might not have), but depending on the market your targeting it might make your game overly complicated. I feel like especially on the iPhone simple seems to do well (fart app) vs. complicated (Rolando).

    While having good beta testers is always plus I think more important is for the developers to get good at filtering what's important, not as important, and irrelevant when it comes to looking at feedback.
     
  15. Hug_a_Panda_Ben

    Hug_a_Panda_Ben Active Member

    Aug 11, 2009
    28
    0
    0
    Janitor
    California
    Thanks for sharing your experience on testing using TA. What you reported I kind of suspected (ie people all want to test so they can play a free game before others) which is what made me a little wary of finding testers on TA or other game related forums.

    I haven't played around with the Social Groups on TA...you said it was a bad idea in hindsight. Exactly how does a social group work and why do you think it was bad?
     
  16. GlennX

    GlennX Well-Known Member

    May 10, 2009
    761
    0
    0
    UK
    The social groups are kind of like a single threaded closed forum. They are cool but the beta tester who set it up dissapeared so we couldn't add the last few testers.

    It would have been better if I'd set it up myself I suppose, at least I could have added members.

    The problem was that I had also set up a mailing list on Google Groups. I'd expected everyone to be happy to do it this way, it would certainly be my preference to pipe the mails into a folder and read them in a nice threaded email client such as GMail but as I added people I watched them change their settings to no email and moan about having to log on to Google Groups. It seems TA users like to do as much as possible here without logging off! Maybe some people aren't as comfortable as I am with mailing lists, I'm a member of at least 10, all piped into nicely threaded folders/labels in GMail.

    The Google Group would have also let me include non TA members.

    The real problem was that I had to maintain both, every time I announced a new release or asked for feedback I had to do it twice. I also had testers in each place who couldn't see responses from each other.

    With hindsight I wish I'd either insisted people use the Google Group or maybe, gone with the social group from day one and never set up the list.

    I'd love to know what regular beta testers prefer before next time.
     
  17. da shiz wiz 19

    da shiz wiz 19 Well-Known Member

    Sep 24, 2009
    7,318
    9
    0
    I'm not a game dev. but I have beta tested many times.
    I was one of those people who didnt like the google wave groups and such. I prefered using a TA group to get all my thoughts said. But I do see what Glennx is saying. A lot of people just sign up and dont do anything with the beta. There could be a group of 40 and only about 10 would check back regularly.
    If you want a successful beta I'd recommend you contact the bigger members of this site. Starball did that and they had paragraph long previews for it. Also if the game is long maybe set goals for the testers? Get to level 3, purchase all these units, test the sensitivity, etc. while you are fine tuning the beta. Then when you finish fixing up last time's issues send out another beta with a new set of questions.

    Just a thought.
     
  18. GlennX

    GlennX Well-Known Member

    May 10, 2009
    761
    0
    0
    UK
    One big downside to the TA group is that people seem able to forget about it completely. You have to remember to go check it. Of course this is also the downside to a mailing list. People don't want to see 20+ mails a day littering their inbox. This isn't really an issue in an email client that can send mails to folders and store them in threads (I've only ever really used outlook and gmail which both do a great job of this) but I should have realised that many of our testers don't get the volume of mail I do and weren't set up like this or were maybe using email clients that simply don't do it well. Teaching people how to use their email clients was kind of beyond the scope of the testing!

    Maybe the social groups are a better option but I really don't like the lack of threading.
     
  19. D-Cypher

    D-Cypher Well-Known Member

    How about just creating a google group? That seems to work well for me. It's basically just a mass email with archiving.

    [rant]
    I personally find Wave annoying and worthless. I'm sure somebody smarter than me will figure out how to make it palatable but right now it's a mess if you get multiple discussions going with various groups of people. Information gets lost in the chaos. And currently it is pretty darn slow. The developer's Waves are also full of annoying bots. Spammers are going to abuse the heck out of it.
    [/rant]
     
  20. EssentialParadox

    EssentialParadox Well-Known Member

    Sep 21, 2009
    602
    0
    0
    UK / Toronto
    Oh okay, great!

    I do agree with you as well in regard to casual beta testers. When it's a casual game I think it's a good idea to have a balance between both experienced gamers and your target audience. You can counterbalance the improvements with the casual tester feedback, as they are the ones who need to enjoy the game, but the experienced gamers will be able to troubleshoot problems.
     

Share This Page