★ TouchArcade needs your help. Click here to support us on Patreon.

How Many Beta Testers Do You Need? What's Ur Experience?

10-22-2009, 04:37 PM
#1
Joined: Aug 2009
Location: California
Posts: 28
How Many Beta Testers Do You Need? What's Ur Experience?

Alright so my team and I are getting ready to launch a playable version of our first iPhone game, Q Racer (I'd call it a casual learning game for adults...I can't stand how most of the "educational" games out there seem to be made for children).

Anyway, we have some people signed up to help us test the game, but I was wondering how you other developers handle the testing phase of an app/game.

Specifically, I was wondering...
  • How many testers is enough to get good feedback?
  • What's your experience been with testing?
  • Do you guys find testing helpful?
  • Any other tips or advice on beta testing a new game?

Would love to hear your experiences on this...thanks in advance!

Hug a Panda: New School Edutainment
Game in Dev: Q Racer - More You Know, Farther You Go

Stalk Us (Facebook) | Follow Us (Twitter) | Grock Us (Blog)
10-22-2009, 05:14 PM
#2
Joined: Oct 2009
Location: Dunedin, NZ
Posts: 22
In terms of numbers, you'll get diminishing returns on adding more testers. Say each tester finds on average 10 bugs / issues. With one tester, you've found 10 bugs. With two testers it may be, say, 15 bugs due to overlaps in what they find. Three testers may find a total of 18 bugs and so on.

In other words, small groups can be very effective. I'd recommend a small group of, say, 5 testers to start with. Get their feedback, make adjustments and then release an updated test version to a new set of five testers, plus the original group of testers. repeat until you're happy.

This way you get continuous feedback from fresh users and those who can rate the value of the changes.

10-22-2009, 05:20 PM
#3
Joined: Sep 2009
Location: THE MATRIX
Posts: 611
One,two,three, (Crunch), the world may never know!)

Quote:
Originally Posted by gunxsword View Post
If you check this guy's post records you see:
1. Has interests in RPG games in general and: 2. Trolls iPads & PSP 3. Abusive use of the word 'dumb' and its variants
10-22-2009, 06:28 PM
#4
Joined: Sep 2009
Location: UK / Toronto
Posts: 602
Quote:
Originally Posted by GeorgeSealy View Post
In terms of numbers, you'll get diminishing returns on adding more testers. Say each tester finds on average 10 bugs / issues. With one tester, you've found 10 bugs. With two testers it may be, say, 15 bugs due to overlaps in what they find. Three testers may find a total of 18 bugs and so on.

In other words, small groups can be very effective. I'd recommend a small group of, say, 5 testers to start with. Get their feedback, make adjustments and then release an updated test version to a new set of five testers, plus the original group of testers. repeat until you're happy.

This way you get continuous feedback from fresh users and those who can rate the value of the changes.
I think that's really good advice actually.
10-22-2009, 06:31 PM
#5
Joined: Aug 2009
Location: California
Posts: 28
Quote:
Originally Posted by GeorgeSealy View Post
In terms of numbers, you'll get diminishing returns on adding more testers. Say each tester finds on average 10 bugs / issues. With one tester, you've found 10 bugs. With two testers it may be, say, 15 bugs due to overlaps in what they find. Three testers may find a total of 18 bugs and so on.

In other words, small groups can be very effective. I'd recommend a small group of, say, 5 testers to start with. Get their feedback, make adjustments and then release an updated test version to a new set of five testers, plus the original group of testers. repeat until you're happy.

This way you get continuous feedback from fresh users and those who can rate the value of the changes.
Thanks for the advice. This is great. I know some people have told me to subscribe to the "more is better" line of testing philosophy and while tempting, your logic just makes much more sense.

How many rounds of testing do most people go through? I'm just wondering when is the right time to pull the plug on testing and launch the game. It seems like theoretically you could keep going since I feel like there's always something that could be improved or tweaked.

Hug a Panda: New School Edutainment
Game in Dev: Q Racer - More You Know, Farther You Go

Stalk Us (Facebook) | Follow Us (Twitter) | Grock Us (Blog)
10-22-2009, 06:47 PM
#6
Joined: Oct 2009
Location: Dunedin, NZ
Posts: 22
Glad to be of help. I'm just regurgitating other people's good advice. If you haven't yet come across Joel on Software, then it's worth having a look at some of his older articles (see the 'reading lists' in the sidebar).

As for how many rounds of testing, I think this is where more is better. In an ideal world, iterate until people run out of reasonable complaints In reality, it's purely a judgement call.
11-03-2009, 12:46 PM
#7
For my new game I got 5 beta testers. 2 of them never filed their report at the end. I attached an excel spreadsheet and told everyone to fill in what they wanted. That way they wouldn't be overwhelmed.

The 3 that submitted feedback were very useful. They gave me some feedback confirming things I knew needed to be fixed, as well as suggested new ideas.

Indie Game Developer
Homepage: mattripston.com
Available now: Block Busting Planet Panic Commerical
Follow me on Twitter
11-03-2009, 01:25 PM
#8
We didn't have any official Beta Testers at all. Maybe it shows, maybe it doesn't, but our game has been regarded as 'very polished'.

We had a 2nd Gen iPod, iPhone 3G and iPhone 3GS at our disposal.

My girlfriend is probably the closest thing we had to a beta tester, and gave me real insight into the mind of a gameplayer. Her initial attempts to play the game were very short erratic strokes, which were ineffective for Flickitty. Her questions also led to the simple in-game instructions. They are so simple that the game has a big presence outside the US, despite not being localized (English only).

The Engine Designer used his kids for beta testers. If you want REAL feedback, use a kid. They are brutal when it comes to telling the truth and trying to figure things out logically. Of course this won't work if your game is more advanced and complex. But kids sure do know the definition of 'fun'.

Just be wary when it comes to Beta Testers. A lot of people just want a free game, and put no effort into actual testing nor providing feedback. The iPhone and iPods are similar enough in hardware that you don't need a large base of Beta Testers (for Pocket PC we needed hundreds of testers for the different devices).

A ragdoll physics platformer:Flickitty
The artist: randall schleufer
Twitter: @FlickittyiPhone
12-22-2009, 06:23 AM
#9
Joined: Aug 2009
Location: California
Posts: 28
Beta Testing Update

Hey guys,

Just wanted to post an update to my original post on asking how many beta testers and how you guys test your games. Got a bunch of great ideas from the forum members that I wanted to give back by posting my experience with testing.

Ended up with 5 testers all of which I personally knew (one of which was my girlfriend...I know Flickitty mentioned how he used his gf as a tester...wasn't copying him...was just looking for people to test). At this early stage I wanted to make sure I got feedback back so I opted not to go with random strangers (for people who have done that I'm curious to hear what your experience has been like). Among the testers we had a good mix of iPod touches and iPhones (3G and 3GS).

The game was in pretty rough shape (ie we didn't spend any time polishing the game) and calling it a beta was probably a little premature. Alpha was more like it. What I wanted to test was the basic game interaction and whether people thought it was fun or not. The whole thinking being why waste a bunch of time implementing a bunch of features when the core gameplay sucks.

Rather than doing an excel spreadsheet I put together a google document/form (very handy and something I recommend) to put together a quick 20 question survey. All the testers responded so we got some good feedback and thanks to the google document system they were all consolidated onto one spreadsheet.

How was the feedback? All of them did not like the game. Haha. It actually wasn't that surprising cause even before the test even I was not digging our own game as well...so like others have mentioned the testing confirmed a lot of things that my team and I had already suspected. The nice thing though was we uncovered some other things that on our own without testing we probably wouldn't have uncovered.

So now we've delayed our game probably about another 2 months to work on a new interaction/gameplay system which we hope to test again in about a month...trying to keep to the iterate iterate iterate philosophy.

I posted some more details about our 1st game beta testing experience.

Anyway, thanks again for the responses. If you guys have any questions/feedback just give me a holler.

Hug a Panda: New School Edutainment
Game in Dev: Q Racer - More You Know, Farther You Go

Stalk Us (Facebook) | Follow Us (Twitter) | Grock Us (Blog)
12-22-2009, 06:57 AM
#10
We have also used the "Girlfriends and Kids as Testers-Method" which gave us quite good results. Then we advanced to the Beta-Phase were we asked a rather small group of friends and acquaintances for feedback.
Regarding the comments on bugs and general gameplay this was a very good way.
I wonder though if a bigger Beta-tester Group might help in creating some sort of small buzz for your game, as it is anyways so hard to get attention for your product. Has anybody used testing as a Marketing-Tool?

Developer of Ringo and Mobileguide Iceland

Follow us on Twitter