Beta-Testing Multiplayer Games

[MarkFromBitmenStudios]

Hi,

I feel the need to go out of the lurker mode I've been in for months to ask some advice on a topic that is really bugging me. Basically, I'm wondering if any of you fellow devs have any experience with running a beta test for multiplayer games.

I mean, it can be hard to find beta testers for offline games that really stick with your game for more than just some initial time and who also provide valuable and honest feedback, but for multiplayer games, there's the additional challenge that people can't just test whenever they have the time to do so but you need to socialize the testers and coordinate "game nights" where they get the chance to play each other.

Since the number of beta testers is limited by Apple's UUID restrictions, you can't hope to have people just press "Automatch" and get an opponent during beta because chances are nobody else might be playing at that time.

Obviously, the game also has a single-player component but that one is not really in my beta test focus right now.

Since everybody has limited time and testing needs coordination and committment, and we live in different timezones, I really fear that this can be a huge challenge.

I'd be really happy to share some thoughts with others who have been in the same situation on how to cope with that. I was also wondering whether it is possible to get TA regulars for beta testing who maybe happen to already know each other from the forums.

In particular, I've seen that TA has social groups for various topics, including a beta tester alliance. Is there some socialization among beta testers already? All this would help tremendously.

I'm also counting a bit on GameCenter to connect testers so they can send invites but I dunno if that will be enough.

To make things worse, the game I'm working on is an RTS game and this genre is well-known to require an extensive balance test in addition to traditional bug finding or network code testing. That means, if I could have it my way, I'd do some 2-3 months of beta but I'm having my doubts that I can get ppl to stick with it over several balance patches and it's also hard to point out imbalances in the first place (which requires at least some basic RTS history, I guess).

Anyway, any input from your side is more than welcome, hands-on experience with multiplayer betas or even some suggestions.

Thanks,
Mark.

TLDR: plz share your experiences on multiplayer beta testing esp. w.r.t. getting testers to play multiplayer games against each other over a prolonged period of time.

[jeremy.provost]

We did extensive multiplayer beta testing for our first game, Cricket Words. I think we were pretty successful in getting people to play. There was no secret sauce, though. It was mainly just good communication. We were fortunate that most of our test players already knew each other, but even for those that didn't we would take the lead in introducing players to each other.

Do you have 2 player only? Or do you have more than 2 players? If you have more than 2 players: we had success in introducing people by just adding strangers to the same match. From there people could friend each other if they wanted to keep playing.

We weren't successful in getting all of our beta testers to keep playing, but we had a good pool of about 20 people who played pretty regularly during development.

A final note: I don't know what stage of development you are in, but I would hold off on involving a large group of beta testers until you have a solid foundation for your game. Don't want to abuse their goodwill.

_____________________________
Cricket Words by Think Tap Work

[MarkFromBitmenStudios]

Thanks jeremy for your valuable feedback, much appreciated. I've a couple of additional questions though:

• Communication Channels: what kind of communication channels worked best for you to build up a strong communication between you and the testers and - equally important - among the testers? Mail, Forums, Game Center, Skype, ...?
• Where did you recruit your testers?
• Did you collect statistics about the game sessions?
• How long did you manage to beta?
• I've tried out your game and it looks like it uses a turn-based mode where players don't have to be around at exactly the same time. Is there some real-time mode too? My game is a real-time multiplayer game, so you can't submit your turn and return to the game as soon as the other has caught up. All players really need to be playing the game simultanously.

[Blackharon]

When we were testing Go Native! (Not exactly as complicated as most RTS) we ended up just programming bots.

Funny thing is, after releasing we didn't have a large enough pool of players to keep the game running, so we used the bots to make the game more fun for those who were playing.

[jeremy.provost]

Communication Channels: what kind of communication channels worked best for you to build up a strong communication between you and the testers and - equally important - among the testers? Mail, Forums, Game Center, Skype, ...?

Honestly just through email and iMessages. For our game it wasn't super-important that the gamers be in touch with each other. Only that they could play with others and be in touch with us to provide feedback.

We used testflightapp.com, as many have, for our distributions. Users can report issues there, but we found that most people would just contact us directly. Always be quick to respond and appreciative. Remember that they're doing you a huge favor.

Where did you recruit your testers?

Almost all of or testers were people that we knew personally. This certainly made it easier. A few of our testers requested that we add their friends to the list. We did so. While we didn't get much direct feedback from these individuals it was still great because it increased the pool of players.

Did you collect statistics about the game sessions?

Yes, we collected information using Flurry. This helped us see how quickly people were able to finish matches, how many games were being played, which difficulty level people were choosing. Things like that. None of it was tied to specific players, just general knowledge about the entire pool of testers.

How long did you manage to beta?

I believe we beta tested for close to 2 months. Maybe a little less. Something to note there, don't overwhelm people with too many updates too frequently. We always made sure that we had substantial updates to the game before sending another build to our testers. They can easily get update fatigue.

I've tried out your game and it looks like it uses a turn-based mode where players don't have to be around at exactly the same time. Is there some real-time mode too? My game is a real-time multiplayer game, so you can't submit your turn and return to the game as soon as the other has caught up. All players really need to be playing the game simultanously.

This last bit is where I can't help you. You're right, we are talking about two completely different types of multiplayer gaming. Undoubtedly, simultaneous multiplayer is much harder to beta test than turn-based gaming. Once you figure out your strategy please do share.
__________________________
Cricket Words by Think Tap Work

[jbsib]

Simply allow and infinite number of people join the beta at, and then when the game is released, make the beta version stop working if you are worried that people will not need to buy the proper version. Like sky gnomes!

[MarkFromBitmenStudios]

I've also thought about that, an open beta which means basically releasing the beta for free on the AppStore (with only multiplayer and a tutorial), probably in one country only to make it manageable. Then, at the end of the beta, shut it down by pulling it off game center which will make it impossible to play any multiplayer games anymore.

A bit trickier to update (have to go through Apple approval each time instead of just releasing on testflightapp) but pure balance changes are pulled from my servers anyway.

The game would be clearly labelled beta though and I'm not sure Apple allows this.

Hmm, what country to pick...

Anybody been doing an open beta already?

[Foursaken_Media]

You can allow unlimited testers if you have an Enterprise Account. We used Testflight to host our beta with 200+ people on Sky Gnomes. Testflight also allows you to send out a link which people use to join, so you don't have to manually do anything. You also have the power to accept or decline anyone joining, just in case...

Also with an enterprise build, your app will automatically be unusable after 12 months, but you can alternatively delete the certificate anytime to make it instantly unplayable. Its pretty cool actually.

You do have to drop some cash to sign up for the Enterprise account though...

[Jorge.]

Hi,

I will be doing Beta testing for my app also, and I am planning to try TestFlight, but I also found the "Beta Family" http://thebetafamily.com

I wonder if anybody has tried it and if you would recommend it, or if I should just stick to TestFlight.

Thanks,

Jorge

[cmo]

Hey Mark -- we have been beta testing a multiplayer-only app for a while. Here's what I've learned:

• Definitely choose a country for open beta, but be wary of how you pick it. You need high volume, but you don't want to focus on a country like China unless you're testing specifically for Asian users. Behaviors do vary between countries. That's why all the big companies tend to test in Canada. We opened in a bunch of countries excluding the US and got some decent organic traffic over time.
• Spend some time building up a small userbase in your primary test country (for us it's the UK). If you are able to spend some money, Tapjoy is cheap to use before you hit large volumes, and has good penetration in a lot of international markets.
• Assuming you go to all the effort of doing the above, don't screw up your analytics. Make sure you're tracking all your events and have the right cohorts, funnels etc set up so that you can see any improvement or decline in the numbers when you change a feature. Do some reading and research if you're not sure about how to do these things. It's actually not too difficult.
• Don't mark anything in your test app as Beta, test, "Coming Soon", etc. Apple will (inconsistently) reject apps for appearing incomplete. We've also been rejected for leaving non-public testing APIs in the code. Oops!
• Don't ignore 1 on 1 usability testing or small groups that you can watch in person! We still did usability studies to check the effectiveness of our UI, tutorials, etc. and got a lot of milage from them.
• For whatever reason, The Beta Family was a flop for us. But I've read elsewhere that it has been useful for some people.

All that said I'm still not 100% happy with how we go about testing new features, because it's difficult for us to test any prototype / in-development feature given Apple's review regime. If I could do it all over again I'd dump Testflight early on and use HockeyApp to slowly grow a group of trusted users. Then I'd switch between testing prototypes with the core group, and then balancing and improving released features with the production build live on the App Store.

Sadly all of this takes a lot of time and effort and can't necessarily be balanced perfectly with budget / timeline etc.