The indie elephant in the room, Minecraft, has changed the perception of what a "beta" truly entails in the mind of the average gamer. No longer do people hearing the word "beta" expect the buggy, frustrating, horribly broken, and not-fun-at-all experience they should. Instead it is now viewed as a supposedly incomplete early development milestone with a high degree of playability and quite a bit of polish. "Who does the actual beta testing of these milestones, then?" you ask. Most probably not the people coming to an indie-game dev forum stating that they are qualified beta testers because they've "beta-tested Minecraft."
Let us get some definitions clear-cut first. A beta version is a feature-complete, but not yet bug-fixed, nor polished version of a given working package in development. That might be a single little feature in a game, or a full software suite. As the example above already has shown, this definition is far from being universal, but herein thou shalt see it as the undeniable, utmost truth. With this definition, Notch's brilliant creation was released in no more (and no less) than well-polished milestones, probably internally alpha and beta tested prior to the release to the self-proclaimed "beta testers."
How have these recent changes in mentality affected the beta testing requirements of future game-related products? The answer to that question very much depends on target audience, but in general, the freely available workhorses out there drift increasingly towards the "I just play around" category.
While Blizzard Entertainment has found a perfect use for these people, they could become a potentially harmful ingredient in any small- or medium-scale testing team. Overestimating how much (quality) work will be done per unit time and tester quickly leads to frustration and delays, causing, in turn, loss of trust and ultimately, sales.
Many indie teams seem to have solved this issue by not planning much at all and/or testing everything by themselves -- a rather workable, but far from optimal, solution.
This blissful ignorance comes as no big surprise - there is a lot of work involved in setting up proper testing infrastructure, finding the right people to do the testing, and communicating the ever-changing aims of the tests. This leads us to the gist of this write-up: POMMS-based beta testing.
The acronym stands for Project-Oriented Modular Motivational System and was incubated by the author as a tool to bring structure and clarity into the chaos that was called beta testing at our little indie game dev studio Camshaft Software, currently working on Automation: The Car Company Tycoon Game. This system implements modern game design elements into the testing process itself, a concept usually connected with the buzzword "gamification." The POMMS-based approach to testing attacks some of the biggest challenges in beta testing: the ability to plan ahead, tester coordination, tester motivation, testing focus, tester flexibility and individuality.
POMMS: What It Is, and How it Works in Testing
The system itself is quickly explained: Well-defined subprojects are outlined, and a specific amount of "points" is awarded for every task a tester completes towards realizing the subproject goals. This score is the so-called tester power level, and when it's over 9000, your subprojects are either too long or the amount of work per point too small.
Depending on the contents of a subproject, a meaningful subproject could be anything between 4-16 weeks long, with a single point being the equivalent of around 15 minutes of work. Within any given subproject, each tester has to reach a certain minimum score to continue participating in beta testing. The highest scoring testers are awarded stars that are cumulative across subprojects. It is up to the developers to decide what the rewards for attaining these stars are. For example, they could give testers special privileges, a place in the software's credits, provide them with special forum statuses, etc.
If a tester achieved more than the minimum score in a subproject, but didn't reach star rankings, the final score is carried over to the next subproject. This carryover does not count towards reaching the next subproject's minimum score, but towards the placement in the star rankings. This rewards testers who continually deliver solid work over several subprojects, as they will grab a star eventually, and by the end of the project, they will not feel like they have been anonymous testing machines after all.