I started out playing MUDs (text-based MMOs) and developed a knack for breaking games. I quickly became a go-to playtester, which I then transitioned into QA work with first SCEA/PlayStation and then third party work for Microsoft Game Studios.

Playing this expansion, and previous expansions, has been fun and exciting, but there is always a jarring sense of removal from the immersion or a frustrating barrier to gameplay when a bug, glitch or issue is encountered. This may be as trivial as a typo, and it may be as far reaching as the gyrocopter issue the Alliance faced at the launch of MoP. I am referring specifically to issues covered by the QA department, versus the design or balance teams.

This issue is hardly unique to WoW. I can’t play GW2 without encountering a crash every few hours due to a glitch involving instancing with parties. Hell, when I played MUDs, I once found a way to use the game’s mail system to physically mail myself to an admin. There are always bugs that go live. However, the QA department’s role is to ensure as few of these happen as possible. Beyond that, their job is also to ensure that the most LIKELY scenarios are all fully tested.

QA teams use a process called test casing to cover these most common potential circumstances. They can’t blindly, blithely and happily prance around the game just checking out what suits their fancy. Sure, in the early stages of a game’s development, most testing is exploratory, but, as it gets closer to release, testing is narrowed down to systematic examination of various elements. Test cases direct QA teams to rigorously check off functionality for a variety of conditions.

For a highly simplified example, a test case for Warsong Gulch might be broken down into various components such as joining the game, winning the game and losing the game. Test teams would then check the various conditions applicable to each category. Joining might include testing joining the game solo, with a party, with the game thumbed down, via random queue, via specific queue, etc. This structured testing ensures that all of the likely scenarios are fully tested for functionality before the game goes live, versus just assuming it will work in a party because it worked solo.

This is not good QA.

I explain all of this so I can make it very clear that video game QA is not a fun and games job. There is a lot of work involved, including time spent planning out the testing before the QA team even sees the game elements they will be working on. This is a serious career and talents beyond just “leet gaming” are required. A good tester should be meticulous, pay close attention to detail and be capable of looking outside the expected norms of gameplay.

Unfortunately, the video game industry suffers from its own history. What was once a cowboy industry started in basements is now one of the most profitable aspects of the entertainment industry; millions of dollars are spent on video games each quarter. Video games showed INCREASED revenue during the recession, while other aspects of entertainment suffered from a decline in sales.

The internal structure of the video game industry, however, does not seem to recognize its own success. Video game QA practices are outdated, following the same model initially developed out of need: toss a bunch of low paid “testers” at the product and hope their enthusiasm for the industry itself balances out all of the downsides inherent in the job. A decade ago, this practice was not only money-savvy, it was necessary. Video games were not a serious industry, so you had to hire what you could and scramble with the resources that gave you. This meant that, inevitably, you ended up with a crunchtime environment as “showstopper” bugs (ie, issues that made the game nigh unplayable) emerged only in the very latest stages of testing.

When you aren’t working with a highly experienced test corps, major issues can lurk undetected until late in the game, and then overtime is requisite to iron out these bugs, as the testers need to regress bugs, that is, test the issues again to see if the bugs have been fixed. Here is a good (if slightly outdated) blog explaining why this is inefficient (and fiscally wasteful):  http://romsteady.blogspot.com/2006/04/testing-hidden-costs-of-testing-at-end.html

This is also not good QA.

Having worked in QA, I know that it’s more than just inexperience that results in these kind of lurking issues. First, many QA departments pick up speed later in the test cycle and then have the bulk of their work staffed by entry level testers, versus retaining a smaller fulltime team of more experienced testers. Many companies claim it saves money (again, see the breakdown in the link above), but it really does seem to be a relic of the earlier days of the industry. Not only do you have to deal with the initial acclimation and training period, this sort of mass hiring (often through a staffing agency) picks up a…wide…caliber of applicants. But why should video game QA teams expect anything else? The industry’s focus on late-cycle testing means that unskilled hires are more desired because they are cheaper…and skilled testers end up migrating outwards to other QA fields. This also means that the workload demand for non-sustainment games (eg games like console games that don’t have ongoing development) fluctuates from a trickle to a flood – ie very little work (sup, unemployment) and then way too much for a small team to handle. In essence, there is very little career mobility in video game testing, and the initial hire process is inclusive enough, often due to to not distinguishing between stellar applicants and mediocre ones.

Here are some numbers to illustrate this (all income is done with location based in Los Angeles, as SoCal is a large gaming hub):

Salary.com:
Software QA lead $50k/yr
Video game QA lead: $15/hr (unsalaried)

Payscale.com (using 0 years experience, BS for both):
Entry level software QA tester: $47k/yr
Entry level video game tester: $31k/yr

This drastically lower pay for video game QA combines with the inevitable crunch time to make the job undesirable for workers on an actual career path. Why take constant overtime when you are getting older, perhaps starting a family – that is not ideal for a skilled tester who wants to maintain a career. Due to these factors, many solid, skilled testers eventually transition to straight software QA.

One of the biggest problems, however, is the job hiring process itself. The video game industry, while profitable beyond belief, still has a very silly stigma applied to it. When jobs – like tester jobs – are outsourced, the hires are often done by HR people who don’t really work in the industry, so they advertise the job as “cool” and “fun” rather than focusing on the skills you will develop. This will inevitably lead to a certain type of employee, and it’s not the one who is doing the best work. In my opinion, the best QA testers are the ones who are unfamiliar or don’t have a direct passion for the project – they won’t be sidetracked by just playing the game for fun or get caught up in how the game “should” be and fail to see the forest through the trees. True, the employees who are slacking off and just playing around may eventually get fired for not doing the job well, but think of the wasted time hiring/training/firing them takes, when it could have been avoided from the outset with proper recruitment.

So, to bring this post to a conclusion…

I’ve been on the test floor. For years. I know what goes on there. Sometimes it’s testing. I am not pretending to know the full extent of all the QA department’s hiring practices, but the Blizzard employment website does make it clear that testers are contract workers, so it’s implied it’s quite similar to other large video game test companies. I know PlayStation is the same.

In that case, the fact is, the amount of GOOD testing you get is miles away from what you COULD get. I’ve seen how test passes ACTUALLY go. Factor in everything: the low pay, the lack of career mobility, the poor recruitment, and here is the truth – testers lie. Especially low paid, recently hired, unskilled testers. I’ve seen an incredible amount of bugs in MoP and I doubt any of them are the fault of the QA leads. For example, if two teams reach max points in Silversong Mines at the same time (one through escort, one through capture), the battleground bugs out. Forever. People have been in matches that have lasted nearly 2 REAL LIFE days. This is a win condition that a test pass would have testers checking, no question – but, oh man, getting the timing right is so HARD, so you get them just passing it as functional.

This is best QA.

Maybe I’m entirely off-base. Blizzard is fairly close to the chest with its internal affairs. But, like I initially stated, I’ve worked in QA and I’ve seen, firsthand, a frustrating level of unprofessional behavior and goofing off from employees.

I like working in video game QA. It’s a constant challenge and the job makes me think. I do, however, lament it sometimes when I see stupid bugs that I know should not have made it past testing. I can only hope that one day we’ll see changes to the industry itself that create an entirely new QA system for video games – and maybe we’ll all see the benefits. Hell, maybe some of my friends will be the ones implementing them. I think the effects would be profound for the video game industry.