Feb. 24, 1995 To: Sam Tramiel Jon Correll John Skruch From: Leonard Tramiel Re: Testing I have been quite proud over the past years of the quality of the software that our company has released. By "quality," I mean the very low bug rate and the lack of stupid mistakes. Lately, this has been more and more difficult to maintain. Often the bugs are not found by the test department but by testing done by the more informal testers we have always used. Some bugs have gotten out. The reasons that we have let out software with either obvious bugs or stupid mistakes are: We need software and this is better than being late enough to fix it. This is not a problem. (In my judgement it is not worth fixing) This is not a problem. (It is too painful to fix) No one noticed the bug. The bug was noticed but not considered important. The recent examples of bugs and stupid mistakes are many, they include: Lack of driving control in Chequered Flag Network Failure in Doom Poor fighting control in Kasumi Ninja Not enough difference in the difficulties in Iron Soldier Bad level arrangement in Iron Soldier (The third one is HARD!) The save game is too easy in Wolfenstein 3D The cheats are too easy in Doom The difficulty progression in Bubsy is wrong These range from minor annoyances to major problems. They all should have been caught by test. In at least one case (Doom) the problems were known to the test department and the game went out anyway! This may have been a conscious decision like the release of Club Drive but I don't believe they all were. The reason for this memo is not to remind you of what most of you already know. The reason is to make sure you know what happened with Hover Strike. Like all games Hover Strike has special things that can go wrong. Many games have their own special weakness and many share a weakness with an entire genre. Hover Strike has such a weakness. It is the danger of "escaping" from the world. Each Hover Strike mission takes up part of a very large possible universe and it is important for the player to not escape from planned mission. If the player does escape the best thing that is likely too happen is he will get totally frustrated because he cannot finish the level. The worst thing that could happen is the game would crash. This weakness is not a surprise and has been known for a long time. The testers have reported the possibility of escape from some missions in the past so there can be no doubt that the testers knew about the issue. Hover Strike was signed off by the test department today. The test report makes no mention of any problems with "going over the edge" of a mission. After the game was signed off I decided to try the final level (an Ice world) because I have never beaten the game. While playing I went over the edge of the world and I became basically impossible for me to finish the level. This happened as a result of fairly normal game play. I did not try to go over the edge. Next I tried to repeat this problem, it was very easy. I then told the lead tester what had happened and he repeated the problem in less than 30 seconds. This problem occurred in almost EVERY mission. After this problem was discovered one of the testers said something like "It was no big surprise that the Ice world had the problem, they were notorious. But when I tried I got out of just about ALL of the worlds". This tells me that a known weakness of the game was not adequately tested. It is not a matter of the testing requirements being unknown. It is not a matter of the bug being obscure or difficult to reproduce. It is a matter of the entire test department never checking to see if a previously reported bug had resurfaced. The test department has gone thru many changes of personnel lately. It now consists entirely of people with very little experience in testing and a test leader with virtually no experience in leading. It seems that the test department no longer understands how to test. I fear that the test department no longer knows what test is. Testing is not just playing the game, or even beating the game. Testing is breaking the game. Finding the weaknesses and pounding on them without mercy. Testing is doing all the sick things that people might try. Testing is a creative task not a rote one. BTW: John, I actually missed Juli. One of the many functions of test that I did not mention above is to guard against typos. It seems that no one does this anymore. The last bug fixed before Friday's retracted release was a repeated word on two consecutive lines. .