FEATURE
Refactoring in the Real World
by D. Keith Casey, Jr. REQUIREMENTS
PHP: 5.2+ We all talk about how to best Background on web2project While most people haven’t heard of Web2project, Other Software: structure our projects and how many have heard of our parent, dotProject. In late • Phing - http://phing.info/trac/ • phploc - http://github.com/sebastianbergmann/ 2007, dotProject faced a crisis. We had over 200,000 to write ’clean code’, but how phploc lines of code spanning over 7 years of develop- • phpcpd - http://github.com/sebastianbergmann/ do we clean it up? How do we ment all the way back to PHP 3. That legacy was phpcpd find problems in a large, legacy reflected in poor object models, a lack of coding • phpmd - http://github.com/manuelpichler/phpmd standards and a mixed up, pseudo-MVC, structure. codebase and clean them up The codebase was difficult to debug and even harder Related URLs: responsibly? to extend. Installing some modules required a skill • http://blueparabola.com/blog/subversion- for creative copy/pasting and generally, a high commit-hooks-php • http://github.com/caseysoftware/web2project/ blob/master/unit_tests/build.xml
October 2010 30 www.phparch.com Refactoring in the Real World
tolerance for pain. As a result, half of the dotProject rebuilds permissions when they change; instead of code and eliminate numerous issues and behavior team wished to start over the “right way” and set on every request. As a result, we eliminated nearly oddities on different screens. It was a starting point out on that path. The other half wanted to refactor 98% of the time required. Now that our pages were and allowed us to make sure ACLs were respected the current codebase and formed the Web2project loading in less than 1/10th of the original time, we and queries were well-structured, but it was difficult team. I was recruited into the latter and have served could focus on stability. and slow going. on that team ever since. For stability and general debugging, we had to take a completely different approach. We knew that Applying Tools to the Process Starting Point no one thing would solve 50% of our problems. We needed a new strategy that would actually work When we set out to refactor the system, we knew Therefore, we started by exploring the tools avail- long term. Luckily about that time, we added a new there was a huge effort in front of us. There were able; at the time, there weren’t many. CodeSniffer team member, Trevor Morse, who brought another coding standard problems, hundreds of bugs, major was - and continues to be - useful for checking our perspective into the group. He dedicated himself to performance issues, HTML validation problems, code against coding standards, like PEAR, but it building a solid Unit Test suite. Since the Companies cross-browser JavaScript problems and errors in how just identifies sloppiness, not necessarily problems. class had been problematic but still relatively permissions and security worked. Since we couldn’t Therefore, we started a completely manual process to straightforward with few external dependencies, he tackle everything at once, we decided to focus on detect, review, and refactor duplicate-but-slightly- started there. In this case, the key was starting with performance and stability first and foremost. It is different code. To be honest, we didn’t have a clue, one and only one class to get things rolling. Once he not important how many features we have nor how and we didn’t know where to find one. had a basic framework for running the tests, others XHTML compliant the system is if it’s not reliable. While the manual process worked, it was time- joined in writing Unit tests for areas in which they On the performance front, we added some simple consuming, prone to errors, and slightly less pain- were having trouble. In a span of 60 days, we went debugging and found that nearly 50% of page load ful than stubbing your toe at 2am while you’re half from zero to nearly ninety Unit Tests; however, we time was spent retrieving our database objects, asleep. Scratch that. It was more painful and result- ran into problems with executing them easily. nearly 40% was permissions checking and only the ed in similar expletives. The bug reports for the v1.0 In order to run the tests easily and consistently, remaining 10% was spent processing the data and Release Candidate demonstrate this. Regardless, we I created the Phing build script visible in Listing rendering it for the user. Our first step was exploring started by looking at areas with similar functionality 1. Phing is like Ant or make but specifically for and MySQL’s Explain command and the Slow Query Log. As and refactoring where we could. Not surprisingly, the extensible with PHP. Using Phing, you can wire a result, we found there to be a total lack of data- most common actions - like retrieving a list of active together a set of tasks to run in sequence to per- base indexation and numerous, redundant joins. By projects or getting a list of the users’ assigned tasks, form tasks such as Subversion checkouts, deleting adding indexes to the proper fields, we immediately were implemented a half dozen different ways. Each directories, running unit tests, or even creating zip/ eliminated 95% of the database time. way was slightly different than the one before in tar.gz files. The key feature for us was its ability to The permissions checking was quite a bit more dif- terms of query structure and respecting system-wide recursively explore a directory structure, retrieve ficult and required a team member, Pedro Azevedo, permissions. The worst part is that most of them all the unit tests, and execute them. In a matter of to build a permissions caching layer that only were just a little wrong. By focusing on these areas minutes, we had a script that would run all of our first, we were able to trim a few thousand lines of
October 2010 31 www.phparch.com Refactoring in the Real World
tests as often as we needed with a single command. average number of lines of code per/class or per/ again, we have specific places to focus our refac- As an added benefit, many IDEs like Eclipse PDT and method. To run it, from the phploc directory, type: torings. Following the bulk of our cleanup effort, NetBeans can execute build scripts automatically as the results were amazing. We had removed nearly needed. php phploc ../web2project/ 15,000 lines of code and reached a 2% duplication. Next, we used one of the single most underrated In general, we look at these metrics on a project Unfortunately, the project as a whole isn’t the im- tools in the PHP world: php lint. Lint simply vali- level, but by specifying a module or class in the portant part. Applying the same detection to specific dates PHP files and throws an E_Fatal if there are command, we can analyze a smaller set of files. This areas of a project is more informative. Further, when syntax errors. The best part is that it’s built into PHP is where the we can draw some interesting conclu- duplicate lines are found, it is useful to explore the itself, so anyone can run it with: sions. area around the samples to determine if there are near-duplicate lines surrounding it. In that case, it php -l filename.php At a project level, the average class is about 800 lines with an average method size of about 80 lines may make the most sense to refactor those areas Ideally, this would be part of our Subversion pre- and Cyclomatic Complexity of about 8. Overall, these and increasing the duplication before refactoring it commit hook to prevent bad code from even making numbers should come down, but they’re not a bad away. it into the repository, but we took what we could starting point. Unfortunately, when we look at a few While using phpcpd regularly, that pattern has get. More importantly, Phing implements lint na- specific classes, the story is completely different. emerged time and time again. Obviously, as we tively as seen below: The Tasks module has nearly 8000 lines with an aver- eliminate duplicate code, the percentage goes down. age method size of about 120 lines and Cyclomatic More importantly, as we clean up code to be more
October 2010 32 www.phparch.com Refactoring in the Real World
similar to eventually refactor away, our percent- to detect and triage problems by size or complexity, of those steps are successful, the zip and tar.gz files age increases. Therefore, throughout development, we can apply standard refactoring techniques. This are created. Although this did absolutely nothing for we see this percentage drift up as a big refactor- has allowed us to eliminate tens of thousands of code quality, the 2.5 minutes required is well worth ing approaches and drops immediately thereafter. lines of code while making the system less complex. it as the files come out exactly as planned each and At a project level, this drift is hidden by the sheer More importantly, through refactoring these dupli- every single time. More importantly, this process amount of code but within individual modules, it is cate snippets, we’ve closed literally dozens of bugs allows us to create a release at almost any time and a visible and dramatic sign of changes required. and eliminated subtle behavior differences of differ- have some confidence in the actual system. The full The most important tool that reached matu- ent parts of the system. But more than anything, the Phing script for this step is included in the ’Related URLs’ rity during this effort was phpmd or the “PHP Mess most important part has been the improved stability section. Detector”. While it is the most powerful tool, it in the development process itself. tends to be the most complicated. By running: Finally, once all of these pieces were in place and the development process was smoother, the next php phpmd.php ../web2project html codesize,design,nami logical step was to simplify the packaging and dis- ng,unusedcode > ../reports.html tribution. While it is trivial to generate zip and tar. LISTING 2 gz files manually, automating it via Phing ensures it 1.
October 2010 33 www.phparch.com Refactoring in the Real World
Results every 32 hours. again, I would start using these tools earlier to help Not surprisingly, the effort started slowly, gained One of the side benefits has been completely prioritize our own efforts better. The important part momentum as we refactored, and has slowed unexpected. With a class hierarchy that is better to remember is that none of these are one-time use again now that the most visible and easiest prob- structured both logically and file-wise, editors such tools. At minimum, unit tests should be run before lems are resolved. Regardless, after two years of as Eclipse PDT and NetBeans can interpret the files and after each commit, though ideally, they could being an independent project, we had closed over properly and support code completion and features be part of a continuous integration system. The rest 150 issues, added 40 features, eliminated nearly like “Go to Declaration”. Previously, these functions of the tools don’t need to be used continuously, 65,000 lines of code, reducing our duplication from would function sporadically at best. This has made but should be run regularly. I collect the output of 8% to 2.1%. This is the equivalent of closing an issue debugging and refactoring become a task measured each tool around the first of each month and when every five days, adding a new feature every other week, in minutes instead of hours or days. we make a release. So far, it has been an excellent and deleting 89 lines of code every day...for two years. And finally, now that we practice coding standards way to track our progress and determine which areas In addition, in the last 16 months, we’ve added and use class structures conventions instead of just need attention. nearly 400 unit tests which works out to a new test talking about them, we can raise our expectations One of the things not discussed here is the value of for the community. The ability to hold everyone to code reviews. Random people throughout the PHP com- the same standards is freeing and can simplify many munity have assisted in one way or another, but specific potentially contentious disagreements. In some people have gone above and beyond all expectations. cases, we’ve had to clean up a module to show the People such as Sebastian Bergmann (author of many of process and give clear examples, but the community the tools above), Arne Blankerts, Stefan Priebsch, and in general is starting to practice the same. After all, Matthew Turland have contributed specific and tangible no one wants to be the guy writing the sloppy code. recommendations, many of which we’ve taken in the last year. Conclusion Overall, the effort has been large, sometimes dif- ficult, and has slowed our development of new features. That said, the code is faster, cleaner, more logical, and there’s less code, period. Since the cleanup, though, new features, functionality and entirely new modules have been easier and faster to To pay the bills, D. KEITH CASEY, JR. works with Blue Parabola, build. Most importantly, the extension modules can LLC and has developed large-scale PHP-based systems for organizations ranging from news channels to small reuse more of the core system logic which means non-profits. In his spare time, he is a core contributor to less code to maintain and the sometimes-complex web2project, blogs regularly at http://CaseySoftware. ACL system is respected. While I would do it all over com/blog and is completely fascinated by monkeys.
October 2010 34 www.phparch.com