The Rock and the Hard Place: Parallel and Agile Development — Part 1: Parallel Development
Note about the series: As we are reach for the best way to deliver the highest quality of software and services, we find ourselves considering parallel development and pondering the whole concept of agile. It may be that there is a sweet-spot between the two, and for each company that might be a slightly different spot. These articles will explore the perils of parallel development in part one, the angst of agile in part two, and how to pick and choose to find the right balance for your organization in part three.
Achieving preposterous results with negligible resources is simply an IT standard these days. We are developing across multiple platforms, industries, and teams which often include offsite or even offshore members. Yet we must bring it all together — cohesively and quickly. Our challenge is made more difficult by the common IT realities that author Johanna Rothman 1 identifies as "Pants on Fire" and "Split Focus" management. Familiar to most of us, "Pants on Fire," describes how IT resources are rushed from one objective to another, never able to stay on any one until it is "done right." "Split Focus" is what we end up with when, in an effort to overcome this chaos, objectives are parceled out in portions to the team. A team may be admonished to spend "half" of their time on this and to juggle three or four other projects in some precise proportion. Since neither of those methods is making stars, the natural logical progression is to split the development into teams. The idea is that each team (or individual) has the luxury of remaining entirely focused on their project until it is (somewhat) completed. In reality this leads to yet another management strategy we can call "Herding Ducks." Once you let these teams and the software scatter off in all directions how will you ever gather them together again?
As scary as we may find it, the lure of parallel development is powerful even for single team environments. Even in that scenario we may be developing for a future release but find that we must make a "quick fix" to a version that's already in the field. We need access to a copy of the source code for that released version — without disrupting (or pre-delivering!) the new code that's under development. So we make our change to the field-release copy and get it to the customer quickly with no interference. Now that fix has to be reintegrated into what is coming from development or it will be lost and the bug will be reintroduced. Even in small-volume, slow-moving environments (and find one of those!) difficulties arise with conflicts and bottlenecks. Taking that to the next level where we have teams making different changes to different code streams and we end up with a proliferation of branches that can cause developers to make inadvertent changes to the wrong code. Even if we don't, we have to remerge and retest everything repeatedly. Do we spend more time cleaning up than we saved? Advocates against parallel development are quick to point out that nine women cannot make a baby in one month. It takes one woman nine months and there is no way around this. For these naysayers, the very idea of having change occurring in more than one stream is a ticking time bomb.
"Branching is a distraction and serves to decrease productivity and code robustness." contends Steve Berczuk in his article Branching to Distraction. 2 Going back to that "fix," regardless of its nature, we are now splitting our attention from the current project work. We may be fixing a problem that has already been addressed on the active development line, and we are constrained to using versions of frameworks and tools with which the released code was delivered, even though using the newer versions of those would make the work easier. No matter what, developers have to deal with the frustration and cognitive costs of working with a code base that is different than what they work with daily. And that's just to make the change, before even thinking about how and what to merge after it is completed. Merging a complex change to a heavily changed main line runs the risk of becoming a branch in and of itself.
On the other hand, companies have grown frustrated with the waiting game of "serial development." The whole idea of coding something then sitting on our hands for it to be tested and accepted before coding the next thing is "old school." As a result, most of us are under pressure to consider parallel development despite our reservations. So let's consider it. Parallel development can be implemented in many ways but has in common the underlying idea that two or more "code steams" are being developed. The copies are referred to as branches and they also have in common that ultimately they must be merged. Beyond that the implementations can vary wildly. Here are some common practices.
- Just Say No. Many organizations still forbid branching entirely. They've weighed the cost and benefits and decided against it. They do all development in one main trunk. If a fix has to be delivered to an old release, it is peeled out of the main development area and hand-integrated on an individual basis. This is a monumental effort for which sacrifices are made all around. It can be very costly in a ways that are not obvious. What often ends up happening is that programmers have their own "sandboxes" — separate work arenas. These isolated sandboxes are not sanctioned, integrated, nor automated. In actual practice, no doesn't mean no. It just means not managed.
- No Merging. Branching is accommodated in limited circumstances with no merging. You can split off development in some situations, but you never bring it back. For example, after a fix is made to a field release it is then opened as a new bug report for the new release and handled separately. Of course, these fixes have to have a high priority in the main trunk because they must be completed before the associated programs go live with the old bug still in place! This is practicable when there is truly minimal branching and the main trunk is released frequently (if you begin to hear the echoes of agile practices, you have good hearing.)
- Critical fixes only — just wing it. Because we don't branch very often, we don't have to plan for it. In a crisis, do whatever is necessary. This is a small step from "Just say no," so the branch management may be less hidden, but is no less damaging. Remarkably common, this method grows like weeds in your garden — genetically modified weeds that are nearly impossible to eradicate.
- Cinderella Method. Go off in your own little development corner and bring back something finished. This is cycle branching and merging where the merge is done at a set point in the life-cycle, often after release to production. It has a structure that can be planned for and accommodated with procedures and tools, but has its own inherent risks. Often work is lost on both sides of the divide. Time is always lost. Sometimes the work to merge the branches becomes its own branch. This multiplies the time lost exponentially.
- Continuous integration. A pillar of agile practices, the idea of frequent integration is taken to extreme heights with daily builds in some environments. Whatever the interval period, the idea is that we don't wait for projects to be completed, but rather do the merge work on a predetermined periodic basis. Frequent integration has the advantage of identifying integration issues quickly, but the disadvantage of requiring the person doing the merge to shift context frequently. Plus, if the idea was to not make the two developers interfere with each other, this is completely defeated — worse than defeated — by making them merge into each other's work almost continuously! This is a particularly difficult concept in the MultiValue world. While a particular program is compiled — and this might be similar to a build in that it quickly reviews the syntax and we can make sure the program at least runs — most of our work to complete a software change requires changes to several types of items, and these are not "compiled" together into a unique, executable deliverable. The idea that you could run a "build" and make sure the whole application still works is ludicrous in our environment. The regression testing involved in this would be astronomically larger than the development effort. It's hard enough to get testing done as it is!
If your mind is already connecting the Cinderella-off-in-a-corner-until-its-done and the continuous integration approaches and seeing where identifying smaller tasks that are completed and integrated frequently could work, go to the head of the class. This is where parallel development meets agile on ground solid enough for MultiValue.
There are several approaches to parallel development that are practiced in MultiValue. Each has some benefits and some risks. The one that will be best for any given environment depends on a lot of factors. In fact it would vary by project, by team, by day — perhaps with the weather.
Changing item names
Since our MultiValue environment is "granular," we can actually manage branch copies right in the same development arena (or account). If you make a change to a program that is already checked out to another developer, you can rename your copy. However, when you would like to test it you find that it is called from a menu that does some necessary pre-work so it cannot run standalone. You can't change the menu to call your copy so you make a copy of the menu, too. Then a subroutine that you modified requires that you also branch the calling program… and so on. You can end up with a proliferation of renamed items, many renamed just to call a renamed item for testing. Still, it is do-able and ultimately when you complete testing and are ready to merge your work back into the main trunk, those items will be easy to revert back. This method works best with frequent integration / small projects because probably multiple projects that change the same items should be tested and delivered together. If a major project is under development in the same arena it can make it very difficult to get the renamed copies past it — that situation lends itself better to separate arenas.
Changing file names
Some of the renaming just to call a renamed item can be avoided by leaving a program with the same name, but move it into a separate file. When you are ready to test you can catalog your version of the program. This method really only accommodates one branch being tested in the account at a time — and even worse, it is not at all obvious which version of the program is cataloged at any given moment.
Separate arenas (accounts)
The idea of creating a whole separate "development account" for each project begins to appeal. No renamed components, no mystery catalogs. Work any way that you need to work and then roll your work back into the main trunk. Space can be an issue, unfortunately, because assuming you already have a development copy of the environment, we are now talking about creating a second one — or more. These are whole, working copies of the environment — perhaps each one comprised of multiple MultiValue "accounts." Some companies like to have a whole copy for each programmer or team. We find in practice that this ends up being more complex than it needs to be — mainly because all of the accounts have to be updated with everything that is completed in all of the other accounts. (Picture braiding Rapunzel's hair.) The notion of separate arenas seems to work better when they are for a purpose, and for a set period of time, not for a person or team and not indefinitely. Having a main arena that is for ongoing fixes and small changes, then a separate one for some major development project can be a good compromise. If you can tip-toe your way toward agile practices with that bigger project — integrating it into the mainline at regular intervals — this is even better. The next article in this series will explore what agile practices are and how they translate to what we really do on a day-to-day basis.
1 Johanna Rothman, author of "Pants on Fire or Split Focus" published on Tech Well (www.techwell.com).
Steve Berczuk, author of "Branching to Distraction", published on Tech Well (www.techwell.com).