The Rock and the Hard Place: Parallel and Agile Development Part 3: Must We Choose?
Note about the series: As we reach for the best way to deliver the highest quality of software and services we find ourselves considering parallel development and pondering the whole concept of agile. It may be that there is a sweet-spot between the two — and for each company that might be a slightly different spot. These articles will explore the perils of parallel development in part 1; the angst of agile in part 2; then how to pick and choose to find the right balance for your organization in part 3.
Welcome to part three in which Pooh and his MultiValue friends gather the best honey from various trees to mix the perfect jar of software development lifecycle practices. To help in this endeavor we've compiled some nuggets of conventional wisdom, a few reference materials and some key terms to help in the search.
After examining the perils of parallel and the angst of agile, many of us may be looking for a third alternative. But we know from Fred Brooks' renowned paper No Silver Bullet 1 that, well, there is no silver bullet. There are no simple answers to our complex questions. Maybe we should just stick with what is tried and true. That may be a curmudgeonly, change-resistant attitude, but it is not entirely without merit. Obviously if we are still here developing software, we have found a good way to do it. We don't have to change our methods just for the sake of changing our methods. We are not trend-followers in the world of MultiValue! And unlike some of our IT brethren, we are not chumps . 2
Yet, we are feeling pressure to accommodate parallel development and struggling with how to manage it. Our technology and our business is changing and growing more complex almost daily. No matter your intentions, you may find yourself staring down the neck of a bottle of "Agile." While some of the advertising may be tempting, the general consensus among practitioners is that trying to plug in some boilerplate set of practices, even with a cute sports-related or foreign sounding name, rarely brings sufficient returns to justify the effort and expense. And if we can't show those returns, we won't have the critical backing of upper management. The last thing we want to do is squander our credibility. Just like a fad diet where you will lose weight the first week, no matter the rules, then gain it all back in the second week, fad development practices are not sustainable. Yeah, you can hear it coming: It's a lifestyle change.
IT people on the ground have little time for theory and many have demanding business users that don't care how results are achieved; they simply expect them to be achieved, immediately. We've trained them, spoiled them that way. It is easier to push back when you are armed with solid understanding (that you can explain), believable metrics (that you can prove) and real world examples (that you can point to). We must be objective about current procedures and about the options. Get away from "What am I comfortable with?" and "What have we always done?" and move toward "What have we always done that is still working for us?" and "What have we been doing out of habit that may be costing us?" On one hand we don't want to be chumps, but we can't afford to put our heads in the sand, either.
You can't swing a cat in the world of software development without hitting a branded Agile Methodology. Of greater note is that you can't find a discussion about software lifecycle management these days that isn't about Agile in some form or another. Some of the terms and approaches that come from the Agile community have been so successful as to become commonplace. Those folks are definitely on to something. The trick is separating the baby from the bathwater — distilling what is useful and helpful for your organization from the overblown fluff or platform-specific dogma. We can help ourselves by lower-casing the word agile and eliminating the term "methodology" from our vocabulary. Japanese terms and sports analogies might best be avoided, too. Of course we need to be agile — fast moving, light on our feet, and flexible. But we don't have to adhere to some contrived set of practices in order to remain that way. We are ready to move into Chapter Three. Mark Kennally represents in his book "SDLC 3.0" that this is the rationalization of modern software engineering methods into a complex adaptive system of practices .
Remember when we were talking about parallel development and we looked at the "Cinderella" approach (go off in a corner with your part of the project and bring it back when it's done)? We looked at that idea from an agile perspective and the idea of interval merging came into view. It was an obvious way to connect the real requirement for parallel development with a faster-moving agile environment. Along that same line we could get away from the need for parallel development in many cases by breaking our change requests down into smaller deliverables and taking turns modifying a component instead of each taking our own copy off to our own little corner. This might be a staggered or overlapping style of agile implementation. Just as in Kindergarten, sometimes we share the toy, sometimes we take turns. (Author's note: The notion of staggered, alternating or overlapping projects is particularly appealing to this developer, who sometimes late at night likes to think of it as Staggering Agility.)
In reality, to move forward with smaller projects it may be necessary to have smaller programs . This means that those couple of multi-thousand line source programs that are central to the application and take heavy modification should be refactored. This is the most common argument in the MultiValue world for not being able to adapt agile practices and not being able to manage parallel development. Further to the argument is the notion that there is simply no time to go back and re-write those central programs into modularized functions. If one were to step back and count the hours wasted because of not doing it, the time could likely be justified. What stops us is not a cold, rational thought but a feeling - white-hot fear that comes from the belly. As in all things that are both scary and necessary, it is better to face it down, break it down. If you plan for it and take precautions, the very real risks can be mitigated. Not taking the time to refactor becomes a road block to all further attempts to streamline the lifecycle, modernize the interface or increase agility and productivity. There isn't a strong enough way to express the idea that we need to clean house a bit with some of our older programs. They are weighing us down like Granny's army boots.
Another bad habit that is holding us back is treating all projects as if they have the same levels of risk and complexity. This is happening on both sides of the coin — IT departments that still do things in a risky, uncontrolled way — even for projects that really ought to be done more carefully. And some IT departments that have gone the other way and created intense checks and balances and onerous procedures that have to be stepped through even for minor changes. Great returns can be realized from an early investment in identifying the programs, files, modules and projects that should be treated with more care and separating the ones that can be managed with a shortened or simplified life-cycle. Once you quantify them, mark them, document them, even the auditors will be happy with the shortened procedures.
After you do a little housekeeping you will find that you are better able to experiment with interval development. Intervals — time boxes — are a central thread of agile practices. Different brands define shorter or longer periods or other criteria for an interval. But in all cases we are admonished to move away from long, unwieldy development projects in a waterfall/cascading life-cycle and toward short term iterations with small, working deliverables.
This idea of timeboxes is so universally accepted that it has to be a great idea, right? Certainly it's good for 3rd party development companies who are quite happy to sign flexible contracts allowing them to deliver "whatever" in "2 weeks" on a recurring basis until the client goes bankrupt. Okay, even if no one is really dumb enough to go along with that, there are other downsides to time-boxing. For one thing it is often an artificial, forced timeline. It has little to do with what is being delivered and nothing to do with who is delivering it. That is the idea, actually. That's what makes it work, when it works. But people have up days and down days. There is an internal rhythm to development that maybe can't be forced to march in step. The impedance mismatch can drive great engineers to mediocrity. But — now that you've identified areas of higher and lower risk, stratified your software repository — you can apply different timeboxes or lifecycles to different projects, to different programmers, to different periods of time.
This common theme of doing a little of an overall project, putting it out there, waiting for feedback, and then creating the next timeboxed deliverable has a lot of merit. It gets some resistance in our community because, let's face it, allowing this flexibility in requirements is the very definition of feature-creep. <little devil> It feels like it would lead to uncontrollable costs. For many organizations, the idea of letting go of the effort to block changes to requirements may feel like letting the dam break and being washed away. The idea lends itself to an all or nothing approach: The lifestyle change.
Let's get familiar with the history and terminology and dabble in a few of the variations. This can lead to a greater level of comfort with the idea, and it may point you to just the right book or set of practices to cherry pick whatever fits with your company culture.
To begin at the beginning, arguably the granddaddy of the agile movement is Scrum . A scrum is a way of restarting play in the sport of rugby after the ball has gone out of play. In 1986, a holistic or rugby approach was applied to product development (manufacturing), then later picked up mainly by Ken Schwaber in the 1990's who called it Scrum. When Schwaber collaborated with Mike Beedle on the book Agile Software Development with Scrum in 2001 , the name and the idea became firmly planted in our IT ideology. Scrum projects make progress in a series of sprints, which are timeboxed iterations no more than a month long. At the start of a sprint, team members commit to delivering some number of features that were listed on the project's scrum product backlog. At the end of the sprint, these features are done --they are coded, tested, and integrated into the evolving product or system. At the end of the sprint, a sprint review is conducted during which the team demonstrates the new functionality to the product owner and other interested stakeholders who provide feedback that could influence the next sprint. Over the last couple of years in particular, Scrum has garnered increasing popularity in the software community due to its simplicity, proven productivity, and ability to act as a wrapper for various engineering practices promoted by other agile methodologies.
The term lean software development originated in a book by the same name, written by Mary and Tom Poppendieck, which presents the traditional lean principles of manufacturing in a modified form. Lean Software Development focuses the team on delivering Value to the customer, and on the efficiency of the "Value Stream," the mechanisms that deliver that Value. Lean eliminates waste through such practices as selecting only the truly valuable features. Applying YAGNI (You Ain't Gonna Need It) for a system, prioritizing those selected, and delivering them in small batches. Like any agile approach, it emphasizes the speed and efficiency of development workflow, and relies on rapid and reliable feedback between programmers and customers. Lean uses the idea of work product being "pulled" via customer request.
Kanban is a process designed to help teams work together more effectively. A philosophical difference is that where Scrum places more emphasis on the project management practices, Kanban places its emphasis on business and value flow practices. If you love sticky-notes you may find yourself drawn to this set of practices.
The emphasis in Extreme Programming (XP) is in the technical practices. Some of these supporting practices have been controversial — and are not widely accepted. While no one can argue that having test plans is a good idea and that having the users involved in developing acceptance tests is obvious, many developers have complained that too much time is spent developing tests, taking away from the real task of developing software. Extreme Programming can be … well, extreme. Pair Programming came from this ideology and has taken a lot of criticism. Yeah, it is just what it sounds like — two people working at one station. The success of this approach may be dependent upon the personalities of the programmers.
Then we come to the Crystal family of methodologies. This is one of the most lightweight and adaptable approaches to software development. It predates the other agile methods and is more of a collection of meta-methodologies with the idea being that different projects and teams warrant different approaches. This is a people-centric approach and bears an underlying familiarity to MultiValue. Philosophically speaking. This Crystal family addresses the realization that each project may require a slightly tailored set of policies, practices, and processes in order to meet the project's unique characteristics. If you are looking for a place to dig deeper, Crystal might be a good starting point.
This is by no means a comprehensive list or complete history. But in all of the agile approaches the idea of allowing requirements to evolve over time and delivering bite-sized nuggets of working software mostly feels like a good idea. The differences in the various agile brands are often subtle but meaningful. It is quite reasonable to examine them and come up with your own workflow, your own criteria for an interval or a project, your own methodology for interfacing with users, empowering your team, for branching and merging and for delivering working software. Within your organization these components may vary from project to project, programmer to programmer, month to month.
It is, however, critical to create a workflow from start to finish so that there is guidance — a road map — to get software to the users. Probabilities for success are also greatly enhanced by having the right tools. Whether you consider a tool that imposes a methodology — but on the upside of that, supports it completely — or you settle on a few tools to help manage the rough spots, you want to make the procedure do-able as well as know-able .
The bottom line is that yes, you can manage parallel development in an agile way. Every organization has a sweet-spot, a balanced place between branching and agility. Working toward finding that balance will result in more satisfied users and developers.
1 Reproduced from: Frederick P. Brooks, The Mythical Man-Month, Anniversary edition with 4 new chapters. Originally printed for 1986 IFIP Tenth World Computing Conference. The paper "No Silver Bullet" likens software development to Werewolves.
2 "Stevey's Blog Rants", a highly entertaining blog by Steve Yegge, a developer at Google . In the 09/27/06 article "Good Agile, Bad Agile", Steve suggests that agile methods are a meme gone viral and its adherents all case studies for Barnum's adage that there is a sucker born every minute.