The Rock and the Hard Place: Parallel and Agile Development - Part 2: Agile Practices 2
Note about the series: As we are reach for the best way to deliver the highest quality of software and services we find ourselves considering parallel development and pondering the whole concept of Agile. It may be that there is a sweet-spot between the two — and for each company that might be a slightly different spot. These articles will explore the perils of parallel development in part 1, the angst of Agile in part 2, then how to pick and choose to find the right balance for your organization in part 3.
We are all aware of Agile practices. We may, in fact, be in the process of implementing or at least considering Agile development methods. Or we may be practicing our own artful techniques in dodging them. Frequently those of us in MultiValue view mainstream software lifecycle practices as something that doesn't apply to us. That somehow we can't adopt them, or that we don't need to adopt them. It is true that when we consider mainstream models and processes, there is a certain amount of adaption and translation that must sometimes occur. But one thing we should all do — if we do nothing else — is recognize that our similarities outweigh our differences. Mainstream approaches can absolutely be applied and their benefits realized within our community. Realizing benefits from something like Agile practices is a cha-cha … a dance step that may take you two steps back before you can lunge forward. But those little steps back are small and rhythmic and the lunge forward can be a thrill. Most importantly, this is true for any platform or IT culture, not just "ours."
The first step is, as always, understanding. There are some commandments. Sometimes there are ten, sometimes more, depending on the blogger and the methodology. There is even a manifesto (really). But let's start by going back further, to what Agile isn't.
There actually is a difference between Agile practices and cowboy coding. Sure, there is this romantic counter-culture image of a coding cowboy that is appealing. Strong, silent, independent… but snatch the harmonica and pull off that sheepskin jacket and all you have is a sloppy, intractable jerk. In fact, the term "cowboy coding" is often used derogatorily by Agilests. That is why it is particularly ironic that there is a school of thought — shared by many MultiValue users — that cowboy coding is Agile.
The Cowboy Manifesto
Calling someone a coding cowboy is an insult. Referring to one's self as a coding cowboy (outside of a self-help group) is simply arrogant. Nothing good can come from it in either case. What leads to this misconception that MultiValue is somehow intrinsically Cowboy and that Cowboy is intrinsically Agile? In Agile, teams are empowered to make their own decisions — to work independently. And one of the core components of Agile development is the call for rapid delivery of small pieces of working software. So if you scan a page about Agile practices and see the words "rapid" and "independent," you might think of Cowboy Coding. But, of course, you'd be missing the point.
Most of us with any experience in MultiValue proudly point to our ability to rapidly deliver change. And it is true that we have an advantage over some of the stodgier platforms. That doesn't mean that we must continue with this outrider image we've built up for ourselves. The reputation for too much change with too little control could be — and should be — shed. We can afford to apply some practices, some methodology, some consistency and we will still be the fastest. It would be a powerful thing if our environment came to be known as fast and good and cheap. You know, instead of the traditional two out of three.
So Agile isn't cowboy and MultiValue doesn't have to mean cowboy. Then what does it mean to be Agile? It should be confessed that the Cowboy Manifesto was made up for this article. But there really is an Agile Manifesto. A bunch of (smart, important, industry leading) geeks went on a ski holiday and came up with it. It is accepted and referenced with somber formality.
The Agile Manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
That is, while there is value in the items on the right, we value the items on the left more.
The key to the whole idea is in that last line. That becomes the mantra, the litmus test. Agile has a common description that you will find in some close incarnation most everywhere you look: Agile development is a methodology (or a group of them) that is based on iterative and incremental development. In this approach, requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. Agile development promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change.
We all probably feel the same two biological responses fighting for dominance when we see the word "methodology": The yawn and the groan. At best we think it might be good for us in the same way that fistful of vitamins and supplements we are supposed to take every day is good for us. And just as hard to swallow. Part of our involuntary negative reaction to that word is because it is the harbinger of the next word we know is coming — discipline. Ugh. But we might feel differently about Agile methodologies if we get to know them.
Most methods impose discipline on software development. They do this by developing a detailed process with a strong emphasis on planning inspired by engineering strategies. Contrary to this idea, Agile methods are adaptive rather than predictive. Engineering methods tend to try to plan out a large part of the software process in great detail for a long span of time. This works well until things change. So their nature is to resist change. The Agile methods, however, welcome change. The goal is to adapt and to thrive on change, even to the point of changing themselves. So, in that spirit, the first thing that must change is our own attitude. We must start to think in terms of what we can do well in a pre-defined time(box) instead of how much time do we need to complete some pre-defined task. See the difference? One consultant that we all know and love attempted to implement Agile processes at a MultiValue site once. It was going pretty well, he says. Until the Director demanded a Gantt chart showing the target dates of all of the timeboxes with specific deliverables for the next year and a half. The sleek, glamorous timebox morphed into a square and stodgy task. The key here is that it is not just about delivering smaller completed tasks more frequently — it is about not planning (much) past each of them. The requirements can change. The priority can change. The scope can change. The team can change. That's the whole idea. That's what makes it different and valuable. It's what makes it work. You don't really think about the next thing you are going to do until you see where you are when this thing is completed.
In order to think in an Agile way, we have to let go of a lot of planning — a lot of predictability . We have to <choke, stutter, gasp> relinquish control . How do we cope in such an uncontrolled and unpredictable environment? By focusing on where we are now. We need to know where we are at any given moment. Then where we go next is a simpler question. So we look for an honest feedback mechanism which can accurately tell us what the situation is at frequent intervals. The key to this feedback is iterative development. This is not a new idea (intl-spectrum/s1046). Iterative development has been around for a while: incremental, evolutionary, staged, spiral... lots of names. The key to Agile iterative development is to frequently produce working versions of the final system that have a subset of the required features.
Iterative development makes sense in predictable processes, but it is essential in adaptive processes because an adaptive process needs to be able to deal with changes in required features. This leads to a style of planning where long term plans are very fluid, and the only stable plans are short term plans that are made for a single iteration. Iterative development gives you a firm foundation within each iteration that you can base your later plans around. So the dream is to make an iteration as short as you can. Break things down into the smallest possible deliverable. This provides more frequent feedback, so you know where you are more often.
These short time-periods, timeboxes, are predefined, even arbitrary. They have little to do with the specific project or team at hand. We must fit completed, working deliverables into the time frame — not fit a timeframe around some notion of complete that has anything to do with the software we are building. Along these same lines, we must let go of the notion of requirements to a large degree in order to be Agile. Instead we have user stories, which are less formal and often given in face-to-face communication. Telling the user story can make it easier to define the iteration — what we will fit into the timebox.
It feels very risky to think about taking verbal, flexible requirements into the lab. But that's because we are used to trying to pin people down — limit the scope, fight feature-creep. When you turn the tables and embrace the evolution of the requirement, this isn't risky after all. But it is (sorry for the overused phrase) a paradigm shift. An important part of this is that the 'stakeholders' — the business users who have requested the software change — must be intrinsically involved in the timebox. They should help to define the deliverable and participate routinely in the meetings.
So far we've been talking about "Agile" as a general philosophy of software development. Under this broad umbrella are specific approaches such as Extreme Programming, Scrum, Lean Development, etc. Each of these more particular approaches has its own ideas, communities, and leaders. Each community is a distinct group of its own, but to be correctly called Agile it should follow the same broad principles. Each community also borrows ideas and techniques from each other. Many practitioners move between different communities spreading different ideas around. All in all it's a complicated but vibrant ecosystem. In truth many organizations create their own hybrid approach. What we want are quickly adoptable practices that provide measurable and sustainable improvements. We want to see these improvements in the quality of our software, in the satisfaction of our customers, and in the productivity and even happiness of our developers. This idea of cherry-picking from various methodologies and finding our own middle ground has come to be referred to as "Real World Agile," and more recently the term SDLC 3 is gaining some traction. SDLC 3 — the 3rd incarnation of the software development life-cycle — is an even broader mixture of both Agile and traditional approaches along with some new ideas.
How do you figure out what will work for your organization? How do you get started? Probably the very first thing that you do in working toward adopting Agile practices is to think about being Agile in that very adoption. Don't think too far down the road. Try something out and then see how it fits and what looks like the next thing to try after that. Here are some specific aspects of Agile that you can usually do right away:
Initial timeboxing: More frequent releases or less frequent releases. First, especially in environments where a lot of parallel development occurs, delivering smaller releases more frequently will enable more frequent integration and minimize the amount of merge work being done.
On the other side of that same coin, organizations that tend to hit production with changes on an ongoing basis spend a certain percentage of their time just dealing with the thrash. In this situation moving to scheduled releases can improve many aspects of the process. Create a regular timebox by saying "we release on Thursday," for example. Then deliver to production at the end of each timebox — like clockwork. A one week time box is very short for Agile development in the mainstream — but for an organization that is hitting the production environment with continuous change, it is a step in the right direction.
Next, take a look at a project in the backlog and think about applying a timebox to it. What can I deliver in a month with my resources? Work with the users and developers to define and deliver just that. It is hard to think about delivering unfinished stuff, so the trick is to think in encapsulated chunks of "finished stuff." A nice bonus is that users are more likely to get to the testing in a more timely fashion when it is a smaller set of changes that they must test. And defining the test is part of the user story.
User Stories: A user story is a requirement that defines what functionality a user really needs. A story might be a variation on this basic statement: In order to <Business Value> as a <User Role> I want <Functionality>. During release planning, user stories are given a rough size estimate using a relative scale like points. During iteration planning, the story will be broken down into tasks and tests.
STORY: In order to make sure my package arrives at my house as a website customer, I want to validate my shipping information.
Call web service to confirm address — estimate 8 hours
Write tests — estimate 4 hours
(Virtual) Colocation/ Face to Face communication: One of the tenets of Agile development that may feel contrary is the frequent face-to-face communication. Most companies have outsourced contracts, off-shore teams and members who work at home. This trend is not likely to reverse, but technology can overcome this dilemma. You can have those frequent face to face meetings — over Skype or other similar technology. Begin to let go of long complicated requirements and email threads and replace them with daily face to face meetings — even if those meetings are virtual.
Requirements exchange: When timeboxes are introduced along with the notion that it is okay for requirements to change the user or product owner must remove a comparable amount of work from the project in order to accommodate a change. This actually does a lot to avoid what we call feature creep. It empowers the users. They begin to feel freer to change their mind about what is important, to try out different ideas, but at the same time to take responsibility for the deadline.
Empower your teams. One of the hall marks of Agile development, what makes it work, is that the team members are empowered to make decisions on their own. Rather than monolithic management and monumental meetings, each team is responsible for their contribution. Project Management in an Agile environment relies more on the project manager's skills in communication, facilitation, coordination, and less on planning and control.
Involve the user community. The users must be stakeholders in the development effort, included in every phase. They must help develop the specs for each timebox, stand ready to test and accept bite-sized projects as they are completed, and provide continuous feedback in the form of daily meetings.
Take baby steps. Massive detailed specs, long lists of upfront requirements, and long term commitments can cost you dearly. Large projects that are difficult to manage ultimately result in software no one wants. So the word of the day is "incremental." Make small changes to the way work is done. Just like the resulting software, the change to our process can be incremental. We learn more with each iteration so that we arrive at a finished picture with a finer grain, a smaller pixel.