Software Craftsmanship: Project Workflow

In working through the topic of Software Craftsmanship for the past few weeks on this blog we've talked about how to approach software development from a code perspective.  I'd like to take a short break this week and talk about optimizing your project workflow in step with optimizing your code.

Over the past decade the Agile development movement has really taken over in the technology industry. You'll find many different forms of agile development used throughout the software industry. You'll even find many different flavors of the same form of agile used differently from organization to organization.

While there's no single one-size fits all solution when it comes to managing your project, it is important to have a base template to use as a frame of reference. My go to template is made of up a hodgepodge of different agile methodologies. The template is made up of three parts, and like any agile process, involves constant iteration through the parts.  This template is driven around three main concepts. First, the beginning of any project involves more unknowns and more risk than the end. Second, the process should involve creating features that are useful and complete. Third, communication, preferably face to face, is needed throughout the process.

Ideally this template is used in conjunction with some sort of Kanban process. A lot of people make the mistake of centering their process around Kanban. That in my opinion is a recipe for failure as Kanban helps you understand how to move work through a system but doesn't do a good job of helping you identify the correct work to move through the system.

Part 1: Identify The Problem Space

In this part you want to have an Initial Design Meeting. This meeting would consist of the project team, the key stake holders of the project, and any other outside tech advisers (like an architect or team lead) which may be able to provide insight into how this problem space fits in with other problem spaces throughout the organization.

In this meeting you want to Identify The Unknowns. The first unknown is what we're building. A quick discussion about what the problem is or what features the system is missing. Once we know what the problems are that we're trying to solve we can start creating straw-men for possible solutions. A good way to flesh this out is to create a few user scenarios that represent the desired workflow for solutions to our problem space. Questions to ask when evaluating the straw-men are:

1. What are the big open questions?
2. What risk do the unknowns represent?
3. What's in scope?
4. What's out of scope?

Part 2: Identify The Features

This next phase is all about defining the Minimal Viable Products. This is key as we want/need to know how we can provide some sort of business value as early in the project as possible. There are a couple of characteristics that I use when trying to determine if something is an MVP that I think are useful:

1. Can the MVP stand on it's own. I.e. if the project were to end or the company were to switch gears for a time, would the MVP still provide value to the business or would it need subsequent development in order to be of use?
2. Is success clearly defined for the MVP? Do we have a clear idea of what the definition of done means for the MVP?
3. Is the MVP achievable in a reasonable time frame?
4. Have we identified the external dependencies of the MVP?

When we're done identifying the MVP's we can organize them by things we can work on now and things that need more information to get started. Each MVP or a set of MVP's can constitute a project milestone.

Part 3: Create stories

During this step in the project workflow the team will get together and identify the stories associated with each milestone (or at a minimum the first milestone).  Take a gander at this article if you're unfamiliar with user stories. Breaking work out into individual stories helps us to keep focus on delivering on our milestone.

Once we've identified our stories we need to identify their dependencies and possibly associate some of the stories with other milestones if they have dependencies that can't be resolved early on in the process.

Lastly, in this part we create "bite sized" tasks for each story. This can be done all at once or, more preferably, at the beginning of each milestone.

Starting the work

Once we've gone through these three parts our teams can start their work. In order to be truly effective and stay agile, the team should work on one milestone at a time and not move on to next milestone till all stories for the current milestone are complete. During the teams daily stand-ups you'll want to report the status of each task/story and identify any of the blocking items that need to be resolved.

Software Craftsmanship: Going Dark (Or why you need a flashlight)

Over the last few places I've been employed at I've given a similar presentation on the topic of Going Dark.  I think it's time to unleash this topic to all three of you out on the interwebs. The term Going Dark may or may not be new to you. Hopefully by the end of this post you'll understand why it's bad, how best to identify it, and how best to avoid it.

Taking a look back at the last few posts I've done on Software Craftsmanship you've probably noticed a pattern emerge in what it means to treat software as a craft. I'll lump them into a category called Software Development Ideals.  So what are those ideals?

• Develop software for maintainability first
• Reduce the waste in our development cycle
• Reduce the complexity of the software
• Introduce less bugs as more features are introduced
• Write better code faster

Before we talk about how Going Dark is the antithesis of the ideals I think it's important to understand the symptoms associated with Going Dark. Those symptoms are:

• Working in isolation on a specific feature or problem for long periods of time
• Waiting to check in code till the feature is “complete"
• Failing to communicate or collaborate on software features as feature development “starts going bad”
• Constantly having to explain your work before and after you commit it to your repository
• Leads, management, or other teams members feeling like what was built wasn't quite what they expected

“Developers who work for long periods -- and by long I mean more than a day -- without checking anything into source control are setting themselves up for some serious integration headaches down the line” Jeff Atwood – Coding Horror

“Developers often put off checking in. They put it off because they don't want to affect other people too early and they don't want to get blamed for breaking the build. But this leads to other problems such as losing work or not being able to go back to previous versions.”
Damon Poole– Author of “Do It Yourself Agile”

Okay so now that we're able to identify the symptoms of someone who has gone dark. Let's talk about why not treating the problem is actually costing you lots of time, energy, and wasted cycles.  There are several problems associated with someone who has gone dark.

• There's more code to integrate on each check-in
• Code that is written in isolation is full of assumptions about the other software with which it will be integrated
• Each assumption is a potential issue that will only be discovered when the software is integrated
• The more code you have to integrate the higher the likelihood of introducing bugs
• The longer you wait to check in your code the higher the likelihood of breaking a piece of NEW code you've written before you check it in
• When you're on the wrong path you end up spending more time going down that path rather then being able to have your path corrected sooner
• There's no way to "gut check" your progress
• Closes your code off from scrutiny (which may seem uncomfortable but ALWAYS makes your code better)

That list alone should scare you. My guess is that you can probably relate to it. You've probably participated in "merge parties" or spent half a day or a full day trying to re-merge your changes after someone else beat you to committing and has changed the topology of the code. Maybe you've come in one day to find that there's a new bug in the system because a merge went bad and something got missed or accidentally added.

The great thing is that there are some pretty simple changes that can be made. The first is to start working out how to make lots of little small changes in the system which can be verified along the way. One good method for doing this is using some form of Test Driven Development. Once you've started to follow down the path of thinking about your code in bite sized pieces, all of which have an associated test, you can start to think about checking in those bite sized pieces frequently. I like to think about this as Checking In Early and Often. There are a lot of benefits of checking in early and often, and you'll notice that they are solutions to the problems that are presented by Going Dark.

• Change set is much smaller which means it's much more manageable and has a lower impact on the code base as a whole
• When you start integration of your changes the chances of having changed the same method or function as someone else is much smaller
• Less of a chance of having to reapply your changes on top of someone else's just because they beat you to the check-in
• When others refactor, you obtain the benefits quicker with less integration work
• Reduces the likelihood of anyone's changes being lost or overwritten by a bad integration
• Easier to find bugs because there's less code to search through
• Easier to fix bugs because there's less code that could be possibly causing the bugs
• Helps to reduce unnecessary code (YAGNI) as you're just writing the code you need to do the task you're on

Go forth into the light and help others come out of their darkness!

Software Craftsmanship: Don't Repeat Yourself

In the software world copy and paste is the enemy. It causes bugs. It slows down development. It duplicates functionality. But more importantly it decreases maintainability.

It causes bugs and slows down development because you have multiple places to maintain the same source of knowledge. If you apply just one change to one place and not the others you will have a bug in your system. And the difficulty of finding and fixing this bug increases with every new permutation as there become more and more code paths that utilize that source of knowledge from different sources.

The antidote to our copy and paste problem can be summed up with  Andy Hunt and Dave Thomas's acronym DRY (don't repeat yourself). The DRY principle states that "every piece of knowledge must have a single, unambiguous, authoritative representation within a system." That statement is chock full of goodness that can be overlooked so let's break it down.

The simplest part of that statement is single. That one word in and of it's self sounds simple but what it really means is that we have a responsibility to make sure that our source of knowledge only has one place it exists. This doesn't mean that you create a "tools" library for all common code like is often the (misunderstood) case with people trying to implement the DRY principle. It means that you need to make an effort to constantly refactor your code such that you increase the reusability of each source of knowledge.

Each source of knowledge needs to be associated with a clear understanding of how and when to be used. When a source of knowledge in your code can be interpreted to do many things it is an ambiguous source of knowledge. Each source of knowledge should be unambiguous.

When a source of knowledge in your code is one of many ways to do the same thing it is not authoritative. What this means practically is that the writers of code can not come to rely on the outcome of your code. Each implementation may come with different nuances or bugs. This will require the consumer of your code to also understand the differences in how your code executes from how the other code that does the same thing executes. Having one representation of the source of knowledge in your code means that source is authoritative whose outcome can be relied upon.

One of the positive side effects of the DRY principle is that it will force you to organize your code better. You'll start to organize your code such that a change in one area of code will not cause other non-related areas to also change.

Software Craftsmanship: You aren't going to need it

One enemy of software maintainability is the tendency of software engineers to add functionality in the system which does not provide any direct business value or is not part of the requirements. This functionality can be user facing but is often functionality that comes about because the engineer goes through some sort of "what if" scenarios.

For instance if an engineer is writing a piece of software that processes PDF documents they may be tempted to add some hooks that will allow it to process yet to be decided upon documents in the future. Often the reasoning for this is that it provides greater flexibility for the software and you should do that work while you're already in that particular module.

I honestly think that one big reason we see engineers do this is that they're afraid to only provide the business value they're being asked to provide. I believe that often these engineers feel that if they're not building their software to do more than it was originally designed to do that they're not a great developer.

I'm not 100% sure where this mentality came from but my hunch is that it was exasperated in the late 90's and early 2000's when the majority of companies were using the Waterfall design process to build their software. This was due to the fact that a lot of software was being packaged and shipped rather than being distributed as a service.

The waterfall process takes the approach that there is a start and end for a software project and once the end is reached the software design doesn't change. One reason this process fails for software is that requirements are always changing. The more a software product is used the more it's users refine their understanding of how they need to use the software and what they can do with it.

Because software was being packaged and shipped using the waterfall process this promoted the need to get as much functionality into the code as possible in order for the software to be able to provide the unknown functionality to the user. The problem with this is that the software no longer does a few things really well but instead does several things not so well.

Now that we're well into an age in software where even packaged software can be updated automatically online there's really no excuse or reason to hold on to this fallacy. Yet we still see it just as prevalent in today's software industry as it was back in the late 90's and early 2000's.

There's a philosophy that's been created to try to combat this archaic way of thinking which is summed up in the acronym YAGNI (you aren't gonna need it). YAGNI is your permission to build something that does only what it's supposed to do and does it well.

YAGNI is built on the assumption that if you need it you'll know enough about it to build it out fully. You'll have enough information to write some tests to document it's functionality. It'll be valuable to the business which means it will be used and relied on.

Software Craftsmanship: Simplicity

In my previous post Software Craftsmanship: The need to understand maintainability I made the case for needing to understand software maintainability. One of the cornerstones to maintainability is simplicity. The simpler a design or implementation is the less difficult it is to maintain as its intent and purpose is clear to the maintainer.

In the software world we have an acronym which jovially sums up the desire to keep a design or implementation simple. That acronym is KISS which stands for keep it simple stupid. While this is an oft used acronym I rarely run across a user that describes what they mean as they're using it.

So what does it mean to keep it simple? While most people I've worked with over the years have had slightly different opinions on simplicity I do think it's possible to come up with some suggestions that most of us can agree on.

• Use meaningful names
• Reduce the number of lines of code and nested statements
• Test your code

Meaningful Names

Take this snippet of code as an example

for (int i=0; i < j; i++)
{
    this.process(a[i]);
}

What's that code doing? No matter how much time you spend looking at it your guess is just that... a guess. How comfortable would you be if I asked you to change it?

Now let's just change the names of things and see how much simpler we can make this code.

for (int dinnerGuest=0; dinnerGuest < numberOfGuestsToNotify; dinnerGuest++)
{
    this.notifyDinnerGuestOfVenueChange(guests[dinnerGuest]);
}

Now tell me what that code is doing. I bet you figured it out in less than five seconds. What if I asked you to modify this code now. Would you feel more comfortable modifying it?

Reducing The Number Of Lines Of Code And Nested Statements

The more lines of code you have to read at any one time means that you have to keep the entire context of that code in mind when trying to modify it. This is especially difficult to do when the code is doing multiple things. Every loop and every conditional forces the modifier to have to try to keep state in his/her mind while tracing through all the possible code paths. The more conditionals and loops in any set of code the more possible paths the maintainer is going to have to juggle.

One way to keep a complex algorithm simple is to encapsulate the pieces of the algorithm into their own methods. Separate out each conditional or body of a loop into it's own method and you're helping the maintainer to understand the flow of the algorithm without having to already know the algorithm. The benefit of this is that when going into a piece of code to maintain it the code provides a more readable and understandable blueprint for the maintainer. This increases the likelihood of a successful modification without any new bugs.

Test Your Code

One of the best ways I know of to keep code simple is to test it. Using a technique called Test Driven Development (TDD) allows you to think about your code from the standpoint of the business value it's supposed to provide first and it's technical merit second. Technical implementation is EXTREMELY important but it is secondary to providing business value.

Once your code can provide business value and you have a way to ensure that it always provides that business value (i.e. your tests) you are free to improve upon the technical implementation using a technique called refactoring.

While this has not been an exhaustive list of all the things you can do to keep your software simple my hope is that it provides you with direction on where to start.