Generic Software Solutions Are Unicorns

How many times have you been working on a platform and recognized the need for some underlying technology that ties together several disparate components in your system. The conversation usually goes something like this.

We've been building component A for some time and we need to build component B and C. A's been doing well but has a lot of warts and tech debt that we'd like to fix. If we only had a generic system X then we could build B and C faster and reduce redundancy between A, B, and C. 

On the surface that reasoning seems sound. You're not dreaming up an ideal system and looking for problems it can solve. You have real world problems you're trying to solve for components A, B, and C. So what's wrong with setting out to build a generic system to solve your problems?

You aren't providing business value along the way

By building a new generic system instead of refactoring your existing system you're increasing you're opportunity loss costs. Every day that you spend building software that doesn't get into customers hands is a day of missed opportunity for feedback. You aren't able to get feedback on whether your solution actually fits the customer need. You aren't able to get feedback on what the unknowns in your system are that only come out in production. You aren't able to get feedback to validate the many assumptions you've had to make along the way. Building successful software requires a closed feedback loop between you and your customers. The longer you go without closing that loop the more risk you are adding that you're building the wrong thing.

Generic systems come with too many unknowns

A generic system needs to be able to solve problems it doesn't even know about. But often, these aren't problems that you have today or will likely have in the near future. Building your system in such a way that it can be delivered in small pieces that provide immediate business value allows you to make sure you're solving the correct problems today, and building a foundation that can be refactored and abstracted tomorrow to solve different problems that are unknown today.

It's easier to do the wrong thing than it is to do the right thing

When building a system for which you don't know the full requirements it's easier to do the wrong thing than it is the correct thing. That's because you're possibilities are almost infinite on what you could build, and at best you just have a educated guess as to what you will need to build in the future. This means that you're going to have to make assumptions that can't be validated. It's very possible that you make design decisions based on these assumptions that are difficult to go back and change. Building only the components you need today ensures that your system only needs to change for the problems you have, not the ones you think you're going to have.

Refactoring existing systems allows you to get the more generic solution that you need 

As you build the systems and components you need, you'll be able to start identifying the parts of your system that are needed across components. These common components are the real foundation of the more generic solution you need. Building them to solve your specific problems today means that they will only do what they need to do (i.e. you won't have spent time building features that aren't used). This will create a virtuous cycle of refactoring and adding more functionality as you need it. You'll get the correct level of abstraction for your components because you'll only be abstracting them when you have a real world need for the abstraction.

Transitioning to a professional software development role: part 3

In my first post in this series, Transitioning to a professional software development role: part 1, I started to outline some of the gaps I've seen in people's preparation for entering a career in the software development industry. I started off by focusing on what software development is not about.

In my second post in this series, Transitioning to a professional software development role: part 2, I took a look at what software development IS about. In the final post in this series I'd like to talk about the tools available that make us more efficient.

Being a good software developer means understanding how to apply agile

For a long time developing software was very much like developing a product on an assembly line. Assembly lines are very rigid and not well suited to respond to change. They run on the assumption that what happens upstream in the assembly line can be built upon and won't change. The moment change is introduced most of the product on the assembly line is ruined and must be thrown away.

Software's assembly line is called Waterfall. Overtime we've come to understand the downfall of waterfall and it's major flaw is that it's very rigid to change. Rigidity to change was okay when the primary delivery mechanism for software was the compact disk. But as software has grown to allow near real time delivery of features and functionality Waterfalls rigidity to change has become a hindrance to delivering high quality software in smaller but more frequent updates and features.

That's where Agile come in. Agile software development is about being able to respond to change in a rapid manner. It teaches us to think about software in a less monolithic manner but instead as a group of features that can be delivered in small chunks frequently over time.

I wrote a post several months ago called Software Craftsmanship: Project Workflow. If you're new to agile it's a good introduction to the anatomy of a project and what I've found useful. While the project workflow I've outlined isn't something you'll see in official Agile books, it is something that I have found extremely useful.

Being a good software developer means understanding how to use Lean

The concept of Lean Manufacturing was invented at Toyota. The primary goal was to reduce waste in the manufacturing cycle. This was done by re-thinking the manufacturing process to identify and remove waste. On example of waste could is parts sitting in a queue waiting to be processed. Toyota was able to show that by re-engineering their manufacturing process they could improve quality, efficiency, and overall satisfaction of customers.

The concepts behind Lean Manufacturing can also be applied to software development. Unfortunately these concepts often are applied incorrectly and have lead to many misconceptions and misunderstandings of Lean Software development. I wrote a post several months ago which outlined common misunderstandings in applying Lean to software development. 

As a professional software developer it's important to understand Lean and how to apply it to developing software.

Being a good software developer means understanding how to make trade-offs 

The last area I want to briefly cover is understanding how to make trade-offs. As a professional software developer you're going to be asked to make trade-offs all the time. Sometimes it will come in the form of quality (a bad trade-off IMO). Other times it will come in terms of features.

The key to understanding how to make trade-offs is learning to ask a few questions.

• What am I gaining by making this trade-off?
• What do I not get that I would gotten if the trade-off was not made?
• What downstream affects will this decision have on my long term strategy or road map?
• What additional work will be required later as a result of this trade-off?

The ultimate goal in software development is to provide business value in every part of the process. Understanding how to make trade-offs will help you provide the right business value at each step in the process.

The fallacy of the re-write

I've been in the software industry for a decade and a half and have worked on dozens of projects. Many of the systems that I have worked on were considered legacy systems. As with any system, but even more so with legacy systems, developers will get frustrated with the systems inflexibility. And inevitably this will lead to the developers decreeing that if they could only re-write the system all the problems will be solved. Unfortunately most product owners will eventually give in to these cries and will commission a re-write.

I'm here to tell you today (as both a developer and a manager) giving in to this urge IS NOT going to solve your problems. What it is going to do is grind your production to a halt and make your customers unhappy. This will have downstream effects on the team as the pressure to produce builds and builds and builds.

So why is a re-write not a viable solution?

Re-writes are usually based on a few commonly held (but false) beliefs in the software industry.

• If we start the project over from scratch we won't carry the problems from the old system into the new.
• If we start the project over from scratch we can use the latest and greatest technologies that are incompatible with our current technology stack.
• If we start the project over from scratch we can move faster and produce results quicker.

Why are these fallacies? If we dig a little deeper we will see that a ground up re-write means you are more likely to introduce problems in the new system than you are to solve problems in the old system. What is typically glossed over is the fact that the current architecture is doing a lot of stuff correct. How do I know this? Because it's the architecture that is in production right now running your business.

Let's take them at each of these fallacies one by one.

If we start the project over from scratch we won't carry the problems from the old system into the new.

This statement can really be broken down into two parts. The first part says that there are problems in the architecture that prevent you from extending the code and because you're now aware of those problems you can re-architect the software so that those problems no longer exist. The second part says that you won't carry over existing bugs into the new system. The second part of this statement is really related to the second fallacy, so we'll cover it when we cover that fallacy.

Because it is true that re-writing a system with knowledge of the current architectural problems can help you avoid current pain points most people are quick to accept this statement without challenge. There are many different times in the life-cycle of a product when problems arise. Some arise as bugs when writing the software. These can typically be rooted out with some sort of unit testing. The next class of problems crop up when integrating each of the pieces of the system together. You can create integration tests to help reduce the amount of integration bugs but often there are integration bugs that don't show up in pre-production environments. These tend to be caused by the dynamic nature of content. Because the new system is a re-write of the old system it will be more difficult to use real inputs/outputs from the old system to test the integration of the new system. Because of this you're likely to introduce problems in the new system that don't already exist in the old system. Because the new system won't be in production till it's done, these new architectural problems are not likely to be found till your new system is in production.

If we start the project over from scratch we can use the latest and greatest technologies that are incompatible with our current technology stack.

On the surface this statement is likely true. What this statement hides is similar to what's hidden in the previous statement. New technologies mean new bugs and new problems. Again it is likely that many of these problems won't surface till the new system is in production because, as anyone who has worked in the industry for at least a few years knows, production traffic is always different from simulated traffic. You run into different race conditions and bugs simply because of the random nature of production traffic.

If we start the project over from scratch we can move faster and produce results quicker.

The final fallacy is usually the one that most companies hang their hat on even if they acknowledge that a re-write from the ground up will introduce new bugs and problems and re-introduce existing bugs and problems. The reason is because they believe that their knowledge of the existing system should help them to only solve problems that need to be solved which leads to the system being built much faster.

The fallacy in this statement is more subtle but much more severe than the others. The reason is because until your new system performs all functions of your old system, the old system is superior from a business value perspective. In fact it isn't untill the new system has 100% feature parity with the old system that it starts to provide the same business value as the legacy system, not to mention more business value. Some will try to gain business value from the new system earlier by switching over to the new system before there is 100% feature parity with the old system. But by doing this you're offering your customers less value for the same amount of money, time, and/or investment.

This visual does a good job of illustrating the feature parity problem.

What is the solution then?

Are you saying I'm stuck with my current architecture and technology stack? NO! The best way to upgrade your technology stack is to do an in-place re-write. By doing this you help mitigate the problems presented in a ground up re-write. What does an in-place re-write look like?

By segregating and replacing parts of your architecture you're reducing the surface area of change. This allows you to have a well defined contract for both the input and output of the system as well as the workflow.

In-place re-write has another huge benefit over ground up re-write. It allows you to validate your new system in production as you would any new feature of the system. This allows you to find bugs sooner as well as validate the workflow and feature parity.

Another benefit of an in-place re-write is that you can decommission parts of the legacy system as you go without ever having to do a big (and scary) "flip of the switch" from the old system to the new system.

Most importantly, your customers do not suffer when you do an in-place re-write as you are not ever taking away features from your customers. Even better, you can prioritize giving your customers new features earlier by implementing them on the new system even before you've finished porting the entire old system over.

Continuous Delivery

Continuous Delivery is an important topic in the software world, and particularly the DevOps movement. A lot of the talk around continuous delivery centers around the technical side of achieving it. But there's another aspect that needs to be considered first before it can truly be achieved. That aspect is the cultural changes that need to happen within an organization in order to truly achieve continuous delivery. These cultural changes are not a pre-requisite for continuous delivery, but you should be aware that they do accompany it.

The follow are the five areas that I believe are required for any organization to be successful at continuous delivery/deployment. Most organizations won't be starting at ground zero but being aware of the cultural change that will come can help accelerate a move to and adoption of continuous delivery.

Cross Discipline Teams

Development and operations need to work together. Ideally, they're both represented on the same team. One of the main goals of continuous delivery is to identify defects earlier in the pipeline and to provide business value as early in the pipeline as possible. The best way to achieve this is to think about delivery from the beginning. The project team should be planning the deployment even before the code is written that they'll be deploying.

Raise visibility

For every change the project team should understand how the change can be measured. Here are a few simple questions that you can ask as part of the dev cycle:

• What metrics are associated with this change and how?
• What are the risks?
• Is the performance profile expected to change?
• How do we measure success for this change?
• What are the signs that the change is not effective?

Build trust

Operations and development need to know that enabling the business is the #1 priority for BOTH units. No change should make it's way into the deployment pipeline unless it provides business value. If both units understand this then we have a shared understanding of the importance of each change and a shared desire to get that change out.

In order to build this trust the operations folks need to know and believe that the development folks will include them in feature discussions. Having operational discussions early and often in the conversation will help ensure successful deployment of the change.

On the flip side the development folks need to know that the operation folks are including them infrastructure discussions early and often as well. Again ideally, these folks work on the same team and these discussions are part of the normal standup discussions.

Minimize risks 

We need to understand that failure happens. It's how you deal with failure that leads to success or failure for the change. It's naive to believe that failure can be prevented 100% of the time.  Because we know it can't our goal is to minimize the risk of any change should it fail.

In order to minimize the risk to the overall system for any given change should focus on reducing the overall surface area of the change. The smaller the change the less risk there is of a failure associated with that change being catastrophic. The longer the time between releases the larger the risk of that change. The more code that's released with every change the larger the risk of failure or bugs.

In order to respond to change effectively we need to build a culture that responds to failure well instead of trying to prevent it. The easiest way to respond to failure well is to practice it. Build failure tolerance into your system. Understand how your system responds to failure and identify the expected and appropriate user experience when failure happens.

Shared responsibility

We all succeed or fail together. It's not possible for operations to succeed and development to fail (or vice versa) and have the product succeed. Both disciplines need to succeed in order for the business to truly succeed.

Once we've grasped the concept that we succeed or fail together we can create a culture where it's okay to talk about and address our mistakes. We try to avoid finger pointing and are able to do this because people own up to their mistakes and take responsibility for them. They participate in postmortems where they create action items to address the failure and prevent failure of that kind in the future.

Having shared responsibility also means that we treat rollback as a last resort. When a problem arises with a particular change we stop the line. We work together to figure out the problem when it happens and we work together to fix the problem before we move on to making additional changes in the system.

Sharing responsibility means that both the operations folks and the development folks should be on-call for incidents. Nothing provides better incentive to fix a problem than getting woken up in the middle of the night or interrupted on a Saturday.

Coding Standards Revisited: My Language Agnostic Coding Standards

In my previous post, Coding Standards Revisited: Writing Code That Lasts, I talked about how to approach a code review from a slightly different perspective. Today I'd like to talk about approaching coding standards from a slightly different perspective.

As I mentioned in my previous post I find a lot of value in framework, language, or platform specific coding standards. But I do not believe they tell the whole story. Here's a short list of non-traditional coding standards that I believe apply to all code on all (modern) frameworks, languages, and platforms.

Naming


Use meaningful variable, method, parameter, and class names. DO NOT use shorthand. A person that does not know how to read code should be able to tell you what the variable, method, parameter, or class is doing just from the name.

Separate out your concerns

A method or class should have one reason to change and ONLY one reason to change. If a particular method in that class has more than 10 - 20 lines it's probably doing too much. There are very few exceptions to this. Methods and classes should be distinct features that overlap in functionality as little as possible.

Cohesiveness

Write highly cohesive classes. The methods in a class need to have a lot in common. The methods in a class should act on the same data. The methods in a class should all be related to the data. I.e. an image manipulation class that has code to physically save the image to disk is not cohesive… it's coupled. There should be a class that deals with saving things to disk and a class that deals with image manipulation.

Dependency Management

Dependencies should be added by reference as much as possible and injected into the classes that depend on them. This allows the dependencies to be created at the correct level of abstraction. 

High-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend upon details. Details should depend upon abstractions.

Don't Start By Abstracting

Only abstract as you need too. The first (and I would argue second) implementation of something should come in the form of concrete classes. Only when you run into a scenario where you need to interchange these classes should you then abstract it.

Code Duplication

Follow the Don't Repeat Yourself (DRY) principle. Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

Commented Out Code

Don't comment code out and leave it. Delete it as you can always go back to it using source control.

Only Write Enough Code To Satisfy What You Need Now

Don't write a single line of code for something in the future (i.e. I'm gonna need a class that….). Only write code that is actually used right now for what you're doing.  Usually for me, this means starting with a test class. In the case of a UI I usually start by writing static UI Code and then working backwards making the code dynamic as I go and ensuring that each piece of dynamic code as a set of tests associated with it.

Software Craftsmanship: Project Workflow

In working through the topic of Software Craftsmanship for the past few weeks on this blog we've talked about how to approach software development from a code perspective.  I'd like to take a short break this week and talk about optimizing your project workflow in step with optimizing your code.

Over the past decade the Agile development movement has really taken over in the technology industry. You'll find many different forms of agile development used throughout the software industry. You'll even find many different flavors of the same form of agile used differently from organization to organization.

While there's no single one-size fits all solution when it comes to managing your project, it is important to have a base template to use as a frame of reference. My go to template is made of up a hodgepodge of different agile methodologies. The template is made up of three parts, and like any agile process, involves constant iteration through the parts.  This template is driven around three main concepts. First, the beginning of any project involves more unknowns and more risk than the end. Second, the process should involve creating features that are useful and complete. Third, communication, preferably face to face, is needed throughout the process.

Ideally this template is used in conjunction with some sort of Kanban process. A lot of people make the mistake of centering their process around Kanban. That in my opinion is a recipe for failure as Kanban helps you understand how to move work through a system but doesn't do a good job of helping you identify the correct work to move through the system.

Part 1: Identify The Problem Space

In this part you want to have an Initial Design Meeting. This meeting would consist of the project team, the key stake holders of the project, and any other outside tech advisers (like an architect or team lead) which may be able to provide insight into how this problem space fits in with other problem spaces throughout the organization.

In this meeting you want to Identify The Unknowns. The first unknown is what we're building. A quick discussion about what the problem is or what features the system is missing. Once we know what the problems are that we're trying to solve we can start creating straw-men for possible solutions. A good way to flesh this out is to create a few user scenarios that represent the desired workflow for solutions to our problem space. Questions to ask when evaluating the straw-men are:

1. What are the big open questions?
2. What risk do the unknowns represent?
3. What's in scope?
4. What's out of scope?

Part 2: Identify The Features

This next phase is all about defining the Minimal Viable Products. This is key as we want/need to know how we can provide some sort of business value as early in the project as possible. There are a couple of characteristics that I use when trying to determine if something is an MVP that I think are useful:

1. Can the MVP stand on it's own. I.e. if the project were to end or the company were to switch gears for a time, would the MVP still provide value to the business or would it need subsequent development in order to be of use?
2. Is success clearly defined for the MVP? Do we have a clear idea of what the definition of done means for the MVP?
3. Is the MVP achievable in a reasonable time frame?
4. Have we identified the external dependencies of the MVP?

When we're done identifying the MVP's we can organize them by things we can work on now and things that need more information to get started. Each MVP or a set of MVP's can constitute a project milestone.

Part 3: Create stories

During this step in the project workflow the team will get together and identify the stories associated with each milestone (or at a minimum the first milestone).  Take a gander at this article if you're unfamiliar with user stories. Breaking work out into individual stories helps us to keep focus on delivering on our milestone.

Once we've identified our stories we need to identify their dependencies and possibly associate some of the stories with other milestones if they have dependencies that can't be resolved early on in the process.

Lastly, in this part we create "bite sized" tasks for each story. This can be done all at once or, more preferably, at the beginning of each milestone.

Starting the work

Once we've gone through these three parts our teams can start their work. In order to be truly effective and stay agile, the team should work on one milestone at a time and not move on to next milestone till all stories for the current milestone are complete. During the teams daily stand-ups you'll want to report the status of each task/story and identify any of the blocking items that need to be resolved.

Software Craftsmanship: Going Dark (Or why you need a flashlight)

Over the last few places I've been employed at I've given a similar presentation on the topic of Going Dark.  I think it's time to unleash this topic to all three of you out on the interwebs. The term Going Dark may or may not be new to you. Hopefully by the end of this post you'll understand why it's bad, how best to identify it, and how best to avoid it.

Taking a look back at the last few posts I've done on Software Craftsmanship you've probably noticed a pattern emerge in what it means to treat software as a craft. I'll lump them into a category called Software Development Ideals.  So what are those ideals?

• Develop software for maintainability first
• Reduce the waste in our development cycle
• Reduce the complexity of the software
• Introduce less bugs as more features are introduced
• Write better code faster

Before we talk about how Going Dark is the antithesis of the ideals I think it's important to understand the symptoms associated with Going Dark. Those symptoms are:

• Working in isolation on a specific feature or problem for long periods of time
• Waiting to check in code till the feature is “complete"
• Failing to communicate or collaborate on software features as feature development “starts going bad”
• Constantly having to explain your work before and after you commit it to your repository
• Leads, management, or other teams members feeling like what was built wasn't quite what they expected

“Developers who work for long periods -- and by long I mean more than a day -- without checking anything into source control are setting themselves up for some serious integration headaches down the line” Jeff Atwood – Coding Horror

“Developers often put off checking in. They put it off because they don't want to affect other people too early and they don't want to get blamed for breaking the build. But this leads to other problems such as losing work or not being able to go back to previous versions.”
Damon Poole– Author of “Do It Yourself Agile”

Okay so now that we're able to identify the symptoms of someone who has gone dark. Let's talk about why not treating the problem is actually costing you lots of time, energy, and wasted cycles.  There are several problems associated with someone who has gone dark.

• There's more code to integrate on each check-in
• Code that is written in isolation is full of assumptions about the other software with which it will be integrated
• Each assumption is a potential issue that will only be discovered when the software is integrated
• The more code you have to integrate the higher the likelihood of introducing bugs
• The longer you wait to check in your code the higher the likelihood of breaking a piece of NEW code you've written before you check it in
• When you're on the wrong path you end up spending more time going down that path rather then being able to have your path corrected sooner
• There's no way to "gut check" your progress
• Closes your code off from scrutiny (which may seem uncomfortable but ALWAYS makes your code better)

That list alone should scare you. My guess is that you can probably relate to it. You've probably participated in "merge parties" or spent half a day or a full day trying to re-merge your changes after someone else beat you to committing and has changed the topology of the code. Maybe you've come in one day to find that there's a new bug in the system because a merge went bad and something got missed or accidentally added.

The great thing is that there are some pretty simple changes that can be made. The first is to start working out how to make lots of little small changes in the system which can be verified along the way. One good method for doing this is using some form of Test Driven Development. Once you've started to follow down the path of thinking about your code in bite sized pieces, all of which have an associated test, you can start to think about checking in those bite sized pieces frequently. I like to think about this as Checking In Early and Often. There are a lot of benefits of checking in early and often, and you'll notice that they are solutions to the problems that are presented by Going Dark.

• Change set is much smaller which means it's much more manageable and has a lower impact on the code base as a whole
• When you start integration of your changes the chances of having changed the same method or function as someone else is much smaller
• Less of a chance of having to reapply your changes on top of someone else's just because they beat you to the check-in
• When others refactor, you obtain the benefits quicker with less integration work
• Reduces the likelihood of anyone's changes being lost or overwritten by a bad integration
• Easier to find bugs because there's less code to search through
• Easier to fix bugs because there's less code that could be possibly causing the bugs
• Helps to reduce unnecessary code (YAGNI) as you're just writing the code you need to do the task you're on

Go forth into the light and help others come out of their darkness!

Software Craftsmanship: Don't Repeat Yourself

In the software world copy and paste is the enemy. It causes bugs. It slows down development. It duplicates functionality. But more importantly it decreases maintainability.

It causes bugs and slows down development because you have multiple places to maintain the same source of knowledge. If you apply just one change to one place and not the others you will have a bug in your system. And the difficulty of finding and fixing this bug increases with every new permutation as there become more and more code paths that utilize that source of knowledge from different sources.

The antidote to our copy and paste problem can be summed up with  Andy Hunt and Dave Thomas's acronym DRY (don't repeat yourself). The DRY principle states that "every piece of knowledge must have a single, unambiguous, authoritative representation within a system." That statement is chock full of goodness that can be overlooked so let's break it down.

The simplest part of that statement is single. That one word in and of it's self sounds simple but what it really means is that we have a responsibility to make sure that our source of knowledge only has one place it exists. This doesn't mean that you create a "tools" library for all common code like is often the (misunderstood) case with people trying to implement the DRY principle. It means that you need to make an effort to constantly refactor your code such that you increase the reusability of each source of knowledge.

Each source of knowledge needs to be associated with a clear understanding of how and when to be used. When a source of knowledge in your code can be interpreted to do many things it is an ambiguous source of knowledge. Each source of knowledge should be unambiguous.

When a source of knowledge in your code is one of many ways to do the same thing it is not authoritative. What this means practically is that the writers of code can not come to rely on the outcome of your code. Each implementation may come with different nuances or bugs. This will require the consumer of your code to also understand the differences in how your code executes from how the other code that does the same thing executes. Having one representation of the source of knowledge in your code means that source is authoritative whose outcome can be relied upon.

One of the positive side effects of the DRY principle is that it will force you to organize your code better. You'll start to organize your code such that a change in one area of code will not cause other non-related areas to also change.

Software Craftsmanship: You aren't going to need it

One enemy of software maintainability is the tendency of software engineers to add functionality in the system which does not provide any direct business value or is not part of the requirements. This functionality can be user facing but is often functionality that comes about because the engineer goes through some sort of "what if" scenarios.

For instance if an engineer is writing a piece of software that processes PDF documents they may be tempted to add some hooks that will allow it to process yet to be decided upon documents in the future. Often the reasoning for this is that it provides greater flexibility for the software and you should do that work while you're already in that particular module.

I honestly think that one big reason we see engineers do this is that they're afraid to only provide the business value they're being asked to provide. I believe that often these engineers feel that if they're not building their software to do more than it was originally designed to do that they're not a great developer.

I'm not 100% sure where this mentality came from but my hunch is that it was exasperated in the late 90's and early 2000's when the majority of companies were using the Waterfall design process to build their software. This was due to the fact that a lot of software was being packaged and shipped rather than being distributed as a service.

The waterfall process takes the approach that there is a start and end for a software project and once the end is reached the software design doesn't change. One reason this process fails for software is that requirements are always changing. The more a software product is used the more it's users refine their understanding of how they need to use the software and what they can do with it.

Because software was being packaged and shipped using the waterfall process this promoted the need to get as much functionality into the code as possible in order for the software to be able to provide the unknown functionality to the user. The problem with this is that the software no longer does a few things really well but instead does several things not so well.

Now that we're well into an age in software where even packaged software can be updated automatically online there's really no excuse or reason to hold on to this fallacy. Yet we still see it just as prevalent in today's software industry as it was back in the late 90's and early 2000's.

There's a philosophy that's been created to try to combat this archaic way of thinking which is summed up in the acronym YAGNI (you aren't gonna need it). YAGNI is your permission to build something that does only what it's supposed to do and does it well.

YAGNI is built on the assumption that if you need it you'll know enough about it to build it out fully. You'll have enough information to write some tests to document it's functionality. It'll be valuable to the business which means it will be used and relied on.

Putting your Software Development Lifecycle on a diet

I was re-reading Mary Poppendieck's Lean Programming Essay and I thought I would take the opportunity while it was fresh in my mind to walk through a few points that really stuck out to me that are commonly misunderstood. I've often seen developers that want to write LEAN software struggle when trying to apply OOP principles like abstraction.

Lean Rule #3: Maximize Flow (Drive Down Development Time)

The basic premise of iterative development is that small but complete portions of a system are designed and delivered throughout the development cycle, with each iteration adding an additional set of features.

In LEAN abstractions aren't bad.  But they're a by product of building small complete portions of a system.  In one iteration you build a complete feature. At this point in the process the YAGNI principle (You Aren't Going To Need It) is applied to any abstractions. There isn't a business case which says that a feature is incomplete without the abstraction. At some later date (either right after the first iteration or some other point) in another iteration you build a feature that, either in design, analysis, or some other point, brings to light that an abstraction is necessary in order to apply the DRY (Don't Repeat Yourself) and YAGNI principles. In order to build the "complete" second feature it  becomes a requirement to refactor and add the abstraction layer. At this point you only add things to the abstraction which are 100% used and required by the first and second iteration features. Adding any additional functionality or abstraction would be a violation of YAGNI. The key principle from LEAN, refactoring, becomes the driving force for the abstraction during the second iteration.

Lean Rule #4:Pull from Demand (Decide as Late as Possible)

Software systems should be designed to respond to change, not predict it.

In the context of abstractions, building an abstraction before you have at least two uses is an attempt to predict change. When you only have one required use case for something you don't have enough information to make a decision about how the second, third, fourth, or Nth use case will use the abstraction. In my experience I would actually argue that you don't really have enough information to abstract until you have a third case, but definitely not before you have a second.

Any abstraction that's built not as a direct result of refactoring would either be built to guide use of the functionality or feature (which goes against the LEAN principle of allowing requirements to change) or to anticipate future use. Anticipating future use is tantamount to trying to predict change which also makes it so that you're putting a limit on how the requirements can change. The ability to constantly change requirements is something that LEAN values tremendously.

Lean Rule #8: Abolish Local Optimization (Sub-Optimized Measurements are the Enemy) “Do it Right” requires that we provide for change.

Lean Programming employs two key techniques that make change easy. Just as Lean Manufacturing builds tests into process so as to detect when the process is broken, Lean Programming builds tests into the development process in order to ensure that when changes don’t inadvertently break the code. In fact, the best approach is to write the tests first, and then write the code. An excellent unit and regression testing capability is the best way to encourage change late in the development process.

The second technique for allowing change to happen late in development is refactoring, or improving the design of existing software in a controlled and rapid manner. When refactoring is an accepted practice, early designs can focus on the issue at hand rather than speculate as to what additional design elements will be needed. As the additional features are actually added, refactoring provides a new, simplified design to handle the new reality. When refactoring is a part of the process, we reduce speculation as to what will be needed in the future by making it easy to accommodate the future if and when it becomes the present.

This last rule really ties #3 and #4 together.  If we have tests for the functionality and regression then we have the ability to change the system at will (including the need to add abstractions) while only having to refactor the tests for the functionality that's changing (adding abstraction is a change).  It allows you to be explicit about the change and the need for the change. If you use TDD then you're ensured that not only does your functionality after the refactor work as expected, but you've also guaranteed that the system is interacted with as expected. The second technique mentioned is really key to the guidance on when/how to add abstractions. The sentence "As the additional features are actually added, refactoring provides a new, simplified design to handle the new reality." really becomes key.

Hopefully this helps you feel good about pushing off adding an abstraction into your system until after you actually have a need for that abstraction. Doing so will allow you to write software that is more adaptable to change.