Generic Software Solutions Are Unicorns

How many times have you been working on a platform and recognized the need for some underlying technology that ties together several disparate components in your system. The conversation usually goes something like this.

We've been building component A for some time and we need to build component B and C. A's been doing well but has a lot of warts and tech debt that we'd like to fix. If we only had a generic system X then we could build B and C faster and reduce redundancy between A, B, and C. 

On the surface that reasoning seems sound. You're not dreaming up an ideal system and looking for problems it can solve. You have real world problems you're trying to solve for components A, B, and C. So what's wrong with setting out to build a generic system to solve your problems?

You aren't providing business value along the way

By building a new generic system instead of refactoring your existing system you're increasing you're opportunity loss costs. Every day that you spend building software that doesn't get into customers hands is a day of missed opportunity for feedback. You aren't able to get feedback on whether your solution actually fits the customer need. You aren't able to get feedback on what the unknowns in your system are that only come out in production. You aren't able to get feedback to validate the many assumptions you've had to make along the way. Building successful software requires a closed feedback loop between you and your customers. The longer you go without closing that loop the more risk you are adding that you're building the wrong thing.

Generic systems come with too many unknowns

A generic system needs to be able to solve problems it doesn't even know about. But often, these aren't problems that you have today or will likely have in the near future. Building your system in such a way that it can be delivered in small pieces that provide immediate business value allows you to make sure you're solving the correct problems today, and building a foundation that can be refactored and abstracted tomorrow to solve different problems that are unknown today.

It's easier to do the wrong thing than it is to do the right thing

When building a system for which you don't know the full requirements it's easier to do the wrong thing than it is the correct thing. That's because you're possibilities are almost infinite on what you could build, and at best you just have a educated guess as to what you will need to build in the future. This means that you're going to have to make assumptions that can't be validated. It's very possible that you make design decisions based on these assumptions that are difficult to go back and change. Building only the components you need today ensures that your system only needs to change for the problems you have, not the ones you think you're going to have.

Refactoring existing systems allows you to get the more generic solution that you need 

As you build the systems and components you need, you'll be able to start identifying the parts of your system that are needed across components. These common components are the real foundation of the more generic solution you need. Building them to solve your specific problems today means that they will only do what they need to do (i.e. you won't have spent time building features that aren't used). This will create a virtuous cycle of refactoring and adding more functionality as you need it. You'll get the correct level of abstraction for your components because you'll only be abstracting them when you have a real world need for the abstraction.

What's the measure of success for your project?

When I first started in the software industry I believed that the measure of success for any project was whether or not it shipped anywhere. One of the biggest surprises to me transitioning into the industry was how much software never actually ships. It baffled my mind (and still does) that people could work for weeks, months, or even years on software that would never ship.

What I've learned though in my career is that shipped software is not the right measure of success. USED software is the correct measure of success. No, not re-sold software. I mean software that is being used in the real world by real people/processes to solve real world problems.

Simply shipping software is not enough. Why? Because shipping software that is not used is the same as not shipping the software in the first place. Anecdotally, the top three reasons for software not shipping in my experience have been:

1. The team never reaching feature complete.

2. The project running out of money.

3. The requirements changing so fundamentally that it made sense to just build something else.

#1 and #2 are closely correlated in that #1 eventually leads to #2. All 3 of those situations are symptoms of having a broken process. So how do you fix your process?

Get a product owner

All good (i.e. used) software has a product owner. Someone who is responsible for making sure that the software being built is solving an actual problem and has people who need the problem solved. A good product owner doesn't search for a customer for the software. Instead a good product owner builds software that has people who already need it and figures out how to get it in front of them.

Focus on building individual features rather than a breadth of features 

All too often in the software industry features get built because of some gut feeling of some higher up in the organization. Don't build software this way, it's the easiest way not to have real users. Instead focus on a small feature set (one feature if at all possible) and build it, ship it, and iterate on it.  

Once you've got people using a feature you'll know the correct next feature to build because your customers will ask you for it. The key is, your existing customers. The ones that rely on your software to accomplish some task, will ask you to build something that they need/want. 

Release early, and often

I've talked about this many times but it bears repeating. The only way to build the correct thing is to get feedback on what's being built as soon as possible. The only way to do that is get something in your customers hands as soon as possible, gather feedback, and pivot in response to customers needs. The only real way to do that is to release early and often.

The Unlearning Of Technology

I grew up in the Washington D.C. area. I grew up a Redskins fan (whose name I DO believe should be changed). That also meant I grew up hating the Cowboys. My family (and extended family) had season tickets and I could go to 2 - 8 games a year. When not at the games I would watch them at home. I was almost religious in my dedication. I could tell you the names of almost every starter and a good portion of the bench.

I moved to Seattle 10 years ago and now I couldn't name 5 people who play for the Redskins. I'm not quite as fanatic a Seahawks fan as I was a Redskins fan, but I watch every game, go to as many as I can, and can tell you their record, their standing in the league and who the key players are.

Did I set out to become a Seahawks fan and not a Redskins fan? No, I just like football and fell pray to convenience and repetition. Not having Redskins games readily available made it hard to stay up with the current happenings. Not watching games each week I wasn't able to learn what the team was doing well today, learning where their defense was weak and their offense strong. Essentially, I was spending time unlearning what I knew about the Redskins.

The same is true in the software industry. You're only going to know the ins and outs of platforms, frameworks, and code that you actually spend time with. If you don't spend time with it you'll evolve and change on separate paths.

Given enough time, eventually you'll grow apart from the technology. You'll stop recognizing the subtle changes from version to version. You'll stop being able to speak to the differentiation of the platform or framework. You'll make stupid, or novice mistakes when you do dive in. You'll miss important aspects of design, architecture, or code quality.

This isn't always a bad thing. Sometime, in order to learn something new, you need to unlearn something you knew well. But it's important to recognize that if you're not actively learning and growing with a platform, framework, or code base, then you're actively unlearning it.

What are you un-learning today that you shouldn't be?

Agile Development: Demo Day

So far in this Agile Development series I've:

• Made the case for Scrum
• Defined who and what a sprint team is
• Walked through the anatomy of a sprint
• Provided tools for determining sprint duration
• Detailed what daily stand-up looks like
• Provided the typical workflow for planning a sprint
• Outlined the sprint retrospective

In my final post in this series I'll explain demo day and how it relates to agile development.

What Demo Day Is

As I've been saying all along in this series, one of the main objectives of Agile is to reduce the time it takes to complete the feedback loop. The loop is complete when the stakeholders have reviewed the feature(s) and signed off on them. Demo day closes the loop and provides the stakeholders with an opportunity to affect change.

I like to think of demo day as the bow that is tied around the sprint. It brings closure to the sprint process which started with sprint planning. At it's core the sprint demo is the teams opportunity to showcase what they've built during the sprint. Beyond that sprint demos can also be a useful tool leading into the next sprint planning phase as the product manager may change the plan for the next sprint based on the progress of the team.

There are two main parts to demo day. First is to review the work completed and not completed during the sprint. Second is to demonstrate the completed work to the stakeholders.

Reviewing Work

Reviewing the sprint work should be comprised of three things. First the team should very briefly call out what stories the team accepted into the sprint during planning. Second the team identifies what stories were added to the sprint after the sprint started. Finally the team goes over what stories were not completed during the sprint.

It's important to note that the review is not an exercise in excuse making or justifying not getting everything done. It's main purpose is to close the feedback loop with stakeholders and set expectations prior to heading into sprint planning.

The Demonstrations

Demonstrations are just that, demoing what the team built during the sprint. Typically the demo is limited to the user impacting aspect of the features the sprint team worked on during the sprint. The team reviews the features to ensure they are complete as well as provide product management the opportunity to get clarifications on behalf of the customer.

One mistake I've often seen people make is trying to "demo" spreadsheets or powerpoint bullets. This usually happens because the team was mostly in bug fix mode or works on non-user facing features. I would encourage teams in this situation to get creative and tell a story using charts, graphs, and metrics instead of showing data in excel or a list of powerpoint bullets.

Agile Development: Sprint Retrospective

In my previous post in this Agile Development series I detailed the typical workflow for planning a sprint. In this post I will outline the process of the sprint retrospective. One helpful way to think about the sprint retrospective is as a mini postmortem. During the retrospective the team goes over what went well, what didn't go well, and what the team could have done better.

What Went Well

The reason we ask what went well is not so that we can pat each other on the back and inflate our egos. One of the keys to a successful agile project is consistency in the velocity of the team sprint over sprint. One of the best ways to do this is to identify what lead to success so that the team can capitalize on it in the upcoming sprint(s).

What Didn't Go Well

As with most growth being realistic, open and honest about our mistakes and failures leads to not repeating them. The purpose of identifying what didn't go well is NOT to lay blame on any particular person or people. Identifying what didn't go well helps us to avoid or correct them those mistakes.

What Could Be Done Better

At first this may not seem different from identifying what didn't go well. The real difference is that we recognize things that may have gone well but could have gone better. Asking what we could do better is an opportunity to identify gaps in our planning, our process, and our estimates. This question should lead directly to actions that can be taken to increase or maintain the teams current velocity.

Because the sprint review typically takes place before the planning of the next sprint asking these three questions provides the team with the opportunity to make changes that give them a higher likelihood for success before the next sprint starts.

Agile Development: Planning A Sprint

In my previous post in this Agile Development series I walked you through the daily stand-up. In this post I'll go outline the typical workflow for planning a sprint.

Identify The Features

The product owner, who is serving as the voice of the customer, proposes the highest priority features. The scrum team is responsible for getting clarification, pushing back on features, and proposing features or work that should be accomplished. Features should be prioritized based on their overall business value. It's important to note that working on tech debt can often provide down stream business value by enabling new feature work.

Prioritize and Estimate

Once the scrum team has identified the features to work on in the sprint, the team should then pull all of the stories associated with that feature into the sprint backlog. The product owner and scrum team should negotiate the priority of the stories. Once the priority of the stories has been determined the team needs to estimate the level of effort for each individual story.

One typical way of estimating stories during planning is to pick a story that does not have a lot of ambiguity which represents a medium (tee-shirt sized) amount of work. The team then assigns that story a point value (the middle value of their pointing system). The team then moves on to point the rest of the stories based on them being either more effort or less effort than the original story. Stories that are more effort are assigned a higher point value while stories with lower effort are assigned a lower point value.

Determine What Stories Make It Into The Sprint

Just because the product owner or scrum team has identified a feature (and it's related stories and bugs) as high priority doesn't mean that the team can actually complete that feature in a sprint. The team needs a way to estimate the amount of work that can be accomplished by the team during the sprint. This is typically done through velocity tracking.

Velocity in agile is typically determined by the amount of points the team was able to accomplish in the previous sprint. Historical sprints aren't to be taken into account as each sprint the teams goal is to get better and better as estimating the level of effort for story work.

The sprint backlog should only contain enough stories to meet the teams velocity. Choosing to few story points means that the team will likely have to poach more work (which hasn't been vetted) from the product backlog. Choosing too many story points means the team is not likely to finish all the work in the given sprint, setting the team up for failure from the start.

At first the teams velocity will be all over the place as they work out their estimation techniques. But over time the team should start to settle in to a consistent velocity. It's important to be aware that there are several common practices that can negatively affect a teams velocity. It's important for the sprint team to guard against these in order to accurately gauge their velocity sprint over sprint.

Under or over estimating the level of effort of a story

Often this is the result of ambiguity that is not resolved priority to planning. If the team doesn't have enough information to accurately gauge the level of effort for a particular story the product owner and scrum master should first work to resolve the ambiguities before a feature or story is considered for a sprint. This is an example of a valid reason for the sprint team to push back on a particular feature or story.

Unplanned sprint work or re-prioritization of the sprint deliverables mid-sprint

This is an Agile no-no and should be avoided at all costs. In Agile our sprint duration is typically short enough that when new work comes up we can prioritize it at the next sprint planning meeting. If new high priority work or re-prioritization mid-sprint is truely unavoidable the best way to make sure it doesn't impact the overall sprint deliverables is for the whole team to understand what the lowest priority work is and have that work drop out of the sprint.

Unaccounted for absence

People are going to take vacation, get sick, or be off for a holiday. It's difficult to account for sick time but it is very easy to account for holiday's and vacation time. Make sure in your sprint planning you understand how many days during the sprint people know they will be out and reduce that sprints velocity accordingly.

Other regular duties

Many software organizations use the DevOps model where the developer is on-call for the software and services they own. Not accounting for this type of work can have a negative impact on planning as the sprint team will sign-up for more work than they can actually accomplish.

Break The Stories down into tasks

Once the sprint backlog has been determined the team should break the stories down into smaller more granular tasks. This allows for more story work to be done in parallel. The team can swarm on a story together ensuring that the highest priority work is done first.

Once the tasks are identified the team is ready to start the sprint!

Agile Development: Stand-up

In my previous post in this Agile Development series I explained two ways to determine sprint duration. In this post I'll go into detail on what an agile stand-up typically looks like.

One thing you've heard me mention over and over in this series is that one of the primary goals of agile is to decrease the total time it takes to complete the feedback loop. One key way to accomplish this is with a daily stand-up.

In it's most basic form the stand-up is a time-boxed meeting (typically 15 minutes) where the scrum team gets together and answers three simple questions:

1. What did I do yesterday?
2. What am I going to do today?
3. What am I blocked on?

What Did I Do Yesterday?

In it's most basic form this question is intended to determine if progress was made towards the overall sprint goals. Knowing, as a team, what has already been accomplished helps the team formulate a plan for what needs to be accomplished over the next day.

I've seen this question often turn into a check-list of everything you did during the day. But it's important to keep in mind that your goal is not to provide an overview of everything you did. Instead, the purpose of this question is to make sure that others are aware of how much progress was made towards the work you committed to the day before. This is especially important when work items in the sprint are interdependent.

What Am I Going To Do Today?

There are two reasons that telling the team what you plan to accomplish for the day is important. First, it allows the team to validate whether the work is really the top priority for the sprint. During sprint planning many teams will prioritize the backlog for the sprint. But because our feedback loop is so short the priorities of any individual task may change throughout the sprint or even daily.

The second reason this question is important is that you are making a commitment to the team to accomplish a particular task. You can think of it as your daily contract with the team. And in turn the other members of the team are making a commitment to you to get other work done. This is how the team is able to increase and maintain a particular velocity. The team is able to time and coordinate work at a granular level that allows the team to maintain a constant throughput. 

What Am I Blocked On?

Making sure that the team is aware of any potentially blocking issues allows the team to react quicker and not allow blocking issues to affect the teams throughput. Because the scrum team talks about this daily, it is able to address potentially blocking or blocking issues immediately.

If a particular team member is blocked on a technical issue, I've often seen the scrum team swarm on the issue until it gets resolved. This has the benefit of making sure any blocking issue that is an upstream dependency of other sprint tasks gets resolved quickly and the team is able to dedicate more of the sprint to work that was planned.

If an issue cannot be resolved due to some unforeseen circumstance the sprint team has the ability to adjust and pull in additional tasks into the sprint while the blocking issue is being resolved. Because we talk about blocking issues daily, there shouldn't be a significant impact on the sprint schedule.

Parking Lots

It's often the case that during stand-up an issue comes up that requires a deeper dive than the 15 minutes allotted allows for. Fight the urge to go into detail during stand-up on the issue. Instead, announce that you have a parking lot and who should attend and then table the issue until after the stand-up is over. This allows less of the team to be randomized by an issue that isn't directly related to their daily commitment. 

Agile Development: Determining Sprint Duration

In my previous post in this Agile Development series I explained the anatomy of a sprint in terms of the three different sprint phases. In this post I'll detail the two main ways sprint duration is determined.

Fixed Time

Most sprint teams you come across will use a fixed time for their sprints. In my experience 2 to 3 week sprints are the most common duration. The goal of a fixed time sprint is to shorten the feedback loop with the customer in order to deliver the right features to them. By limiting the time to 2 or 3 weeks you can easily change course as the requirements change and provide continued business value to the customer without spending a lot of time on features that won't be used or whose requirements have changed.

Having a fixed time sprint allows the team to set expectations with your customer on what new features they will be building and when they will be available to the customer. I would strongly suggest that you don't have sprints that are longer than 3 weeks in duration. 2 to 3 weeks keeps the feedback loop small while also giving the sprint team enough time to build a feature out end to end. Increasing the amount of time it takes to complete the feedback loop means that the sprint team could spend more time building the wrong product and less time building the correct product should the requirements change.

It is occasionally the case with a fixed time sprint where the sprint will end and the team will have been unable to complete a particular feature. In this case the retrospective should identify the cause of the decreased velocity. Typical causes for a decrease in velocity (in my experience) have been:

• Failure to account for vacation or known leave of sprint team members.
• Failure to account for time spent on external work (like on-call duties).
• Failure to account for the unexpected work that grows the sprint backlog.
• Undefined or not well defined definitions of done.
• Stories that are too large and hide dependencies.
• Unknown external dependencies. 

Fixed Feature

Another, less common, way to determine sprint duration is to use a fixed feature approach. This approach says that a sprint is complete only when a particular feature is complete regardless of time. This is typically used when it is known that dependencies are hidden but that it is not easy to uncover those dependencies without first doing some work towards the end goal of the feature. In a system without a well defined or understood architecture, a fixed feature sprint duration is often used as it allows for the unknowns that go with the not well defined architecture.

Earlier in my career this was a more appealing way to determine sprint duration but as I've grown in my Agile planning I've come to realize that open-ended sprint duration's actually detract from shipping software more than they contribute to shipping complete software. Even with the most ambiguous problems it is usually better to first root out the ambiguity in a fixed time sprint and then plan and work on the feature.

Another reason that I've found for fixed feature sprints being less than ideal is that you have a harder time gauging your velocity as your velocity is determined by the amount of work you can complete in a fixed amount of time. Since knowing you're velocity is key for estimating when a feature is complete without it you cannot set your customers expectations as to when they will get a particular feature.

Fixed feature sprints also increase the time it takes to complete the feedback loop. And, as I mentioned earlier, increased feedback loops mean that it takes longer to respond to customer requests.

Agile Development: The Anatomy Of A Sprint

In my previous post in the Agile Development series I walked through the makeup of a sprint team. I explained the roles of Product, Dev, QA, Operations, and the Scrum Master. In this post I will go into detail on the anatomy of a sprint.

Planning

The first phase of the sprint life-cycle is planning. During the planning phase sprint velocity is determined, features are identified, features are estimated, and the backlog is created.

At it's most basic level the velocity of the sprint is determined by the number of points that were completed during the last sprint. When a sprint team first forms their velocity is very volatile. But as the team learns to estimate their story points more accurately there is less oscillation between the number of points completed sprint over sprint.

Determining which features are candidates for inclusion usually starts with product as the voice of the customer. Product will advocate for what the customer needs. The team will also identify what bugs and tech debt need to be addressed during the sprint.

Once the candidate features are identified the team will estimate the the level of effort for each story. At the formation of the sprint team a point system is determined and that point system is used for estimating work. The two most common point systems I've come across are using Fibonacci numbers (with 8 or 13 being the top end of a sprint) and using 1, 5, 10, 25, 50, and 100 as story points. Whatever your system of pointing there needs to be a way to determine small, medium and large stories.

There are many different estimating techniques but most teams I've seen use some sort of t-shirt size technique. They start by picking a story and assigning it an arbitrary point value. The rest of the stories are then pointed by determining if the story is larger or smaller than the initial story. When you have more than 3 assignments that you can use for point values you can get more granular and determine if the story is larger or smaller than the other stories in it's t-shirt size bucket.

Once the velocity is determined, potential features are identified, and stories are estimated you will be able to create your sprint backlog. Backlog creation begins by stack ranking the stories (taking dependencies into account) and cutting the sprint off after the last story that keeps the total number of sprint points at or less than the pre-determined sprint velocity.

Implementation

The second phase of the sprint life-cycle is implementation. During the implementation phase the team meets daily at stand-up and goes over what they worked on yesterday, what they're working on today, and any potential blocking issues. When a blocking issue is identified it's up to the scrum master to try to unblock the team member. During the implementation phase the software should be developed, tested, and integrated.

Review

The last phase of the sprint life-cycle is review. The review phase takes on two forms. The first is the sprint demo and the second is the sprint retrospective.

The sprint demo is the teams opportunity to showcase what they've built during the sprint. Sprint demos can actually be a very useful tool leading into the next sprint planning phase. Often when we think about planning with regards to software we think about what we're going to build from an architectural perspective. The problem with this line of thinking is that it typically leaves out integration, testing, and deployment. If we instead think about what we're going to demo during planning we'll have to take into account how we're going to demo it.

A note of caution about sprint demos. One mistake I've often seen teams make during sprint demos is to walk everyone through a bullet list of work that was performed. Walking others through a bullet list isn't a demo. If you find that your team is working on features that don't have a user facing component then figure out a creative way to tell the story of how what you worked on matters. Use an animation, a use case scenario, or maybe even a chart or some graphs to help illustrate what you worked on. 

The sprint retrospective is a mini postmortem. During the retrospective the team goes over what went well, what didn't go well, and what the team could have done better. The retrospective is a dedicated time for us to identify the issues that caused us to lose velocity, to call out our successes during the sprint, as well as identify what we could have done differently to have improved the sprint. Because the sprint review typically takes place before the planning of the next sprint we have an opportunity to make changes that give us a higher likelihood for success before the next sprint starts.

The retrospective IS NOT a finger pointing exercise. It's NOT a witch hunt or an opportunity to push blame around. The retrospective IS an opportunity to identify gaps in our planning, our process, or our estimates. The point of the retrospective is to maintain or increase our velocity.

Isn't there a phase missing?

You may be wondering where the "release" phase is. There isn't one. Releasing your software should be done as part of the implementation phase. Your sprint planning should account for what it takes to ship the software, if that's part of the definition of done for the sprint.

Agile Development: Sprint Teams

Sprints are the way that Scrum teams iterate over software. The goal of a sprint is to produce a working set of software that is potentially shippable. I say potentially because it's really up to the business to decide when the software is shippable. As long as the feature development is complete there is a choice. You can evaluate the number and severity of open bugs as well as the other features and make a call on whether you ship or not.

Sprint Teams

Sprint teams are comprised (at minimum) of a representative of product and the software development engineers. Though in practice you'll really want someone from QA (quality assurance, if your org has a separate QA group) and someone from operations represented.

Products Role

In Agile we favor collaboration over contract negotiation. The role of product management as part of the scrum team is to represent the voice of the customer. The product manager should be working directly with the customer to understand their needs. The product manager helps collaborate with the rest of the team on the requirements of the feature(s) being developed each sprint.

Developments Role

The developers are the ones actually building out the features. The role of the developer is to collaborate with product to turn the requirements into feature stories and tasks. The developer is also the advocate for the health of the code and architecture. The developer is responsible for making sure that unnecessary tech debt isn't accrued sprint over sprint. Often this means collaborating with product on the scope of work for each sprint to account for taking on bug fixes and technical debt.

Quality Assurances Role

Ideally all software we write includes unit, functional and integration tests. But that doesn't mean that all testing can be automated. There are some aspects of the user experience that may require manual testing. Or if your organization is new to agile you may have creating automated tests as tech debt. Whatever the reason, our goal of building potentially shippable software each sprint means including QA as part of the sprint team.

Operations Role

One other often overlooked member of the sprint team is operations. While continuous delivery and continuous integration are the ideals for deployment, many organizations have not shifted to that model yet. If your org is one that hasn't, including operations will help accurately account for the sprint work necessary to release the software.

Scrum Master

Each sprint has a scrum master. It is the role of the scrum master to facilitate scrum. This is done by managing the product backlog, facilitating the scrum meetings, facilitating the sprint planning meeting, facilitating the retrospective meeting, helping to define what done means for a given story or task, and removing roadblocks to the team. The scrum master is also responsible for removing roadblocks to the scrum team. This is usually done by fostering collaboration between teams and product.

Agile Development: The Need For Scrum

Over the last two decades the internet has become ubiquitous in our lives. The speed of our internet connections and performance of our computers have increased several orders of magnitude. These changes have had a cascading effect on the software industry. Software used to be run on hardware to expensive for the average person (or even company) to purchase and maintain. Bandwidth limitations required software to be distributed via a high cost and immutable media like the compact disk. As hardware prices decreased and the bandwidth and speed of our connections increased it became easier to distribute software over the web or even run it entirely in the browser using a combination of HTML, JavaScript and CSS. 

As software became easier to distribute our ability to innovate and provide our customers with fixes and new features became paramount to the success of a software product. The birth of the mobile computing industry with devices powerful enough to run both native and web based software intensified our need to respond to the ever changing needs of our customers.

One of the side affects of software distribution evolving has been that our processes, originally created to manage change over a long period of time, became inflexible and costly in a world where change was frequent. The software industry was plagued with process that couldn't keep up with the rapid multi-platform change the industry was going through.

This inflexibility in the software development process gave birth to the Agile movement. Agile recognized that responding to the changing needs of our customers has become as important or more important than following a long term plan. Collaboration on product requirements needed to take precedence over processes or tools. Agile understood that rather than formalizing requirements and managing those requirements as they flowed downstream that it needed to embrace iterative development and distribution to keep up with the ever changing needs of it's customers.

Agile understood that the cost of downstream change in a rigid process is greater than identifying the change upstream and pivoting earlier. In order to manage this change agile embraced the concept of Scrum. In traditional software development information flowed downstream from product to development, development to test, test to operations, and operations to the customer. If a change was needed it required a renegotiation of requirements and schedule due to the fact that any change restarted the process at the beginning.

Scrum increases the speed and flexibility of software development by bringing product, development, test and operations together into a cross-functional team responsible for adapting to change. Instead of requirements flowing downstream in a rigid process, requirements are collaborated on with product and test. Requirements are turned into features which are developed iteratively. At the end of each iteration the next set of requirements is defined. These requirements can build upon a previous iteration or completely change the direction of the software. Each iteration is time-boxed in order to allow for greater speed and flexibility in change.

Over the next few weeks I'll go in-depth into the different pieces of Agile Scrum with the goal of providing you a framework by which you can apply agile as part of your development process.

Minimizing the risk of bugs

Software development is a craft that requires practice, hard work and dedication. It's a craft that involves many edge cases and unintended consequences. As awful as it sounds, we've all got bugs in our code. The goal of writing software should not be to write software with no bugs as this is unattainable. Instead the goal should be to minimize the risk of high severity bugs.

Every change to your software is an opportunity to introduce new bugs. Following these tips will help you minimize the risk of introducing bugs into your system.

Test Your Software

Duh, right? Testing your software may sound like a no-brainer but you'd be surprised by how many times people break builds and introduce buggy software just because they didn't test their code. In my previous post on Testing Your Software Properly I provided a checklist of tests that you should run before you commit your code.

Without proper tests you can not have confidence that you aren't regressing an old bug or introducing a new bug into the system. Tests are the foundation for confidence in any change you make.

Reduce the surface area of your change

The more lines of code you change with every commit the higher the risk of bugs. This is where encapsulation, SOLID principles, and refactoring become so important.

It's important to encapsulate your software into individual loosely coupled pieces. If you make a change in a class and it causes cascading changes throughout the rest of your software in code un-related to your change then you're likely introducing new bugs.

If you follow the Single Responsibility Principle in SOLID your classes will have one reason and only one reason to change. This reduces the surface area of your change because you will not be changing code in unrelated modules.

Following principles like DRY and YAGNI will lead to more robust code that is flexible and easy to change.

Keeping your code simple is one of the keys to reducing the surface area of your change.

Reduce the complexity of the code

Overly complex code leads to bugs for many reasons:

• The code is fragile because the learning curve is steep.
• It's easier to do the wrong thing than it is the correct thing when making a change in the code.
• The code is not readable often causing you to make multiple context switches to understand a single workflow.

What are some signs that your code is too complex?

• The patterns in the code are not clear, obvious, and/or discoverable.
• You have multiple levels in indirection that support one workflow or use case.
• You have an abstraction that fronts a single concrete implementation.
• Your code does not have clear boundaries.
• You have highly interdependent modules.
• Your code is not highly cohesive.
• Your code has rigid rules that are not enforceable in their individual units but only as a whole.

Testing your software properly

Here's a general checklist of the type of tests that you can use to ensure that you're testing your software properly before you ship it. This isn't an exhaustive list but can be used as a starting point for you to write a more exhaustive list that's right for your software.

Compile before you commit

Good testing starts with making sure your code actually compiles. Yes, people actually check in code without compiling it first. This is just a dumb mistake and is 100% avoidable. 

Run a clean build before you commit

A somewhat non-obvious thing to make sure when compiling is that you're not using a cached compiled object that has changed. Often compilers will cache objects and attempt to track when the dependency changes and only recompile the dependent object when the compiler believes it has actually changed. Cached objects will then be linked against your code and make your software appear to work but those objects may have actually changed and broken functionality. Doing a clean build before you commit will ensure that an object you depend on hasn't changed in such a way as to break your code integration.

Happy Path Tests

Happy path tests should be your minimum bar when committing code. Happy path tests ensure that your code works as intended when used in the way it was designed. These tests can be thought of as functional tests. They test the functionality of the software and ensure that the software meets the business requirement.

Negative Path Tests

Negative path tests ensure that your software is resilient to change. Negative path testing includes using your code in ways for which it was not intended. Common tests include sending in null object parameters and testing upper and lower bounds of parameters. Negative path testing also includes testing that your software properly handles exceptions and throws the proper exceptions.

White Box Tests

White box tests ensure that your code works from the outside in. These are a set of tests that ensure your objects work from the consumers perspective. These tests include making sure the object can be created and initialized, that method calls work according to spec, and that the code does not misbehave from the callers perspective.

Black Box Tests

Black box tests ensure that your code works from the inside out. These tests require access to the internals of the object. Black box tests usually test the fitness of particular private methods and algorithms. 

Life-cycle Tests

You should test how your objects function in various aspects of the objects life-cycle. The key to life-cycle tests is to make sure that your objects mange state properly. Life cycle tests are also useful in making sure that you don't have any memory leaks in your code due to life-cycle changes.

Life-cycle testing includes testing the creation, destruction, concurrency and serialization of your objects. Two life-cycle areas that tend to cause bugs are not properly testing when the object state is saved or restored or when the object is used in a multi-threaded environment.

Integration Tests

Do you understand how your software works in the context of the larger system of components that use and are used by your software? Integration tests allow you to make sure that your software works end to end in the system as a whole. 

When Not To Refactor

Refactoring software is a crucial part of extending the life of software. Refactoring contributes to enhancing the maintainability of the software by incrementally improving the design, readability and modularity of the components. But not much has been said about when not to refactor software.

Don't refactor code unless you need to change the code for a business reason.

One of the common mistakes I often see with regards to refactoring is when people refactor code that doesn't need it under the guise of making it better. The argument usually goes something like "this needs to be more abstract", "I wrote this code a long time ago and it is crappy", "this code is too complex" or something along those lines.

You should only refactor code when you are already in the code to make a change to support the business. That may sound counter intuitive but one of the worst things we can do is change code, however crappy, unreadable or complex that doesn't have a reason to change.

Valid business reasons to change code include (but are not limited to):

• Adding new functionality
• Extending existing functionality.
• Making measurable performance improvements.
• Adding a layer of abstraction in order to support a new use case.
• Modularizing a particular object so that it can be reused in another part of the system

Adding new functionality or Extending existing functionality

This is where the boyscout rule comes into play. If you are in already in the code for another reason then you should clean up the code even if you didn't make the mess.

Making measurable performance improvements

This one is probably self explanatory but it's important to note that performance improvements will usually require some level of refactoring. 

Adding a layer of abstraction in order to support a new use case

This is an important one to understand. Often people will over generalize code at the beginning. This leads to overly complex designs and less readable code. If we follow the rule of not creating a layer of abstraction until we have at least two or three use cases for the code then there will come a point when you need to refactor the code in order to provide a layer of abstraction that doesn't already exist.

Until that second or third use case comes about the code should not be generalized. You don't have enough information about future uses of the code to get the abstraction correct. You may get lucky and guess at the future abstraction but you don't want to run your business on guesses and luck.

Modularizing a particular object so that it can be reused in another part of the system

Code reuse is one of the most important tenets of object oriented programming. When we identify code that is not specific to a particular object or package AND is needed in some other part of the system we should refactor this code into it's own module. Its important to ONLY do this when the code is actually needed in another part of the system.

Don't refactor code without tests

In order to refactor code safely you should have unit and integration tests for the existing functionality. I would also argue that you should write tests for the new functionality as well before you refactor. This will help you to understand the proper way to refactor the code as it helps you define how the refactored code should be used from a consumers standpoint.

If the tests don't exist for the the existing functionality you should write them first before you start refactoring. This helps ensure that you don't cause a new bug in the code or regress an old bug when refactoring.

Software Estimation: How to estimate software accurately part 3

In this series on software estimation I defined an accurate estimate as something that will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

In part 1 I explained how to account for risk and unknowns in your software estimation. In part 2 I explained how to quantify the work that is known. In the final post of this series I explain the last pieces needed to give an accurate software estimation.

Identifying Opportunity Cost

Let's say we estimate project X and determine that it will take 2 developers 2 months to complete. After estimating project Y we know that it will take 2 developers 1 month to complete (or 1 developer 2 months). Finally, when estimating project Z we determine that it will take 1 person 1 month. We now have a starting point from which to determine, based on cost to build and return on investment, if there is an opportunity loss associated with starting project X before projects Y or Z.

Refining As New Information Becomes Available

Most software estimates will start out as a range. Typically larger projects will have a wider range. This is because there are always things we know we don't know about the project as well as things that we don't know that we don't know.

It is easier to reduce ambiguity with information we know we don't know because we know where to start. As you practice software estimation you'll start to learn the correct questions to ask to identify the information you don't know you don't know.

Identifying new information should result in refining your previous estimate. The more ambiguity you are able to remove the more accurate your software estimate can be. The goal should be to continually narrow the range of the estimate.

More Accurate When Accompanied By A Confidence Interval

A confidence interval is a range of numbers from which we expect the real value to be contained. Having a good confidence interval allows us to determine the range of our estimate. This range can then be examined to validate our assumptions and to identify ambiguity.

For example, if I asked you to estimate how many marbles fit in a mason jar the actual number of marbles that would fit would be contained in your estimate. My 90% confidence interval may be 25 - 100 marbles. The actual number of marbles may be 87, in which case my interval contained the real value.

If you're interested in learning more about how to compute a confidence interval you should read How to Measure Anything by Douglas Hubbard.

Identify And Clarify Ambiguity

I'll wrap up this post with some questions you can ask to identify and clarify ambiguity in your software estimates.

• What assumptions does your estimate rely on?
• What are the risks associated with your estimate? Identify the things that could cause your estimate to be wrong.
• What external dependencies does your project have? For example OS updates, product launches, planned outtages, SDK updates, etc.
• What data do you need? Is it already available in a consumable format?
• What services are required? Do they exist? Do they talk the correct protocol?
• Are there any User Interface (UI) or User Experience (UX) dependencies that need to be resolved?

Software Estimation: How to estimate software accurately part 2

In this series on software estimation I defined an accurate estimate as something that will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

In part 1 I explained how to account for risk and unknowns in your software estimation. In this post I'll explain how to quantify the work that is known.

Quantifying The Known Work

All software has to be designed, architected, implemented, tested, debugged, and deployed. Each of these areas needs to be accounted for in the software estimate. Here are a set of questions that should be asked for each area to help you understand the level of effort associated with building your software.

Using these questions you can break out the known work into smaller buckets of work. You can then estimate each of these smaller pieces independently.

Design

Has the user experience and user interaction been defined? Do people interact directly via a user interface you provide or indirectly from another user interface? Have all the transitions and states been defined?

Will your software be available in multiple regions and/or locales? Does the design take regionalization and localization into account? Who provides the translations?

Architecture

What platform will the software run on? Are there platform specific requirements or constraints? Does hardware need to be purchased or provisioned?

What abstractions need to be defined? What is the hierarchy and/or composition of the abstractions?

Does the software rely on persistent data? How is it generated? How is it updated? What data specific workflows are there?

What metrics do you need to track?

How is the software going to scale? Are there single points of failure? Is the software going to be memory bound and/or CPU bound?

What is the software NOT going to do?

Versioning

Is your software released incrementally? How will the system handle data structure changes?

How are functionality updates handled in your system? Do updates need to be isolated from each other or can they co-exist. Is there a need to maintain backwards compatibility?

Implementation

What is the source control mechanism? Is it already setup? Does everyone have access? Is there a branching strategy?

Who is going to implement the software? Do they have all the hardware and software they need? Do you they know the platforms or technologies that your project uses? Do they require time to ramp up on the platform or tools? Is external training required?

Testing

Is your project going to use continuous integration? If not, why? Is your project going to use continuous deployment? If so, how do you deal with a failed deployment?

How are bugs going to be tracked?

How are the units of software defined? Are there tests for all units? A good software project has tests for each unit of work as well as tests for the integration of multiple units.

Are your tests automated? How often are the tests run and by whom?

Do you need black box testing, white box testing, or a combination of both for your software? Are you testing more than just the happy path or are you doing negative case testing as well? Does your software need stress or load tests?

Does your testing require specific hardware or software? Do these need to be procured?

Debugging

What tools are going to be used to debug the software? How are you going to profile performance?

What needs to be logged? How often and by what components? How are logs collected?

How are errors reported? What work is required to make errors in the system reproducible?

Deployment

How is the software going to be released? How is your software going to be monitored? How is your infrastructure going to be built out? Can you automate the infrastructure build out using tools like Chef or Puppet?

How is the software documented? How is the documentation updated?

What's Next?

In my next post I'll explain the last pieces needed to give an accurate software estimation.

Software Estimation: How to estimate software accurately part 1

In my last post Software Estimate: What you're doing wrong I explained why people tend to create inaccurate software estimates. In this post I'll outline what's necessary to create an accurate software estimate.

Previously I defined an accurate estimate as something that will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

Let's take a deeper dive into each of these areas with the goal of helping you to understand what information you need to gather to accurately estimate your software.

Identifying Risk and Unknowns

It is inevitable that there is risk associated with the software you're building. Risk in a software project is typically associated in one or more of the following areas:

• Blocking another project.
• Being blocked by another project.
• Requirements or scope that haven't been defined.
• Attrition.

If we understand the areas that pose risk to our project we can use that information to inform our estimate. Your project may need additional time to coordinate with another team. You may need to account for another projects schedule and/or risk.

Blocking another project

While blocking another project many not sound like something that needs to be taken into account when estimating software it can actually be the source of unexpected work. The other project may have requirements for your software that you're unaware of. You may have to make last minute changes to your software to account for these requirements. This is especially true if the other project is a higher priority project for the company than yours.

It's important from an estimation standpoint to understand if there are other projects that need to consume your software. If so, how well is the dependency between your project and theirs defined? Does the other project take direct dependencies on your software (for example by compiling against your library)? If you're providing a service how well is the service contract defined?

When you update your software what are the downstream effects on consumers of your software? Does your software require versioning?

Being blocked by another project

What other components, libraries, or services does your software depend on? Have they already been shipped? If so how do you deal with updates to the software? If the software hasn't already shipped, are their milestones known?

What happens if they can't deliver or aren't able to deliver in time for your project to meet it's goals? Do features in your software get removed? How long will it take to remove or update the affected components?

Requirements that haven't been defined

Blocking another project or being blocked by another project are not the only sources of undefined requirements. Additional requirements many come from your product owners due to under-defined user experiences or user interactions.

Are there mock ups of the user experience and user interaction? Have static web based mocks that people can touch and feel been built? Are all the transitions defined? What state does your application need or maintain?

Attrition

While attrition is often difficult to account for in the actual estimate, asking yourself some attrition related questions will help you identify work that should be accounted for in your estimate.

Does your software require a specialized skillset or tools? How long does it take to aquire the tools? What's the ramp up time to learn the skillset? What's the lead time that it takes to hire someone with the needed skills or platform knowledge?

What's Next?

In my next post I'll explain how to quantify the work that is known.

Software Estimation: What you're doing wrong

During the beginning of my career as a software development engineer one of things that I dreaded the most was the beginning of any project where I knew I was going to be asked to give an estimate for the software I was going to write. Often I thought "how can I estimate something I have no clue how I'm even going to build?"

Ultimately the dread I felt came down to this one simple fact. I had no clue how you could accurately estimate software. Over the years, as I've learned to estimate software development much more accurately, I've learned that what I thought of as software estimation was built on several assumptions that were incorrect.

Fallacies of Estimation

• Estimates require giving a specific date
• Estimates should be given on the spot
• Estimates can't change

Estimates Require Giving a Specific Date

This fallacy was probably the heart of the anxiety I felt about software estimation. I always assumed that an estimate was a fixed amount of time like 2 days or 1 month. It never occurred to me that an estimate could be a range. Fixed dates are rigid and inflexible and are easy to get wrong. Fixed dates cannot account for the fact that there are unknowns associated with the project. 

An accurate estimate will be a range. Ranges are more flexible and can help account for the unknown and any risks associated with the project.

Estimates should be given on the spot

This fallacy is based on the assumption that some big decision needs to be made right now and doesn't have time to wait for you to think about what unknowns you're dealing with. First off, if you're making a big decision on the spot based on a back of the hand estimate you shouldn't be in your role and shouldn't have the responsibility that you do. It's careless and dangerous to you, your product, and your customers to expect some engineer to accurately give you an estimate on the spot.

An accurate estimate will require you to do some research.

Estimates can't change

It may be true that the original estimate needs to be something that you can make a big decision or commitment based on. It's not true however that estimation stops once that decision is made or a project is put into motion. An estimate needs to be a living thing.

Estimates are about approximations. They're about dealing with ambiguity. As things become less ambiguous your estimate should be refined.

Where to go from here?

Now that we know the fallacies of software estimation we can define the characteristics of an accurate software estimate. An accurate estimate will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

In my next post I'll talk about the truths of software estimation and give you a set of resources to help you create accurate software estimations.

Android WebView: HTML5 Video and Rotation

In the last few posts I've shown you have to add a Web View to your Android application. Web View's are useful for showing external content without leaving your app, adding dynamic content to your app allowing change without an app update, or simply display legal pages (terms of service, end user license agreements, etc) directly from your site.

I wanted to end this brief series talking about HTML5 video and rotation. There are some gotchas that it's important to be aware of in these areas that I've come across over the past few years.

Stopping Video

In some versions of Android the audio of video that was playing in a Web View will continue to play in the background even after exiting the Fragment or Activity that is hosting the web view. This appears to be a bug where onPause isn't being called on the web view in those versions of Android.

Handling this isn't simply a matter or calling onPause explicitly on the Web View if your app is targeting devices running Gingerbread or below. This is because those versions of Android don't have a public onPause method on the Web View.

The easiest way to handle this in a platform version independent way is to call the Web View's onPause method via reflection.

Class.forName("android.webkit.WebView")
         .getMethod("onPause", (Class[])null)
         .invoke(yourWebView, (Object[])null);

Playing Arbitrary HTML5 Videos

Web video players are very very fragile when it comes to playing arbitrary HTML5 video. Sites that host HTML5 video often rely on nuances that are present in some browsers or javascript engines but not in others. You may find that some HTML5 video just will not play in your embedded Web View.

Playing HTML5 Video Using A Native Video Player

While it is possible to create a custom WebChromeClient to intercept HTML5 video and play it using a native video player I would not recommend this. There are several problems with this approach that are very difficult to overcome.

• You need to intercept ALL media playback events from javascript and handle them natively.
• Handling all video related javascript events can only be accomplished by injecting javascript in the page or having direct access to the javascript engine.
• Some pages dynamically load videos on the page which means you have to be monitoring the DOM for changes.

Rotation

Depending on how you handle rotation in your Activities and Fragments you may be accidentally reloading web pages on rotation. This happens when the call to loadUrl is made during the Activity or Fragment setup, since the rotation workflow causes the Activity or Fragment to be recreated.

This can be made even worse when the user had navigated several links deep into a web page. When the app is rotated and the Url is reloaded the user will find themselves back on the initial web page and not where they expect.

Handling rotation with a WebView can be done by:

• Only calling loadUrl when creating the WebView when the saved instance state in null.
• Call the WebView's saveState method in your Activity's onSaveInstanceState method.
• Call the WebView's restoreState method in your Activity's onRestoreInstanceState method.

Android WebView: Interacting with your apps Java from the Web Page.

In my previous post, Android WebView: Interacting with the Web Page from Java, I showed you how to call javascript functions from your native Java code. I also briefly mentioned that on Android KitKat and above you have the option of receiving the result directly from the javascript call using the evaluateJavascript method. This is an easy way to handle any result that may have been returned from the javascript call.

The evaluateJavascript method is very helpful but does not handle all use cases in which you may need to execute native Java code initiated by your web page. One use case would be handling web events (like a button click) or for returning data from a method on Android Jelly Bean or below.

One word of warning before we begin, allowing an HTML page to execute native Java code is inherently insecure. Web pages are isolated from the app they're running in by design. Creating a bridge between the web page and the app means that javascript on the page may have the ability to do anything on the device for which the app that's hosting the web page has permissions for. It is extremely important that you keep this in mind when creating the bridge from the web page to your app.

There are three main things that you need to do in order to call native Java methods from javascript:

• Create a javascript interface object which serves as the bridge between javascript and Java.
• Add the interface to the Web View so that any page that it hosts can use the bridge.
• Update your javascript to use the bridge.

Creating a Javascript Interface Object

In this very basic class I am declaring a listener interface which I have the caller pass in via the constructor. This is one way to make sure that you only allow javascript to interact with your app in the way you expect. By using a well defined interface you are making it more difficult for malicious javascript to execute arbitrary code.

To make a method visible to javascript the method needs to be decorated with the @JavascriptInterface attribute.

Notice that I'm calling the listener back using a handler and runnable. This is because the bridge class methods aren't executing on the main thread. We need to execute the listener callback on the main thread.

public class WebViewNativeBridge
{
    public interface OnSomeChangeListener {
        public void onSomeChange(String text);
    } 

    private OnSomeChangeListener listener = null;

    public WebViewNativeBridge(OnSomeChangeListener listener) {
        this.listener = listener;
    } 

    @JavascriptInterface @SuppressWarnings("unused")
    public void someChangeFromWeb(String text)
    {
        final String newText = text; 

        // This code is not executed in the UI thread
        // so we must force it to happen
        new Handler(Looper.getMainLooper()).post(new Runnable()
        {
            @Override
            public void run()
            {
                if(listener != null)
                    listener.onSomeChange(newText);
            }
        });
    }
}

Add The Javascript Bridge Object To Your WebView

The way you wire up your native javascript bridge is to add the javascript interface to your Web View using the addJavascriptInterface method. This method takes in two parameters, your bridge object and the key word you want to use to access the bridge from within javascript.

WebViewNativeBridge.OnSomeChangeListener listener
                                                        = this.getOnSomeChangeListener();
WebViewNativeBridge bridge = new WebViewNativeBridge(listener);
browser.addJavascriptInterface(bridge, "NativeBridge");

I created the bridge object with a custom listener I created:

private WebViewNativeBridge.OnSomeChangeListener getOnSomeChangeListener() {
    return new WebViewNativeBridge.OnSomeChangeListener() {
        public void onSomeChange(String val) {
            // do something with the value passed in
        }
    };
}

Update The Javascript To Use The Native Bridge Object

The last step is to call into your native Java code from javascript. Using the NativeBridge object we registered with the Web View we can call the someChangeFromWeb method from our javascript. Note, the name of the object you use in javascript has to be exactly the same as the second parameter you passed into the addJavascriptInterface method.

function notifyNativeOfChange() {
    NativeBridge.someChangeFromWeb("some change");
}