Writing code that will last: Identifying Dependencies

In my previous post, writing code that will last: understanding bottlenecks, I talked about how bottlenecks can come in the form of vertical scalability, horizontal scalability, or statefulness. In this post I'll explore how identifying upstream and downstream dependencies in your software can make or break it from a longevity perspective.

Upstream Dependencies

There are several upstream dependencies that you need to consider in your code. These come in the form of your operating system, the platform your application or service runs on, the external applications and services you use, and the contracts you depend on for data.

Operating systems and software platforms can affect the performance, security, and ability of your service. Writing code that lasts means abstracting your code enough from the OS and underlying platform such that you can upgrade the OS or platform without changes to your application or service. This allows you to protect your self and your customers from security vulnerabilities as well as allows you to take advantage of performance enhancements that come from kernel or library updates.

You should also consider the dependencies you have an external applications or services. Do you depend on them in a version specific way? How will your application need to change if they change the schema of the contract you have or deprecate an API you use. Building code that will last doesn't mean prematurely optimizing your code to try to defend against these changes, but instead means understanding how to encapsulate them such that the effect these changes have on your code are minimal.

Downstream Dependencies

Downstream dependencies are people, applications, and services that depend on your software. From a downstream dependency perspective writing code that will last means understanding when you're making a choice that you can't go back and change after the fact. In order to write code that lasts we have to minimize the number of these choices that we make. These choices often come in the form of the APIs we expose, the contracts we create with other services, the way we represent objects, and etc. Once these choices are made, sometimes they can't be taken back without having some sort of negative side effect.

Writing code that will last: Understanding Bottlenecks

In my previous post, writing code that will last: avoiding potential roadblocks, I talked about how roadblocks can come in the form of cost, re-inventing the wheel, or under-defined requirements. In this post I'll explore how understanding potential bottlenecks in your software can make or break it from a longevity perspective.

Understanding Bottlenecks

The first bottleneck you should be considering is how your application scales vertically. Vertical scale typically has direct correlation with your applications performance characteristics. Vertical scale is achieved by increasing CPU, Memory, Network Capacity, or Disk your application has available to it. Is your application going to be CPU or Memory bound? Will you require a lot of Disk I/O? How much data will you be reading/writing over the network? You need to understand each of these areas in order to know how your application will scale vertically.

How your application scales horizontally is the next potential bottleneck you should understand. Horizontal scaling is the ability to add additional machines running your software into an existing pool to add additional computing. Horizontal scaling allows you to increasing the throughput of your application by having more capacity to process requests or jobs that the application performs. Your software needs to take this into account from the beginning or it won't last without requiring rework.

Another area to consider when understanding your applications bottlenecks is how stateful it is. Does your application require some external state in order to run? How the application gets and maintains this state can lead to potential bottlenecks. E.g maybe your application needs some specific information about the user making the request that it has to lookup somewhere. Or maybe it needs information about previous requests. An example of a stateful application is one that uses a shared database. The reason state cases bottlenecks is because your application is now dependent on something external that it does not control.

Writing code that will last: Avoiding Potential Roadblocks

I've written about writing code that will last before from the context of coding standards. Today I'd like to look at it from the non-code perspective. 

As a software engineer one of the best feelings was wrapping up a complex solution and feeling like I had written beautiful code. Code that sailed through peer review unscathed. Code that adhere'd to the standards of my team. Code that used best practices. Code that was simple and elegant.

One of the worst feelings as as software engineer was realizing that your beautiful code wasn't standing the test of time. Maybe it wasn't flexible enough to adapt to changing business requirements. Or possibly it created an unnecessary bottleneck. Or, despite the peer reviews, it wasn't as performant as expected.

  

Writing code that will last is about being able to look around corners and see the non obvious. Aside from the obvious, writing SOLID code, it's about avoiding potential roadblocks, about understanding bottlenecks, and identifying upstream and downstream dependencies. In this post I'll explore avoiding potential roadblocks.

Avoiding Potential Roadblocks

One of the biggest roadblocks to software engineering is cost.  Cost can come in many different forms; cost of development, cost of maintenance, cost of infrastructure. There are two areas to think about when considering the cost of software. First is ROI (return on investment). Does putting the software into production and keeping it in production save or make you more money in the long run than it will cost you to build and maintain? Make sure you're considering the operational overhead when determining ROI.

The second are to consider with regards to cost is opportunity loss costs, i.e. what software are you NOT building so that you can build this software. Your team could be building many different things. Why is this project the right one to build now? Is there something else you could be building that gets you a competitive advantage, a higher ROI, or reduces operational overhead?

Another roadblock to long lasting software is re-inventing the wheel. At it's simplest form it means writing software that already exists. It could exist as open source software, COTS (commercial off the shelf) software, or some internal library in your company's software portfolio. The latter is more often the case at larger companies as is having two different teams writing similar code at the same time. Sometimes in order to not re-invent the wheel, you need to change how you're approaching solving your problem so that you can re-use existing software. 

The last major roadblock to avoid is under-defined requirements. There's nothing worse than building a solution and realizing you it doesn't solve your problem because it didn't uncover the need for feature X. This is especially demoralizing to the team if they've just spent months building the software. The easiest way to avoid this is to have very clearly defined user scenarios up front coupled with short end to end milestones that allow you to get your software into production quickly. This allows you to reduce your feedback loop time and adapt to changing or unrecognized requirements.

You Shouldn't Backfill Unit Tests

Recently I've found myself in several conversations about unit tests, and the lack of them on a given project. It seems that almost everyone on the team knows that writing unit tests is a good thing and it seems that most of them also wished they'd written them from the start. But, as is the case with most fast moving projects, the team finds themselves in the position where they don't have adequate test coverage. This leads to reduced confidence in the code and not being able to catch bugs far enough upstream to effectively prevent them from ever being released to customers.

What this usually leads to is people on the team making comments, in response to a bug or regression someone is working, that if they only had unit tests they would have found this bug earlier. What I've seen most often in that situation is someone finally getting fed up enough with bugs and/or regressions and picking up the unit test torch and leading a crusade to backfill all the missing tests. While I appreciate the spirit of this, it's not the right thing to do.

Why's it not the right thing to do? First, you're not actually solving your problem. Your actual problem is that you can't ensure that the code you're currently changing 1. does what it's meant to do and 2. doesn't do something it's not meant to do.  Second, depending on how old the given code is, you may not know how it was actually supposed to work vs how it actually does work. You may end up writing unit tests are aren't correct, and in the worst case scenario cause you to change working code. Lastly, when the team is done backfilling the tests, they haven't learned new behavior (i.e. to write the tests when you write the code) but, more likely, have grown a disdain towards writing unit tests.

Now before you start to think I'm advocating against writing unit tests I'll flat out tell you I'm not. In fact I've come to realize in my career that writing your tests first is the only way to assure that you have working code. So what am I advocating for then?

Make a decision as a team to write unit tests and then write unit tests for all the code the team writes from that point forward. You'd be surprised at how much better your code coverage will be after just a short amount of time. Why? Because you'll be writing unit tests for code that you're fixing bugs in. You'll be writing them for new features or changes to existing features. Essentially, you'll be writing tests in the hot spots of your code that are requiring change. The very code that is either causing bugs, or is likely to have bugs introduced in.

If you never change a particular set of code again then you don't really need unit tests for that code (remember I'm talking about unit tests and not integration, system, or other types of tests).  If you do change that code, then because you've decided as a team to write tests, you'll add the missing tests when you go in to make the code change.

So how do you get started? It's easy, just follow my 3 step plan.

1. Make a decision to not put up with having incomplete code (i.e. code without corresponding tests).
2. When you write your next set of code, write tests!!!! (preferably first)
3. Enjoy the fact that you now have less buggy code that you can have more confidence in.

Agile Development: Demo Day

So far in this Agile Development series I've:

• Made the case for Scrum
• Defined who and what a sprint team is
• Walked through the anatomy of a sprint
• Provided tools for determining sprint duration
• Detailed what daily stand-up looks like
• Provided the typical workflow for planning a sprint
• Outlined the sprint retrospective

In my final post in this series I'll explain demo day and how it relates to agile development.

What Demo Day Is

As I've been saying all along in this series, one of the main objectives of Agile is to reduce the time it takes to complete the feedback loop. The loop is complete when the stakeholders have reviewed the feature(s) and signed off on them. Demo day closes the loop and provides the stakeholders with an opportunity to affect change.

I like to think of demo day as the bow that is tied around the sprint. It brings closure to the sprint process which started with sprint planning. At it's core the sprint demo is the teams opportunity to showcase what they've built during the sprint. Beyond that sprint demos can also be a useful tool leading into the next sprint planning phase as the product manager may change the plan for the next sprint based on the progress of the team.

There are two main parts to demo day. First is to review the work completed and not completed during the sprint. Second is to demonstrate the completed work to the stakeholders.

Reviewing Work

Reviewing the sprint work should be comprised of three things. First the team should very briefly call out what stories the team accepted into the sprint during planning. Second the team identifies what stories were added to the sprint after the sprint started. Finally the team goes over what stories were not completed during the sprint.

It's important to note that the review is not an exercise in excuse making or justifying not getting everything done. It's main purpose is to close the feedback loop with stakeholders and set expectations prior to heading into sprint planning.

The Demonstrations

Demonstrations are just that, demoing what the team built during the sprint. Typically the demo is limited to the user impacting aspect of the features the sprint team worked on during the sprint. The team reviews the features to ensure they are complete as well as provide product management the opportunity to get clarifications on behalf of the customer.

One mistake I've often seen people make is trying to "demo" spreadsheets or powerpoint bullets. This usually happens because the team was mostly in bug fix mode or works on non-user facing features. I would encourage teams in this situation to get creative and tell a story using charts, graphs, and metrics instead of showing data in excel or a list of powerpoint bullets.

Agile Development: Sprint Retrospective

In my previous post in this Agile Development series I detailed the typical workflow for planning a sprint. In this post I will outline the process of the sprint retrospective. One helpful way to think about the sprint retrospective is as a mini postmortem. During the retrospective the team goes over what went well, what didn't go well, and what the team could have done better.

What Went Well

The reason we ask what went well is not so that we can pat each other on the back and inflate our egos. One of the keys to a successful agile project is consistency in the velocity of the team sprint over sprint. One of the best ways to do this is to identify what lead to success so that the team can capitalize on it in the upcoming sprint(s).

What Didn't Go Well

As with most growth being realistic, open and honest about our mistakes and failures leads to not repeating them. The purpose of identifying what didn't go well is NOT to lay blame on any particular person or people. Identifying what didn't go well helps us to avoid or correct them those mistakes.

What Could Be Done Better

At first this may not seem different from identifying what didn't go well. The real difference is that we recognize things that may have gone well but could have gone better. Asking what we could do better is an opportunity to identify gaps in our planning, our process, and our estimates. This question should lead directly to actions that can be taken to increase or maintain the teams current velocity.

Because the sprint review typically takes place before the planning of the next sprint asking these three questions provides the team with the opportunity to make changes that give them a higher likelihood for success before the next sprint starts.

Agile Development: Planning A Sprint

In my previous post in this Agile Development series I walked you through the daily stand-up. In this post I'll go outline the typical workflow for planning a sprint.

Identify The Features

The product owner, who is serving as the voice of the customer, proposes the highest priority features. The scrum team is responsible for getting clarification, pushing back on features, and proposing features or work that should be accomplished. Features should be prioritized based on their overall business value. It's important to note that working on tech debt can often provide down stream business value by enabling new feature work.

Prioritize and Estimate

Once the scrum team has identified the features to work on in the sprint, the team should then pull all of the stories associated with that feature into the sprint backlog. The product owner and scrum team should negotiate the priority of the stories. Once the priority of the stories has been determined the team needs to estimate the level of effort for each individual story.

One typical way of estimating stories during planning is to pick a story that does not have a lot of ambiguity which represents a medium (tee-shirt sized) amount of work. The team then assigns that story a point value (the middle value of their pointing system). The team then moves on to point the rest of the stories based on them being either more effort or less effort than the original story. Stories that are more effort are assigned a higher point value while stories with lower effort are assigned a lower point value.

Determine What Stories Make It Into The Sprint

Just because the product owner or scrum team has identified a feature (and it's related stories and bugs) as high priority doesn't mean that the team can actually complete that feature in a sprint. The team needs a way to estimate the amount of work that can be accomplished by the team during the sprint. This is typically done through velocity tracking.

Velocity in agile is typically determined by the amount of points the team was able to accomplish in the previous sprint. Historical sprints aren't to be taken into account as each sprint the teams goal is to get better and better as estimating the level of effort for story work.

The sprint backlog should only contain enough stories to meet the teams velocity. Choosing to few story points means that the team will likely have to poach more work (which hasn't been vetted) from the product backlog. Choosing too many story points means the team is not likely to finish all the work in the given sprint, setting the team up for failure from the start.

At first the teams velocity will be all over the place as they work out their estimation techniques. But over time the team should start to settle in to a consistent velocity. It's important to be aware that there are several common practices that can negatively affect a teams velocity. It's important for the sprint team to guard against these in order to accurately gauge their velocity sprint over sprint.

Under or over estimating the level of effort of a story

Often this is the result of ambiguity that is not resolved priority to planning. If the team doesn't have enough information to accurately gauge the level of effort for a particular story the product owner and scrum master should first work to resolve the ambiguities before a feature or story is considered for a sprint. This is an example of a valid reason for the sprint team to push back on a particular feature or story.

Unplanned sprint work or re-prioritization of the sprint deliverables mid-sprint

This is an Agile no-no and should be avoided at all costs. In Agile our sprint duration is typically short enough that when new work comes up we can prioritize it at the next sprint planning meeting. If new high priority work or re-prioritization mid-sprint is truely unavoidable the best way to make sure it doesn't impact the overall sprint deliverables is for the whole team to understand what the lowest priority work is and have that work drop out of the sprint.

Unaccounted for absence

People are going to take vacation, get sick, or be off for a holiday. It's difficult to account for sick time but it is very easy to account for holiday's and vacation time. Make sure in your sprint planning you understand how many days during the sprint people know they will be out and reduce that sprints velocity accordingly.

Other regular duties

Many software organizations use the DevOps model where the developer is on-call for the software and services they own. Not accounting for this type of work can have a negative impact on planning as the sprint team will sign-up for more work than they can actually accomplish.

Break The Stories down into tasks

Once the sprint backlog has been determined the team should break the stories down into smaller more granular tasks. This allows for more story work to be done in parallel. The team can swarm on a story together ensuring that the highest priority work is done first.

Once the tasks are identified the team is ready to start the sprint!

Agile Development: Stand-up

In my previous post in this Agile Development series I explained two ways to determine sprint duration. In this post I'll go into detail on what an agile stand-up typically looks like.

One thing you've heard me mention over and over in this series is that one of the primary goals of agile is to decrease the total time it takes to complete the feedback loop. One key way to accomplish this is with a daily stand-up.

In it's most basic form the stand-up is a time-boxed meeting (typically 15 minutes) where the scrum team gets together and answers three simple questions:

1. What did I do yesterday?
2. What am I going to do today?
3. What am I blocked on?

What Did I Do Yesterday?

In it's most basic form this question is intended to determine if progress was made towards the overall sprint goals. Knowing, as a team, what has already been accomplished helps the team formulate a plan for what needs to be accomplished over the next day.

I've seen this question often turn into a check-list of everything you did during the day. But it's important to keep in mind that your goal is not to provide an overview of everything you did. Instead, the purpose of this question is to make sure that others are aware of how much progress was made towards the work you committed to the day before. This is especially important when work items in the sprint are interdependent.

What Am I Going To Do Today?

There are two reasons that telling the team what you plan to accomplish for the day is important. First, it allows the team to validate whether the work is really the top priority for the sprint. During sprint planning many teams will prioritize the backlog for the sprint. But because our feedback loop is so short the priorities of any individual task may change throughout the sprint or even daily.

The second reason this question is important is that you are making a commitment to the team to accomplish a particular task. You can think of it as your daily contract with the team. And in turn the other members of the team are making a commitment to you to get other work done. This is how the team is able to increase and maintain a particular velocity. The team is able to time and coordinate work at a granular level that allows the team to maintain a constant throughput. 

What Am I Blocked On?

Making sure that the team is aware of any potentially blocking issues allows the team to react quicker and not allow blocking issues to affect the teams throughput. Because the scrum team talks about this daily, it is able to address potentially blocking or blocking issues immediately.

If a particular team member is blocked on a technical issue, I've often seen the scrum team swarm on the issue until it gets resolved. This has the benefit of making sure any blocking issue that is an upstream dependency of other sprint tasks gets resolved quickly and the team is able to dedicate more of the sprint to work that was planned.

If an issue cannot be resolved due to some unforeseen circumstance the sprint team has the ability to adjust and pull in additional tasks into the sprint while the blocking issue is being resolved. Because we talk about blocking issues daily, there shouldn't be a significant impact on the sprint schedule.

Parking Lots

It's often the case that during stand-up an issue comes up that requires a deeper dive than the 15 minutes allotted allows for. Fight the urge to go into detail during stand-up on the issue. Instead, announce that you have a parking lot and who should attend and then table the issue until after the stand-up is over. This allows less of the team to be randomized by an issue that isn't directly related to their daily commitment. 

Agile Development: Determining Sprint Duration

In my previous post in this Agile Development series I explained the anatomy of a sprint in terms of the three different sprint phases. In this post I'll detail the two main ways sprint duration is determined.

Fixed Time

Most sprint teams you come across will use a fixed time for their sprints. In my experience 2 to 3 week sprints are the most common duration. The goal of a fixed time sprint is to shorten the feedback loop with the customer in order to deliver the right features to them. By limiting the time to 2 or 3 weeks you can easily change course as the requirements change and provide continued business value to the customer without spending a lot of time on features that won't be used or whose requirements have changed.

Having a fixed time sprint allows the team to set expectations with your customer on what new features they will be building and when they will be available to the customer. I would strongly suggest that you don't have sprints that are longer than 3 weeks in duration. 2 to 3 weeks keeps the feedback loop small while also giving the sprint team enough time to build a feature out end to end. Increasing the amount of time it takes to complete the feedback loop means that the sprint team could spend more time building the wrong product and less time building the correct product should the requirements change.

It is occasionally the case with a fixed time sprint where the sprint will end and the team will have been unable to complete a particular feature. In this case the retrospective should identify the cause of the decreased velocity. Typical causes for a decrease in velocity (in my experience) have been:

• Failure to account for vacation or known leave of sprint team members.
• Failure to account for time spent on external work (like on-call duties).
• Failure to account for the unexpected work that grows the sprint backlog.
• Undefined or not well defined definitions of done.
• Stories that are too large and hide dependencies.
• Unknown external dependencies. 

Fixed Feature

Another, less common, way to determine sprint duration is to use a fixed feature approach. This approach says that a sprint is complete only when a particular feature is complete regardless of time. This is typically used when it is known that dependencies are hidden but that it is not easy to uncover those dependencies without first doing some work towards the end goal of the feature. In a system without a well defined or understood architecture, a fixed feature sprint duration is often used as it allows for the unknowns that go with the not well defined architecture.

Earlier in my career this was a more appealing way to determine sprint duration but as I've grown in my Agile planning I've come to realize that open-ended sprint duration's actually detract from shipping software more than they contribute to shipping complete software. Even with the most ambiguous problems it is usually better to first root out the ambiguity in a fixed time sprint and then plan and work on the feature.

Another reason that I've found for fixed feature sprints being less than ideal is that you have a harder time gauging your velocity as your velocity is determined by the amount of work you can complete in a fixed amount of time. Since knowing you're velocity is key for estimating when a feature is complete without it you cannot set your customers expectations as to when they will get a particular feature.

Fixed feature sprints also increase the time it takes to complete the feedback loop. And, as I mentioned earlier, increased feedback loops mean that it takes longer to respond to customer requests.

Agile Development: The Anatomy Of A Sprint

In my previous post in the Agile Development series I walked through the makeup of a sprint team. I explained the roles of Product, Dev, QA, Operations, and the Scrum Master. In this post I will go into detail on the anatomy of a sprint.

Planning

The first phase of the sprint life-cycle is planning. During the planning phase sprint velocity is determined, features are identified, features are estimated, and the backlog is created.

At it's most basic level the velocity of the sprint is determined by the number of points that were completed during the last sprint. When a sprint team first forms their velocity is very volatile. But as the team learns to estimate their story points more accurately there is less oscillation between the number of points completed sprint over sprint.

Determining which features are candidates for inclusion usually starts with product as the voice of the customer. Product will advocate for what the customer needs. The team will also identify what bugs and tech debt need to be addressed during the sprint.

Once the candidate features are identified the team will estimate the the level of effort for each story. At the formation of the sprint team a point system is determined and that point system is used for estimating work. The two most common point systems I've come across are using Fibonacci numbers (with 8 or 13 being the top end of a sprint) and using 1, 5, 10, 25, 50, and 100 as story points. Whatever your system of pointing there needs to be a way to determine small, medium and large stories.

There are many different estimating techniques but most teams I've seen use some sort of t-shirt size technique. They start by picking a story and assigning it an arbitrary point value. The rest of the stories are then pointed by determining if the story is larger or smaller than the initial story. When you have more than 3 assignments that you can use for point values you can get more granular and determine if the story is larger or smaller than the other stories in it's t-shirt size bucket.

Once the velocity is determined, potential features are identified, and stories are estimated you will be able to create your sprint backlog. Backlog creation begins by stack ranking the stories (taking dependencies into account) and cutting the sprint off after the last story that keeps the total number of sprint points at or less than the pre-determined sprint velocity.

Implementation

The second phase of the sprint life-cycle is implementation. During the implementation phase the team meets daily at stand-up and goes over what they worked on yesterday, what they're working on today, and any potential blocking issues. When a blocking issue is identified it's up to the scrum master to try to unblock the team member. During the implementation phase the software should be developed, tested, and integrated.

Review

The last phase of the sprint life-cycle is review. The review phase takes on two forms. The first is the sprint demo and the second is the sprint retrospective.

The sprint demo is the teams opportunity to showcase what they've built during the sprint. Sprint demos can actually be a very useful tool leading into the next sprint planning phase. Often when we think about planning with regards to software we think about what we're going to build from an architectural perspective. The problem with this line of thinking is that it typically leaves out integration, testing, and deployment. If we instead think about what we're going to demo during planning we'll have to take into account how we're going to demo it.

A note of caution about sprint demos. One mistake I've often seen teams make during sprint demos is to walk everyone through a bullet list of work that was performed. Walking others through a bullet list isn't a demo. If you find that your team is working on features that don't have a user facing component then figure out a creative way to tell the story of how what you worked on matters. Use an animation, a use case scenario, or maybe even a chart or some graphs to help illustrate what you worked on. 

The sprint retrospective is a mini postmortem. During the retrospective the team goes over what went well, what didn't go well, and what the team could have done better. The retrospective is a dedicated time for us to identify the issues that caused us to lose velocity, to call out our successes during the sprint, as well as identify what we could have done differently to have improved the sprint. Because the sprint review typically takes place before the planning of the next sprint we have an opportunity to make changes that give us a higher likelihood for success before the next sprint starts.

The retrospective IS NOT a finger pointing exercise. It's NOT a witch hunt or an opportunity to push blame around. The retrospective IS an opportunity to identify gaps in our planning, our process, or our estimates. The point of the retrospective is to maintain or increase our velocity.

Isn't there a phase missing?

You may be wondering where the "release" phase is. There isn't one. Releasing your software should be done as part of the implementation phase. Your sprint planning should account for what it takes to ship the software, if that's part of the definition of done for the sprint.

Agile Development: Sprint Teams

Sprints are the way that Scrum teams iterate over software. The goal of a sprint is to produce a working set of software that is potentially shippable. I say potentially because it's really up to the business to decide when the software is shippable. As long as the feature development is complete there is a choice. You can evaluate the number and severity of open bugs as well as the other features and make a call on whether you ship or not.

Sprint Teams

Sprint teams are comprised (at minimum) of a representative of product and the software development engineers. Though in practice you'll really want someone from QA (quality assurance, if your org has a separate QA group) and someone from operations represented.

Products Role

In Agile we favor collaboration over contract negotiation. The role of product management as part of the scrum team is to represent the voice of the customer. The product manager should be working directly with the customer to understand their needs. The product manager helps collaborate with the rest of the team on the requirements of the feature(s) being developed each sprint.

Developments Role

The developers are the ones actually building out the features. The role of the developer is to collaborate with product to turn the requirements into feature stories and tasks. The developer is also the advocate for the health of the code and architecture. The developer is responsible for making sure that unnecessary tech debt isn't accrued sprint over sprint. Often this means collaborating with product on the scope of work for each sprint to account for taking on bug fixes and technical debt.

Quality Assurances Role

Ideally all software we write includes unit, functional and integration tests. But that doesn't mean that all testing can be automated. There are some aspects of the user experience that may require manual testing. Or if your organization is new to agile you may have creating automated tests as tech debt. Whatever the reason, our goal of building potentially shippable software each sprint means including QA as part of the sprint team.

Operations Role

One other often overlooked member of the sprint team is operations. While continuous delivery and continuous integration are the ideals for deployment, many organizations have not shifted to that model yet. If your org is one that hasn't, including operations will help accurately account for the sprint work necessary to release the software.

Scrum Master

Each sprint has a scrum master. It is the role of the scrum master to facilitate scrum. This is done by managing the product backlog, facilitating the scrum meetings, facilitating the sprint planning meeting, facilitating the retrospective meeting, helping to define what done means for a given story or task, and removing roadblocks to the team. The scrum master is also responsible for removing roadblocks to the scrum team. This is usually done by fostering collaboration between teams and product.

Agile Development: The Need For Scrum

Over the last two decades the internet has become ubiquitous in our lives. The speed of our internet connections and performance of our computers have increased several orders of magnitude. These changes have had a cascading effect on the software industry. Software used to be run on hardware to expensive for the average person (or even company) to purchase and maintain. Bandwidth limitations required software to be distributed via a high cost and immutable media like the compact disk. As hardware prices decreased and the bandwidth and speed of our connections increased it became easier to distribute software over the web or even run it entirely in the browser using a combination of HTML, JavaScript and CSS. 

As software became easier to distribute our ability to innovate and provide our customers with fixes and new features became paramount to the success of a software product. The birth of the mobile computing industry with devices powerful enough to run both native and web based software intensified our need to respond to the ever changing needs of our customers.

One of the side affects of software distribution evolving has been that our processes, originally created to manage change over a long period of time, became inflexible and costly in a world where change was frequent. The software industry was plagued with process that couldn't keep up with the rapid multi-platform change the industry was going through.

This inflexibility in the software development process gave birth to the Agile movement. Agile recognized that responding to the changing needs of our customers has become as important or more important than following a long term plan. Collaboration on product requirements needed to take precedence over processes or tools. Agile understood that rather than formalizing requirements and managing those requirements as they flowed downstream that it needed to embrace iterative development and distribution to keep up with the ever changing needs of it's customers.

Agile understood that the cost of downstream change in a rigid process is greater than identifying the change upstream and pivoting earlier. In order to manage this change agile embraced the concept of Scrum. In traditional software development information flowed downstream from product to development, development to test, test to operations, and operations to the customer. If a change was needed it required a renegotiation of requirements and schedule due to the fact that any change restarted the process at the beginning.

Scrum increases the speed and flexibility of software development by bringing product, development, test and operations together into a cross-functional team responsible for adapting to change. Instead of requirements flowing downstream in a rigid process, requirements are collaborated on with product and test. Requirements are turned into features which are developed iteratively. At the end of each iteration the next set of requirements is defined. These requirements can build upon a previous iteration or completely change the direction of the software. Each iteration is time-boxed in order to allow for greater speed and flexibility in change.

Over the next few weeks I'll go in-depth into the different pieces of Agile Scrum with the goal of providing you a framework by which you can apply agile as part of your development process.

Minimizing the risk of bugs

Software development is a craft that requires practice, hard work and dedication. It's a craft that involves many edge cases and unintended consequences. As awful as it sounds, we've all got bugs in our code. The goal of writing software should not be to write software with no bugs as this is unattainable. Instead the goal should be to minimize the risk of high severity bugs.

Every change to your software is an opportunity to introduce new bugs. Following these tips will help you minimize the risk of introducing bugs into your system.

Test Your Software

Duh, right? Testing your software may sound like a no-brainer but you'd be surprised by how many times people break builds and introduce buggy software just because they didn't test their code. In my previous post on Testing Your Software Properly I provided a checklist of tests that you should run before you commit your code.

Without proper tests you can not have confidence that you aren't regressing an old bug or introducing a new bug into the system. Tests are the foundation for confidence in any change you make.

Reduce the surface area of your change

The more lines of code you change with every commit the higher the risk of bugs. This is where encapsulation, SOLID principles, and refactoring become so important.

It's important to encapsulate your software into individual loosely coupled pieces. If you make a change in a class and it causes cascading changes throughout the rest of your software in code un-related to your change then you're likely introducing new bugs.

If you follow the Single Responsibility Principle in SOLID your classes will have one reason and only one reason to change. This reduces the surface area of your change because you will not be changing code in unrelated modules.

Following principles like DRY and YAGNI will lead to more robust code that is flexible and easy to change.

Keeping your code simple is one of the keys to reducing the surface area of your change.

Reduce the complexity of the code

Overly complex code leads to bugs for many reasons:

• The code is fragile because the learning curve is steep.
• It's easier to do the wrong thing than it is the correct thing when making a change in the code.
• The code is not readable often causing you to make multiple context switches to understand a single workflow.

What are some signs that your code is too complex?

• The patterns in the code are not clear, obvious, and/or discoverable.
• You have multiple levels in indirection that support one workflow or use case.
• You have an abstraction that fronts a single concrete implementation.
• Your code does not have clear boundaries.
• You have highly interdependent modules.
• Your code is not highly cohesive.
• Your code has rigid rules that are not enforceable in their individual units but only as a whole.

Testing your software properly

Here's a general checklist of the type of tests that you can use to ensure that you're testing your software properly before you ship it. This isn't an exhaustive list but can be used as a starting point for you to write a more exhaustive list that's right for your software.

Compile before you commit

Good testing starts with making sure your code actually compiles. Yes, people actually check in code without compiling it first. This is just a dumb mistake and is 100% avoidable. 

Run a clean build before you commit

A somewhat non-obvious thing to make sure when compiling is that you're not using a cached compiled object that has changed. Often compilers will cache objects and attempt to track when the dependency changes and only recompile the dependent object when the compiler believes it has actually changed. Cached objects will then be linked against your code and make your software appear to work but those objects may have actually changed and broken functionality. Doing a clean build before you commit will ensure that an object you depend on hasn't changed in such a way as to break your code integration.

Happy Path Tests

Happy path tests should be your minimum bar when committing code. Happy path tests ensure that your code works as intended when used in the way it was designed. These tests can be thought of as functional tests. They test the functionality of the software and ensure that the software meets the business requirement.

Negative Path Tests

Negative path tests ensure that your software is resilient to change. Negative path testing includes using your code in ways for which it was not intended. Common tests include sending in null object parameters and testing upper and lower bounds of parameters. Negative path testing also includes testing that your software properly handles exceptions and throws the proper exceptions.

White Box Tests

White box tests ensure that your code works from the outside in. These are a set of tests that ensure your objects work from the consumers perspective. These tests include making sure the object can be created and initialized, that method calls work according to spec, and that the code does not misbehave from the callers perspective.

Black Box Tests

Black box tests ensure that your code works from the inside out. These tests require access to the internals of the object. Black box tests usually test the fitness of particular private methods and algorithms. 

Life-cycle Tests

You should test how your objects function in various aspects of the objects life-cycle. The key to life-cycle tests is to make sure that your objects mange state properly. Life cycle tests are also useful in making sure that you don't have any memory leaks in your code due to life-cycle changes.

Life-cycle testing includes testing the creation, destruction, concurrency and serialization of your objects. Two life-cycle areas that tend to cause bugs are not properly testing when the object state is saved or restored or when the object is used in a multi-threaded environment.

Integration Tests

Do you understand how your software works in the context of the larger system of components that use and are used by your software? Integration tests allow you to make sure that your software works end to end in the system as a whole. 

When Not To Refactor

Refactoring software is a crucial part of extending the life of software. Refactoring contributes to enhancing the maintainability of the software by incrementally improving the design, readability and modularity of the components. But not much has been said about when not to refactor software.

Don't refactor code unless you need to change the code for a business reason.

One of the common mistakes I often see with regards to refactoring is when people refactor code that doesn't need it under the guise of making it better. The argument usually goes something like "this needs to be more abstract", "I wrote this code a long time ago and it is crappy", "this code is too complex" or something along those lines.

You should only refactor code when you are already in the code to make a change to support the business. That may sound counter intuitive but one of the worst things we can do is change code, however crappy, unreadable or complex that doesn't have a reason to change.

Valid business reasons to change code include (but are not limited to):

• Adding new functionality
• Extending existing functionality.
• Making measurable performance improvements.
• Adding a layer of abstraction in order to support a new use case.
• Modularizing a particular object so that it can be reused in another part of the system

Adding new functionality or Extending existing functionality

This is where the boyscout rule comes into play. If you are in already in the code for another reason then you should clean up the code even if you didn't make the mess.

Making measurable performance improvements

This one is probably self explanatory but it's important to note that performance improvements will usually require some level of refactoring. 

Adding a layer of abstraction in order to support a new use case

This is an important one to understand. Often people will over generalize code at the beginning. This leads to overly complex designs and less readable code. If we follow the rule of not creating a layer of abstraction until we have at least two or three use cases for the code then there will come a point when you need to refactor the code in order to provide a layer of abstraction that doesn't already exist.

Until that second or third use case comes about the code should not be generalized. You don't have enough information about future uses of the code to get the abstraction correct. You may get lucky and guess at the future abstraction but you don't want to run your business on guesses and luck.

Modularizing a particular object so that it can be reused in another part of the system

Code reuse is one of the most important tenets of object oriented programming. When we identify code that is not specific to a particular object or package AND is needed in some other part of the system we should refactor this code into it's own module. Its important to ONLY do this when the code is actually needed in another part of the system.

Don't refactor code without tests

In order to refactor code safely you should have unit and integration tests for the existing functionality. I would also argue that you should write tests for the new functionality as well before you refactor. This will help you to understand the proper way to refactor the code as it helps you define how the refactored code should be used from a consumers standpoint.

If the tests don't exist for the the existing functionality you should write them first before you start refactoring. This helps ensure that you don't cause a new bug in the code or regress an old bug when refactoring.

How To Become A Great Software Engineer

I've been in the software industry for well over a decade now and today I thought I would reflect on the qualities I've seen in great software engineers that aren't just about the code they write.

Great software engineers are curious about everything. Because great software engineers are curious about everything they tend to dive much deeper into learning technologies. They're not satisfied with just learning how to use something but want to know how it works as well. This helps the engineer to better pick the *correct* solutions for the task at hand. It's important to note that diving too deep can also be a crux as you get stuck in analysis paralysis. Great software engineers recognize when they've learned enough to move on a start delivering.

Great software engineers are passionate about what they build. Not matter what they're building they want it to be great. Whether it's a script, a site, a process, or an application they want it to be used and want it to be a positive reflection of their ability. Because of this they give 100% on everything they build.

Great software engineers don't want to reinvent the wheel. Because these engineers are so curious they naturally become aware of what software is out there. They play with lots of different technologies, frameworks, and platforms. This gives them better judgement about what the correct solutions are to integrate with and what software they need to build themselves.

Great software engineers understand that software engineering is about more than just code. Software is part of a larger system. A great software engineer thinks about hardware, security, operations, deployment, and the maintainability of code.

Great software engineers are willing to do the stuff no one else wants to do. This doesn't mean that they volunteer for all the crappy work. But they understand that sometimes you have to pay your dues. They also understand that the crappy work is a good way to practice their skills.

Great software engineers have learned to make others better. These engineers have learned that you can't do it by yourself. They've also learned that not having a cohesive system is an anti-pattern. Because of this they *want* those around them to be better. Therefore they invest in teaching and mentoring others. In turn they learn more about their knowledge gaps and become better engineers themselves.

Transitioning to a professional software development role: part 3

In my first post in this series, Transitioning to a professional software development role: part 1, I started to outline some of the gaps I've seen in people's preparation for entering a career in the software development industry. I started off by focusing on what software development is not about.

In my second post in this series, Transitioning to a professional software development role: part 2, I took a look at what software development IS about. In the final post in this series I'd like to talk about the tools available that make us more efficient.

Being a good software developer means understanding how to apply agile

For a long time developing software was very much like developing a product on an assembly line. Assembly lines are very rigid and not well suited to respond to change. They run on the assumption that what happens upstream in the assembly line can be built upon and won't change. The moment change is introduced most of the product on the assembly line is ruined and must be thrown away.

Software's assembly line is called Waterfall. Overtime we've come to understand the downfall of waterfall and it's major flaw is that it's very rigid to change. Rigidity to change was okay when the primary delivery mechanism for software was the compact disk. But as software has grown to allow near real time delivery of features and functionality Waterfalls rigidity to change has become a hindrance to delivering high quality software in smaller but more frequent updates and features.

That's where Agile come in. Agile software development is about being able to respond to change in a rapid manner. It teaches us to think about software in a less monolithic manner but instead as a group of features that can be delivered in small chunks frequently over time.

I wrote a post several months ago called Software Craftsmanship: Project Workflow. If you're new to agile it's a good introduction to the anatomy of a project and what I've found useful. While the project workflow I've outlined isn't something you'll see in official Agile books, it is something that I have found extremely useful.

Being a good software developer means understanding how to use Lean

The concept of Lean Manufacturing was invented at Toyota. The primary goal was to reduce waste in the manufacturing cycle. This was done by re-thinking the manufacturing process to identify and remove waste. On example of waste could is parts sitting in a queue waiting to be processed. Toyota was able to show that by re-engineering their manufacturing process they could improve quality, efficiency, and overall satisfaction of customers.

The concepts behind Lean Manufacturing can also be applied to software development. Unfortunately these concepts often are applied incorrectly and have lead to many misconceptions and misunderstandings of Lean Software development. I wrote a post several months ago which outlined common misunderstandings in applying Lean to software development. 

As a professional software developer it's important to understand Lean and how to apply it to developing software.

Being a good software developer means understanding how to make trade-offs 

The last area I want to briefly cover is understanding how to make trade-offs. As a professional software developer you're going to be asked to make trade-offs all the time. Sometimes it will come in the form of quality (a bad trade-off IMO). Other times it will come in terms of features.

The key to understanding how to make trade-offs is learning to ask a few questions.

• What am I gaining by making this trade-off?
• What do I not get that I would gotten if the trade-off was not made?
• What downstream affects will this decision have on my long term strategy or road map?
• What additional work will be required later as a result of this trade-off?

The ultimate goal in software development is to provide business value in every part of the process. Understanding how to make trade-offs will help you provide the right business value at each step in the process.

Transitioning to a professional software development role: part 2

In my previous post, Transitioning to a professional software development role: part 1, I started to outline some of the gaps I've seen in people's preparation for entering a career in the software development industry. I started off by focusing on what software development is not about.

In this post I want to take a look at what software development IS about.

Being a good software developer is about understanding data structures

The foundation of a good software developer is understanding data structures and object oriented programming. Data structures like Binary Trees, Hash Tables, Arrays, and Linked Lists are core to writing software that is functional, scalable, and efficient.

It's not just good enough to understand what the data structures are and how they're used. It's crucial that you also understand WHEN to use them. Understanding when to use particular data structures properly comes with a few benefits. First, it helps others intuitively understand your code. Others will be able to understand your frame of reference better. Second, it helps you avoid "having a hammer and making everything a nail" syndrome. That's when you're learning something new and looking for places to apply your new knowledge, often shoehorning it in to places it doesn't belong.

Being a good software developer is about being able to estimate your work

I can't stress enough how important this is. Your team, your managers, and your customers are going to rely on you for consistency. They're going to make plans around what you do. And because of this learning to estimate your work is crucial in helping you and them meet commitments. Understanding how to estimate your software well also helps you build a regular cadence in what you deliver which is helpful for your customers.

There are a three concepts that I've found that really helped me learn to estimate my work well.  The first is the Cone of Uncertainty. This concept is really helpful because it helps you tease out what you know you don't know as well as what you don't know you don't know. Understanding the cone of uncertainty helps you remove ambiguity in what you're working on which in turn helps you better understand the level of effort it will take.

Once you've teased out the uncertainty in your work you can use Planning Poker as a way to quantify how much work something is. It's important that you try not to tie your poker points to a time scale as it will tend to skew your pointing exercise. Instead, as you get better about learning to quantify how much work something is relative to your other work you'll start to naturally see how much time it takes. For instance let's say you use fibonacci numbers 1, 2, 3, 5, 8, and 13 to quantify you're work. Over time as you get better at pointing your work, you'll also see a trend in how much time certain points take. Only then can you accurately associate a timescale with your pointing.

The last concept that I've found very helpful in learning to estimate how much work I can do in any given period is by tracking my velocity. If you're using planning poker to determine how big the chunks of work are and you're using agile to set a cadence or rhythm for when you deliver your work, then velocity tracking can help you be more predictable in how much work you can deliver in any given agile sprint. Understanding your velocity helps you to set reasonable expectations on what you can deliver and helps those that are planning for the future understand what it would take to decrease the time of a project or make sure that a project is on track and will meet it's deliverable dates.

Being a good software developer is about re-use in order to avoid re-inventing the wheel

As newer engineers we want to solve problems that we find interesting and a challenge. Often as we get into the depths of a particular problem space it will be evident that you're trying to solve an already solved problem. At this point you're at a cross roads where you can continue down the path of solving the problem yourself and re-invent the wheel. Often this is the result of both curiosity and mistrust. You're curious about how to solve a particular problem or curious about whether you could solve the problem better than those that have come before you. This also happens when we don't trust that a particular library actually solves the problem you're trying to solve. Or because another solution solves a slightly different, but compatible problem, we don't trust that our problem is in the same problem space.

This is very detrimental to a project for a few reasons. First, the problem has already been solved so you're going to waste time solving an already solved problem. Second, it's likely the case that the problem is more nuanced than you're aware of. It's also likely the case that the people who have already solved the problem have dedicated themselves to solving that problem. I.e. it's the entirety of their problem domain. This means that they're going to be the subject matter experts in this area. Because this is only one part of your overall problem you won't be able to dedicate the required amount of time solving the problem as well.

I would encourage you to first look to see if someone has already solved your problem either in part or in whole. There's plenty of high quality open source projects on GitHub and SourceForge. These projects have people who are eager for you to use and incorporate their projects into your project.

Being a good software developer is about knowing the limits of your understanding

There are several aspects to understanding the limits of your understanding. One aspect is to know that knowledge about any particular domain has both a breadth and a depth to it. It is impossible to gain both a breadth and depth of understanding in all areas of software development amongst all subject domains. Because if this it's important to be aware of what you have a breadth of understanding in but are lacking depth and what you have a depth of understanding in but don't have a breadth of understanding. Over time you'll develop both a depth and a breadth of understanding in a few particular subject areas. But it's important to know that this takes time, theory, and practice. Without all three of those you won't gain the breadth and the depth.

Knowing the limits of your understanding also involves being able to say you were wrong. There are going to be plenty of times when you thought you had a depth of understanding or breadth of understanding of something only to find out you didn't fully understand or misunderstood the subject. Being able to say you were wrong is the first step to correcting your understanding and being able to build on your new knowledge.

Transitioning to a professional software development role: part 1

I've spent 14+ years in the software industry in either an IC (individual contributor) role, as an engineering lead, or as a manager. I've worked in both the public and private sector. I've worked at companies as large as 100,000+ people and as small as 19 people. One thing that's been pretty consistent over time is that people first entering into the software industry are ill-prepared for what it means to be a professional software developer. This is equally as true for those coming out of college as it is for those transitioning to software from another industry.

What I'd like to do in this post is outline some of the gaps I've seen in people's preparation and try to pave the way toward helping those interested in software development understand what's expected of them in the industry and how to be prepared.

While the following post will focus on being a good software developer, most of what I outline is applicable to other roles in the software industry such as project/program/product management.

This will be a mult-part post. In part one I will focus on what software development is not.

Being a good software developer is not just being able to code

You're part of a team. Software development isn't just about solving problems with efficient algorithms. You're part of a team which is part of a larger ecosystem. There are product people trying to manage the vision of the software. Their are project people trying to manage the cadence of the software life-cycle. There are other engineers consuming the output of your work. There are internal and external customers trying to use your software to make their lives more meaningful either by being more efficient, participating in some sort of community, or just goofing off playing a game you've written.

Because of this people are relying on you to be an effective communicator. They're relying on you to be effective with time management. They're relying on you to ask for help when you get stuck. They expect you not to go dark. And they're relying on you to help them out when they get stuck.

Essentially you're part of a new tribe, each person having different but overlapping responsibilities. It's important to remember to grow your skills both technically AND with soft skills.

Being a good software developer is not about being clever

One of the biggest mistakes I see newer folks in the software industry make is trying to be too clever in their solutions. Writing software that lasts is about simplicity. Learning to write simple code that clearly communicates it's intentions and intended purpose(s) means that it will be used effectively. Writing code that is clear means that it's readable.

In the software industry you're going to spend more of your time reading other peoples code than you will actually writing code. It's important to learn what it means to write readable code.  I would highly recommend you read the book Clean Code: A Handbook of Agile Software Craftsmanship.

Being a good software developer is not about personal style

Every industry has it's own DSL (domain specific language). That DSL helps people to communicate more effectively within the industry by removing ambiguity and subjectivity. Software development has several different layers of DSLs that it is important to learn.

There are language specific idioms and standards that it's important to be familiar with. There are platform specific standards. For instance standard *nix programs tend to do one thing that can be chained (or composed) with other programs (by piping) to serve some larger purpose. Whereas on the other hand, Windows programs tend to be monolothic in nature and self contained. It's important to know what the standards are for the platform you're working on.

In the same way there are going to be general coding standards that are industry accepted as well as coding standards that are specific your new organization. Your organization will also likely have it's own set of standard tooling for development, deployment, and distribution.

Conditional logic In Ant

Every so often I find myself needing some conditional logic in my ant build files based on either an automated build property or some property set by current state.

There is an open source library, ant-contrib, which gives you if/else statements in your Ant build files but I tend to not use ant-contrib for three reasons. First, it adds bloat to my project because of the requirement to include it's jar in my projects classpath. Second, you have to mess around with defining tasks in your build files which I just don't feel are very intuitive. Lastly, Ant already includes the ability to perform conditional logic by taking advantage of the Ant target's if attribute.

Performing conditional logic in Ant without an additional library is pretty easy. You simply need to define three targets. The first is the target you want to run in the TRUE scenario. The second is the target you want to run in the FALSE scenario. The third is the target that sets a property (or properties) based on some condition and calls the other targets.

Let's take a look at a very simple build file. This will print This build IS *nix if the isUnix property is set to true otherwise it will print This build is NOT *nix.

<?xml version="1.0" encoding="UTF-8"?>
<project name="example">
    <condition property="isUnix"><os family="unix"/></condition>

    <target name="-unix-build" if="performUnixBuild">
        <echo>This build IS *nix</echo>
    </target>

    <target name="-non-unix-build" if="performNonUnixBuild">
        <echo>This build is NOT *nix</echo>
    </target>

    <target name="build">
        <condition property="performUnixBuild"><istrue value="${isUnix}" /></condition>
        <condition property="performNonUnixBuild"><isfalse value="${isUnix}" /></condition>

        <antcall target="-unix-build" />
        <antcall target="-non-unix-build" />
    </target>

</project>

You can see this in action by copying that file to a machine with Ant on it and running:

$ ant build.

If you're on a Unix like machine it will print This build IS *nix otherwise it will print This build is NOT *nix.

You can test the else logic by overriding the isUnix property at the command line using:

$ ant -DisUnix=false build.