A brief overview of The Cloud

In my first post on The Cloud I explained how The Cloud was born. In this post I want to give a high level overview of The Cloud and explain why you should care about it.

At a very high level The Cloud can be broken up into two over lapping but distinct groups denoted by their use cases. I'll refer to the first group as The Consumers Cloud and the second as The Engineers Cloud. My next post in this series will go into more detail about The Consumers Cloud, while my last post will go over The Engineers Cloud.

The commonality between both groups is that The Cloud is a set of servers distributed in multiple regions throughout the world. Having multiple servers is what allows The Cloud to handle a lot of data. Having those servers distributed regionally around the world allows The Cloud to be fast by reducing the distance between you and the data that is stored in The Cloud.

The Infrastructure of The Cloud

Cloud providers like Amazon AWS, Microsoft Azure, Google Cloud Computer, and etc provide a set of services that abstract interacting with their servers so that the software their customers build don't have to worry about the details of each individual machine.

Most Cloud providers take concepts like file storage, complex computation, geographic regionalization, and security and make them scalable with demand. If a service has high demand it can have lots of resources to meet that demand. If the demand is low it can use less resources. This is very different than the data center model which required a fixed number of servers. If you needed to scale your service you had to purchase additional hardware. When your demand was low those extra servers were sitting around idle. One of the goals for Cloud providers is to optimize use. This allows customers to pay for only what they use when they need it and to optimize for their particular usage pattern.

Why you should care about The Cloud

The Cloud is what enables so much of what we've come to rely on in our every day lives. When you purchase music from one of the many streaming services, you're investing in The Cloud. When you watch Netflix or Amazon Prime, you're investing in The Cloud.  In fact, you wouldn't have applications like Dropbox, Instagram, Facebook, iTunes, YouTube or etc without The Cloud.

How The Cloud Was Born

The Cloud has become a phrase with so much meaning that it has become meaningless. Very much like what Web 2.0 was in the early 2000's. Often two people talking about The Cloud can be talking about two completely different things.

This series is not meant to teach you how to use The Cloud or how to get your start-up going on The Cloud. The purpose of this series is to provide a brief overview of some of the typical uses and meanings of The Cloud to help make sense of how it fits into your everyday life.

I'd like to start this series on The Cloud by explaining at a high level how The Cloud was born. I'm only going to deal with the concepts of The Cloud and not on specific milestones and companies that contributed to the birth of The Cloud.

The Data Center

The Data Center can be thought of as the predecessor of The Cloud. I say predecessor because for most The Cloud replaced the traditional data center. But predecessor is really a misnomer because The Cloud couldn't exist with the data center. In fact The Cloud is a series of data centers interconnected with a set of services that abstract interacting with the servers and services running in them. The Cloud is really just a commoditization of the traditional data center.

Running a data center meant that your company had to have more than one core competency. In addition to the core competency of building whatever it is that your company built you also had to have expertise in infrastructure, operations, and network engineering (at a minimum).

Infrastructure

Running a data center meant you had to have experts on your payroll that understood server hardware. Running your software and services in a data center meant you either owned the data center yourself or you were renting space in someone else's data center (more likely the case). In either case your primary goal was making sure the servers stayed up and running. Your secondary goal was to try to reduce your capital expense and increase the efficiency of your data center by purchasing less and running more on what you purchased.

Operations

IT Operations specializes in the deployment and maintenance of the software and services you run. In the data center world an IT Ops engineer will plan and execute the deployment of software either in the traditional model with a hand-off from software engineering to operations or with the now more common model of DevOps.

The Cloud doesn't totally negate the role of operations but instead augments the software engineers role to also include deployment and maintenance of the software. This is usually done using the Continuous Delivery model of software development.

Network Engineering

Running a data center also required your company to have expertise in network engineering. Network engineers are responsible for the communications infrastructure within the data center. They manage how data gets in and out of the data center as well as between servers and services in the data center. They maintain the physical and virtual network infrastructure.

And Then Came The Cloud

As companies focused more and more on the delivery of their own software a market emerged where companies provided the ability for you to run your software and services on their servers and offloading the management your IT infrastructure, operations, and networks to them. Thus The Cloud was born.

Is it a job or a career?

Every so often I'll be having a conversation with someone who is not in the technology industry and the conversation will turn to the fact that people in the software industry are "always on". They note that we don't work a typical 9-5, that we're often working nights and weekends either checking email, coordinating with someone about a project we're working on, or simply writing some code.

I would venture to guess, at least from the engineers perspective, that this has a little to do with the fact that a large number of software engineers are in the industry because it was a hobby before it was a job. A lot of us decided to write software as our day job simply because we could get paid for doing something we were already doing in our free time. But that's tangential to the "always on" appearance we seem to have.

I'd like to posit that it's not something unique to the software industry but points to the difference between a job and a career. In fact I would argue that the difference between a job and a career has nothing to do with what your role is or what title is currently listed on your business card (or if you have a business card at all).

The dictionary definitions of job and career don't seem to vary very much. A job is a place you get paid for work you perform. A career is an occupation you've had for an extended period of your life. Really, you could say that the dictionary defines a career as a job that you've had for a long time. But I think those definitions miss the point. There's more to a career in my opinion, or at least a successful one.

Many people have jobs that they go to day in a day out for the majority of their lives. They work long hours, nights, and/or weekends. They stay in these jobs for many many years earning a paycheck and checking out when they're not in the office. This may be a very satisfying job but to me this is not a career.

To me, a career starts with a job that you want to be at. Not just because that job pays or has the ability for you to "turn off" when you get home. Instead, it's a job that gives you an opportunity to grow both as a person and in your field. A job that stretches you both mentally and (if you're lucky) physically. A career is challenging, causing you to navigate uncharted territory. A career is something you invest in and at the end have been changed by. 

I don't think a career is about whether you're "always on" or able to "check out" at the end of the day, it's about whether or not you're growing, changing, and being challenged.

If you're in a job that you thought was a career, what's stopping you from the career you want?

Software Estimation: How to estimate software accurately part 3

In this series on software estimation I defined an accurate estimate as something that will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

In part 1 I explained how to account for risk and unknowns in your software estimation. In part 2 I explained how to quantify the work that is known. In the final post of this series I explain the last pieces needed to give an accurate software estimation.

Identifying Opportunity Cost

Let's say we estimate project X and determine that it will take 2 developers 2 months to complete. After estimating project Y we know that it will take 2 developers 1 month to complete (or 1 developer 2 months). Finally, when estimating project Z we determine that it will take 1 person 1 month. We now have a starting point from which to determine, based on cost to build and return on investment, if there is an opportunity loss associated with starting project X before projects Y or Z.

Refining As New Information Becomes Available

Most software estimates will start out as a range. Typically larger projects will have a wider range. This is because there are always things we know we don't know about the project as well as things that we don't know that we don't know.

It is easier to reduce ambiguity with information we know we don't know because we know where to start. As you practice software estimation you'll start to learn the correct questions to ask to identify the information you don't know you don't know.

Identifying new information should result in refining your previous estimate. The more ambiguity you are able to remove the more accurate your software estimate can be. The goal should be to continually narrow the range of the estimate.

More Accurate When Accompanied By A Confidence Interval

A confidence interval is a range of numbers from which we expect the real value to be contained. Having a good confidence interval allows us to determine the range of our estimate. This range can then be examined to validate our assumptions and to identify ambiguity.

For example, if I asked you to estimate how many marbles fit in a mason jar the actual number of marbles that would fit would be contained in your estimate. My 90% confidence interval may be 25 - 100 marbles. The actual number of marbles may be 87, in which case my interval contained the real value.

If you're interested in learning more about how to compute a confidence interval you should read How to Measure Anything by Douglas Hubbard.

Identify And Clarify Ambiguity

I'll wrap up this post with some questions you can ask to identify and clarify ambiguity in your software estimates.

• What assumptions does your estimate rely on?
• What are the risks associated with your estimate? Identify the things that could cause your estimate to be wrong.
• What external dependencies does your project have? For example OS updates, product launches, planned outtages, SDK updates, etc.
• What data do you need? Is it already available in a consumable format?
• What services are required? Do they exist? Do they talk the correct protocol?
• Are there any User Interface (UI) or User Experience (UX) dependencies that need to be resolved?

Software Estimation: How to estimate software accurately part 2

In this series on software estimation I defined an accurate estimate as something that will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

In part 1 I explained how to account for risk and unknowns in your software estimation. In this post I'll explain how to quantify the work that is known.

Quantifying The Known Work

All software has to be designed, architected, implemented, tested, debugged, and deployed. Each of these areas needs to be accounted for in the software estimate. Here are a set of questions that should be asked for each area to help you understand the level of effort associated with building your software.

Using these questions you can break out the known work into smaller buckets of work. You can then estimate each of these smaller pieces independently.

Design

Has the user experience and user interaction been defined? Do people interact directly via a user interface you provide or indirectly from another user interface? Have all the transitions and states been defined?

Will your software be available in multiple regions and/or locales? Does the design take regionalization and localization into account? Who provides the translations?

Architecture

What platform will the software run on? Are there platform specific requirements or constraints? Does hardware need to be purchased or provisioned?

What abstractions need to be defined? What is the hierarchy and/or composition of the abstractions?

Does the software rely on persistent data? How is it generated? How is it updated? What data specific workflows are there?

What metrics do you need to track?

How is the software going to scale? Are there single points of failure? Is the software going to be memory bound and/or CPU bound?

What is the software NOT going to do?

Versioning

Is your software released incrementally? How will the system handle data structure changes?

How are functionality updates handled in your system? Do updates need to be isolated from each other or can they co-exist. Is there a need to maintain backwards compatibility?

Implementation

What is the source control mechanism? Is it already setup? Does everyone have access? Is there a branching strategy?

Who is going to implement the software? Do they have all the hardware and software they need? Do you they know the platforms or technologies that your project uses? Do they require time to ramp up on the platform or tools? Is external training required?

Testing

Is your project going to use continuous integration? If not, why? Is your project going to use continuous deployment? If so, how do you deal with a failed deployment?

How are bugs going to be tracked?

How are the units of software defined? Are there tests for all units? A good software project has tests for each unit of work as well as tests for the integration of multiple units.

Are your tests automated? How often are the tests run and by whom?

Do you need black box testing, white box testing, or a combination of both for your software? Are you testing more than just the happy path or are you doing negative case testing as well? Does your software need stress or load tests?

Does your testing require specific hardware or software? Do these need to be procured?

Debugging

What tools are going to be used to debug the software? How are you going to profile performance?

What needs to be logged? How often and by what components? How are logs collected?

How are errors reported? What work is required to make errors in the system reproducible?

Deployment

How is the software going to be released? How is your software going to be monitored? How is your infrastructure going to be built out? Can you automate the infrastructure build out using tools like Chef or Puppet?

How is the software documented? How is the documentation updated?

What's Next?

In my next post I'll explain the last pieces needed to give an accurate software estimation.

Software Estimation: How to estimate software accurately part 1

In my last post Software Estimate: What you're doing wrong I explained why people tend to create inaccurate software estimates. In this post I'll outline what's necessary to create an accurate software estimate.

Previously I defined an accurate estimate as something that will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

Let's take a deeper dive into each of these areas with the goal of helping you to understand what information you need to gather to accurately estimate your software.

Identifying Risk and Unknowns

It is inevitable that there is risk associated with the software you're building. Risk in a software project is typically associated in one or more of the following areas:

• Blocking another project.
• Being blocked by another project.
• Requirements or scope that haven't been defined.
• Attrition.

If we understand the areas that pose risk to our project we can use that information to inform our estimate. Your project may need additional time to coordinate with another team. You may need to account for another projects schedule and/or risk.

Blocking another project

While blocking another project many not sound like something that needs to be taken into account when estimating software it can actually be the source of unexpected work. The other project may have requirements for your software that you're unaware of. You may have to make last minute changes to your software to account for these requirements. This is especially true if the other project is a higher priority project for the company than yours.

It's important from an estimation standpoint to understand if there are other projects that need to consume your software. If so, how well is the dependency between your project and theirs defined? Does the other project take direct dependencies on your software (for example by compiling against your library)? If you're providing a service how well is the service contract defined?

When you update your software what are the downstream effects on consumers of your software? Does your software require versioning?

Being blocked by another project

What other components, libraries, or services does your software depend on? Have they already been shipped? If so how do you deal with updates to the software? If the software hasn't already shipped, are their milestones known?

What happens if they can't deliver or aren't able to deliver in time for your project to meet it's goals? Do features in your software get removed? How long will it take to remove or update the affected components?

Requirements that haven't been defined

Blocking another project or being blocked by another project are not the only sources of undefined requirements. Additional requirements many come from your product owners due to under-defined user experiences or user interactions.

Are there mock ups of the user experience and user interaction? Have static web based mocks that people can touch and feel been built? Are all the transitions defined? What state does your application need or maintain?

Attrition

While attrition is often difficult to account for in the actual estimate, asking yourself some attrition related questions will help you identify work that should be accounted for in your estimate.

Does your software require a specialized skillset or tools? How long does it take to aquire the tools? What's the ramp up time to learn the skillset? What's the lead time that it takes to hire someone with the needed skills or platform knowledge?

What's Next?

In my next post I'll explain how to quantify the work that is known.

Software Estimation: What you're doing wrong

During the beginning of my career as a software development engineer one of things that I dreaded the most was the beginning of any project where I knew I was going to be asked to give an estimate for the software I was going to write. Often I thought "how can I estimate something I have no clue how I'm even going to build?"

Ultimately the dread I felt came down to this one simple fact. I had no clue how you could accurately estimate software. Over the years, as I've learned to estimate software development much more accurately, I've learned that what I thought of as software estimation was built on several assumptions that were incorrect.

Fallacies of Estimation

• Estimates require giving a specific date
• Estimates should be given on the spot
• Estimates can't change

Estimates Require Giving a Specific Date

This fallacy was probably the heart of the anxiety I felt about software estimation. I always assumed that an estimate was a fixed amount of time like 2 days or 1 month. It never occurred to me that an estimate could be a range. Fixed dates are rigid and inflexible and are easy to get wrong. Fixed dates cannot account for the fact that there are unknowns associated with the project. 

An accurate estimate will be a range. Ranges are more flexible and can help account for the unknown and any risks associated with the project.

Estimates should be given on the spot

This fallacy is based on the assumption that some big decision needs to be made right now and doesn't have time to wait for you to think about what unknowns you're dealing with. First off, if you're making a big decision on the spot based on a back of the hand estimate you shouldn't be in your role and shouldn't have the responsibility that you do. It's careless and dangerous to you, your product, and your customers to expect some engineer to accurately give you an estimate on the spot.

An accurate estimate will require you to do some research.

Estimates can't change

It may be true that the original estimate needs to be something that you can make a big decision or commitment based on. It's not true however that estimation stops once that decision is made or a project is put into motion. An estimate needs to be a living thing.

Estimates are about approximations. They're about dealing with ambiguity. As things become less ambiguous your estimate should be refined.

Where to go from here?

Now that we know the fallacies of software estimation we can define the characteristics of an accurate software estimate. An accurate estimate will:

• Give you an understanding of risk and unknowns.
• Quantify the known work.
• Be something that you can base a big decision on.
• Be refined as new information becomes available.
• Be a range with a 90% confidence interval.

In my next post I'll talk about the truths of software estimation and give you a set of resources to help you create accurate software estimations.