Transitioning to a professional software development role: part 3

In my first post in this series, Transitioning to a professional software development role: part 1, I started to outline some of the gaps I've seen in people's preparation for entering a career in the software development industry. I started off by focusing on what software development is not about.

In my second post in this series, Transitioning to a professional software development role: part 2, I took a look at what software development IS about. In the final post in this series I'd like to talk about the tools available that make us more efficient.

Being a good software developer means understanding how to apply agile

For a long time developing software was very much like developing a product on an assembly line. Assembly lines are very rigid and not well suited to respond to change. They run on the assumption that what happens upstream in the assembly line can be built upon and won't change. The moment change is introduced most of the product on the assembly line is ruined and must be thrown away.

Software's assembly line is called Waterfall. Overtime we've come to understand the downfall of waterfall and it's major flaw is that it's very rigid to change. Rigidity to change was okay when the primary delivery mechanism for software was the compact disk. But as software has grown to allow near real time delivery of features and functionality Waterfalls rigidity to change has become a hindrance to delivering high quality software in smaller but more frequent updates and features.

That's where Agile come in. Agile software development is about being able to respond to change in a rapid manner. It teaches us to think about software in a less monolithic manner but instead as a group of features that can be delivered in small chunks frequently over time.

I wrote a post several months ago called Software Craftsmanship: Project Workflow. If you're new to agile it's a good introduction to the anatomy of a project and what I've found useful. While the project workflow I've outlined isn't something you'll see in official Agile books, it is something that I have found extremely useful.

Being a good software developer means understanding how to use Lean

The concept of Lean Manufacturing was invented at Toyota. The primary goal was to reduce waste in the manufacturing cycle. This was done by re-thinking the manufacturing process to identify and remove waste. On example of waste could is parts sitting in a queue waiting to be processed. Toyota was able to show that by re-engineering their manufacturing process they could improve quality, efficiency, and overall satisfaction of customers.

The concepts behind Lean Manufacturing can also be applied to software development. Unfortunately these concepts often are applied incorrectly and have lead to many misconceptions and misunderstandings of Lean Software development. I wrote a post several months ago which outlined common misunderstandings in applying Lean to software development. 

As a professional software developer it's important to understand Lean and how to apply it to developing software.

Being a good software developer means understanding how to make trade-offs 

The last area I want to briefly cover is understanding how to make trade-offs. As a professional software developer you're going to be asked to make trade-offs all the time. Sometimes it will come in the form of quality (a bad trade-off IMO). Other times it will come in terms of features.

The key to understanding how to make trade-offs is learning to ask a few questions.

• What am I gaining by making this trade-off?
• What do I not get that I would gotten if the trade-off was not made?
• What downstream affects will this decision have on my long term strategy or road map?
• What additional work will be required later as a result of this trade-off?

The ultimate goal in software development is to provide business value in every part of the process. Understanding how to make trade-offs will help you provide the right business value at each step in the process.

Transitioning to a professional software development role: part 2

In my previous post, Transitioning to a professional software development role: part 1, I started to outline some of the gaps I've seen in people's preparation for entering a career in the software development industry. I started off by focusing on what software development is not about.

In this post I want to take a look at what software development IS about.

Being a good software developer is about understanding data structures

The foundation of a good software developer is understanding data structures and object oriented programming. Data structures like Binary Trees, Hash Tables, Arrays, and Linked Lists are core to writing software that is functional, scalable, and efficient.

It's not just good enough to understand what the data structures are and how they're used. It's crucial that you also understand WHEN to use them. Understanding when to use particular data structures properly comes with a few benefits. First, it helps others intuitively understand your code. Others will be able to understand your frame of reference better. Second, it helps you avoid "having a hammer and making everything a nail" syndrome. That's when you're learning something new and looking for places to apply your new knowledge, often shoehorning it in to places it doesn't belong.

Being a good software developer is about being able to estimate your work

I can't stress enough how important this is. Your team, your managers, and your customers are going to rely on you for consistency. They're going to make plans around what you do. And because of this learning to estimate your work is crucial in helping you and them meet commitments. Understanding how to estimate your software well also helps you build a regular cadence in what you deliver which is helpful for your customers.

There are a three concepts that I've found that really helped me learn to estimate my work well.  The first is the Cone of Uncertainty. This concept is really helpful because it helps you tease out what you know you don't know as well as what you don't know you don't know. Understanding the cone of uncertainty helps you remove ambiguity in what you're working on which in turn helps you better understand the level of effort it will take.

Once you've teased out the uncertainty in your work you can use Planning Poker as a way to quantify how much work something is. It's important that you try not to tie your poker points to a time scale as it will tend to skew your pointing exercise. Instead, as you get better about learning to quantify how much work something is relative to your other work you'll start to naturally see how much time it takes. For instance let's say you use fibonacci numbers 1, 2, 3, 5, 8, and 13 to quantify you're work. Over time as you get better at pointing your work, you'll also see a trend in how much time certain points take. Only then can you accurately associate a timescale with your pointing.

The last concept that I've found very helpful in learning to estimate how much work I can do in any given period is by tracking my velocity. If you're using planning poker to determine how big the chunks of work are and you're using agile to set a cadence or rhythm for when you deliver your work, then velocity tracking can help you be more predictable in how much work you can deliver in any given agile sprint. Understanding your velocity helps you to set reasonable expectations on what you can deliver and helps those that are planning for the future understand what it would take to decrease the time of a project or make sure that a project is on track and will meet it's deliverable dates.

Being a good software developer is about re-use in order to avoid re-inventing the wheel

As newer engineers we want to solve problems that we find interesting and a challenge. Often as we get into the depths of a particular problem space it will be evident that you're trying to solve an already solved problem. At this point you're at a cross roads where you can continue down the path of solving the problem yourself and re-invent the wheel. Often this is the result of both curiosity and mistrust. You're curious about how to solve a particular problem or curious about whether you could solve the problem better than those that have come before you. This also happens when we don't trust that a particular library actually solves the problem you're trying to solve. Or because another solution solves a slightly different, but compatible problem, we don't trust that our problem is in the same problem space.

This is very detrimental to a project for a few reasons. First, the problem has already been solved so you're going to waste time solving an already solved problem. Second, it's likely the case that the problem is more nuanced than you're aware of. It's also likely the case that the people who have already solved the problem have dedicated themselves to solving that problem. I.e. it's the entirety of their problem domain. This means that they're going to be the subject matter experts in this area. Because this is only one part of your overall problem you won't be able to dedicate the required amount of time solving the problem as well.

I would encourage you to first look to see if someone has already solved your problem either in part or in whole. There's plenty of high quality open source projects on GitHub and SourceForge. These projects have people who are eager for you to use and incorporate their projects into your project.

Being a good software developer is about knowing the limits of your understanding

There are several aspects to understanding the limits of your understanding. One aspect is to know that knowledge about any particular domain has both a breadth and a depth to it. It is impossible to gain both a breadth and depth of understanding in all areas of software development amongst all subject domains. Because if this it's important to be aware of what you have a breadth of understanding in but are lacking depth and what you have a depth of understanding in but don't have a breadth of understanding. Over time you'll develop both a depth and a breadth of understanding in a few particular subject areas. But it's important to know that this takes time, theory, and practice. Without all three of those you won't gain the breadth and the depth.

Knowing the limits of your understanding also involves being able to say you were wrong. There are going to be plenty of times when you thought you had a depth of understanding or breadth of understanding of something only to find out you didn't fully understand or misunderstood the subject. Being able to say you were wrong is the first step to correcting your understanding and being able to build on your new knowledge.

Transitioning to a professional software development role: part 1

I've spent 14+ years in the software industry in either an IC (individual contributor) role, as an engineering lead, or as a manager. I've worked in both the public and private sector. I've worked at companies as large as 100,000+ people and as small as 19 people. One thing that's been pretty consistent over time is that people first entering into the software industry are ill-prepared for what it means to be a professional software developer. This is equally as true for those coming out of college as it is for those transitioning to software from another industry.

What I'd like to do in this post is outline some of the gaps I've seen in people's preparation and try to pave the way toward helping those interested in software development understand what's expected of them in the industry and how to be prepared.

While the following post will focus on being a good software developer, most of what I outline is applicable to other roles in the software industry such as project/program/product management.

This will be a mult-part post. In part one I will focus on what software development is not.

Being a good software developer is not just being able to code

You're part of a team. Software development isn't just about solving problems with efficient algorithms. You're part of a team which is part of a larger ecosystem. There are product people trying to manage the vision of the software. Their are project people trying to manage the cadence of the software life-cycle. There are other engineers consuming the output of your work. There are internal and external customers trying to use your software to make their lives more meaningful either by being more efficient, participating in some sort of community, or just goofing off playing a game you've written.

Because of this people are relying on you to be an effective communicator. They're relying on you to be effective with time management. They're relying on you to ask for help when you get stuck. They expect you not to go dark. And they're relying on you to help them out when they get stuck.

Essentially you're part of a new tribe, each person having different but overlapping responsibilities. It's important to remember to grow your skills both technically AND with soft skills.

Being a good software developer is not about being clever

One of the biggest mistakes I see newer folks in the software industry make is trying to be too clever in their solutions. Writing software that lasts is about simplicity. Learning to write simple code that clearly communicates it's intentions and intended purpose(s) means that it will be used effectively. Writing code that is clear means that it's readable.

In the software industry you're going to spend more of your time reading other peoples code than you will actually writing code. It's important to learn what it means to write readable code.  I would highly recommend you read the book Clean Code: A Handbook of Agile Software Craftsmanship.

Being a good software developer is not about personal style

Every industry has it's own DSL (domain specific language). That DSL helps people to communicate more effectively within the industry by removing ambiguity and subjectivity. Software development has several different layers of DSLs that it is important to learn.

There are language specific idioms and standards that it's important to be familiar with. There are platform specific standards. For instance standard *nix programs tend to do one thing that can be chained (or composed) with other programs (by piping) to serve some larger purpose. Whereas on the other hand, Windows programs tend to be monolothic in nature and self contained. It's important to know what the standards are for the platform you're working on.

In the same way there are going to be general coding standards that are industry accepted as well as coding standards that are specific your new organization. Your organization will also likely have it's own set of standard tooling for development, deployment, and distribution.

What I've learned in a year

I have now been blogging for one full year. In the past year I've done one installment of the Starting From Scratch series on Android, a series about building a mobile app, 2 Back To The Basics installments covering Binary Trees and Hash Tables, and a whole lot of random posts about software development. So what have I learned over the past year of writing a technology blog.

Ideas aren't as hard to come by as I would have thought

Over the past year I've learned that I have a lot more to talk about than I originally thought I would. When I originally started this blog in Oct 2013 I wasn't sure what the heck I was going to talk about each week. While I wouldn't say I have a book of ideas just laying around, I haven't had trouble coming up with a topic each week. I've probably got about a dozen or so post ideas sitting in the queue just waiting to be written.

A weekly post is a good pace

When I set out to start this blog I wasn't sure on which end of the spectrum my blog was going to fall. One side of the spectrum is a Twitter like blog. The type where you have a bunch of frequent but short (as short as one sentence sometimes) posts. On the other side of the spectrum you have article like blogs. These are blogs that read like a magazine or newspaper article. They're usually chock full of information and other require multiple sittings to read through.

I've found myself somewhere in the middle, slightly skewed more toward article length. I really like doing the multi-part series as well as the little nuggests of things I've learned.

Write a lot and then take time to think

I will usually write three or four blog posts at a time and then stew on my thoughts for a couple of weeks. I feel like it really helps me understand HOW I want to write about WHAT I write. Often I'll have an idea which I think about one way, and then after writing about it will come back and edit it from a different perspective. It's almost been like a conversation with myself.

I still don't know who my target audience is

And I'm okay with that. Sometimes I want to write a really technical article. I'll go deep into an algorithm and feel great about it. Sometimes I want to write a high level about something that's applicable to life outside of software development (even if it's a post about software).

I'm just happy to be writing.

PC marketshare

Over the life of my blog 38% have visited from Windows, 37% have visited from Mac, a smaller than I expected 6% have visited from Linux, and then a hodgepodge of OS's make up the rest (including mobile).

I consume the blogs I follow almost entirely on my mobile devices (90% phone and 10% tablet). So it was a bit surprising to me that the vast majority of blog readers were doing so on desktop machines. I think a lot of this has to do with the sources of my traffic. But I'm not wholly convinced.

I'm just glad to be here

Whether you've been reading this blog from the beginning or this is your first week, I hope you're enjoying what you're finding here. Writing is a form of art to me. I enjoy it, it relaxes me, and it makes me feel connected to humanity.

Thanks for taking the time out of your day to read my blog :)

Conditional logic In Ant

Every so often I find myself needing some conditional logic in my ant build files based on either an automated build property or some property set by current state.

There is an open source library, ant-contrib, which gives you if/else statements in your Ant build files but I tend to not use ant-contrib for three reasons. First, it adds bloat to my project because of the requirement to include it's jar in my projects classpath. Second, you have to mess around with defining tasks in your build files which I just don't feel are very intuitive. Lastly, Ant already includes the ability to perform conditional logic by taking advantage of the Ant target's if attribute.

Performing conditional logic in Ant without an additional library is pretty easy. You simply need to define three targets. The first is the target you want to run in the TRUE scenario. The second is the target you want to run in the FALSE scenario. The third is the target that sets a property (or properties) based on some condition and calls the other targets.

Let's take a look at a very simple build file. This will print This build IS *nix if the isUnix property is set to true otherwise it will print This build is NOT *nix.

<?xml version="1.0" encoding="UTF-8"?>
<project name="example">
    <condition property="isUnix"><os family="unix"/></condition>

    <target name="-unix-build" if="performUnixBuild">
        <echo>This build IS *nix</echo>
    </target>

    <target name="-non-unix-build" if="performNonUnixBuild">
        <echo>This build is NOT *nix</echo>
    </target>

    <target name="build">
        <condition property="performUnixBuild"><istrue value="${isUnix}" /></condition>
        <condition property="performNonUnixBuild"><isfalse value="${isUnix}" /></condition>

        <antcall target="-unix-build" />
        <antcall target="-non-unix-build" />
    </target>

</project>

You can see this in action by copying that file to a machine with Ant on it and running:

$ ant build.

If you're on a Unix like machine it will print This build IS *nix otherwise it will print This build is NOT *nix.

You can test the else logic by overriding the isUnix property at the command line using:

$ ant -DisUnix=false build.

The fallacy of the re-write

I've been in the software industry for a decade and a half and have worked on dozens of projects. Many of the systems that I have worked on were considered legacy systems. As with any system, but even more so with legacy systems, developers will get frustrated with the systems inflexibility. And inevitably this will lead to the developers decreeing that if they could only re-write the system all the problems will be solved. Unfortunately most product owners will eventually give in to these cries and will commission a re-write.

I'm here to tell you today (as both a developer and a manager) giving in to this urge IS NOT going to solve your problems. What it is going to do is grind your production to a halt and make your customers unhappy. This will have downstream effects on the team as the pressure to produce builds and builds and builds.

So why is a re-write not a viable solution?

Re-writes are usually based on a few commonly held (but false) beliefs in the software industry.

• If we start the project over from scratch we won't carry the problems from the old system into the new.
• If we start the project over from scratch we can use the latest and greatest technologies that are incompatible with our current technology stack.
• If we start the project over from scratch we can move faster and produce results quicker.

Why are these fallacies? If we dig a little deeper we will see that a ground up re-write means you are more likely to introduce problems in the new system than you are to solve problems in the old system. What is typically glossed over is the fact that the current architecture is doing a lot of stuff correct. How do I know this? Because it's the architecture that is in production right now running your business.

Let's take them at each of these fallacies one by one.

If we start the project over from scratch we won't carry the problems from the old system into the new.

This statement can really be broken down into two parts. The first part says that there are problems in the architecture that prevent you from extending the code and because you're now aware of those problems you can re-architect the software so that those problems no longer exist. The second part says that you won't carry over existing bugs into the new system. The second part of this statement is really related to the second fallacy, so we'll cover it when we cover that fallacy.

Because it is true that re-writing a system with knowledge of the current architectural problems can help you avoid current pain points most people are quick to accept this statement without challenge. There are many different times in the life-cycle of a product when problems arise. Some arise as bugs when writing the software. These can typically be rooted out with some sort of unit testing. The next class of problems crop up when integrating each of the pieces of the system together. You can create integration tests to help reduce the amount of integration bugs but often there are integration bugs that don't show up in pre-production environments. These tend to be caused by the dynamic nature of content. Because the new system is a re-write of the old system it will be more difficult to use real inputs/outputs from the old system to test the integration of the new system. Because of this you're likely to introduce problems in the new system that don't already exist in the old system. Because the new system won't be in production till it's done, these new architectural problems are not likely to be found till your new system is in production.

If we start the project over from scratch we can use the latest and greatest technologies that are incompatible with our current technology stack.

On the surface this statement is likely true. What this statement hides is similar to what's hidden in the previous statement. New technologies mean new bugs and new problems. Again it is likely that many of these problems won't surface till the new system is in production because, as anyone who has worked in the industry for at least a few years knows, production traffic is always different from simulated traffic. You run into different race conditions and bugs simply because of the random nature of production traffic.

If we start the project over from scratch we can move faster and produce results quicker.

The final fallacy is usually the one that most companies hang their hat on even if they acknowledge that a re-write from the ground up will introduce new bugs and problems and re-introduce existing bugs and problems. The reason is because they believe that their knowledge of the existing system should help them to only solve problems that need to be solved which leads to the system being built much faster.

The fallacy in this statement is more subtle but much more severe than the others. The reason is because until your new system performs all functions of your old system, the old system is superior from a business value perspective. In fact it isn't untill the new system has 100% feature parity with the old system that it starts to provide the same business value as the legacy system, not to mention more business value. Some will try to gain business value from the new system earlier by switching over to the new system before there is 100% feature parity with the old system. But by doing this you're offering your customers less value for the same amount of money, time, and/or investment.

This visual does a good job of illustrating the feature parity problem.

What is the solution then?

Are you saying I'm stuck with my current architecture and technology stack? NO! The best way to upgrade your technology stack is to do an in-place re-write. By doing this you help mitigate the problems presented in a ground up re-write. What does an in-place re-write look like?

By segregating and replacing parts of your architecture you're reducing the surface area of change. This allows you to have a well defined contract for both the input and output of the system as well as the workflow.

In-place re-write has another huge benefit over ground up re-write. It allows you to validate your new system in production as you would any new feature of the system. This allows you to find bugs sooner as well as validate the workflow and feature parity.

Another benefit of an in-place re-write is that you can decommission parts of the legacy system as you go without ever having to do a big (and scary) "flip of the switch" from the old system to the new system.

Most importantly, your customers do not suffer when you do an in-place re-write as you are not ever taking away features from your customers. Even better, you can prioritize giving your customers new features earlier by implementing them on the new system even before you've finished porting the entire old system over.

Starting From Scratch: Android - Creating A Release Build

This week we're finishing the Starting From Scratch series with a look at how to create a release build of our app. I'll show you how to create a release key for your app, secure your release key via encryption, and how to integrate the automatic decryption (and clean up) of your encrypted key during the normal Android Ant build process

Creating a release key for your app

Your key is what identifies your app as being published by you. This is what ensures that only official versions of your app can be released. It's ABSOLUTELY important that noone gets access to your key. DO NOT commit this keystore to your source control repo as is.

Create a release keystore

$ keytool -genkey -v -keystore my.release.keystore -alias myalias -keyalg RSA -keysize 2048 -validity 10000

Securing your release key

Not having the keystore in source control doesn't create a pit of success as you have to manage your key separately from your project. Furthermore, anyone with access to your key can sign an app as you. In order to safely create a pit of success we're going to encrypt our keystore and delete the original so it's not lying around anywhere for someone to abuse.

To encrypt the keystore we'll use openssl and DES3 encryption.

$ openssl des3 -salt -in my.release.keystore -out my.release.keystore.encrypted
$ rm my.release.keystore

The next thing you want to do is put your encrypted keystore in the provisioning directory.

$ mkdir provisioning
$ mv my.release.keystore.encrypted provisioning/

Integrating into the Android Ant build process

Now that we have a key that can be used to sign our applicaiton and we've secured that key from unauthorized access we now need to integrate into the standard Android Ant build process.

The first thing we need to do is create an Ant target that will decrypt the keystore. We also want to create a target to clean up the decrypted keystore immediately after the build. Note that the -decrypt-keystore target supports both prompting the builder for the password or getting the password from an Ant property in the case of an automated release build.

Here's what our encryption.xml file looks like. Create this file in the same directory as your projects build.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<project name="encryption">
    <target name="-decrypt-keystore" depends="" if="isRelease">
        <echo>Decrypting keystore</echo>
        <if>
            <condition>
                <and>
                    <isset property="key.store.password"/>
                </and>
            </condition>
            <then>
                <exec executable="openssl">
                    <arg value="des3"/>
                    <arg value="-d"/>
                    <arg value="-salt"/>
                    <arg value="-in"/>
                    <arg value="provisioning/${assets.keystore}"/>
                    <arg value="-out"/>
                    <arg value="provisioning/release.keystore"/>
                    <arg value="-pass"/>
                    <arg value="pass:${key.store.password}"/>
                </exec>
            </then>
            <else>
                <exec executable="openssl">
                    <arg value="des3"/>
                    <arg value="-d"/>
                    <arg value="-salt"/>
                    <arg value="-in"/>
                    <arg value="provisioning/${assets.keystore}"/>
                    <arg value="-out"/>
                    <arg value="provisioning/release.keystore"/>
                </exec>
            </else>
        </if>
    </target>
    <target name="-clean-keystore" depends="">
        <echo>Cleaning up decrypted keystore</echo>
        <delete file="provisioning/release.keystore"/>
    </target>
</project>

In order to support automated release builds we need to add a few Ant properties to our projects local.properties file. DO NOT CHECK THIS IN TO YOUR SOURCE CONTROL. This file should be restricted as much as possible because it contains the password used to decrypt your keystore. You do not have to put your password in this file. If you don't you'll be prompted to enter your password during the release build.

assets.keystore=my.release.keystore.encrypted
key.store=provisioning/release.keystorekey.alias=myalias
key.alias.password=PASSWORD_YOU_USED_WHEN_CREATING_YOUR_KEYSTORE
key.store.password=PASSWORD_YOU_USED_WHEN_CREATING_YOUR_KEYSTORE

The last thing we need to do is wire up the decryption and cleanup of our key into the existing Android Ant build process. To do this we'll implement the -pre-build, -pre-clean, and -post-build build targets in our custom_rules.xml file. Note that we only want our decryption to happen during a release build. So we're going to define an isRelease property. Our -decrypt-keystore target checks for this property before execution.

<?xml version="1.0" encoding="UTF-8"?><project name="custom_rules">
      <condition property="isRelease"><contains string="${ant.project.invoked-targets}" substring="release"/></condition>
    <target name="-pre-build">
        <antcall target="-decrypt-keystore" />
    </target>
    <target name="-pre-clean" depends="-clean-keystore"></target>
    <target name="-post-build" depends="-clean-keystore"></target></project>

Finally, the last thing we need to do is update our projects build.xml file to include our encryption.xml and custom_rules.xml files.  Add the following two import statements ABOVE the existing ant/build.xml import. For example:

<import file="encryption.xml" optional="false" />
<import file="custom_rules.xml" optional="false" />

<import file="${sdk.dir}/tools/ant/build.xml" />

You can now build a signed release version of your app on the command line with the following command.

$ ant release