What I've learned in a year

I have now been blogging for one full year. In the past year I've done one installment of the Starting From Scratch series on Android, a series about building a mobile app, 2 Back To The Basics installments covering Binary Trees and Hash Tables, and a whole lot of random posts about software development. So what have I learned over the past year of writing a technology blog.

Ideas aren't as hard to come by as I would have thought

Over the past year I've learned that I have a lot more to talk about than I originally thought I would. When I originally started this blog in Oct 2013 I wasn't sure what the heck I was going to talk about each week. While I wouldn't say I have a book of ideas just laying around, I haven't had trouble coming up with a topic each week. I've probably got about a dozen or so post ideas sitting in the queue just waiting to be written.

A weekly post is a good pace

When I set out to start this blog I wasn't sure on which end of the spectrum my blog was going to fall. One side of the spectrum is a Twitter like blog. The type where you have a bunch of frequent but short (as short as one sentence sometimes) posts. On the other side of the spectrum you have article like blogs. These are blogs that read like a magazine or newspaper article. They're usually chock full of information and other require multiple sittings to read through.

I've found myself somewhere in the middle, slightly skewed more toward article length. I really like doing the multi-part series as well as the little nuggests of things I've learned.

Write a lot and then take time to think

I will usually write three or four blog posts at a time and then stew on my thoughts for a couple of weeks. I feel like it really helps me understand HOW I want to write about WHAT I write. Often I'll have an idea which I think about one way, and then after writing about it will come back and edit it from a different perspective. It's almost been like a conversation with myself.

I still don't know who my target audience is

And I'm okay with that. Sometimes I want to write a really technical article. I'll go deep into an algorithm and feel great about it. Sometimes I want to write a high level about something that's applicable to life outside of software development (even if it's a post about software).

I'm just happy to be writing.

PC marketshare

Over the life of my blog 38% have visited from Windows, 37% have visited from Mac, a smaller than I expected 6% have visited from Linux, and then a hodgepodge of OS's make up the rest (including mobile).

I consume the blogs I follow almost entirely on my mobile devices (90% phone and 10% tablet). So it was a bit surprising to me that the vast majority of blog readers were doing so on desktop machines. I think a lot of this has to do with the sources of my traffic. But I'm not wholly convinced.

I'm just glad to be here

Whether you've been reading this blog from the beginning or this is your first week, I hope you're enjoying what you're finding here. Writing is a form of art to me. I enjoy it, it relaxes me, and it makes me feel connected to humanity.

Thanks for taking the time out of your day to read my blog :)

Conditional logic In Ant

Every so often I find myself needing some conditional logic in my ant build files based on either an automated build property or some property set by current state.

There is an open source library, ant-contrib, which gives you if/else statements in your Ant build files but I tend to not use ant-contrib for three reasons. First, it adds bloat to my project because of the requirement to include it's jar in my projects classpath. Second, you have to mess around with defining tasks in your build files which I just don't feel are very intuitive. Lastly, Ant already includes the ability to perform conditional logic by taking advantage of the Ant target's if attribute.

Performing conditional logic in Ant without an additional library is pretty easy. You simply need to define three targets. The first is the target you want to run in the TRUE scenario. The second is the target you want to run in the FALSE scenario. The third is the target that sets a property (or properties) based on some condition and calls the other targets.

Let's take a look at a very simple build file. This will print This build IS *nix if the isUnix property is set to true otherwise it will print This build is NOT *nix.

<?xml version="1.0" encoding="UTF-8"?>
<project name="example">
    <condition property="isUnix"><os family="unix"/></condition>

    <target name="-unix-build" if="performUnixBuild">
        <echo>This build IS *nix</echo>
    </target>

    <target name="-non-unix-build" if="performNonUnixBuild">
        <echo>This build is NOT *nix</echo>
    </target>

    <target name="build">
        <condition property="performUnixBuild"><istrue value="${isUnix}" /></condition>
        <condition property="performNonUnixBuild"><isfalse value="${isUnix}" /></condition>

        <antcall target="-unix-build" />
        <antcall target="-non-unix-build" />
    </target>

</project>

You can see this in action by copying that file to a machine with Ant on it and running:

$ ant build.

If you're on a Unix like machine it will print This build IS *nix otherwise it will print This build is NOT *nix.

You can test the else logic by overriding the isUnix property at the command line using:

$ ant -DisUnix=false build.

The fallacy of the re-write

I've been in the software industry for a decade and a half and have worked on dozens of projects. Many of the systems that I have worked on were considered legacy systems. As with any system, but even more so with legacy systems, developers will get frustrated with the systems inflexibility. And inevitably this will lead to the developers decreeing that if they could only re-write the system all the problems will be solved. Unfortunately most product owners will eventually give in to these cries and will commission a re-write.

I'm here to tell you today (as both a developer and a manager) giving in to this urge IS NOT going to solve your problems. What it is going to do is grind your production to a halt and make your customers unhappy. This will have downstream effects on the team as the pressure to produce builds and builds and builds.

So why is a re-write not a viable solution?

Re-writes are usually based on a few commonly held (but false) beliefs in the software industry.

• If we start the project over from scratch we won't carry the problems from the old system into the new.
• If we start the project over from scratch we can use the latest and greatest technologies that are incompatible with our current technology stack.
• If we start the project over from scratch we can move faster and produce results quicker.

Why are these fallacies? If we dig a little deeper we will see that a ground up re-write means you are more likely to introduce problems in the new system than you are to solve problems in the old system. What is typically glossed over is the fact that the current architecture is doing a lot of stuff correct. How do I know this? Because it's the architecture that is in production right now running your business.

Let's take them at each of these fallacies one by one.

If we start the project over from scratch we won't carry the problems from the old system into the new.

This statement can really be broken down into two parts. The first part says that there are problems in the architecture that prevent you from extending the code and because you're now aware of those problems you can re-architect the software so that those problems no longer exist. The second part says that you won't carry over existing bugs into the new system. The second part of this statement is really related to the second fallacy, so we'll cover it when we cover that fallacy.

Because it is true that re-writing a system with knowledge of the current architectural problems can help you avoid current pain points most people are quick to accept this statement without challenge. There are many different times in the life-cycle of a product when problems arise. Some arise as bugs when writing the software. These can typically be rooted out with some sort of unit testing. The next class of problems crop up when integrating each of the pieces of the system together. You can create integration tests to help reduce the amount of integration bugs but often there are integration bugs that don't show up in pre-production environments. These tend to be caused by the dynamic nature of content. Because the new system is a re-write of the old system it will be more difficult to use real inputs/outputs from the old system to test the integration of the new system. Because of this you're likely to introduce problems in the new system that don't already exist in the old system. Because the new system won't be in production till it's done, these new architectural problems are not likely to be found till your new system is in production.

If we start the project over from scratch we can use the latest and greatest technologies that are incompatible with our current technology stack.

On the surface this statement is likely true. What this statement hides is similar to what's hidden in the previous statement. New technologies mean new bugs and new problems. Again it is likely that many of these problems won't surface till the new system is in production because, as anyone who has worked in the industry for at least a few years knows, production traffic is always different from simulated traffic. You run into different race conditions and bugs simply because of the random nature of production traffic.

If we start the project over from scratch we can move faster and produce results quicker.

The final fallacy is usually the one that most companies hang their hat on even if they acknowledge that a re-write from the ground up will introduce new bugs and problems and re-introduce existing bugs and problems. The reason is because they believe that their knowledge of the existing system should help them to only solve problems that need to be solved which leads to the system being built much faster.

The fallacy in this statement is more subtle but much more severe than the others. The reason is because until your new system performs all functions of your old system, the old system is superior from a business value perspective. In fact it isn't untill the new system has 100% feature parity with the old system that it starts to provide the same business value as the legacy system, not to mention more business value. Some will try to gain business value from the new system earlier by switching over to the new system before there is 100% feature parity with the old system. But by doing this you're offering your customers less value for the same amount of money, time, and/or investment.

This visual does a good job of illustrating the feature parity problem.

What is the solution then?

Are you saying I'm stuck with my current architecture and technology stack? NO! The best way to upgrade your technology stack is to do an in-place re-write. By doing this you help mitigate the problems presented in a ground up re-write. What does an in-place re-write look like?

By segregating and replacing parts of your architecture you're reducing the surface area of change. This allows you to have a well defined contract for both the input and output of the system as well as the workflow.

In-place re-write has another huge benefit over ground up re-write. It allows you to validate your new system in production as you would any new feature of the system. This allows you to find bugs sooner as well as validate the workflow and feature parity.

Another benefit of an in-place re-write is that you can decommission parts of the legacy system as you go without ever having to do a big (and scary) "flip of the switch" from the old system to the new system.

Most importantly, your customers do not suffer when you do an in-place re-write as you are not ever taking away features from your customers. Even better, you can prioritize giving your customers new features earlier by implementing them on the new system even before you've finished porting the entire old system over.

Starting From Scratch: Android - Creating A Release Build

This week we're finishing the Starting From Scratch series with a look at how to create a release build of our app. I'll show you how to create a release key for your app, secure your release key via encryption, and how to integrate the automatic decryption (and clean up) of your encrypted key during the normal Android Ant build process

Creating a release key for your app

Your key is what identifies your app as being published by you. This is what ensures that only official versions of your app can be released. It's ABSOLUTELY important that noone gets access to your key. DO NOT commit this keystore to your source control repo as is.

Create a release keystore

$ keytool -genkey -v -keystore my.release.keystore -alias myalias -keyalg RSA -keysize 2048 -validity 10000

Securing your release key

Not having the keystore in source control doesn't create a pit of success as you have to manage your key separately from your project. Furthermore, anyone with access to your key can sign an app as you. In order to safely create a pit of success we're going to encrypt our keystore and delete the original so it's not lying around anywhere for someone to abuse.

To encrypt the keystore we'll use openssl and DES3 encryption.

$ openssl des3 -salt -in my.release.keystore -out my.release.keystore.encrypted
$ rm my.release.keystore

The next thing you want to do is put your encrypted keystore in the provisioning directory.

$ mkdir provisioning
$ mv my.release.keystore.encrypted provisioning/

Integrating into the Android Ant build process

Now that we have a key that can be used to sign our applicaiton and we've secured that key from unauthorized access we now need to integrate into the standard Android Ant build process.

The first thing we need to do is create an Ant target that will decrypt the keystore. We also want to create a target to clean up the decrypted keystore immediately after the build. Note that the -decrypt-keystore target supports both prompting the builder for the password or getting the password from an Ant property in the case of an automated release build.

Here's what our encryption.xml file looks like. Create this file in the same directory as your projects build.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<project name="encryption">
    <target name="-decrypt-keystore" depends="" if="isRelease">
        <echo>Decrypting keystore</echo>
        <if>
            <condition>
                <and>
                    <isset property="key.store.password"/>
                </and>
            </condition>
            <then>
                <exec executable="openssl">
                    <arg value="des3"/>
                    <arg value="-d"/>
                    <arg value="-salt"/>
                    <arg value="-in"/>
                    <arg value="provisioning/${assets.keystore}"/>
                    <arg value="-out"/>
                    <arg value="provisioning/release.keystore"/>
                    <arg value="-pass"/>
                    <arg value="pass:${key.store.password}"/>
                </exec>
            </then>
            <else>
                <exec executable="openssl">
                    <arg value="des3"/>
                    <arg value="-d"/>
                    <arg value="-salt"/>
                    <arg value="-in"/>
                    <arg value="provisioning/${assets.keystore}"/>
                    <arg value="-out"/>
                    <arg value="provisioning/release.keystore"/>
                </exec>
            </else>
        </if>
    </target>
    <target name="-clean-keystore" depends="">
        <echo>Cleaning up decrypted keystore</echo>
        <delete file="provisioning/release.keystore"/>
    </target>
</project>

In order to support automated release builds we need to add a few Ant properties to our projects local.properties file. DO NOT CHECK THIS IN TO YOUR SOURCE CONTROL. This file should be restricted as much as possible because it contains the password used to decrypt your keystore. You do not have to put your password in this file. If you don't you'll be prompted to enter your password during the release build.

assets.keystore=my.release.keystore.encrypted
key.store=provisioning/release.keystorekey.alias=myalias
key.alias.password=PASSWORD_YOU_USED_WHEN_CREATING_YOUR_KEYSTORE
key.store.password=PASSWORD_YOU_USED_WHEN_CREATING_YOUR_KEYSTORE

The last thing we need to do is wire up the decryption and cleanup of our key into the existing Android Ant build process. To do this we'll implement the -pre-build, -pre-clean, and -post-build build targets in our custom_rules.xml file. Note that we only want our decryption to happen during a release build. So we're going to define an isRelease property. Our -decrypt-keystore target checks for this property before execution.

<?xml version="1.0" encoding="UTF-8"?><project name="custom_rules">
      <condition property="isRelease"><contains string="${ant.project.invoked-targets}" substring="release"/></condition>
    <target name="-pre-build">
        <antcall target="-decrypt-keystore" />
    </target>
    <target name="-pre-clean" depends="-clean-keystore"></target>
    <target name="-post-build" depends="-clean-keystore"></target></project>

Finally, the last thing we need to do is update our projects build.xml file to include our encryption.xml and custom_rules.xml files.  Add the following two import statements ABOVE the existing ant/build.xml import. For example:

<import file="encryption.xml" optional="false" />
<import file="custom_rules.xml" optional="false" />

<import file="${sdk.dir}/tools/ant/build.xml" />

You can now build a signed release version of your app on the command line with the following command.

$ ant release