Backing up using git

Over the years I've found that as I move from machine to machine, whether adding a new machine to my collection or replacing an old one, it's a pain to constantly copy files from one place to the next. I've also found that the backup process tends to be error prone as I either forget to back something up, don't backup often enough because of having to plug in the extra drive, needing to retrieve something that I've deleted, or I accidentally overwrite something in the backup.

There have been several projects, both open source and commercial, that help solve these problems by doing incremental backup. But I've found that they tend to be platform specific or not supported on the wide variety of machines I have in my mix. Most of them also still require you to use an external drive in order to protect against harddrive failure.

A few years ago I switched to using git as my backup mechanism. It doesn't require me to hassle with an external drive while still solving the problem of protecting against disk failure. It's supported on all major platforms. There are cloud hosting options as well as options to host locally. It provides incremental backup. I can easily exclude things from backup as well as retrieve deleted items.

Initial creation (one-time steps)

Create repository to back up to on backup machine

$ ssh your.backup.machine
$ mkdir -p /path/to/backup/location/my_backup_repository.git
$ cd /path/to/backup/location/my_backup_repository.git
$ git --bare init
$ exit

Add repository location on machine to backup

$ cd /path/to/backup
$ git init
$ git remote add origin USER@your.backup.machine:/path/to/backup/location/my_backup_repository.git
$ echo "backup using git" > README
$ git add README
$ git commit -m "Initial Commit"
$ git push origin master

Performing the backup

It's important that BEFORE you back anything up, you encrypt anything that you don't want backed up in the clear. I like to encrypt things using openssl because of it's wide range of platform availability.

Make sure that you .gitignore anything you don't want backed up. Often I backup my OS X ~/Documents folder. But this is the default location for some application configuration that I don't want backed up. It's important that you add these folders to your .gitignore so you don't unnecessarily back them up and use up unwanted space.

On backup machine

$ cd /path/to/backup

$ # encrypt any files you want encrypted

$ git status

$ git add .
$ git commit -m "backup $(date +'%Y-%m-%d_%H%M')"
$ git push

You now have a secondary backup of your important files. But even better you have a simple way to restore from a backup.

Restoring a backup

$ cd /path/to/restore/to
$ git clone USER@your.backup.machine:/path/to/backup/location/my_backup_repository.git ./

Done!

Are you teaching anyone anything new?

If I had to pick only one piece of advice my grandfather gave me to share with others it would be that the best way to learn something is to teach it.

You may be thinking to yourself that this doesn't make any sense. How can you teach something to someone that you don't already know? The answer is simple, you start by learning yourself.

Teaching someone else something new is not about being exhaustive and not always about being comprehensive. It's really about setting them up to understand the problem domain and to be able to intuitively answer, or find the answers themselves, to questions about that domain.

Here are some questions you can ask yourself in order to prepare for teaching something you don't already know about.

What is the context?

When do people use this approach/tool/etc? What's different about it than it's alternatives? Does it require a certain platform, design, or methodology? Who is it's target audience? How is it consumed? How is it distributed? What's it's history? What problem is it trying to solve?

These are all basic questions that help you understand why this thing exists. Understanding it's purpose is one of the first steps in understanding how to use it.

What are the fundamental assumptions?

Almost everything you interact with makes some type of assumption. A t.v. assumes you can see or hear. A microwave assumes you have electricity. A car assumes you have have a license.

What assumptions exist in your problem domain? How have they influenced the decision making in that domain?

What are the tools?

Are there particular tools that a "standard ..." would use? For example a plumber uses a monkey wrench. A Java developer uses Eclipse. A .Net developer uses Visual Studio.

What are the tools that are used to aid in solving problems in that domain? How are they used and in what context? Understanding what tools to use will save you time that you would otherwise be spending re-inventing the tool and it's use-case.

What are the gotchas?

Every problem domain has some area or areas that people get hung up on. C++ has pointers. Object-oriented has polymorphism and inheritance. Java has the final keyword and it's many different meanings.

These are good places to start. Understanding where people get hung up is usually a good place to learn what people are missing or misunderstanding that has caused them to get hung up.

What does your development community look like?

What does your development community look like? No, I'm not asking what you're development environment looks like. And I'm not asking what development frameworks or platforms you're using. I'm asking what does the tribe of developers of which you are apart look like?

Growing as a software engineer is not just about learning new algorithms, platforms, or tools. It's also about learning systems and methodologies. Here's some questions you should ask about your development community to make sure you're putting yourself in a place where you are able to grow and grow others.

Are the people in your community able to challenge each others ideas?



Does your development community foster the idea of critical review? When ideas or plans are presented does the community enable constructive criticism? Do egos get in the way of someone pointing out flaws in designs or approaches? Is dissent about an approach fostered or is it taken (or given) as a personal attack?



Exposing your ideas, approaches, and designs to a larger group is a great way to expose holes, risks, and errors in approach. But only if the community embraces and fosters an environment where criticism is constructive. It has to be okay to be wrong. And it has to be okay to change your mind.



If your community encourages healthy debate and constructive criticism you will grow as an engineer. Your designs will be better and less error prone, simply because it will account for more than you could have come up with on your own and it will account for context that you do not have.



Are you being exposed to new or different paradigms?



Have you ever switched between a statically typed language and a dynamic one? Or procedural programming and object-oriented programming? What you notice when you do is that the assumptions are not the same. Nor are the trade-offs or optimizations. Being exposed to a wider range of development paradigms helps you to start asking new or different questions about your current development paradigm.



For example if you've been a hard core java programmer and never been exposed to Ruby you may not have been exposed to lambdas or closures. You won't have experiential knowledge of the problems they solve or their usefulness.



Does your community include people with a range of backgrounds?

Does everyone in your community use the same tools and frameworks?

Does everyone in your community use the same tools, frameworks, and platforms? Each tool, framework, and platform comes with it's own set of idioms and standards. They all have their pros and cons, but understanding the trade-offs will help you to think more critically about the problems you are trying to solve. It will foster choosing the correct tools to solve your problem rather than shoehorning your problem into the tools you know.

What's your measure of success?

In the software industry, or any industry for that matter, you're constantly making decisions as to where you're going to invest your time and resources. Maybe you're building a greenfield project and deciding where to start. Maybe you're 12 months into a project that isn't going as planned. Maybe your company or organization has pivoted it's business objectives and you're being asked to pivot with it.  Whatever the case, I've too often seen these decisions made without fully understanding what you're committing to and without having a clear plan to measure whether where you're investing your time and resources is actually being successful.

Measuring success isn't simply a matter of checking a box that something has been delivered. There's much more too it than that. You need to make sure you're delivering the right thing. You need to make sure that you're able to fail quickly if/when you discover that you're not on the right track. You need to have a clear definition of what's enough.

Here are five questions I like to ask before I invest in any project that allow me to have a clear understanding of what success looks like.

Do you have well defined user scenarios?

Well defined user scenarios allow you to know if you're building the correct thing or not. A well defined user scenario should clearly articulate what problem you're trying to solve, how you plan on solving it, and it should tease out some of the unknowns or external dependencies.

A great place to start with a user scenario is with a story board. Storyboards allow you to think about your product as it is used in the real world. Too often we start our user scenarios with an interface or UX design. These designs are great but they don't help us tease out the context surrounding our problem. For example, where users are when the problem occurs. Or what type of constraints the environment adds to the mix, like only having one hand available.

Can you show that you're providing business value now?

Does your new investment require that you wait until the product is complete before you can judge it's fitness? This should be a red flag that the structure and milestones of your project are not lined up correctly.

Agile using scrum and Kanban are two great ways to help you answer this question. Agile allows you to break things into smaller more iterable pieces. It gives you a way to measure your velocity as you go to help you adjust your milestones and scope. Kanban allows you to understand what work in progress is going on. It also allows you to understand and account for the unexpected work that inevitably comes down the pipe. Kanban allows you to make more informed resource decisions based on current resource allocations and throughput.

Is what you're investing in structured in a way as to allow you to change scope?

Do you have a way to show incremental progress? Do you have a way to see your work in action in the real world? If not, you should. Having a way to check-in and get a real world feel for how what your building works in the real world is crucial.

One way to do this is by setting up regular demo days. In agile, these tend to fall at the end of a sprint. It gives you a way to quantify what it is you're delivering and a framework from which trade offs can be discussed. I've seen pretty successful scrum teams decide what they're demo'ing at the next demo day BEFORE deciding what stories are in their sprint.

Having demo days allows you to break up larger work into smaller more consumable pieces. It also allows you to identify dependencies earlier in the process. Maybe your demo requires some work from another team? Maybe it requires some additional hardware or resources. Maybe it requires a deployment. Whatever the case, you're able to start asking the correct questions and unblocking your team before any work actually begins.

Are you able to measure success incrementally?

Demo days are one way to measure success incrementally but not the only way. Alpha and beta releases are another way. Getting something in-front of real world users (even just internally) will open your work up to a higher level of scrutiny which will inevitably make your product better. It will also force you to ask yourself the correct questions on how you measure success for individual features. For example, if you're adding a button to make peoples lives easier, how do you know if they're using it? How do you know if it was discoverable? How do you know if it actually solved the problem? Knowing the answers to this class of questions is paramount to the success of your project. Asking that class of questions helps inform the approach you take to solving your problems.

Do you understand the unknowns?

As you start asking the other questions it will become clear that there are always unknowns in your project that need to be fleshed out. It may be understanding the cost associated with the project and how it's going to be funded. It may be intra-team, inter-team, external, or 3rd party dependencies that are not apparent at the inception of the project. It may be that the level of effort is greater than the rewards for building any particular piece of your project.

Whatever the unknown, it should be your goal to root it out. You're process should make finding the unknowns easier than allowing them to crop up at just the wrong time.

Building a mobile app: The tools

Now that we've taken a look at the various aspects of building a mobile app, I thought I would end the series with a helpful resource guide.

FireOS

• Phone
• Getting started resources.
• Dynamic Perspective
• Active Widgets
• Foundation Controls
• FireFly
• Kindle
• Getting started resources.

iOS

• Building for iOS.
• Human interface guidelines.
• Build and test iOS applications using XCTool.
• Building static libraries for iOS.
• Sharing resources across applications using bundles.
• App Store review guidelines.
• Understanding iOS crash reports.

Android

• Build from the command line using Ant.
• Build from the command line or Android Studio using Gradle.
• Avoid dealing with fragmentation by using Google Play Services.
• Https and SSL.
• Tablet app quality checklist.
• Android design.

Windows Mobile

• MSDN resources.
• How to create your first Windows Phone app.
• Windows Phone Emulator.
• Nokia's Windows Phone useful links.

HTML5

• Smashing Magazine: Best of both worlds.
• Design your own mobile game.
• The Architect's Guide to Choosing Between HTML5 or Native Mobile Apps.

Building a mobile app: Picking a strategy

In my last post, Building a mobile app: Mobiles unique challenges, I explained some of the unique challenges that mobile application developers face. In this post I'd like to talk a little about picking a mobile implementation strategy.

As you start thinking about building for mobile you should start by reading a guest post on Martin Fowler's blog entitled "Selecting a Mobile Implementation Strategy" by Giles Alexander. That post is very comprehensive and lays out a lot of the decisions that need to be thought through when deciding what mobile platforms to target and how to approach multi-platform development and feature parity. It weighs the various options ranging from pure HTML to pure native. It also talks gives good insight when thinking about going breadth or depth first in terms of target platforms and features.

The second article you should read is "The Architect's Guide to Choosing Between HTML5 or Native Mobile Apps". While I personally don't believe that HTML5 support is ready for HTML only native applications, it's a good thought exercise to go through. That post talks about the trade offs to doing an HTML5 mobile app (i.e. a mobile app that's basically a browser utilizing HTML5) versus a native mobile app. In my experience people using mobile apps expect native experiences and HTML5 just can't handle that well.  One other thing to consider as well is that it isn't an all or nothing choice.  You can have parts of your app using Native and other parts using HTML5.

One last area to consider in picking a strategy are single language cross compilation tools like RhoMobile, Titanium, Xamarin, and PhoneGap. While these tools offer a lower barrier to entry into the mobile space they come with a few trade offs that are very important to be aware of. I say they come with a lower barrier of entry because they require you to only learn/use one programming language/platform in order to target multiple mobile platforms.

Pros:

• Lower barrier of entry as there is only one tool/framework.
• Single codebase.
• Abstracted interaction with hardware.

Cons:

• Vendor lock in.  Once you start down the path of using a tool like Xamarin's you're essentially locked in to that tool forever for your mobile app unless you re-write the app from the ground up using the native language or another tool.  The cost for this would be tremendous and rises as you add features to your mobile apps.
• In most cases, you're dealing with the "lowest common denominator" issue in your code base.  I.e. in order to maintain a single code base without having to fork (the big sell for these tools/frameworks) you can't use features or sensors that aren't available on ALL target devices.
• The types of developers you hire for these platforms aren't of the same caliber as those that write native code using the native frameworks.  This may not sound important but IMO it's actually the most important.  You want developers that "think" in terms of the platform that you're targeting.  The iOS platform is VERY different from the Android platform with is VERY different from the Windows Mobile platform.  What works well on one, often does not work well on the others.  The patterns used on one platform do not often solve the same problems on the others (this is often the case with iOS and Android).  You want developers that can think in the mindset of the target rather than the framework being used.
• These frameworks remove the "built for this device" feel of your app.  Your app should not only feel native on the device it's being used on but it should feel like the device was built for your app.  When you use a generic one size fits all tool to build your mobile platform you can't take advantage of the nuances of each platform.  For example iOS uses a "tabs at the bottom" metaphor whereas Android uses a tabs at the top metaphor and Windows uses a "don't use tabs because we don't give you the ability to" metaphor.

Building a mobile app: mobile's unique challanges

In my last post, Building a mobile app: Understanding the problem space, I introduced us to the problem space of building a mobile app. In this post I'd like to talk about some of the unique challenges that we face when developing mobile applications. These challenges apply to mobile in general and aren't really specific to any particular platform. It's important to understand these challenges before you start to build a mobile architecture or add mobile into your existing architecture.

Data is one of the most precious parts of any mobile application. Whether your application is a game, a productivity tool, a utility or something else, it's going to be consuming data of some sort from somewhere. You really have three options when it comes to data. First is to bake all necessary data into the app. The second is to download all necessary data from the internet. And the third is to use a hybrid approach and have some data baked into the app and some downloaded.

The first approach is the easiest as the app becomes self contained. Most platforms have even made it easy to bake certain types of data into our apps like localized strings or images. Structured data is more difficult to bake into the app as it tends to increase the apps size and can cause lag in the app as it's read from disk to memory. Using this approach requires an app update anytime the data needs to change.

The second approach solves the app update problem by storing all data outside of the app. Thus, every user always has the most up to date data available. But this decreases the apps usefulness. It means that at a minimum the user must be connected to the internet when they first run the app. It also means that the app needs to be built with a (sometimes complex) caching strategy in order to make the app usable in an offline or disconnected state. Finally, while the application package size will be smaller it means that users will have higher data usage as they use the app because they'll be downloading content for the app at the rate in which the data changes.

The last approach is probably the most common. It allows the app to be designed to work offline while also allowing the app creators to update content without having to provide an over the air update to the app. But this too comes with some challenges. It means having to create a caching and update strategy for your content. If you get that wrong it means that your users may get old or incorrect data while disconnected. This strategy requires more diligence in responding to the connection state. It also means being diligent about making your over the air content backwards compatible to any version of the app that may still be in the wild.

App updates are another unique challenge that mobile developers face. Not only does the app creator not control whether the user will install an app update, they don't control whether the user will install a system update either. What this means is that applications must be feature aware. Being feature aware means being able to update the app to take advantage of new system functionality or new data that is needed by system changes without breaking the existing apps out in the ecosystem. Even in a world where everyone updates to the latest version of your app they don't do it all at once. So there's always going to be a period, however brief, where you will have multiple versions of your app in the wild at any given time.

Interaction is another challenge that, while not unique to mobile, is more difficult on mobile. Each different mobile platform (FireOS, Android, iOS, Windows Phone, and etc..) comes with its own set of interactions, workflows, and user expectations. Your app should feel natural on whatever platform it's running on. Which means either slightly or dramatically tailoring the user experience and interaction design for each platform.

The last mobile challenge I'd like to mention is hardware. There are some types of hardware that are common on most mobile devices like an accelerometer, camera, or WiFi. But then there are some like dynamic perspective, external storage, or gyrascopes which are not going to be available on all devices. And lastly there are some that are going to be common on all devices like GPS or memory whose quality, size, or accuracy may vary greatly.