Backing up using git

Over the years I've found that as I move from machine to machine, whether adding a new machine to my collection or replacing an old one, it's a pain to constantly copy files from one place to the next. I've also found that the backup process tends to be error prone as I either forget to back something up, don't backup often enough because of having to plug in the extra drive, needing to retrieve something that I've deleted, or I accidentally overwrite something in the backup.

There have been several projects, both open source and commercial, that help solve these problems by doing incremental backup. But I've found that they tend to be platform specific or not supported on the wide variety of machines I have in my mix. Most of them also still require you to use an external drive in order to protect against harddrive failure.

A few years ago I switched to using git as my backup mechanism. It doesn't require me to hassle with an external drive while still solving the problem of protecting against disk failure. It's supported on all major platforms. There are cloud hosting options as well as options to host locally. It provides incremental backup. I can easily exclude things from backup as well as retrieve deleted items.

Initial creation (one-time steps)

Create repository to back up to on backup machine

$ ssh your.backup.machine
$ mkdir -p /path/to/backup/location/my_backup_repository.git
$ cd /path/to/backup/location/my_backup_repository.git
$ git --bare init
$ exit

Add repository location on machine to backup

$ cd /path/to/backup
$ git init
$ git remote add origin USER@your.backup.machine:/path/to/backup/location/my_backup_repository.git
$ echo "backup using git" > README
$ git add README
$ git commit -m "Initial Commit"
$ git push origin master

Performing the backup

It's important that BEFORE you back anything up, you encrypt anything that you don't want backed up in the clear. I like to encrypt things using openssl because of it's wide range of platform availability.

Make sure that you .gitignore anything you don't want backed up. Often I backup my OS X ~/Documents folder. But this is the default location for some application configuration that I don't want backed up. It's important that you add these folders to your .gitignore so you don't unnecessarily back them up and use up unwanted space.

On backup machine

$ cd /path/to/backup

$ # encrypt any files you want encrypted

$ git status

$ git add .
$ git commit -m "backup $(date +'%Y-%m-%d_%H%M')"
$ git push

You now have a secondary backup of your important files. But even better you have a simple way to restore from a backup.

Restoring a backup

$ cd /path/to/restore/to
$ git clone USER@your.backup.machine:/path/to/backup/location/my_backup_repository.git ./

Done!

Are you teaching anyone anything new?

If I had to pick only one piece of advice my grandfather gave me to share with others it would be that the best way to learn something is to teach it.

You may be thinking to yourself that this doesn't make any sense. How can you teach something to someone that you don't already know? The answer is simple, you start by learning yourself.

Teaching someone else something new is not about being exhaustive and not always about being comprehensive. It's really about setting them up to understand the problem domain and to be able to intuitively answer, or find the answers themselves, to questions about that domain.

Here are some questions you can ask yourself in order to prepare for teaching something you don't already know about.

What is the context?

When do people use this approach/tool/etc? What's different about it than it's alternatives? Does it require a certain platform, design, or methodology? Who is it's target audience? How is it consumed? How is it distributed? What's it's history? What problem is it trying to solve?

These are all basic questions that help you understand why this thing exists. Understanding it's purpose is one of the first steps in understanding how to use it.

What are the fundamental assumptions?

Almost everything you interact with makes some type of assumption. A t.v. assumes you can see or hear. A microwave assumes you have electricity. A car assumes you have have a license.

What assumptions exist in your problem domain? How have they influenced the decision making in that domain?

What are the tools?

Are there particular tools that a "standard ..." would use? For example a plumber uses a monkey wrench. A Java developer uses Eclipse. A .Net developer uses Visual Studio.

What are the tools that are used to aid in solving problems in that domain? How are they used and in what context? Understanding what tools to use will save you time that you would otherwise be spending re-inventing the tool and it's use-case.

What are the gotchas?

Every problem domain has some area or areas that people get hung up on. C++ has pointers. Object-oriented has polymorphism and inheritance. Java has the final keyword and it's many different meanings.

These are good places to start. Understanding where people get hung up is usually a good place to learn what people are missing or misunderstanding that has caused them to get hung up.

What does your development community look like?

What does your development community look like? No, I'm not asking what you're development environment looks like. And I'm not asking what development frameworks or platforms you're using. I'm asking what does the tribe of developers of which you are apart look like?

Growing as a software engineer is not just about learning new algorithms, platforms, or tools. It's also about learning systems and methodologies. Here's some questions you should ask about your development community to make sure you're putting yourself in a place where you are able to grow and grow others.

Are the people in your community able to challenge each others ideas?



Does your development community foster the idea of critical review? When ideas or plans are presented does the community enable constructive criticism? Do egos get in the way of someone pointing out flaws in designs or approaches? Is dissent about an approach fostered or is it taken (or given) as a personal attack?



Exposing your ideas, approaches, and designs to a larger group is a great way to expose holes, risks, and errors in approach. But only if the community embraces and fosters an environment where criticism is constructive. It has to be okay to be wrong. And it has to be okay to change your mind.



If your community encourages healthy debate and constructive criticism you will grow as an engineer. Your designs will be better and less error prone, simply because it will account for more than you could have come up with on your own and it will account for context that you do not have.



Are you being exposed to new or different paradigms?



Have you ever switched between a statically typed language and a dynamic one? Or procedural programming and object-oriented programming? What you notice when you do is that the assumptions are not the same. Nor are the trade-offs or optimizations. Being exposed to a wider range of development paradigms helps you to start asking new or different questions about your current development paradigm.



For example if you've been a hard core java programmer and never been exposed to Ruby you may not have been exposed to lambdas or closures. You won't have experiential knowledge of the problems they solve or their usefulness.



Does your community include people with a range of backgrounds?

Does everyone in your community use the same tools and frameworks?

Does everyone in your community use the same tools, frameworks, and platforms? Each tool, framework, and platform comes with it's own set of idioms and standards. They all have their pros and cons, but understanding the trade-offs will help you to think more critically about the problems you are trying to solve. It will foster choosing the correct tools to solve your problem rather than shoehorning your problem into the tools you know.

What's your measure of success?

In the software industry, or any industry for that matter, you're constantly making decisions as to where you're going to invest your time and resources. Maybe you're building a greenfield project and deciding where to start. Maybe you're 12 months into a project that isn't going as planned. Maybe your company or organization has pivoted it's business objectives and you're being asked to pivot with it.  Whatever the case, I've too often seen these decisions made without fully understanding what you're committing to and without having a clear plan to measure whether where you're investing your time and resources is actually being successful.

Measuring success isn't simply a matter of checking a box that something has been delivered. There's much more too it than that. You need to make sure you're delivering the right thing. You need to make sure that you're able to fail quickly if/when you discover that you're not on the right track. You need to have a clear definition of what's enough.

Here are five questions I like to ask before I invest in any project that allow me to have a clear understanding of what success looks like.

Do you have well defined user scenarios?

Well defined user scenarios allow you to know if you're building the correct thing or not. A well defined user scenario should clearly articulate what problem you're trying to solve, how you plan on solving it, and it should tease out some of the unknowns or external dependencies.

A great place to start with a user scenario is with a story board. Storyboards allow you to think about your product as it is used in the real world. Too often we start our user scenarios with an interface or UX design. These designs are great but they don't help us tease out the context surrounding our problem. For example, where users are when the problem occurs. Or what type of constraints the environment adds to the mix, like only having one hand available.

Can you show that you're providing business value now?

Does your new investment require that you wait until the product is complete before you can judge it's fitness? This should be a red flag that the structure and milestones of your project are not lined up correctly.

Agile using scrum and Kanban are two great ways to help you answer this question. Agile allows you to break things into smaller more iterable pieces. It gives you a way to measure your velocity as you go to help you adjust your milestones and scope. Kanban allows you to understand what work in progress is going on. It also allows you to understand and account for the unexpected work that inevitably comes down the pipe. Kanban allows you to make more informed resource decisions based on current resource allocations and throughput.

Is what you're investing in structured in a way as to allow you to change scope?

Do you have a way to show incremental progress? Do you have a way to see your work in action in the real world? If not, you should. Having a way to check-in and get a real world feel for how what your building works in the real world is crucial.

One way to do this is by setting up regular demo days. In agile, these tend to fall at the end of a sprint. It gives you a way to quantify what it is you're delivering and a framework from which trade offs can be discussed. I've seen pretty successful scrum teams decide what they're demo'ing at the next demo day BEFORE deciding what stories are in their sprint.

Having demo days allows you to break up larger work into smaller more consumable pieces. It also allows you to identify dependencies earlier in the process. Maybe your demo requires some work from another team? Maybe it requires some additional hardware or resources. Maybe it requires a deployment. Whatever the case, you're able to start asking the correct questions and unblocking your team before any work actually begins.

Are you able to measure success incrementally?

Demo days are one way to measure success incrementally but not the only way. Alpha and beta releases are another way. Getting something in-front of real world users (even just internally) will open your work up to a higher level of scrutiny which will inevitably make your product better. It will also force you to ask yourself the correct questions on how you measure success for individual features. For example, if you're adding a button to make peoples lives easier, how do you know if they're using it? How do you know if it was discoverable? How do you know if it actually solved the problem? Knowing the answers to this class of questions is paramount to the success of your project. Asking that class of questions helps inform the approach you take to solving your problems.

Do you understand the unknowns?

As you start asking the other questions it will become clear that there are always unknowns in your project that need to be fleshed out. It may be understanding the cost associated with the project and how it's going to be funded. It may be intra-team, inter-team, external, or 3rd party dependencies that are not apparent at the inception of the project. It may be that the level of effort is greater than the rewards for building any particular piece of your project.

Whatever the unknown, it should be your goal to root it out. You're process should make finding the unknowns easier than allowing them to crop up at just the wrong time.