Splitting a Git repository into multiple repositories

Today I thought I would pass along a helpful code organization tip. Occasionally I've run across the need to split an existing git repository into multiple repositories and have wanted to keep the histories intact for each split out repository. One common scenario where this arises is when you want to refactor out a piece of code or submodule from an existing project into it's own library for re-use.

Splitting an existing git repository into multiple repositories is actually pretty straight forward if you use git's subtree command. A git subtree is simply a sub folder within the existing repository that you can commit, branch, and merge. The easiest way to explain how to do this is with an example.

Let's pretend we have a project called MyProj that is really made up of two sub-projects ProjA and ProjB that we want to split into their own repositories. The first thing we need to do is make sure we're in the directory of the git repository that we want to split up.

$ cd /path/to/MyProj

I like to remove the origin remote so I don't accidentally push something to origin. This allows me to always start over if I mess something up.

$ git remote rm origin

Now we can split ProjA and ProjB into their own subtrees.  We're going to use the -b argument which tells git to create a new branch for the split subtree with it's own complete history

$ git subtree split -P relative/folder/for/ProjA -b ProjA
$ git subtree split -P relative/folder/for/ProjB -b ProjB

For me the easiest thing to do at this point is to create a new empty git repository for ProjA and ProjB where I can fetch the new branch, add my new remote repository, and then push it to the origin remote's master branch. This presupposes that you've created new empty remote repositories for ProjA and ProjB.

Here I'm going to create the new local repository for ProjA as a sibling folder to the original project. Creating the new ProjB repository is the exact same process.

$ cd ..
$ mkdir ProjARepo
$ cd ProjARepo

Before we do anything with the ProjA subtree we need to initialize our new empty git repository

$ git init

Now that we have an empty git repository we can fetch the ProjA branch from the origin MyProj repository.

$ git fetch ../MyProj ProjA
$ git checkout -b master FETCH_HEAD

The last thing we have to do is add the origin remote for the repository and push our changes to it's master branch.

$ git remote add origin git@github.com:ProjA.git
$ git push -u origin master

And there you have it. You now have separate repositories for ProjA and ProjB.  At this point you can remove them from MyProj or remove MyProj alltogether if ProjA and ProjB were the only things in the original repository.

What's in a title?

Throughout my career I've seen a common detrimental pattern that I think needs some clarification. That pattern is an engineer who has great technical ability focusing on getting a specific title in order to exert more influence in the organization. They seek a specific title in order to justify their place in an organization or amongst a specific peer group.

In essence they're attempting to use a title to define who they are rather than who they have been. What I mean by that is that they need to already be meeting the expectations of a title before they actually earn that title.

Anytime you interact with someone else professionally you need a common understanding of what a reasonable expectation is of that person's capabilities. Many times this starts with the persons role in the organization, which is defined by their title. A persons title sets the bar for what a reasonable level of expectation of duty or performance is.

Because of this many young engineers seek specific titles in order to change or expand their role within the organization. Often they believe their title is a limiting factor in their ability to influence decisions being made on their team or in their organization. They believe that, if they only had a different title, they would have more influence or more ability to set direction within an organization.

A persons title is meant to reflect their current capabilities and their current sphere of influence. In order for them to be successful their title must be a reflection of who they have been not who they are working to become. That's why focusing on achieving a title to provide that clarity is actually detrimental to them actually being successful.

The concrete bar for a title'd position will be different from organization to organization and business unit to business unit, but each title'd position should have a general bar associated with it. That general bar should be determined by the characteristics of the position, the ability of the individual, and the influence the individual should be able to exert amongst their peers and those those that are senior to them.

Characteristics Of The Position

Each position has ideal characteristics associated with it, the depth of which is determined by the seniority of the position. Those characteristics define expectations around communication, self management, team interaction, ability to manage up, and etc.

Ability Of The Individual

Every engineer should be able to implement solutions using industry patterns and best practices. Every engineer should be able to grasp what's needed in order to validate their solutions. The more senior an engineer is the higher the expectations should be that they can identify and implement those patterns and practices, they can design and architect the correct solutions, and they understand and can implement validation of those solutions at the right level.

Influence

Each position has a level of expectation around what their sphere of influence should be. The more senior the position the broader the sphere of influence within the team, the organization, and the business unit.

A title case study: Senior Engineer

Common misconceptions of what it means to be a senior engineer

Being a senior engineer has a general bar associated with it. A common misconception I've seen in our industry associated with the senior title is that it's based on the number of years a person has been in the industry. There are a lot of engineers that believe that after they put in N number of years they're automatically a senior. Another common misconception is that the senior designation is based mostly on technical ability. I've seen countless numbers of engineers that believe that just because they're technically more proficient than their peers, and often as technically proficient as other seniors, that they also deserve a senior title.

Why being a senior isn't only about how much time you've put in

Time doesn't provide any accurate assessment of where a person is at in achieving the characteristics of their position. Time alone doesn't make you a better communicator. It doesn't teach you how to be effective on a team. It doesn't provide you the discipline necessary to be a good self manager. Time doesn't make you more or less successful in managing up.

Time can be helpful and detrimental in helping you build the characteristics needed to be successful at the next level. For example, time can help you to learn to communicate better by giving you more opportunity to practice your communication skills in different scenarios with different audiences. It can also give you more opportunity to reinforce the bad habits of communication that you've been using your entire career.

Time, if used wisely can be a real an opportunity to learn, grow, and start practicing performing at the next level.

Why being a senior isn't only about technical proficiency

Technical proficiency is about a persons ability to solve the problems they face as an individual contributor. These problems may be domain specific, specific to the technology stack they're working on, or may be general to their industry. What technical proficiency doesn't measure is a persons ability influence their peers, their seniors, or those they're senior to. It doesn't provide a measure of how well the person inspires others or communicates. It doesn't provide a measure of their understanding of the business or their ability to manage trade offs or make tough decisions.

What it really means to be a senior

So what does it mean to be a senior?

It means you've grown in the characteristics necessary to make you successful in your current role. You've looked for and taken advantage of opportunities presented to you to practice working with people in all levels within your organization. You've practiced communicating differently and effectively depending on your audience. It means that you're able to manage your time well and able to not get rat-holed into a specific problem or side tracked by the next shiny object.

It means being technically proficient. Being able to identify best practices and patterns in the industry. Being able to implement those patterns and practices yourself but also being able to let someone else implement them differently than you would have. It means being able to understand that there is a difference between the minimal viable product and the ideal product with all the bells and whistles.

It means being able to understand your organizations landscape. You have to be able to identify the key influencers and build alliances with them. It means being able to become one of those key influencers. It means being able to get people to want to follow you without you having to ask or tell them to follow you.

Putting your Software Development Lifecycle on a diet

I was re-reading Mary Poppendieck's Lean Programming Essay and I thought I would take the opportunity while it was fresh in my mind to walk through a few points that really stuck out to me that are commonly misunderstood. I've often seen developers that want to write LEAN software struggle when trying to apply OOP principles like abstraction.

Lean Rule #3: Maximize Flow (Drive Down Development Time)

The basic premise of iterative development is that small but complete portions of a system are designed and delivered throughout the development cycle, with each iteration adding an additional set of features.

In LEAN abstractions aren't bad.  But they're a by product of building small complete portions of a system.  In one iteration you build a complete feature. At this point in the process the YAGNI principle (You Aren't Going To Need It) is applied to any abstractions. There isn't a business case which says that a feature is incomplete without the abstraction. At some later date (either right after the first iteration or some other point) in another iteration you build a feature that, either in design, analysis, or some other point, brings to light that an abstraction is necessary in order to apply the DRY (Don't Repeat Yourself) and YAGNI principles. In order to build the "complete" second feature it  becomes a requirement to refactor and add the abstraction layer. At this point you only add things to the abstraction which are 100% used and required by the first and second iteration features. Adding any additional functionality or abstraction would be a violation of YAGNI. The key principle from LEAN, refactoring, becomes the driving force for the abstraction during the second iteration.

Lean Rule #4:Pull from Demand (Decide as Late as Possible)

Software systems should be designed to respond to change, not predict it.

In the context of abstractions, building an abstraction before you have at least two uses is an attempt to predict change. When you only have one required use case for something you don't have enough information to make a decision about how the second, third, fourth, or Nth use case will use the abstraction. In my experience I would actually argue that you don't really have enough information to abstract until you have a third case, but definitely not before you have a second.

Any abstraction that's built not as a direct result of refactoring would either be built to guide use of the functionality or feature (which goes against the LEAN principle of allowing requirements to change) or to anticipate future use. Anticipating future use is tantamount to trying to predict change which also makes it so that you're putting a limit on how the requirements can change. The ability to constantly change requirements is something that LEAN values tremendously.

Lean Rule #8: Abolish Local Optimization (Sub-Optimized Measurements are the Enemy) “Do it Right” requires that we provide for change.

Lean Programming employs two key techniques that make change easy. Just as Lean Manufacturing builds tests into process so as to detect when the process is broken, Lean Programming builds tests into the development process in order to ensure that when changes don’t inadvertently break the code. In fact, the best approach is to write the tests first, and then write the code. An excellent unit and regression testing capability is the best way to encourage change late in the development process.

The second technique for allowing change to happen late in development is refactoring, or improving the design of existing software in a controlled and rapid manner. When refactoring is an accepted practice, early designs can focus on the issue at hand rather than speculate as to what additional design elements will be needed. As the additional features are actually added, refactoring provides a new, simplified design to handle the new reality. When refactoring is a part of the process, we reduce speculation as to what will be needed in the future by making it easy to accommodate the future if and when it becomes the present.

This last rule really ties #3 and #4 together.  If we have tests for the functionality and regression then we have the ability to change the system at will (including the need to add abstractions) while only having to refactor the tests for the functionality that's changing (adding abstraction is a change).  It allows you to be explicit about the change and the need for the change. If you use TDD then you're ensured that not only does your functionality after the refactor work as expected, but you've also guaranteed that the system is interacted with as expected. The second technique mentioned is really key to the guidance on when/how to add abstractions. The sentence "As the additional features are actually added, refactoring provides a new, simplified design to handle the new reality." really becomes key.

Hopefully this helps you feel good about pushing off adding an abstraction into your system until after you actually have a need for that abstraction. Doing so will allow you to write software that is more adaptable to change.

I finally understand the why of ChromeOS

When I attended Google I/O in May 2013 I had no expectation that I'd be receiving a Chromebook Pixel. I had thoughts (hopes) that they'd announce Android Key-lime Pie (now known to be called Kit-Kat) with a new tablet.  My original Samsung Galaxy Tab 10.1 had been trucking along well since July of 2011 with heavy use. I commute 1 - 1.5 hours each way on the bus to work and I use that time to mostly read Kindle books but also to catch up on blogs, email, and the internet as a whole. It's pretty safe to say that I used my Galaxy Tab 10.1 3 - 5 hours a day every day.

I feel a little privileged saying I expected Google to give away free tablets, and I probably was being a little privileged. But their history had been to give away free phones/tablets at their events (especially Google I/O). Much to my disappointment Google didn't announce Android Kit-Kat and didn't announce a new tablet.  But much to my surprise they did announce that they were giving away Chromebook Pixel's to everyone in attendance.

I was super curious about this because, frankly, I've been really skeptical of Chrome OS since it was first announced. I just couldn't image a world in which Chrome OS and Android could live in the same ecosystem.  At the time I couldn't understand why Google wasn't putting more of an effort into bringing Android to more traditional devices instead of creating a completely new operation system.

Android vs. Chrome OS

After having my Pixel for 6 months I believe I finally understand why Google created Chrome OS as opposed to trying to bring Android to more traditional devices. This realization only came after using my Pixel as my everyday (home) laptop for the last 6 months. The problems that Android is trying to solve are very different from the problems that need to be solved on more traditional, less transient devices.

In my opinion, mobile devices are trying to (primarily) solve these problems:

• Hard-line to the rest of the world (phone, email, text)
• Critical content at a glance (calendar, contacts, events, places, transportation, activities).
• Activity based content with a focus on location and movement (maps, running/biking/etc)
• Bite-sized consumable content (Facebook, Twitter, Pinterest, etc)
• Content consumable in the in-between time (games, apps, news, weather, etc). 
• Media consumption (music, movies, books, etc) typically in a short form. 

In my opinion, traditional devices are trying to (primarily) solve these problems:

• Communication (email, instant message)
• Research and Planning
• Work (documents, spreadsheets, presentations, blogging/writing, development, shared content access, etc)
• Media consumption (music, movies, books, etc) typically in long form.

There are overlaps between the two but for the most part traditional devices tend to be the canonical source of your data whereas mobile devices tend to be a transient source of your data.  For instance, your traditional device may have all your email on it whereas your mobile device may have only your recent email on it.  Your traditional device may have all your music and movies whereas your mobile device may have a subset of them.

Once I started to look at Chrome OS as a replacement for more traditional devices like desktops/laptops and less of a replacement for your mobile device Chrome OS started to make a lot more sense. Chrome OS actually seems more like Google's attempt to redefine traditional devices. The big change not being in how you use the device but in how the device interacts with your content.

Currently traditional devices are the place where you create, store, and consume your content. Your mobile device tends to be a place where your content is transiently stored, occasionally created, and mostly consumed.  This doesn't lend well to an environment where we're constantly switching between our traditional device and our mobile device. We're being forced to be aware of the context switch and in some cases, take active action before and after the context switch. Currently the hand-off between traditional devices and mobile is clunky and awkward.

Google, Apple, Microsoft, and others have all been making strides to make it easier to store your content outside of your traditional device in The Cloud (I still cringe saying that). But that's only part of the solution. It's a solution that's been grafted into an existing ecosystem that wasn't designed to solve that problem.  So it sticks out and forces you to at least be partially aware of it's existence.  Chrome and Firefox are making good strides to blur the lines with their bookmark and tab syncing across devices but again that's only a partial solution.

The real solution must involve changing the culture around traditional devices. It must involve freeing the user from thoughts or knowledge of where their content is and on what devices that content is available.  And that culture change needs to happen on traditional devices first.

I think that's the why of Chrome OS.

Are they there yet?  I think that depends on who you are. If your the average consumer who blogs, surfs the web, possibly uses Netflix, Rdio, Hulu, Office/Google Docs, Facebook, LinkedIn, Twitter, and etc then yes, I think you'd be able to replace your traditional device with a Chromebook.

If you're a power user, a developer, or someone that relies on open standards then no Chrome OS won't work out of the box for you.  I store a lot of my files on a shared WebDav drive encrypted using open-ssl. I use KeePass to store my important account info. There's no elegant solution to getting those three things working out of the box on Chrome OS.  Thankfully David Schneider created crouton which is a Chromium centered set of scripts which allow you to create chroots. With crouton I can get a full Ubuntu Linux chroot running on my Chromebook Pixel.  What that means is that I can write software, access my WebDav/Encrypted/KeePass files, and do everything I'm used to doing on a traditional device.  But there are two critical failures with this.  First is that I have to keep my Chromebook in developer mode which sucks. Second, and most importantly, I have to go back to the traditional paradigm of interacting with a traditional device.  To me that defeats the purpose of Chrome OS and turns it into just another pretty User Experience.

Only time will tell if Google can save this problem for both the average consumer and the techie. I do think it's possible to solve if you really understand the problem.