Jai’s Weblog – Tech, Security & Fun…

Tech, Security & Fun…

  • Jaibeer Malik

    Jaibeer Malik
  • View Jaibeer Malik's profile on LinkedIn
  • Subscribe

  • Feedburner

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 23 other followers

  • Archives

  • Categories

  • Stats

    • 258,980
  • Live Traffic

Svn2Git: Migrating repository from Subversion to Git

Posted by Jai on October 23, 2013

Efficient enough Version Control System has always been a challenge based on your requirements of local, centralized or distributed version controlling. Git a distributed version controlling system helps us to achieve same painlessly and is already out for quite some time now with proven track record. In this post we will cover the steps to migrate from your existing version control system like SVN to Git.

Why Git

Some of the features of Git which makes it to standout,

  • Fixed the pitfalls and learning from SVN
  • Dramatic increase in operation speed (diff, merge, view history etc.)
  • Easy, Cheap and efficient branch operations
  • Full history tree available offline
  • Distributed, peer-to-peer model
  • Git’s repositories are much smaller than Subversions
  • Git branches carry their entire history
  • Git provides better auditing of branch and merge events
  • Git’s repository file formats are simple, so repair is easy and corruption is rare.
  • Backing up Subversion repositories centrally is potentially simpler – since you can choose to distributed folders within a repo in git
  • Git repository clones act as full repository backups
  • Walking through versions is simpler in Subversion because it uses sequential revision numbers (1,2,3,..); Git uses unpredictable SHA-1 hashes. Walking backwards in Git is easy using the “^” syntax, but there is no easy way to walk forward.


For full comparison, refer to GitSvnComparison

Installing Git

Install git from repository on your selected environment, Download.

For Linux environments, Download 

Check which version of git you want to install and which version is by default available in the OS repository on unix environment. To install later version of git on CentosOS,

Install GUI clients

There are quite some GUI available to install, Check GUI clients list 

TortoiseGit

You can use tortoisegit client to have similar behavior like of tortoise svn client on windows.

Download TortoiseGit,

msysgit

Windows Git gui, download msysgit 

SmartGit/HG

Check SmartGit/HG for windows and unix client,

Migration steps from SVN to Git

We will cover different steps to migrate the existing svn repository to a git repository.

Using git-svn

Check the complete list of steps to convert SVN repository to Git repository,

  • Retrieve a list of all Subversion committers
  • Clone the Subversion repository using git-svn
  • Convert svn:ignore properties to .gitignore
  • Push repository to a bare git repository
  • Rename “trunk” branch to “master”
  • Clean up branches and tags

Check also answers for svn migration using git-svn,

For the detailed step by step approach, read from the book Git and SVN 

Read example Atlassian confluence SVN to Git migration approach,

Using svn2git utility

svn2git:

“svn2git is a tiny utility for migrating projects from Subversion to Git while keeping the trunk, branches and tags where they should be.”

The svn2git wrapper internally used the git-svn functionality only.

In this post we will cover the one way migration, full migration. You can set SVN as read only for the time being it is migrating. It can be migrated from svn to git with out read only and can be set read only once fully migrated to git. You can also use different ways to have two way synch and migration between git and svn.

1. Clean up the SVN repository to reduce the size of git repository.

The original svn repository may contain files which are quite big in size or may be unnecessary to keep for future purpose. It is suggested approach to clean up the repository for unnecessary data. The file size and total files data will later decide the size of your git repository and will affect the performance of your final git repository too.

Find svn repository files size and analyze the same,

$ svn list -vR <http://testhost.com/svn/>

2. Get list for SVN users.

To have the readable usernames and proper mapping of user in your git repository, you need to map svn users to git users. The final git log information will also contain the svn user information, which will be displayed as per your provided mapping.

Execute below command to retrieve the list of user names from svn repository,

$ svn log -q --username  <http://testhost.com/svn/> | grep -e '^r' | awk 'BEGIN { FS = "|" } ; { print $2 }' | sort | uniq

Create authorslist.txt file with format,

# svnusername= Full Name
jaibeer.malik = Jaibeer Malik <jaibeer.malik@test.com>
test1 = Test1 User <test1user@test.com>

This file will be used as mapping file for authors in the svn2git command.

3. Adding/Removing SVN metadata in logs

You can choose to include metadata information from svn to git if you want to track back the changes. The svn2git provide option to include the same. The final git log information will include the git-svn-id in the format shown below.

# git-svn-id: <https://path/to/repository@######
$ svn2git http://svn.example.com/path/to/repo --metadata

Not to include the git-svn-id information will make the git log more human readable, don’t include it if you don’t want to refer back the svn repository.

4. Migrate data

We will take an example of a typical svn repository with a structure containing trunk, branches and tags information for a project. Check different options for svn2git if you only want to migrate trunk and not branches and tags.

In below exmaples the repository http://testhost.com/svn/ we are assuming default svn structure for your project, which is like

#Parent dir: http://testhost.com/svn/

trunk
  ...
branches
  1.x
  2.x
tags
  1.0.0
  1.0.1
  1.1.0
  2.0.0

We will be migrating whole svn repository which includes trunk, branches and tags. To start the migration, execute below commands.

$ nohup svn2git --verbose --username  http://testhost.com/svn/ --authors /home/jai/authorslist.txt > /tmp/nohup.out &

#--verbose                    Be verbose in logging
#--authors AUTHORS_FILE       Path to file containing svn-to-git authors mapping

$ tail -f /tmp/nohup.out

Check the “svn2git –help” for various options suiting best to your requirements of your project structure and what data (trunk, branches, tags etc.) you want to migrate.

The process will be started in background, tail the file to see the current status as the process is started in verbose mode. Disable the verbose mode if you want to remove logging overhead.

In case of process hung or to get latest updates from the svn repository, use following command,

$ svn2git --rebase --verbose

The local existing data will be updated with any changes to the svn repository or will continue from the last synch step.

Once the migration process is finished you have local git repository with all the data from svn. The next step is to filter and update the tags and branches information in local git repository.

5. Update Tags information

Execute the below script to update the tags information. You can choose to have regular git tags or annotated tags from svn to git. Annotated tags are objects of their own that require a tag message and record the tag creator and the tag creation date.

# svg-tags-update.sh

git for-each-ref --format='%(refname)' refs/remotes/svn/tags/* | while read r; do
	tag=${r#refs/remotes/svn/tags/}
	#echo "Tag: $tag"

	sha1=$(git rev-parse "$r")
	#echo "SHA1: $sha1"

	#echo "Git Command: git tag -a -m 'Tag: ${tag} sha1: ${sha1}'" "$tag" "$sha1"
	git tag -a -m "Tag: ${tag} sha1: ${sha1}" "$tag" "$sha1"

	# Remove the svn/tags/* ref
	git update-ref -d "$r"
done

6. Update branches information

You can choose to keep only relevant branches in git rather than all unnecessary branches in svn. Execute below script to create branches and update branch references in local git.

# svg-branches-update.sh

# create local branches out of svn branches

git for-each-ref --format='%(refname)' refs/remotes/svn/ | while read branch_ref; do

	branch=${branch_ref#refs/remotes/svn/}
	echo "Creating branch: $branch with Git command: git branch -t '$branch' '$branch_ref'"

	git branch "$branch" "$branch_ref"
	git update-ref -d "$branch_ref"

done

7. Push all data to remote git repository

Let’s say you already have a remote bare git repositroy created to share the repository. We will be covering ways to share a remote git repository in separate post later.

$ ssh git@testhost.com
$ mkdir testproject.git
$ cd testproject.git
$ git init --bare
$ git update-server-info # If planning to serve via HTTP
$ exit

To push all the data from local git repository to remote bare git repository, use the below git commands.

#add remote repository
$ git remote add origin git://testhost.com/testproject.git

# push all the branches to remote repository
$ git push --all

# push all the tags to remote repository
$ git push --tags

8. Clean up git repository and prune huge files.

Run “git gc” to compress the objects data.

Find top 10 heavy objects in the database.

$ git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n --reverse | head -10

#grep the path of sha1

$ git rev-list --objects --all | grep

You may wish to clean up unncesessary branches which may have ended up during svn to git migration.


#delete unwanted trunk branch

#Delete branch in local git repository
$ git branch -D trunk

# Delete branch in remote git repository
$ git push origin --delete trunk

9. Managing ignored repository files

“git svn ” allows you to handle the svn ignored files explicitly. The available options are,

“create-ignore: Recursively finds the svn:ignore property on directories and creates matching .gitignore files. The resulting files are staged to be committed, but are not committed. Use -r/–revision to refer to a specific revision.

show-ignore: Recursively finds and lists the svn:ignore property on directories. The output is suitable for appending to the $GIT_DIR/info/exclude file.”

To add ignore file list to git from svn,

# Append svn:ignore settings to the default git exclude file:
$ git svn show-ignore >> .git/info/exclude

$ git add .gitignore
$ git commit -m 'Converted svn:ignore properties to .gitignore'

If empty directories are allowed and svngitignore is available for each you will land up having too many corresponding gitignore files. Another option to put all ignore information in single .gitignore file.

$ git svn show-ignore > .gitignore

10. Managing SVN keywords functionality

SVN Keywords , is functionality to substitute dynamic information about versioned file into the file contents itself as stated.

For git Keywords expansion is not recommended. You still can achieve it using a custom script.

You can also use gitattributes to achieve the same.

Check out the git-rcs-keywords to add keywords expansion to your git repository.

Conclusion

I hope the post will help you in smooth migration of your existing svn repository to git repository. But once you get started with new technology there are always some learning and gotcha around to get it right.

In further posts, we will also cover the learning and common erros faced while doing the migration and while getting started with git.

Stay tuned for upcoming posts,

  • Gitorious: Ways to share your git repository
  • Git getting started common problems, fixes and lessons learned

One Response to “Svn2Git: Migrating repository from Subversion to Git”

  1. […] Svn2Git: Migrating repository from Subversion to Git […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: