New Subversion Downloader

Beginning today, we’re trying out a new method of importing Subversion repositories. We’ll be using this new method on a limited number of projects, and as the code proves itself we’ll gradually phase it in for all Subversion projects.

If you see an enlistment on your project marked “Subversion (Sync Beta)”, then your project is part of this experiment. Please let us know if you notice anything unusual. We expect the new project reports to be identical to the previous ones.

For those who are curious, here’s a description of what’s changing.

Internally, Ohloh currently stores all project source in Git repositories, and project reports are all prepared based on these Git repositories. This means that source code stored in other formats (Subversion or CVS) must be converted to Git. Ohloh uses an in-house tool to convert a single Subversion or CVS branch to Git.

While this converter has been extremely reliable for us, it has a lot of limitations in its design, and it’s also painfully slow.

We’re trying a new strategy now: we will begin storing Subversion repositories in their native format. We’ll be using the svnsync command to create local mirrors of entire Subversion repositories.

This has a lot of great benefits:

  1. It’s much faster, which means more frequent project report updates for you and less server maintenance for us.
  2. Ohloh’s Subversion converter follows only a single branch, and it cannot follow directory renames (the infamous --stop-on-copy limitation that causes so much forum traffic). The use of svnsync removes these limitations. Note importantly that this does not mean that all of those projects on Ohloh with missing history will suddenly fix themselves — this is just the first step towards that goal.
  3. This code introduces a new abstraction layer in the Ohloh architecture. We now allow multiple native source code formats on our servers. This opens up the ability (finally) for us to add additional source control systems, like Mercurial.

15 Responses to New Subversion Downloader