We’re doing it!

It’s happening!  We’ve started clean fetches of ALL of our repositories using the new SAN! For background, please see the Details on the Infrastructure blog post.

We currently have 497K projects that have 592K distinct repositories that we are going to reprocess from scratch. To do this, we cleared out all the old jobs that have not completed, connected our FISbots (Fetch, Import, SLOC bots) to the new SAN, and started re-scheduling new Fetch jobs.

We’ve completed nearly 100,000 repository fetches and have some 445,000 scheduled with a few more to schedule.  We are also monitoring the failures. Unlike the last set of failures, which could include problems due to the old SAN, these failures should all be actionable.  While there will be some repositories that will just be hard to get because they are large or the servers are slow, most of the failures are turning out to be repositories that are no longer present.  These types of failures are a real opportunity to look at the projects and determine if we can update the enlistments, or if the project has been abandoned and is no longer available any where (in which case, we will remove it from the Open Hub).

So, what does this mean for you, the awesome Open Hub User?

It means that it might take some time to get your project re-fetched and updated. We’ll do our best to respond to requests to get things updated, but please know that there is now a massive backlog that will take some time to process.

And after this is all over, we will have a smaller, leaner set of projects on the Open Hub that fulfill our mandate of monitoring active OSS projects.  And that will be better for all of us!

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor

56 Responses to We’re doing it!