3 Month Update

Hi Folks; Happy 2015. We at Open Hub Central are overdue with an update. Here is the skinny on what has happened, what is happening and what is planned for the Black Duck Open Hub since our last touch point at the end of August:

Done

  • Project Activity Indicator (PAI) level added to the API
  • Updated Project Widgets to use the new Open Hub logo
  • Addressed a number of smaller UI Layout issues
  • Added Organizations to the API.  Available API calls are:
  • Added the necessary documentation for the Organization API  😀
  • Added water marks to our charts. You can seem them on our Project pages and our Account pages. Take them; please.  Oh, and the Organization InfoGraphic, take that too.
  • We added an automated script to generate our language analysis on a monthly basis now that we have confidence in the performance of database.
  • We updated all our back-end crawlers to Ubuntu 14 and Ruby 1.9.3.  Now our entire infrastructure — web servers, utility servers and crawlers — are all running the same OS, Ruby and code.
  • We improved the Organization Portfolio Projects page table with enhanced data — that updated UI will be available with our next push to production!  Before we could roll out the updated user interface, we needed to make sure the data were in place, so first we updated the analysis to generate these new data fields.  That’s been running quietly in the background for a while now.
  • We fixed many, many, many encoding issues.  Thank you for your continued patience as we continue to navigate forward from an encoding-agnostic version of Ruby to passionately encoding-aware version of Ruby.
  • Improved database performance for a number of analytical queries, improved database configurations for others.  We found a tool — pg_repack — that let us rebuild tables without taking the tables offline.  This improved table layout and indices, and reduces storage requirements as well.  We also aggressively monitored the vacuuming of our tables.
  • The improvements to our database performance let us increase our back-end processing thresholds, which ultimately let us get caught up with our analytical processing.  Our apologies to all those who were frustrated to see their projects’ analysis get weeks and then a month or more out of date and our thanks for hanging in there while we addressed the problems.
  • We deployed a series of back-end fixes as well that reduced issues with hanging CVS jobs, fixes for SVN checkout corruption problems, SVN “targets are directory” errors, problems adding new SourceForge CVS projects to the Open Hub.
  • We added more infrastructure to protect the Open Hub from spammer accounts and have some additional ideas as well.
  • We fixed a particularly visible bug that hid the available, unclaimed commits when someone was not logged in
  • We converted the Ohloh SCM library to a gem and published it on the RubyGems.org server.

In Progress

The most exciting news is our current effort, which we have called “Platform Upgrade: Ruby and Rails” or “Project PURR”.  The purpose of this precisely planned project is to provide our platform with the power to promptly produce pertinent performance and practical process improvements.

As we’ve mentioned before, we’ve been running on Rails 2.3, which is currently only getting updates for severe security issues.  Upgrading our Rails version has been a long-overdue task. But there is often significant pain with upgrading from Rails 2 to Rails 3. Plus, Rails 4 has been out for a while and Rails 5 is peeking over the horizon.  We didn’t want to do an expensive migration up to Rail 3 and then have to do another migration up to Rails 4.

Additionally, the current Ohloh code base has been in continuous development since around 2006 and has had the input of 19 different contributors.  All the engineers here will recognize the likelihood that the code today is just not as optimally designed and realized and one might wish.

I would like to be clear that we are grateful to all those developers who worked to improve Ohloh from the conceivers, Robin Lucky and Jason Allen, through to my predecessors, Abhay Mujumdar and Jared Barboza.  Hats off to these and the other engineers!  But there is cruft and dingwitz in that thar code there.  We didn’t want to migrate the code base forward and risk carrying the explorations, dead code, trials and inventions that will inevitably be present in any code base that has been in active development for 8 years.

We are also working on centralizing some services so that the core work of getting code repositories and counting lines of code can be more reliable, scalable and available to other consumers of such data at Black Duck.  This will eliminate a good body of code from the Ohloh code base.  We didn’t want to carry code that will be eliminated forward either.

Therefore we are migrating the functionality of the Ohloh UI to a new code base using Ruby 2.1 (soon to be Ruby 2.2) and Rails 4.  We’re taking all the logic and moving that over, refactoring models and classes, and taking advantage of features present in Rails 4 to build a leaner and more maintainable code base.  The User Interface will look the same and the database schema will remain the same so that the current analytical processes do not have to change.

Coming Up

We have a number of initiatives and plans for 2015.  Here is a subset of them:

  • Have Open Hub account holders earn the privilege of gaining some administrative functionality that will let them delete duplicate projects, mark spammy accounts as such, and manually schedule analytical jobs.
  • Help folks discover open source solutions and share their discoveries with selected individuals.
  • Have different landing page experiences for logged in users from non-logged in users
  • Make Code Search a integrated feature in the Open Hub
  • Take advantage of the new infrastructure for improved repository maintenance.

We are putting together a new Open Hub Community Day for March 2015 and hope many more folks will be able to join us this year.

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor

15 Responses to 3 Month Update