3 Month Update

Hi Folks; Happy 2015. We at Open Hub Central are overdue with an update. Here is the skinny on what has happened, what is happening and what is planned for the Black Duck Open Hub since our last touch point at the end of August:

Done

  • Project Activity Indicator (PAI) level added to the API
  • Updated Project Widgets to use the new Open Hub logo
  • Addressed a number of smaller UI Layout issues
  • Added Organizations to the API.  Available API calls are:
  • Added the necessary documentation for the Organization API  😀
  • Added water marks to our charts. You can seem them on our Project pages and our Account pages. Take them; please.  Oh, and the Organization InfoGraphic, take that too.
  • We added an automated script to generate our language analysis on a monthly basis now that we have confidence in the performance of database.
  • We updated all our back-end crawlers to Ubuntu 14 and Ruby 1.9.3.  Now our entire infrastructure — web servers, utility servers and crawlers — are all running the same OS, Ruby and code.
  • We improved the Organization Portfolio Projects page table with enhanced data — that updated UI will be available with our next push to production!  Before we could roll out the updated user interface, we needed to make sure the data were in place, so first we updated the analysis to generate these new data fields.  That’s been running quietly in the background for a while now.
  • We fixed many, many, many encoding issues.  Thank you for your continued patience as we continue to navigate forward from an encoding-agnostic version of Ruby to passionately encoding-aware version of Ruby.
  • Improved database performance for a number of analytical queries, improved database configurations for others.  We found a tool — pg_repack — that let us rebuild tables without taking the tables offline.  This improved table layout and indices, and reduces storage requirements as well.  We also aggressively monitored the vacuuming of our tables.
  • The improvements to our database performance let us increase our back-end processing thresholds, which ultimately let us get caught up with our analytical processing.  Our apologies to all those who were frustrated to see their projects’ analysis get weeks and then a month or more out of date and our thanks for hanging in there while we addressed the problems.
  • We deployed a series of back-end fixes as well that reduced issues with hanging CVS jobs, fixes for SVN checkout corruption problems, SVN “targets are directory” errors, problems adding new SourceForge CVS projects to the Open Hub.
  • We added more infrastructure to protect the Open Hub from spammer accounts and have some additional ideas as well.
  • We fixed a particularly visible bug that hid the available, unclaimed commits when someone was not logged in
  • We converted the Ohloh SCM library to a gem and published it on the RubyGems.org server.

In Progress

The most exciting news is our current effort, which we have called “Platform Upgrade: Ruby and Rails” or “Project PURR”.  The purpose of this precisely planned project is to provide our platform with the power to promptly produce pertinent performance and practical process improvements.

As we’ve mentioned before, we’ve been running on Rails 2.3, which is currently only getting updates for severe security issues.  Upgrading our Rails version has been a long-overdue task. But there is often significant pain with upgrading from Rails 2 to Rails 3. Plus, Rails 4 has been out for a while and Rails 5 is peeking over the horizon.  We didn’t want to do an expensive migration up to Rail 3 and then have to do another migration up to Rails 4.

Additionally, the current Ohloh code base has been in continuous development since around 2006 and has had the input of 19 different contributors.  All the engineers here will recognize the likelihood that the code today is just not as optimally designed and realized and one might wish.

I would like to be clear that we are grateful to all those developers who worked to improve Ohloh from the conceivers, Robin Lucky and Jason Allen, through to my predecessors, Abhay Mujumdar and Jared Barboza.  Hats off to these and the other engineers!  But there is cruft and dingwitz in that thar code there.  We didn’t want to migrate the code base forward and risk carrying the explorations, dead code, trials and inventions that will inevitably be present in any code base that has been in active development for 8 years.

We are also working on centralizing some services so that the core work of getting code repositories and counting lines of code can be more reliable, scalable and available to other consumers of such data at Black Duck.  This will eliminate a good body of code from the Ohloh code base.  We didn’t want to carry code that will be eliminated forward either.

Therefore we are migrating the functionality of the Ohloh UI to a new code base using Ruby 2.1 (soon to be Ruby 2.2) and Rails 4.  We’re taking all the logic and moving that over, refactoring models and classes, and taking advantage of features present in Rails 4 to build a leaner and more maintainable code base.  The User Interface will look the same and the database schema will remain the same so that the current analytical processes do not have to change.

Coming Up

We have a number of initiatives and plans for 2015.  Here is a subset of them:

  • Have Open Hub account holders earn the privilege of gaining some administrative functionality that will let them delete duplicate projects, mark spammy accounts as such, and manually schedule analytical jobs.
  • Help folks discover open source solutions and share their discoveries with selected individuals.
  • Have different landing page experiences for logged in users from non-logged in users
  • Make Code Search a integrated feature in the Open Hub
  • Take advantage of the new infrastructure for improved repository maintenance.

We are putting together a new Open Hub Community Day for March 2015 and hope many more folks will be able to join us this year.

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor
  • Cpchang Cp

    While i run rtspcamera, a error message pop up as below ,
    “An internal error occurred during: “Launching New_configuration (1)”.
    Path for project must have only one segment.”

  • Hi Peter!

    Congrats on the progress!

    And thanks for the detailed report. Most people have no idea how much work it is. It’s a good example of how important it is to design systems that are future-proof.

    You are doing something unique and challenging, and so important. Keep up the fantastic work that you & Black Duck are doing for the ecosystem!

    Best regards,

    M 😉

    Tiki Suite

    https://suite.tiki.org/Tiki+Suite

  • David Martínez Moreno

    Hello, Peter. I returned to Ohloh/OH since several months of inactivity and the first thing I noticed is that the site is extremely slow.
    Is this code restructure also the reason why even the most active projects like the Linux Kernel are several weeks behind?
    Linux kernel: 18 days
    Mozilla core: 3 months
    OpenOffice: 18 days
    GIMP: 17 days
    VLC: 17 days
    X.Org (no activity available): 3 months
    OpenStack: 1 month

    I’d love to see stats on queue sizes, average time to refresh repos, number of code repos, and many more! Do you have them available?

    Thanks for posting anyway. 🙂

    • Hi David, thanks for coming back. Yes, we are aware of the performance issues and are focused on them. It would seem to be a confluence of vacuums on the database tables, aggressively running analyze jobs to catch up, and our ever-growing database size.

      We like to touch every project every 3 days. It looks like our back end systems can do that now, so we are looking for those projects in a failure state and manually restarting them.

      We have almost 668K projects and it looks like 120K of them are in a failed state that need to be manually restarted. We want to reschedule them in chunks so we don’t swamp the database.

      • Ah; just found this (above) reply. I just created a new topic (https://www.openhub.net/forums/8/topics/9730). It’s been this way for months. You *really* need to take care of this. I know you are aware and working on it, but if this has been lasting for over 3 months then you will lose (and will have lost) a lot of users.
        Maybe cache the html response(s) (more) aggressively?

  • Arno Nym

    What happened to the rank number in list of people?

    • May I ask for some additional details? What is it that you are seeing or not seeing that is surprising? There is likely impact to ranking due to the number of jobs in a failed state from last month.

      • Arno Nym

        I was number 11847 in the people list the days when it was called Ohloh. Now I don’t know my place (or is it called kudo_position?) in the developers chart list anymore. 🙁 Can you bring that feature back, please?

        • Thanks for the details; we’ll look into it.

        • elizabeth wingler

          Who did editing of Elizabeth winglers face book last year and what’s Jeffery Wingler keeping from wife

  • Kaz Nishimura

    Can you make more frequent reports on your performance issues? Now even your Project Update Status page is not updating regularly. All we can do would be writing in your forums these days.

  • Pingback: Progress report: Catching up on outdated analyses | Black Duck Open Hub Blog()

  • Earth Day Facts

    Can you make more frequent reports on your performance issues?What Happened to the rank issues?Earth Day Facts