We’re doing it!

It’s happening!  We’ve started clean fetches of ALL of our repositories using the new SAN! For background, please see the Details on the Infrastructure blog post.

We currently have 497K projects that have 592K distinct repositories that we are going to reprocess from scratch. To do this, we cleared out all the old jobs that have not completed, connected our FISbots (Fetch, Import, SLOC bots) to the new SAN, and started re-scheduling new Fetch jobs.

We’ve completed nearly 100,000 repository fetches and have some 445,000 scheduled with a few more to schedule.  We are also monitoring the failures. Unlike the last set of failures, which could include problems due to the old SAN, these failures should all be actionable.  While there will be some repositories that will just be hard to get because they are large or the servers are slow, most of the failures are turning out to be repositories that are no longer present.  These types of failures are a real opportunity to look at the projects and determine if we can update the enlistments, or if the project has been abandoned and is no longer available any where (in which case, we will remove it from the Open Hub).

So, what does this mean for you, the awesome Open Hub User?

It means that it might take some time to get your project re-fetched and updated. We’ll do our best to respond to requests to get things updated, but please know that there is now a massive backlog that will take some time to process.

And after this is all over, we will have a smaller, leaner set of projects on the Open Hub that fulfill our mandate of monitoring active OSS projects.  And that will be better for all of us!

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor
  • Nice to hear that things get improved. But the status page https://blog.openhub.net/status/ doesn’t show any progress. Is it working? Or is there any other page showing the progress?

    • Hi Vincent;

      Sorry for not replying sooner. The status graph is functioning correctly. The maximum value in the scale is nearly 250K and we’re processing less than 8000 per day due to the massive amount of recrawling and some other issues.

      We’re working on getting the numbers back to where they should be.

  • jtjsowosl

    Could you answer, why my github-bound openhub account was disabled and why have you stopped the conversation in email about this issue?

    • Hi jtjsowosl;

      I’m really sorry for any trouble you’ve had. Unfortunately I cannot find any record of an account with this user name or the email address you provided to post this comment. Nor can I find any email conversation with this name or address.

      May I ask that you please contact us at info@openhub.net so we can look into this issue and resolve your question? Again, I am sorry to put you through this process again.

  • Dan Kohn

    Could you please look at https://www.openhub.net/p/kubernetes which has not been updated in 2 months.

    • Thanks for the ping, Dan, we restarted a blocked job and everything completed successfully

      • Dan Kohn

        Thank you. I greatly appreciate the service.

  • Removing code locations seems broken. I tried to add the actual code location for libnfc (https://www.openhub.net/p/libnfc/), which is pending (any info about the progress of the blast re-indexing?), but trying to remove the two old locations leads to a 404.

    • Hi Romain;

      I think the actual issue was the length of the project description. Quite some time ago the maximum length was set to 800 characters, but existing descriptions were grandfathered in. Then, with the rebuild of the UI, we added a validation check that raises a misleading exception.

      I shortened the description; would you try adding the code location again?

      • Lol, you are right! I could remove the deprecated enlistments and the good one I added a couple days ago started to import the project ! Unfortunately, it failed (I guess that the import was triggered by the removals? BTW, I guess that the system will catch up the next time it tries to fetch the code).

        Do you have some information about the progress of the analysis of all OpenHub projects?


        • I just happened to be examining a problem on the server that was working on your project and I used up all the available disk space. So I just restarted your Import job. So far so good.

          I’ll write a blog post soon with more details, but we are recovering from a bad NFS mount while making a major upgrade to our Fetch, Import and SLOC infrastructure.

          First job is to stabilize the new infrastructure, then we’ll spend some time cleaning up projects that are out of date.

  • Mark Krake

    Hello Peter,
    thanks a lot for the good news.
    I’ve been waiting for our project to be updated but it seems as if it has run into issues again. Has now not been updated for 3 months.

    Can you please give it a bump? https://www.openhub.net/p/metasfresh/enlistments


    • Thanks for the ping, Mark. 170 code locations are in process

      • Mark Krake

        Great! Thanks a lot Peter.
        Our project is up-to-date now.

  • Tofe

    Hello Peter,
    Could you please have a look at https://www.openhub.net/p/webos-ports/ ? it never really finished its first analysis, which was 2 months ago.

    • Hi Tofe;

      Updates are in progress. Thanks for the ping

    • Hi Tofe,

      There were 9 repositories that no longer had a “master” branch. For most of them, the webOS-ports/master was the active branch.

      The old code locations (repository + branch) were removed and replaced with new code locations with the active branch (mostly).

      We should be able to get the project update completed now.

  • Jan

    Hi Peter,

    can you have a look at MPF: https://www.openhub.net/p/mpf/enlistments? Last fetch was 2 month ago.

  • Jean Deruelle

    Could you please look at https://www.openhub.net/p/restcomm which has not been updated in 2 months ?

    • Hi Jean;

      We were able to get the project updated a few days ago. Thanks for the heads-up!

      • Jean Deruelle

        Thanks for the fast turnaround time and great service !

      • Jean Deruelle

        By the way, what is the typical cycle for updating projects ? I see that it has not been updated in almost a month.

        • Prior to the need to re-fetch all 700K code locations, we were updating projects every 1-3 days. After we have re-fetched everything, our goal is to return to that period and actually exceed it with updates every 24-48 hours.

          • Jean Deruelle

            Impressive. Do you have any ETA on when it is planned to arrive at that goal so I stop pinging you every month or so to ask for a re fetch/refresh 🙂 ?

          • I’ve been working on those numbers and will provide as much as I can in a blog post, which is also in progress.

          • Jean Deruelle

            Thanks Peter. I was wondering if by any chance, you would have time to kindly reschedule an analysis of the project as it has fallen behind for about a month now https://www.openhub.net/p/restcomm ?
            Thanks a lot for the time and effort in openhub

          • Hi Jean;

            We just did a nice feature release that had some extended consequences. 🙂

            RestComm has been updated again.

          • Jean Deruelle

            You guys rock. Very nice update ! Just a feedback comment is that on mobile devices such as Chrome for Android, the website and graphs are a bit hard to see. Keep up the great work !

          • Thanks for your kind words and helpful feedback. We know our mobile game is weak right now.

          • Hi Peter, I was wondering if by any chance, you would have time to kindly reschedule an analysis of the project as it has fallen behind for about a month now https://www.openhub.net/p/restcomm?
            Thanks a lot for the time and effort in openhub.net again

      • Jean Deruelle

        Hi Peter, I was wondering if by any chance, you would have time to kindly reschedule an analysis of the project as it has fallen behind for about a month now https://www.openhub.net/p/restcomm. Sorry about the hassle.
        Thanks a lot for the time and effort in openhub.net again

        • Hi Jean; Sorry for the delay in getting back to you. Restcomm is up to date. Thanks for your help and support!

      • Jean Deruelle

        Hi Peter, I was wondering if by any chance, you would have time to kindly reschedule an analysis of the project as it has fallen behind for about 2 months now https://www.openhub.net/p/restcomm
        Thanks a lot for the time and effort in openhub.net again !

        • Hi Jean;

          All up to date! Thanks for the heads up. I promise, we are working on getting updates to projects back on the original tight schedule!

          • Jean Deruelle

            Thanks Peter !

          • Jean Deruelle

            Hi Peter,

            Gentle ping. It’s been more than a month, would you be so kind to trigger another update ? Thanks a ton!

          • Thanks for the ping! Project is up to date.

            We’re still working hard on making our back end screamingly fast!

          • Jean Deruelle

            Hi Peter,

            Another Gentle ping. It’s been more than 2 months, would you be so kind to trigger another update ? Thanks again !

          • Hi Jean;

            Thanks again for the ping! It’s been updated, although at least one code location was refreshed 5 days ago. I’ll run another full update to bring everything up to par.

            The problem was a series of jobs that had been killed, but not auto-rescheduled.

  • Eugene Sandulenko

    Hi Peter, could you please take a look at https://www.openhub.net/p/scummvm Seems to be stuck.

    And thank you for your updates.

    • Hi Eugene;

      Thanks for the ping. New jobs are being blocked by the backlog of existing jobs, so I scheduled a full update, which is in progress and which I will monitor.

      • Eugene Sandulenko

        Thanks a lot, much appreciated

        • Eugene Sandulenko

          Hi Peter, would it be possible to kick the scummvm project once again? It got stuck for 2.5 months now.

  • Milanium

    https://www.openhub.net/p/openchrom/enlistments didn’t really update when I added more repositories. Instead it shows “Ignored Files: All files included.” for the latest repositories added. Is that a bug?

    • Hi Milanium;

      Thanks for the heads up. Yes, there was a job that was blocking the others. Everything has been cleared and the project is up to date.

  • Dan Kohn

    Sorry to bug you again, but could you please unstick Mesos and Docker Swarm. https://www.openhub.net/p/_compare?project_0=Kubernetes&project_2=docker+swarm&project_1=Mesos

  • Uwe Scholz

    Hi Peter! Could you please have a look at https://www.openhub.net/p/gnome-commander? The project has not been updated for one and a half month. Thanks in advance.

  • Cédric Bonhomme

    Hello @peterdegenportnoy:disqus ! Are you still in the process of updating all projects from scratch? Lot of projects I am involved are not updated since one or two months or more.
    For example https://www.openhub.net/p/Stegano, https://www.openhub.net/p/JARR or https://www.openhub.net/p/shelter-database .
    The status of the Stegano project is “Waiting in queue” since weeks.
    Can you please have a look?
    Thank you!

    • Cédric Bonhomme

      Hi @peterdegenportnoy:disqus ! Could you please update just the JARR project ( https://www.openhub.net/p/JARR ) ? An updated was processed yesterday, but on a 2 months old code ! thank you very much!

      • Hi Cédric;

        Sorry for not responding to your earlier message! All three projects have been updated. Thanks for the heads up!

        • Cédric Bonhomme

          No problem! Thanks a lot!

  • Barry Smith

    Could you please update https://www.openhub.net/p/petsc its been over two months. Thanks