Hey, Hey, Hey; What’s Happening Today?

Hail Hubbites!

As we talked about in our Open Hub in 2016 post, we have recently made a major step forward in addressing significant infrastructure concerns. Down in the “More Infrastructure” section, we mentioned, “So we started an effort to virtualize our crawlers and are pilot testing that work now.” The FISbot servers are now out of the pilot test and the old crawlers are being decommissioned and un-racked.  That’s not to say that there are no problems, but the problems we have are not worth switching horses back to the old infrastructure. No, we’d rather take care of the horse we’re riding now.

However, the issues we are having are impacting data on the site and, while we are moving quickly to address them, we’d like to share what we know with you so that everyone can be kept up to date:

  • There is an issue that after a Fetch, Import, SLOC cycle is completed, the follow-on Analysis Job is not always being generated.  This is leaving some projects with fresh raw data, but no updated analysis.
  • There is an issue that the Job Scheduler is not always detecting projects with out of date analysis and scheduling new jobs.  This is leaving some projects with no new fresh raw data.
  • We’ve changed the way we are doing some internal tracking and accounting of when jobs were executed. This switch has resulted in a mismatch between the fields where we are tracking job progress and the data we are presenting on the site so that some projects either show the wrong date the data were collected or do not show that value at all.
  • There are some new low-level issues with local copies of repositories.  Since we’ve switched from 18 crawlers with dedicated local storage to virtual servers with a NFS mount to a SAN, we are seeing new file system level issues.  These issues typically cause Fetch jobs to fail.

To address these, we are combing through project and repositories repeatedly throughout the day and scheduling jobs to try and keep everything up to date. Please let us know if you project has fallen behind so we can address it while we work on the code fixes to bring the new FISbot infrastructure up to snuff.

In other news, the Spammer Cleanup program is also out of the Pilot phase and is chugging through our accounts and inviting account holders to verify their account.  We are focusing on those accounts that were created and then show no activity on the Open Hub.  If you get one of these re-verification emails, please simply log on to the site and provide one of the requested forms of verification. However, if you have been an active member of the Open Hub, then you should not be part of this email re-verification process. However, we will still ask for verification when you log in if you’ve not logged in since these new security checks were put in place.

The “Invention Process” for our new security pages has started and is very exciting. We are looking at what we can produce and deploy quickly that will help illustrate the security landscape for OSS projects. After the initial deployment of fact-based data presentation, we will look towards adding additional elements that provide a broader overview of OSS security.  Oh, and look forward to a new Project page layout that will begin moving throughout the site and will take advantage of the larger screen size of modern day browsers.

Final point: Such Perform. Wow Speed.

In the post GitHub, Performance, and Crawlers (Oh My!) from October 2015, we talked about the People Index page performance improving from 18-60 seconds to less than 1 second,  and the Explore Projects page improving from 100 seconds (!) also improving to less than 1 second, and widget performance improving to 1.5 seconds. We were very pleased that we restored the average web server response times to under 1.2 seconds, or 1200 milliseconds.

Ladies and Gentlemen, Boys and Girls, Things and Its; for the past few months, average web server response time has been under 400 milliseconds — a 3X improvement in speed.  Since the deployment of FISbot, average web server response time has been around 200 milliseconds, a 6X improvement in speed.  With a number of FIS jobs and Analysis jobs going unscheduled, we expect some impact to the site performance when we fix these code defects. Never fear; the next infrastructure project will separate the analysis database from the web application database and result in consistently speedy web application performance.

I know it’s been a tough process and at times the site was nigh unusable.  Thanks for sticking in there with us. You guys are the best (I’m getting teary over here). And we’re continuing to work hard to bring you the unparalleled best set of freely available analysis of ALL the OSS projects.  Thank you so very much for being part of the OSS community and member of the Open Hub.

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor
  • My user data has not updated in almost a year. Can you comment on this topic:

    https://www.openhub.net/topics/9969

  • dankohn

    Docker Swarm is not being updated. Could you please re-crawl. https://www.openhub.net/p/docker-swarm

    • Thanks for the ping! The failed job has been fixed and the project is up to date.

  • Hi Peter. Most projects still have not been updated recently. For instance, for gnu-mpfr: “Analyzed 1 day ago. based on code collected 20 days ago.” And for mutt: “Analyzed about 2 months ago. based on code collected about 2 months ago.” For this one, when I click on “Code Locations”, Update Status says: “(Failed about 2 months ago.)” but I’ve checked that this is working.

    • Hi Vincent, Thanks for the ping! These projects are up to date again.

      We’ve found the problem that was impacting job scheduling, and the fix is going into independent verification today. We are optimistic that we can get a fix into production quickly.

  • Basile B.

    Hello Peter, I think that my project reveals another bug on your site: https://www.openhub.net/p/Coedit

    I don’t know if it’s releavant by what has happended is that I’ve registered Coedit by login via GitHub, then a few weeks later I’ve removed your site from the authorized applications because it looked that if was not useful. But now I realize that it could be possible that you use a GH hook to determine if a project has to be re-analized ? Anyway I don’t know what’s _really_ hapening but the project has not been updated since 2 monthes.

    • Hi Basile, thanks for bringing this to our attention.

      It wasn’t a serious problem — just a job that failed due to a temporary condition on the server, but it wasn’t automatically rescheduled. I restarted the job and it finished correctly and I’ll look for other jobs that may have failed in the same way.

      Our Fetch, Import, Sloc, Bot (FISbot) runs through all code locations every 24 – 48 hours to see if they need to be updated. Having a job in a failed state can be a blocker though, as we saw on your project.

      As we noted in Stepping Forward and Back (https://blog.openhub.net/2016/06/stepping-forward-and-back/), we have some other serious infrastructure issues upon which we are working focusedly

      • Basile B.

        Thx, I’ve read the 29 of July blog post so I’ve waited before posting this but it looks like it’s happening again. If there’s anything odd in the repository that prevent the bot to finish its job then let me known !

        • There was no problem in the project or the repository. We simply have a massive backlog that is blocking the creation of new jobs. You know, it’s time for a new blog post. 🙂