New UI, New Account Creation Mechanism, Project Updates

Hail Fellow Open Hubbarians!  You have been marvelously patient, for which we are eternally grateful, as we have been cleanly implementing the Ohloh UI in a new code base on the latest production versions of Ruby and Rails.  Here is how events transpired over the past half year: We first talked about Project PURR in early January, then we identified there was a sizable problem with spammers in February, and in April we confirmed have indeed closed new account creation.  We also shared how we were having performance issues due to excessively slow database queries.

During this time we worked diligently to keep the site up and running while trying to shrink the backlog out-of-date projects.  We even shut the site down to clean out old cruft in the database.  That was very helpful and dramatically reduced the number of out-of-date analyses and support records, and unburdened the system sufficiently so that we caught up with just about all the projects and were looking to get any outliers cleaned up.

About two weeks ago, we lost disks in two of our 18 crawlers.  It should not have been a problem because we keep a supply of disks on hand and we had two remaining.  These disks are not manufactured any more, are hard to find, are expensive, and when we do find them, they are refurbished.  You may not be surprised to learn that one of those two disks was bad and the RAID array was not able to rebuild. And then we learned something else about our infrastructure.  Our crawler architecture is highly robust in terms of jobs being interrupted, killed, and restarted. The design is sufficiently robust so that should one of the workers on a crawler is not available, the others can proceed without interruption.  Unfortunately, the architecture cannot stand the loss of a full server.

Each server, as part of normal processing, will check code and push code to other servers to ensure we have at least 2 copies of each repository.  We tried running the remaining crawlers with the damaged one shutdown.  Within a few hours, all 17 remaining crawlers were hung while waiting to get status from or push to the missing crawler.  We tried restarting the processes repeatedly, and would get some processing to complete, but the vast majority of work would just block and stay hung.  The upshot is that there has not been any crawling or project updates in over 2 weeks.  This is very disappointing especially since we had just gotten ahead of the database performance problems that had been problematic for a while.

So we shut down all the crawlers.  New disks were on order and because the RAID is configured as 10, all should have been well. Two weeks later when the new disks arrived, the RAID controller could not rebuild the array with the new disk.  We’re looking at full data loss, but most critically, we still cannot do any crawling.

We are working on multiple plans:

  • Remove the missing crawler from the database so that other crawlers don’t look for it
  • Adjust the code so that crawlers are more robust when one of the other crawlers can’t be reached.
  • We are redesigning the crawler code so that we can move it off of the 18 crawlers and into a virtual environment that relies on a SAN for a single, highly reliable copy of a repository (no more duplication across crawlers!)

The key take away is that we cannot run the crawlers until we get past this unanticipated failure.  No crawlers means no new analyses. This goes for projects, people and organizations.

Concurrent with this trouble, we are close to releasing the results of Project PURR.  This should be a near seamless transition, with the exception that you will eventually need to log in again. After PURR is in production, we will re-enable the ability to create new accounts.  New accounts will be created using Twitter Digits and to ensure consistency across the site, existing accounts will be required to update their login credentials using Twitter Digits.  We understand that this may be unpopular with some account holders, but we hope all will understand that external verification of an accounts’ validity will help to reduce the amount of spam on the Open Hub and make the site more useful for legitimate users.

We are running final tests and expect to direct some traffic to the new application in the next few days.

Once again, thank you all so very much for being part of the Open Hub community and for your extraordinary patience while we address critical infrastructure aspects in order to deliver features that will help the Open Hub continue to provide value to the Open Source community.

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor
  • Thanks for the detailed update, and sorry for all the pain.

    And a special “thank you” to Black Duck Software for the dedication to this fantastic project!

    M 😉

    • Hi Marc!

      Thanks for your support. It’s so great to work on something for the Open Source community and have such great support!

      I promise you: Good things *are* coming!! In a few months, we’ll have a completely updated infrastructure, architecture, and set of applications. We will be able to build new features and create new integrations and analyses faster and more reliably. I am so very excited!!

      On other fronts; might you be out in OSCON in July? I’ll be there and it would be great to see you.

      • No OSCON this year, as I have another trip planned. Maybe next year!

        Thanks!

  • Kent Fredric

    Ouch. All that stuff sounds painful :(.

    On a related note, it would be nice if there was some user visible status page showing statistics on crawl rates / crawl servers / etc so instead of us standing around for 2 weeks wondering why a repository isn’t updated, we can just look at the status page and see what the expected time frame is for the next crawl.

    Doesn’t affect what will happen, just makes it more bearable from a user perspective other than “Check every day and see if its updated yet, maybe there’s a bug that needs reporting?” and then get a blog about it 2 weeks later 🙂

    • You’re absolutely correct, Kent. That’s on me: I had been hoping that I’d have good news to share. That said, it looks like we’ll be able to reinstall the OS on the missing crawler without loosing any of the 900+ GB of repository data.

  • This is great.. Wonderful post.. Thanks for sharing.

    Web Development India

  • What’s a Twitter Digits?

    Can I use a Twitter account for that (already annoying), or do I need a mobile phone for that? If the latter, I’d like the request the deletion of my account and all associated data, please.

    • Hi Doc;

      Yes, it will require a mobile phone number. We understand that this will be unpopular with some of our users, but felt that we could not provide the service our users deserve with the volume of spam accounts and projects on the site. Email addresses are insufficient to ensure that only real users are creating accounts for their own appropriate use of the Open Hub.

      We will be very sorry to see you go, but you can delete your own account if you wish. Please go to https://www.openhub.net/accounts/me/edit. At the bottom of the page is a red “Delete Account” button.

      • It’s more than just “unpopular” with me. It makes OpenHub completely, 100% unusable to me. Because I don’t *have* a mobile phone (and I don’t want one neither).

        And if you think that mobile phone numbers are sufficient to stave off spammers, you are unfortunately mistaken. Click farmers have piles and piles of SIM cards to burn through and use for all kinds of phone authentification.

        • I hear you. I wish there was an appropriate balance of authentication that wasn’t too heavy handed that still left the site reasonably clear of junk accounts.

          • Florian Brunner

            This sounds bad. I don’t think I will want to login using my mobile phone. Can you elaborate more about Twitter Digits? Especially regarding privacy. From what I read so far it sounds like you would have to register your mobile phone number with Twitter. Again, not somethink I will likely want.

          • Hi Florian;

            Thanks for your question.

            We will be using Twitter Digits only for account verification in lieu of using a CAPTCHA. When you sign up, or login after the update, you will be prompted for your mobile number. We don’t keep that number and you don’t need a Twitter account. You get a verification code on your phone and enter that code into our site. We check with Twitter that the verification code is correct and, if so, we get an authorization number from Twitter and you’re in.

          • Kent Fredric

            I kinda wish there was a reasonable identity platform that allowed people who had control of a domain to authenticate using their domain, perhaps with extra guarding with a SSL certificate requirement on that domain signed by a respected CA.

            It wouldn’t be bullet proof, but I don’t see how getting a domain/certificate is *easier* than getting a sim card.

            I completely empathize with the “I have no phone and don’t want one” crowd, because that was really me for a long time.

            I would even argue we may be strangely more frequent in OpenSource than we are in other demographics, but I don’t have figures to back that. ( Though I do know autism spectrum is over represented in IT for some reason, and there are papers articulating people on the spectrum being more phone averse than the general public, which could have something to do with that perception )

  • dave_b13754

    Thanks for the update and the hard work … looking forward to seeing the new site.

  • Myriam Schweingruber

    Any news on the crawlers? AFAICS there hasn’t been any updates.

    • Thanks for asking! We tweeted about it on 7/11, but really should have posted something here too.

      They’re up and we think we’re caught up. If you have a project that hasn’t recently been analyzed, please contact us at info@openhub.net or tweet us at @bdopenhub.

      The latest news is that we’re switching over to our new application, so there may be some change-over issues as we ramp up the traffic on the new UI app.

      • Myriam Schweingruber

        Great news, thank you for the update, Peter.
        I will send a mail right away, as sadly KDE is still not updated.

        • Myriam Schweingruber

          apparently sending mails is not very efficient either, I got no reply, and it’s now almost 8 month we face the same problem 🙁

          • Hi Myriam; We scheduled a new Analysis a few weeks ago, after the previous message from you. One of the 364 code locations failed repeatedly: git://anongit.kde.org/calligra. We were able to get a refresh from this location and have scheduled another Analysis. Thanks for the heads up!

          • Per Hedbor

            Hm. our repository (pike) has not been crawled in the last 20 days, although there are few days when there are not commits (that is, it’s actually one of the more actively developed open source projects on the site, but I guess it’s so unpopular it’s seldom crawled). Anyway, it used to be crawled every 2 weeks or so, but recently there has been on average 3-4 weeks between updates.

          • Sorry for the delay in responding. The project has been rescheduled for an update and we’ll monitor the progress.