Update: New Account Creation, PURR, and Project Analyses

Hi Folks; Hope you’re having a great 2015.

Just past the first quarter and we’re cooking with gas at Open Hub Land.

The biggest news is that, as we talked about in February, we shut off new account creation. If you want an account, please contact us at info@openhub.net and we can manually create one for you. We have selected a replacement technology for the ReCaptcha that should help ensure users are legitimate individuals and are creating only one account.  We may save that for the new UI, which will be deployed at the end of May.  More on that in a bit.

We’re exploring Twitter Digits, which will require new accounts to have a phone number that can receive an SMS message and will limit the use of a phone number to a single account. We are also talking about simply re-enabling new account creation on the Open Hub using the current mechanism in the hopes that the spammers have dropped us from their site lists.  However, one suspects their systems have a significant degree of automation. Therefore, on one hand we can wire up a new account verification model and carry it forward if it works as desired even if we only have less than two months on the current platform, or we can restore the old mechanism and hope we don’t get slammed right away, or we can do nothing until the new code is ready (which is the least desirable option since legitimate folks are looking to get accounts on the Open Hub).  Active conversations are happening today.

Beyond re-enabling new account creation, most of our focus remains on Project PURR — Platform Upgrade Ruby and Rails.  This project is a migration of all the User Interface code to the latest stable version of Ruby and Rails. This was a big undertaking and we are executing it well.  Two thirds of the way through the project and we have migrated about 130 models, 50 controllers, 400 views, over 100 asset files, plus a smattering of decorators, validators, null models, mailers, etc.  Additionally, there is over 2K lines of tests in 270 test files with over 99% code coverage for the Ruby and Rails code. We are focused on migrating only those models that are needed to display the UI.  There are models that are used by both the backend analytics as well as the UI, so we are migrating only the UI related code.

We set a six month timeline to complete this work and are well on target to complete on time.  The best part is that there shouldn’t be much of a change that anyone notices. The look and feel of the web application will remain the same, although we should pick up some improved mobile layout by using Bootstrap 3.  That said, we are not spending extra effort at this point to optimize the mobile experience; we still get a tiny percentage of traffic from mobile devices, which can be interpreted both ways as an argument both for and against mobile UI optimization.

On the schedule for this year is a re-architecture of the underlying database. As we talked about in early February, the fact that we run analyses and serve the web site from the same massive database is at the heart of our performance challenges. We decided that PURR would use the same DB as the current web application code, that way we could control the risk associated with the project.  When we re-architect the database, we will divide it into two parts — an analytics DB and a web application DB.  There will be a data publication model from analytics to web app and a messaging system from web app do analytics so that we can trigger re-analysis when an account holder claims a position on a project. While we are doing all that we can to keep the database well tuned, vacuumed and analyzed, the site will continue to suffer until we can separate these two areas of concern.

Directly related to that topic of analysis is the backlog of project analysis.  Shortly after that blog post, we rescheduled a block of 120,000 jobs that had been trapped by a code defect.  At the time, we had over 375K projects to analyze, as we tweeted at the time. We were scheduling those trapped jobs 21,000 at a time and carefully monitoring the database to balance the needs of analysis and web presentation.  It took over two weeks to work our way through that backlog. Since we started that process, we made some configuration changes to the database that improved performance. All the analysis jobs are back to running at regular scheduling priority and we are looking for projects with outdated analysis that are not due to a code repository failure, which is the largest analysis blocker.  Please help us out by tweeting to @bdopenhub about projects with outdated analysis so that the projects important to are sure to get attention.  All of the projects we’ve known about and restored are being processed correctly and are being refreshed well within the targeted 3 day window.

About Peter Degen-Portnoy

Mars-One Round 3 Candidate. Engineer on the Open Hub development team at Black Duck Software. Family man, athlete, inventor

8 Responses to Update: New Account Creation, PURR, and Project Analyses