Lessons Learned Upgrading Harvest to Ruby 1.9.3

We’re thrilled to announce that all of our apps have been upgraded from REE to Ruby 1.9.3. We wanted to share some notes about what went well, what went wrong, and what we learned in the process.

The Payoff

The vertical red line marks our update to Ruby 1.9.3, and as you can see, the results were impressive (lower is better). Our average response time dropped from around 150ms per request to around 50ms.

Our server loads took similar dips:

The first graph shows our server load for our non-core apps (Co-op, our forum, and some internal tools), and the second graph shows the load for our marketing site and for the main Harvest application.

Importantly, during the period shown in all the graphs above, our traffic volume was increasing steadily (and sometimes dramatically), and yet our resource usage still decreased with the upgrade.

Aside from those server-side gains, we enjoyed some local benefits as well. I did some benchmarking of our test suite for Harvest before and after the upgrade, and our suite runs 12.67% faster on Ruby 1.9.3, which saves us a few minutes on every run.

Procedure and Timelines

We upgraded six different apps from REE to 1.9.3. Our goal was to start with the smaller apps first and slowly learn our way up to the main Harvest application. Ultimately, I think this worked out well — we discovered a lot of the smaller gotchas earlier in the process on our simpler apps, and weren’t sent on quite as many wild goose chases in the more complicated ones.

As we moved to each new app, our general procedure stayed roughly the same:

1. Use RVM to jump to 1.9.3 and a clean gemset, fix any errors from bundle install (usually just by simply upgrading gem versions), then attempt to run our test suite. (Note: We’ve since transitioned to rbenv and rbenv-gemset due to some compatibility issues with pow, but the process is the same.)

2. Usually, our suite would crash, and we’d have to upgrade a handful of gems and plugins.

3. Once our tests were running, we’d step through each error and failure and work our way towards a clean run.

4. Once we had a clean run of our test suite, we did some local click testing (hitting what we thought would be pain points), and then checked that app off the list and moved on, saving formal QA for after all apps were upgraded.

To run through these steps for each app was actually a surprisingly quick process. This blog and our forum each took less than one day, our marketing site took less than two days, and Co-op and Harvest each took just a week, although that was with the full-time focus of two developers (myself and prime hacker Barry Hess).

We were able to upgrade Ruby on all of our application servers without any downtime by using Chef and the nginx Healthcheck module (special hat-tip to the dev-ops wizardry of our very own Warwick Poole).

Changes and Pain Points

On the whole, the upgrade was a smooth affair, but we still needed to make a fair number of updates and ran into a couple of problems along the way.

Method Changes

The majority of our test failures and errors were caused by assorted syntax updates and deprecations in 1.9.3. Most of these were pretty minor, but were often hard to hunt down (like the changes to to_s for many-but-not-all classes).

• Array#to_s performed a join in 1.8. In 1.9, it became an alias for inspect. A similar change occurred with Hash#to_s.

• String no longer includes Enumerable, so there’s no more String#each. It’s been replaced by #each_byte, #each_char, #each_codepoint, and #each_line, depending on what you’re after.

• String#starts_with? and String#ends_with? became String#start_with? and String#end_with?, which was a nice and easy find-and-replace fix.

• No more colons with when in case statements.

• Date#parse no longer plays nicely with MM/DD/YYYY-style dates:

    1.8.7 > Date.parse("12/14/1986")
    => Sun, 14 Dec 1986
    1.9.3 > Date.parse("12/14/1986")
    ArgumentError: invalid date
  

• Rational#to_s no longer reduces fractions-over-1 to just their integer representation:

    1.8.7 > Rational(2,1).to_s
    => "2"
    1.9.3 > Rational(2,1).to_s
    => "2/1"
  

  This caused us to briefly inform customers that they had invoices that were “38/1 days late”.

This list is not exhaustive, and we found many more in our pre-upgrade research that didn’t hit us (Hash#key was replaced with Hash#index, Hash#select now returns a Hash instead of an Array, Object#type became Object#class, etc.), so your mileage may vary.

CSV Changes

FasterCSV has been brought into the 1.9 standard library and is now just CSV.

Most of the fixes to handle this were pretty easy: simply update the class name from FasterCSV to CSV, then make some straightforward updates to the new CSV reading and writing methods. That knocked out almost all of our issues.

Two edge cases ended up taking up the majority of time spent on CSV fixes: properly handling imported CSVs with BOMs and with carriage returns. Let’s ignore those particular fixes, though, and focus on why this is another great example of why having an exhaustive test suite is a very good thing.

We probably would have never thought to check for these edge cases in our QA, but luckily, they’re covered by tests in our suite. If those tests weren’t there, we probably wouldn’t have known those problems existed until a customer unsuccessfully tried uploading an Excel-generated CSV, leading to a support ticket and wasted developer time to fix a bug that we’ve seemingly already fixed once before.

So write those tests.

Encoding

Encoding ended up being our biggest real world problem, because it didn’t bite us until we went to production with Co-op. We weren’t the only ones to experience this pain.

If you’re interested in the ins-and-outs of encoding in 1.9, check out James Edward Gray II’s 11 Part Series on Character Encoding in 1.9.

Most of our problems in development were relatively minor and fixed with magic comments.

Our big problems came up in production with data that had been stored as one encoding but now was coming out and assumed to be in UTF-8.

1. First, we had problems with encodings in the shared cache between Harvest and Co-op. Data was coming out of the shared cache in Co-op with a ASCII-8BIT encoding, which was not what the upgraded Co-op was expecting or could handle well with its own strings all in UTF-8. Monkeypatching memcache-client allowed us to force-encode all strings coming out of the cache to UTF-8. Notably, this was only an issue while Co-op and Harvest’s Ruby versions were mismatched — once we upgraded Harvest, we removed the patch and everything worked perfectly between the two apps like before.

2. Our next big encoding problem came from serialized YAML, just like Tobi said it would. Like with the shared cache with Co-op, when the data was serialized after the upgrade, there was no problem getting it back out, so this only affected data serialized before the upgrade that was accessed afterward. We considered a few fixes here — migrate the whole DB to fix the encoding, fix the encoding as the records were individually accessed, monkeypatch ActiveRecord — and ended up going with that last one.

    class ActiveRecord::Base
      def unserialize_attribute_with_utf8(attr_name)
        traverse = lambda do |object, block|
          if object.kind_of?(Hash)
            object.each_value { |o| traverse.call(o, block) }
          elsif object.kind_of?(Array)
            object.each { |o| traverse.call(o, block) }
          else
            block.call(object)
          end
          object
        end
   
        force_encoding = lambda do |o|
          o.force_encoding(Encoding::UTF_8) if o.respond_to?(:force_encoding)
        end
   
        value = unserialize_attribute_without_utf8(attr_name)
        traverse.call(value, force_encoding)
      end
      alias_method_chain :unserialize_attribute, :utf8
    end
   

Worth It?

We think so. There has been plenty written about the theoretical speed increases you’ll see with Ruby 1.9.3, but we’re glad to share that we’ve seen significant wins in our complex real-world applications and in our local environments with just a couple of weeks of development.

Discuss on Hacker News

My First Program: Doug Fales

Hello world! I am Doug Fales, and I work at Harvest as a developer. I help build and maintain features in Harvest, the Harvest for Mac desktop client, and the iPhone app.

My first program was written in AppleSoft BASIC on my family’s Apple IIGS. I was inspired in large part by reading Clifford Stoll’s excellent book The Cuckoo’s Egg. I can remember thinking how cool it would be if one day I understood enough about computers to do what he had done in tracking down a Russian hacker.

My first program did not involve Russian hackers in any way. To my wife’s eternal horror, it involved drawing a clown face on a grid of 40x80 color blocks. I can’t remember what the graphics mode was in AppleSoft, but the pixels were quite large, and the clown face I drew was all the more disturbing because of it.

Though I don’t remember many of the details, I can vividly remember thinking how difficult this programming stuff was as I laboriously typed out line after numbered line (remember to leave space between the line numbers for future edits!). How did the pros ever finish a non-text-based game if they had to draw out each pixel like this?

This was in high school for me, circa 1996, and I was teaching myself on a lonely technological island from boring reference books that shipped with our old IIGS. No programming classes in school. No real text editor aside from the horrendous BASIC mode that came with the IIGS. Control structures and variables? That would have to wait until college.

Needless to say, this program was some of the worst code I’ve ever written. Yet somehow the resulting feeling of having built something that I had mentally conceived and then translated into a computer’s language was so rewarding that I would decide to do it for a living.

Looking back, the thing I appreciate most about my clueless younger self is the enthusiasm that enabled me to power through until I had something resembling my original idea. My technique and tools have evolved since then, but the process is remarkably the same: an idea, an enthusiastic frenzy of building and design, followed by another idea of how it could be better, followed by more enthusiastic refactoring. It’s important for me to remember this–that what we do has remained fundamentally unchanged, in a good way. That building a complex piece of software for thousands of customers is still as much fun (more fun, even) as hacking AppleSoft BASIC without variables or control structures.

My First Program: Barry Hess

Over the coming weeks the Harvest tech team will introduce themselves. We’ve all had a blast recently sharing the story of how we each hacked out our first program. I’ll get things rolling…

In the early 80’s my aunt and uncle sent our family a TI-99/4A for Christmas. It came with a couple code listing books and some games:

TI BASIC, you can’t be a kid without it!

We had no disk drive. I remember my brother trying to write Q-Bert over a couple weeks, never turning off the computer. My first program was keying lines from one of those books. Not sure what it was; probably just printing text or repeating my name in a loop. GOTO!

After that I probably had a stint with The Turtle. I also recall our school having a bunch of PETs when I was in first or second grade.

Sometimes it feels like every developer has a story about how he or she has been programming since the age of five. I enjoyed gaming and messing around with computers, but I never really got into programming until college.

While I don’t believe childhood hacking is a prerequisite for being a great developer, I am interested in showing my daughters a thing or two about coding. I’m proud to say, when I showed my seven-year-old daughter Logo her eyes got wide with excitement. Now she has me scouring the internet for tickets to LogoConf!

How It All Started

Hello there. Welcome to Tech Time: a blog about nerdy stuff, written by the Harvest Engineering Crew.

I’d like to start this off by giving you a quick, cursive picture of how we got here. Bear with me:

Danny and I (we’re co-founders of Harvest) met at Cornell University in 1997, in a class called CS314 (the content of which I can barely recall… something to do with CPUs). We paired up as partners and ended up spending more time designing the final presentation than we did with the actual project. Our professor was visibly confused flipping through the book we delivered. We passed the class. He asked if we could help design the site for his new venture (Ensim). Danny and I became close friends.

It’s been nearly 15 years since Danny and I met. Today, Harvest is 6 years old and there are 22 of us, 9 on the tech team. Here’s a run down on the tech side of things:

• Danny and I graduated. Met years later to form Iridesco, a design studio.
• RoR happened. We found Dee Zsombor, a super hacker, who was living in Romania at the time. He’s now in Budapest, Hungary. Still super, still hacking.
• We built SuprGlu. It got popular. We needed servers, had no money, shut it down.
• The three of us built Harvest in 2006. Danny and I designed and built the UI, Dee built the entire backend system.
• Barry Hess joined us in 2007. He started working on Harvest in September, from his home office in Owatonna, MN. Together, the four of us – from NYC, Romania, and Minnesota – built and maintained Harvest for the first two and a half years.
• In 2009, Doug Fales found us from Livingston, MT. Aside from being a scary good developer, he’s a scary burly hunter.
• Late 2009, we were having trouble scaling and servers were going wonkers every week. Warwick Poole came to the rescue. He whips our system in shape in between his world-wide golf tours.
• Patrick Filler joined us in 2010 on account of his excellent Hobby presentation on pizza. We had no idea at the time that he’d become the Chosen One.
• T.J. Schuck found us in 2011. He’s the best finger dancer of all devs, and he’s the only Rails Developer working out of our NYC office. He works next to me.
• Joschka Kintscher emailed us out of the blue, from Lemgo, Germany, to see if he can intern at Harvest. We had no internship program, but we brought him in anyway.
• Evan Walsh just joined us this week. I don’t really know him yet.
• Ryan will join us next week. I’m going to just include his name here so he won’t feel left out.

There you are. You will hear from these fine people in the coming days. We’re a relatively small tech team, but together we work on a complex and popular web application. Harvest is built on mostly open source software, serving thousands of businesses around the world and around the clock. Our tech team is constantly solving interesting and difficult technical problems, and we’re always experimenting with new technology and building our own tools. We are a curious bunch and always learning, and we have a lot to share on Tech Time.

Thank you for dropping by, and thank you for your time.