Deploy Safely
We’re not very fond of taking unnecessary risks at Harvest — and the easiest way to avoid big risks is to make small, incremental changes. Like a cook tasting a dish, it allows us to make small adjustments as needed to make Harvest the best it can be. Mmmm, salty passwords.
You’re hopefully unaware, but we update Harvest a lot (for example, 13 times yesterday) — and most of the time it’s easy and natural to make small, simple changes. But how do you make small, incremental changes with upgrades or new features?
Minimize Changes
“Is this the minimum set of changes that we need to go live?”
Sometimes when you find yourself in a section of code that hasn’t been touched in a while it’s tempting to make a whole bunch of changes by accident. “Oh, this is using the old hash syntax, I’ll just make a quick update to take out the hash rockets” can quickly become “Wait, why is this even a class anyway?” — and before you know it, you’ve made a set of changes which has nothing to do with what you set out to do.
“Is this the minimum set of changes that we need to go live?”
Thanks to our amazing developer operations team, there’s virtually no cost to make updates to Harvest — so why lump your changes into something that’s impossible to predict how it will react when released?
We try to make our pull requests as boring as possible. They should be simple, plain, and easy to digest. No need to include those hash update changes — you can just as easily follow up afterwards with another pull request.
“Is this the minimum set of changes that we need to go live?”
It Was The Best Of Times, It Was The Worst Of Times
No matter the change, always have a plan for after the pull request is released — a plan for success and a plan for failure.
“We can just cap production deploy:rollback, right?” Well, not always. Some updates can have rippling consequences — sessions expiring and logging everyone out, secure cookies becoming incompatible and unreadable, serialized data suddenly becoming unserializable — and not all of these consequences are immediately apparent.
Deploys should be boring. Rehearsing a revert plan can mean the difference between high-stress downtime and “Whoops, let’s try again tomorrow.” Rehearse what it means to fail, at what point you’ll decide to rollback a release, and what needs to be done (and who will be doing it). We all make mistakes — and we should expect them to happen often, and be prepared for them.
Just as important as a rollback plan is a checkup plan — the plan you follow when an update is successful. You can consider it the definition of success — all the places you’ll need to check once the update is made to make sure things are working the way you expect.
Releasing Large Features
Large features (like our new Projects section) can easily balloon into huge, risky releases — but we’ve adopted using feature flags to break down big features into small, incremental changes.
If you’ve been using Harvest since November of 2013, you’ve been using a copy of Harvest which supports the new Projects section — it’s just been hidden from you (or not, if your account has been in our early access group (thank you!)). Each change we’ve made has been released into our production code, one small change at a time.
Feature flagged releases can be some of the safest releases possible — since we can release a feature to a small subset of our customers and make sure it’s behaving the way we expect (by following the checkup plan).
Real Talk
At Harvest, we like releases to be boring, simple and straightforward, prepared and predictable. Small changes don’t keep you late at work or make you work on the weekends or on holidays. We’ve found that keeping our releases small means constantly asking ourselves and each other “Is this the minimum set of changes that we need to go live?” It means preparing and planning for failure and success. It means using tools like feature flags to break down big, risky releases.
It means we can eat dinner with our families.