At each of the four startups I’ve worked for, the database has been the primary point of scale. This carried through to the early days of Zendesk, where countless hours were spent pulling the databases back from the edge of scale by manually killing queries. A common firefight in those days was:
- Hapless developer writes hopeless query
- Code is shipped.
- World is on fire. Find query. Kill it. Roll back the deploy.
This can be an endless cycle, and the solution the industry seems to be settled on is production monitoring. There’s a ton of great products in that area (Datadog, New Relic, Vivid Cortex), but there’s a couple of problems with the monitoring approach.
Prevention > Monitoring
By the time your monitoring has alerted, there’s a high likelihood you’ve pissed off a customer. Second (and this is the big bet we’re making), the person most likely sitting in front of that performance graph is not the person who can fix it. Shiba is an attempt to connect the problem (bad query) with the person who can actually fix it (the developer), at a time that it’s most likely to be fixed (just after it’s been written).
Shiba uses table-count and index statistics from your production database to analyze queries in your test suite, allowing Shiba to generate a time estimate for each. It then comments on your pull requests to warn you that you’re about to do a bad thing.
Have I sold you? No?
Look, preventative medicine is a nasty sell. But it’s why you write your tests. It’s why you do code review. Let’s talk for a second about a hypothetical outage at a (ahem) hypothetical company named “FitNab”. I suppose it’s a health-tracking website for thieves. The age old story of this outage starts with a commit:
Looks decent, right? We show people a list of thieves. And, being diligent developers, we’ve ensured
people.occupation is indexed, so no problems, right? Well… sort of.
The issue here is we’ve failed to add a
.limit to the
Person.where query, and as the database grows, so does the resource usage of this controller action. This request will appear fast, all the way to production, where it will take down the site in a blaze of glory. Let’s see what Shiba says about this PR:
Armed with this information, we slap a
.limit(20) into our controller and get on with our damned lives.
See for yourself
Shiba is an extremely young project, but every code base we’ve tried it on has had some reasonably easy to detect, dangerous performance problems no one had found before. We’d love for you to give it a whirl on your project and let us know what you find! The core of Shiba is and will always will be open source.
Our early release is currently focused on Ruby applications, but support for other languages will be added.