a bad deploy
I was frustrated Thursday evening, and Friday morning.
On Thursday, I got home from work and saw a message saying a service I owned was spitting out 500's and completely fucked. Great.
I knew exactly why. I'd made a few assumptions, which led to a few mistakes, which led to the issue at hand. I rectified it immediately, and back to normal.
I was frustrated for a few reasons:
- It was my fault
- I'd been given shit by my boss
- How I caused it shouldn't have led to an issue
So, what happened?
On Thursday, I'd finished up some small feature, which needed two things to go live:
- A database table migration (adding a new column with a default value)
- A deployment of the service using that database table
I was on a 4 day work week and it was ~4pm, so figured I was better off leaving the deploy until Monday.
I also decided I would run the migration, adding a column with a default should be fine and it would be quick and ensure it was all ready come Monday. All the lower environments looked good, so I assumed that this was super low risk, because:
- The service would have no issues with the new column being there.
- There had been 0 issues in lower environments when I had migrated + deployed, as it was all working when I checked.
- We had adequate monitoring in place.
All three were bad assumptions. Turns out, whoever built the service (that I owned now but rarely touched), didn’t make it very resilient to change and didn't put any alerts in place.
Fuck me. I completely own it was a lot of bad assumptions on my part, and I messed up by not verifying before leaving, and by not doing the full deploy (I had time realistically).
But also, fuck whoever didn’t do a good job building it.
Ugh!! Lessons learned, frustratingly. I hate when I mess up.