My first database migration crashed our site for 3 hours last week

I was working on this simple update to switch our user data from one SQL table to another. I thought I had it all planned out but I forgot to handle the foreign key constraints. Everything locked up and nobody could log in for the whole afternoon. My boss was not happy and I had to roll back the whole thing. Has anyone else messed up a migration and lost a whole day fixing it?

3 comments

3 Comments

jakel363d ago

Did you test the migration on a copy of the database before hitting production?

martin.felix3d ago

...and honestly that's just how life works man. I've seen it in my own work and everywhere else too. People skip the boring prep step because they're so sure they've got it figured out. Then the whole thing blows up because you missed one dumb little detail that would've taken ten minutes to catch. It's like when you think you can just wing a home repair without measuring twice and then you're staring at a crooked pipe fitting wondering where it all went wrong. The smarter you feel going in the harder you crash when the reality check hits.

riveradams3d ago

Three hours really isn't that long for a production outage (I've seen people take down entire companies for a weekend). People act like a three hour downtime is the end of the world but really it's just a learning moment. Your boss will forget about it in a week, trust me. The real problem is the panic and rushing which makes people do dumb things like not testing on a staging database first. Jakel36 is right that testing on a copy should be step one every time. So yeah it sucks but three hours isn't worth beating yourself up over forever.