|
Yah, the last place I was working we had a mishmash of tools, and so there were a number of occasions where we had to run SQL commands directly on the database. Eventually we got into a good system where we would write scripts that would generate SQL to do the things we wanted. We then saved that SQL, and ran it after peer review. But, every once in a while, you were doing something simple, and BAM! You've just screwed something up.
We were a transaction processing center, so there was a constant flow of transactions into the system. On one particularly sad moment, I tried to run a simple table upgrade with the system still live, thinking that the ALTER TABLE would only take a second. Little did I know, instead of doing the normal update thing, it decided to corrupt the tables. The good news was that I made a backup before I started the alter table, the bad news is that it was 8 minutes later when I discovered that it decided to empty out the tables and corrupt them. Oh, man, was that ever fun restoring from backups, and running through the MySQL bin-log by hand trying to pick out which things needed to be re-entered and which things somehow made it back in after the restore. (JBoss was doing some funky things with the db writes)
And that is my story of my moment of shame.
|