Last Friday night, shortly after midnight, I received an urgent text from one of our champion’s team members: “Houston, we have a huge problem!” he said. What was supposed to be routine dedicated server maintenance, the server, which hosts our site and that of hundreds of our clients, revealed a major malfunction with one of our mirrored disk drives. And as if this was not enough, we also discovered some serious issues with the server hardware.
Our dedicated server is securely parked among thousands of other servers in a huge data center at one of our global partners. I called for a quick emergency meeting at 2:30AM Saturday morning with our team and our hosting partner. We realized, that due to the hardware error, the initial plan A to replace the failed drive, failed, and we had to resort to plan B. We quickly decided to rebuild a new server and restore our sites and data from our backup mirrored drive. A few hours later, plan B was progressing, the new server was rebuilt and working, and we were on our way to restoring everything from our mirrored drive.
While operating on just a few hours of sleep and having restored one third of our server, suddenly there came an urgent text from my team notifying me that the second backup drive had also failed! Someone out there must have been testing my patience and nerves.
I took a deep breath, consulted with my team and we decided to dive into yet a third plan; plan C to restore all that was missing from our nightly backup system. We encountered some hiccups and stressful moments along the way but finally, on Monday, everything was pretty much up and running with few if any glitches.
My top lessons learned:
- We are rarely in complete control of everything.
- Surrounding myself with an amazing team and partners has saved me on many occasions and has been the key to our continued growth and success.
- Even plan B is not always good enough. (Always back it up with a plan C)!
Carefully review your personal and business backup systems and procedures and make sure you are prepared with Plan A, B and even C. It will save you!
Get your copy of my latest new books available now on my Amazon’s author page.