Unusual causes of big problems
 

I'm interested in hearing some stories that people have where they experienced some big crippling problems with websites, servers or networks and the cause of which were obscure things that nobody would think to look for.

Here's an example I have: A large e-commerce website we were running was slow as hell. Traffic was normal, we double and triple checked the queries that the scripts ran, and we set up some nice caching on the server, but the site still crawled.

It turned out to be a rogue search spider that was crawling the site. It totally ignored the robots.txt and it relentlessly loaded fresh pages searching for content to index. We had to ban the spider's IP and once we did, the site ran perfectly... better than before in fact.

I've got a few fun stories like this:

Where I work we've got a bunch of remote sites that we support. One of the sites repeatedly complained about slow network performance but was unable to actually demonstrate that the problem happened. After about three months of repeated network sniffs and lots of angry phone calls back and forth, we tracked down the problem - a fiber cable running from one side of a server room to the other was slowly failing and would cause intermittent network outages. Once we replaced the cable, everything was fine.

For another, we had persistent issues from one of our data centers - the admins working there complained that every time they accessed the SSL servers back at the downtown facility, the connections dropped to modem-era speeds. We were able to demonstrate the problem very rapidly but unable to figure out exactly what the issue was. After a lot of consultation with the company we use for our encryptors as well as another company we use for firewalls, the decision was made to increase the tcp send and receive buffers on the firewalls and encryption devices. There's no legitimate reason in my mind for why we should have had to increase them past the default. Once we did? Problem solved.

Once at the school I worked at, a kid managed to miss the CD drive tray and somehow stick the CD between the top of the drive and the edge of the case. Didn't cause any problems other than it was a pain to open the thing up and get out, but we made fun of him for being such a goober.

one of my coworkers set up a second dhcp server on our network.... that screwed things up for a bit.

Long story ... but big, unusual problems:

About 3 years ago we moved our IT department and server room into a brand-new state of the art building on campus. This building cost around 43 million dollars and has all the bells and whistles.

The server room is nothing different than most server rooms: separate A/C with backup; 15Kva power conditioner and UPS; natural gas generator with diesel backup. Nothing can go wrong ... right?

Well, during the first month we lost power to the room because the utility company was doing some work and severed one of the main feeds to the campus. No biggie right? The UPS and generator will keep us up and running. Nope. It all failed. Nothing like trial by fire. Room goes down hard. We go through and unplug everything because we don't know what's going to happen when power is restored ... good thing too.

We get the techs out here and they "fix" everything so nothing like this will ever happen again.

Power goes out again a few weeks later (rainstorm). Generator kicks in; but the vibration from the generator itself trips the earthquake valve which cuts off natural gas. No big deal - diesel should kick in any minute which sends an alarm to physical plant to call us and give us an hour to get back to campus and shutdown right? Nope. UPS batteries are depleted in 45 minutes. The UPS is supposed to send a graceful shutdown signal to the servers when it reaches 20% ... didn't happen. The room goes down hard.

Power is restored. The room comes up just as hard. We lose three servers because of the subsequent surges. Why is the power conditioner failing; shouldn't it be inline and protect us from surges? You'd think so. But it seems that the power conditioner fails "closed" when batteries are depleted. So when power is restored the servers are NOT protected but the power conditioner itself IS protected from surges.

The technicians for the power conditioner tell us that the device will protect itself from surges after batteries are depleted because it would "cost us a lot of money to replace it."

Hmmm, let's think about the logic here. We have paid around $30,000 for a power conditioner to protect $200,000 worth of equipment; but they have designed the conditioner to protect itself rather than the much more costly equipment downstream?

To make a long story short it took us two more power outages to get everything working the way it should. And I still don't trust any of it.

dating myself... but don't ever put IRMA board machines onto the ethernet network. no tools and having to check each and every machine and double check each and every node on the network.

and the generators... make sure that the pump is pumping the gas INTO the machinery not back out into the main tank. You'll not have a long backup time.

One day our network (both internally and out the DSL line) in our office is slow. Like dead slow. The technical term we used was "mutt dog slow".

I crack open Etherape and see MASSIVE traffic coming off the dark desktop of an employee who had called in sick and was working from home. Something real serious is happening on that machine, and it's saturating all the bandwidth in the router... and nobody's sitting there. Spooky! Turning on the monitor is no help--it shows a normal suse login screen. As a matter of policy, only the developer has root on his own machine and is completely responsible for it, so I can't even log in to see what's running.

I get that developer on AIM from his house. I ask him if something strange is running on his box. He hedges a bit, then confesses that he's got a remote X session running through an SSH tunnel he'd set up the day before (turns out this was a "planned" sick day). Not VNC--a raw networked X session. Running KDE, no less. He's happily working in his work environment from his home machine, sucking up every available bit of office bandwidth. How very convenient for him!

Guess what port got blocked on the router that day?

Similarly to vanblah, though not so critical.

Moved to a massssive building after a merger and filled a room with 50-60 machines, rackmounts and towers. It's got diesel generator and UPS on all the racks, and it's 'guaranteed' by our IT department to not fail. (I work in R&D)

Power grid for the city goes down, generators kick on. All our desktop machines have power, but the server room is completely dark. No lights, no machines up, no UPS' are functioning. But the room is cold as fuck, because the generator has kept the air conditioning running. :)

Machines were down, but at least they didn't overheat?

Needless to say that "guarantee" was challenged.

I once had to put a sniffer on a network to find out why they companies internet was so slow. Basically when they setup their anti-virus up on all the PC, they would each try to update every hour. Simple thing to take care of but I never expected it.