I've got a few fun stories like this:
Where I work we've got a bunch of remote sites that we support. One of the sites repeatedly complained about slow network performance but was unable to actually demonstrate that the problem happened. After about three months of repeated network sniffs and lots of angry phone calls back and forth, we tracked down the problem - a fiber cable running from one side of a server room to the other was slowly failing and would cause intermittent network outages. Once we replaced the cable, everything was fine.
For another, we had persistent issues from one of our data centers - the admins working there complained that every time they accessed the SSL servers back at the downtown facility, the connections dropped to modem-era speeds. We were able to demonstrate the problem very rapidly but unable to figure out exactly what the issue was. After a lot of consultation with the company we use for our encryptors as well as another company we use for firewalls, the decision was made to increase the tcp send and receive buffers on the firewalls and encryption devices. There's no legitimate reason in my mind for why we should have had to increase them past the default. Once we did? Problem solved.
__________________
Eat antimatter, Posleen-boy!
|