Tilted Forum Project Discussion Community  

Go Back   Tilted Forum Project Discussion Community > Interests > Tilted Technology


 
 
LinkBack Thread Tools
Old 07-18-2007, 12:06 PM   #1 (permalink)
Please touch this.
 
Halx's Avatar
 
Owner/Admin
Location: Manhattan
Unusual causes of big problems

I'm interested in hearing some stories that people have where they experienced some big crippling problems with websites, servers or networks and the cause of which were obscure things that nobody would think to look for.

Here's an example I have: A large e-commerce website we were running was slow as hell. Traffic was normal, we double and triple checked the queries that the scripts ran, and we set up some nice caching on the server, but the site still crawled.

It turned out to be a rogue search spider that was crawling the site. It totally ignored the robots.txt and it relentlessly loaded fresh pages searching for content to index. We had to ban the spider's IP and once we did, the site ran perfectly... better than before in fact.
__________________
You have found this post informative.
-The Administrator
[Don't Feed The Animals]
Halx is offline  
Old 07-18-2007, 03:56 PM   #2 (permalink)
I am Winter Born
 
Pragma's Avatar
 
Location: Alexandria, VA
I've got a few fun stories like this:

Where I work we've got a bunch of remote sites that we support. One of the sites repeatedly complained about slow network performance but was unable to actually demonstrate that the problem happened. After about three months of repeated network sniffs and lots of angry phone calls back and forth, we tracked down the problem - a fiber cable running from one side of a server room to the other was slowly failing and would cause intermittent network outages. Once we replaced the cable, everything was fine.

For another, we had persistent issues from one of our data centers - the admins working there complained that every time they accessed the SSL servers back at the downtown facility, the connections dropped to modem-era speeds. We were able to demonstrate the problem very rapidly but unable to figure out exactly what the issue was. After a lot of consultation with the company we use for our encryptors as well as another company we use for firewalls, the decision was made to increase the tcp send and receive buffers on the firewalls and encryption devices. There's no legitimate reason in my mind for why we should have had to increase them past the default. Once we did? Problem solved.
__________________
Eat antimatter, Posleen-boy!
Pragma is offline  
Old 07-18-2007, 06:37 PM   #3 (permalink)
The Computer Kid :D
 
Location: 127.0.0.1
Once at the school I worked at, a kid managed to miss the CD drive tray and somehow stick the CD between the top of the drive and the edge of the case. Didn't cause any problems other than it was a pain to open the thing up and get out, but we made fun of him for being such a goober.
MikeSty is offline  
Old 07-19-2007, 05:22 AM   #4 (permalink)
Devils Cabana Boy
 
Dilbert1234567's Avatar
 
Location: Central Coast CA
one of my coworkers set up a second dhcp server on our network.... that screwed things up for a bit.
__________________
Donate Blood!

"Love is not finding the perfect person, but learning to see an imperfect person perfectly." -Sam Keen
Dilbert1234567 is offline  
Old 07-19-2007, 07:07 AM   #5 (permalink)
Junkie
 
Long story ... but big, unusual problems:

About 3 years ago we moved our IT department and server room into a brand-new state of the art building on campus. This building cost around 43 million dollars and has all the bells and whistles.

The server room is nothing different than most server rooms: separate A/C with backup; 15Kva power conditioner and UPS; natural gas generator with diesel backup. Nothing can go wrong ... right?

Well, during the first month we lost power to the room because the utility company was doing some work and severed one of the main feeds to the campus. No biggie right? The UPS and generator will keep us up and running. Nope. It all failed. Nothing like trial by fire. Room goes down hard. We go through and unplug everything because we don't know what's going to happen when power is restored ... good thing too.

We get the techs out here and they "fix" everything so nothing like this will ever happen again.

Power goes out again a few weeks later (rainstorm). Generator kicks in; but the vibration from the generator itself trips the earthquake valve which cuts off natural gas. No big deal - diesel should kick in any minute which sends an alarm to physical plant to call us and give us an hour to get back to campus and shutdown right? Nope. UPS batteries are depleted in 45 minutes. The UPS is supposed to send a graceful shutdown signal to the servers when it reaches 20% ... didn't happen. The room goes down hard.

Power is restored. The room comes up just as hard. We lose three servers because of the subsequent surges. Why is the power conditioner failing; shouldn't it be inline and protect us from surges? You'd think so. But it seems that the power conditioner fails "closed" when batteries are depleted. So when power is restored the servers are NOT protected but the power conditioner itself IS protected from surges.

The technicians for the power conditioner tell us that the device will protect itself from surges after batteries are depleted because it would "cost us a lot of money to replace it."

Hmmm, let's think about the logic here. We have paid around $30,000 for a power conditioner to protect $200,000 worth of equipment; but they have designed the conditioner to protect itself rather than the much more costly equipment downstream?

To make a long story short it took us two more power outages to get everything working the way it should. And I still don't trust any of it.
vanblah is offline  
Old 07-19-2007, 07:18 AM   #6 (permalink)
Tilted Cat Head
 
Cynthetiq's Avatar
 
Administrator
Location: Manhattan, NY
dating myself... but don't ever put IRMA board machines onto the ethernet network. no tools and having to check each and every machine and double check each and every node on the network.

and the generators... make sure that the pump is pumping the gas INTO the machinery not back out into the main tank. You'll not have a long backup time.
__________________
I don't care if you are black, white, purple, green, Chinese, Japanese, Korean, hippie, cop, bum, admin, user, English, Irish, French, Catholic, Protestant, Jewish, Buddhist, Muslim, indian, cowboy, tall, short, fat, skinny, emo, punk, mod, rocker, straight, gay, lesbian, jock, nerd, geek, Democrat, Republican, Libertarian, Independent, driver, pedestrian, or bicyclist, either you're an asshole or you're not.
Cynthetiq is offline  
Old 07-19-2007, 07:28 AM   #7 (permalink)
Darth Papa
 
ratbastid's Avatar
 
Location: Yonder
One day our network (both internally and out the DSL line) in our office is slow. Like dead slow. The technical term we used was "mutt dog slow".

I crack open Etherape and see MASSIVE traffic coming off the dark desktop of an employee who had called in sick and was working from home. Something real serious is happening on that machine, and it's saturating all the bandwidth in the router... and nobody's sitting there. Spooky! Turning on the monitor is no help--it shows a normal suse login screen. As a matter of policy, only the developer has root on his own machine and is completely responsible for it, so I can't even log in to see what's running.

I get that developer on AIM from his house. I ask him if something strange is running on his box. He hedges a bit, then confesses that he's got a remote X session running through an SSH tunnel he'd set up the day before (turns out this was a "planned" sick day). Not VNC--a raw networked X session. Running KDE, no less. He's happily working in his work environment from his home machine, sucking up every available bit of office bandwidth. How very convenient for him!

Guess what port got blocked on the router that day?
ratbastid is offline  
Old 07-19-2007, 08:20 AM   #8 (permalink)
Lover - Protector - Teacher
 
Jinn's Avatar
 
Location: Seattle, WA
Similarly to vanblah, though not so critical.

Moved to a massssive building after a merger and filled a room with 50-60 machines, rackmounts and towers. It's got diesel generator and UPS on all the racks, and it's 'guaranteed' by our IT department to not fail. (I work in R&D)

Power grid for the city goes down, generators kick on. All our desktop machines have power, but the server room is completely dark. No lights, no machines up, no UPS' are functioning. But the room is cold as fuck, because the generator has kept the air conditioning running.

Machines were down, but at least they didn't overheat?

Needless to say that "guarantee" was challenged.
__________________
"I'm typing on a computer of science, which is being sent by science wires to a little science server where you can access it. I'm not typing on a computer of philosophy or religion or whatever other thing you think can be used to understand the universe because they're a poor substitute in the role of understanding the universe which exists independent from ourselves." - Willravel
Jinn is offline  
Old 07-19-2007, 08:36 AM   #9 (permalink)
People in masks cannot be trusted
 
Xazy's Avatar
 
Location: NYC
I once had to put a sniffer on a network to find out why they companies internet was so slow. Basically when they setup their anti-virus up on all the PC, they would each try to update every hour. Simple thing to take care of but I never expected it.
Xazy is offline  
 

Tags
big, problems, unusual

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -8. The time now is 12:32 AM.

Tilted Forum Project

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Search Engine Optimization by vBSEO 3.6.0 PL2
© 2002-2012 Tilted Forum Project

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360