[C] Debugging and tracking down problems - Tilted Forum Project Discussion Community

Fallon · 01-17-2005, 08:14 AM

Hey everyone
Just as a quick background, I run an online text-based game written in C, right now we're getting some hefty lag spikes which is either 1 or 2 things which would be either the host is getting high volumes of traffic which is a possability or I have some bad functions. What programs do you guys run to track these types of problems down? I've used gdb but that's useful in crashes, or at least how I know it.
Thanks

noodles · 01-17-2005, 10:46 AM

well... first, can you just ask the host if they've been having high volumes of traffic? that kind of strikes that one out quickly, or if its is the case, there's not much that you can do about it.

second, you might want to check to see the resources the game takes up over time. just check up on it over the period of a day or so, from a fresh execution. figure out if there's a memory leak or something. (i don't know if there's any program to help with this, i'm not an expert at developing in c)

third, and this is just an idea i had, you may want to create a timekeeping library, one which is short and quick to use for debugging in this case. at the start of your functions, initiate the timekeeper, at the end of your functions, call the timekeeper and print out the time it took to run the function. this might be an easy way to find functions that are many orders of magnitude too slow, or are taking progressively longer and shouldn't be. (just an idea, again, maybe there's a program that will do this for ya)

RAGEAngel9 · 01-17-2005, 09:12 PM

I know there are a couple of linux based memory checking programs, but I can't remember names right now.
Might be a good place to start.

theFez · 01-17-2005, 09:18 PM

http://www.nagios.org/
is great for server monitoring. if you have the access to install it on the server, it can provide you with TONS of information about server stats.

Rawb · 01-18-2005, 04:29 PM

There are two things I go to when I start with something like that:

1. strace (or truss, or something else, depending on which os you are running on), it will give you a real-time list of every system call that process is making. It is pretty intensive, so it takes a bit, but it will let you know if you are doing a flood of reads or writes or malloc's that are killing your system. I've used it to debug a few things.

2. vmstat (or something else, depending on what system you are running on), this will let you know what part of your system appears to be the bottleneck, it will show you if it is swapping, or if you are running a ton of I/O and your disk can't keep up.

If it's not one of those, or vmstat is showing that your CPU is spiking, odds are that you have some bad code in there. To figure out what kind of code is killing you, you need a profiler, gprof is one I have tried, but it has been ages, so I can't remeber how to use it, I can just tell you to give it a shot. If you are really looking to run some optimizations (hopefully not needed for a text-based game), you can also run valgrind to see where your cache choke points are.

Just remember, try and optimize the largest thing and only the largest thing, one at a time and just keep on moving down the list until you get it good enough.

Kalibah · 01-18-2005, 10:11 PM

MUD?

Care to share link

trache · 01-20-2005, 11:33 AM

Use GNU gprof. It will tell you how long certain portions of your code take to run. While functions themselves don't affect network lag much, this may help a tiny bit. At least it will tell you where you need to fix your code up, if there are certain portions that take time.

Pragma · 01-20-2005, 06:21 PM

I'd recommend using Ethereal to find out if it's a network issue [i.e.: trace all network traffic and see if everything is sending like it should be]. If you don't find any problems there, then I'd recommend using gdb with breakpoints set up around any code you think may have issues and trace through very carefully.

roboshark · 01-21-2005, 12:53 PM

You can profile your code to see if you have bad functions that eat up processor cycles. "man gprof" is what I'm talking about (if you're using Windows, my condolences....)

ergdork · 02-01-2005, 03:05 PM

I've used gprof, but never have had a lot of success with it. In all honestly, its probably my failure to learn to REALLY use it well. But it also requires compilation w/ -pg. What I recommend, if you can get yourself into the situation to use it, is Shark on a Mac. I know thats something of a special case, but it is an INCREDIBLE sampler. It makes it VERY easy to just run some code, look at the output, tweak it and run again. It makes iterative optimizing very easy. It also has a very nice GUI for browsing the time spent in various functions.

Its not installed by default, but part of the free CHUD downloads. I recommend it to anyone who can get their hands on it.

If you can't get that, then another linux suggestion (outside of gprof) is oprofile. Its newer than gprof, requires a kernel module, but its pretty nice. It can sample random running processes.

As far as memory tracking, theres always valgrind (which has a performance tool called cachegrind).

01-17-2005, 08:14 AM	#1 (permalink)
Fallon Junkie Location: RI	[C] Debugging and tracking down problems Hey everyone Just as a quick background, I run an online text-based game written in C, right now we're getting some hefty lag spikes which is either 1 or 2 things which would be either the host is getting high volumes of traffic which is a possability or I have some bad functions. What programs do you guys run to track these types of problems down? I've used gdb but that's useful in crashes, or at least how I know it. Thanks

01-17-2005, 09:18 PM	#4 (permalink)
theFez Crazy Location: here and there	http://www.nagios.org/ is great for server monitoring. if you have the access to install it on the server, it can provide you with TONS of information about server stats. __________________ # chmod 111 /bin/Laden

01-20-2005, 11:33 AM	#7 (permalink)
trache Insane	Use GNU gprof. It will tell you how long certain portions of your code take to run. While functions themselves don't affect network lag much, this may help a tiny bit. At least it will tell you where you need to fix your code up, if there are certain portions that take time. __________________ "You looked at me as if I was eating runny eggs in slow motion." - Gord Downie of The Tragically Hip

01-21-2005, 12:53 PM	#9 (permalink)
roboshark Crazy	You can profile your code to see if you have bad functions that eat up processor cycles. "man gprof" is what I'm talking about (if you're using Windows, my condolences....) __________________ I want no escape.

01-17-2005, 10:46 AM	#2 (permalink)
noodles Psycho Location: sc	well... first, can you just ask the host if they've been having high volumes of traffic? that kind of strikes that one out quickly, or if its is the case, there's not much that you can do about it. second, you might want to check to see the resources the game takes up over time. just check up on it over the period of a day or so, from a fresh execution. figure out if there's a memory leak or something. (i don't know if there's any program to help with this, i'm not an expert at developing in c) third, and this is just an idea i had, you may want to create a timekeeping library, one which is short and quick to use for debugging in this case. at the start of your functions, initiate the timekeeper, at the end of your functions, call the timekeeper and print out the time it took to run the function. this might be an easy way to find functions that are many orders of magnitude too slow, or are taking progressively longer and shouldn't be. (just an idea, again, maybe there's a program that will do this for ya)

01-17-2005, 09:12 PM	#3 (permalink)
RAGEAngel9 Insane Location: Vermont	I know there are a couple of linux based memory checking programs, but I can't remember names right now. Might be a good place to start.

01-18-2005, 04:29 PM	#5 (permalink)
Rawb Crazy Location: Salt Town, UT	There are two things I go to when I start with something like that: 1. strace (or truss, or something else, depending on which os you are running on), it will give you a real-time list of every system call that process is making. It is pretty intensive, so it takes a bit, but it will let you know if you are doing a flood of reads or writes or malloc's that are killing your system. I've used it to debug a few things. 2. vmstat (or something else, depending on what system you are running on), this will let you know what part of your system appears to be the bottleneck, it will show you if it is swapping, or if you are running a ton of I/O and your disk can't keep up. If it's not one of those, or vmstat is showing that your CPU is spiking, odds are that you have some bad code in there. To figure out what kind of code is killing you, you need a profiler, gprof is one I have tried, but it has been ages, so I can't remeber how to use it, I can just tell you to give it a shot. If you are really looking to run some optimizations (hopefully not needed for a text-based game), you can also run valgrind to see where your cache choke points are. Just remember, try and optimize the largest thing and only the largest thing, one at a time and just keep on moving down the list until you get it good enough.

01-18-2005, 10:11 PM	#6 (permalink)
Kalibah Insane Location: Padded Playhouse	MUD? Care to share link

01-20-2005, 06:21 PM	#8 (permalink)
Pragma I am Winter Born Location: Alexandria, VA	I'd recommend using Ethereal to find out if it's a network issue [i.e.: trace all network traffic and see if everything is sending like it should be]. If you don't find any problems there, then I'd recommend using gdb with breakpoints set up around any code you think may have issues and trace through very carefully.

02-01-2005, 03:05 PM	#10 (permalink)
ergdork Tilted	I've used gprof, but never have had a lot of success with it. In all honestly, its probably my failure to learn to REALLY use it well. But it also requires compilation w/ -pg. What I recommend, if you can get yourself into the situation to use it, is Shark on a Mac. I know thats something of a special case, but it is an INCREDIBLE sampler. It makes it VERY easy to just run some code, look at the output, tweak it and run again. It makes iterative optimizing very easy. It also has a very nice GUI for browsing the time spent in various functions. Its not installed by default, but part of the free CHUD downloads. I recommend it to anyone who can get their hands on it. If you can't get that, then another linux suggestion (outside of gprof) is oprofile. Its newer than gprof, requires a kernel module, but its pretty nice. It can sample random running processes. As far as memory tracking, theres always valgrind (which has a performance tool called cachegrind).