toilet_duck is correct, different players have different capabilities. One thing to look for when you are researching this is any mention of "VCD". VCD is the most common format (outside of actual DVDs, of course) that many players can display properly. And a VCD is merely a mpeg or mpg file (of the mpeg-1 variety, the most common type found on the 'net) of a particular size and other properties, burned onto a CD in a particular way.
If you find that your player supports the VCD format, go
here (same site toilet_duck linked to, just the front page) for many tutorials and links to free software that can convert your videos into the correct format and burn them onto CDs.
Of course if you find that your DVD player supports formats like wmv or divx avi or anything like that, conversion of those will not be necessary and you may be able to simply burn those directly to a CD.