For Firefox (the one that doesn't have the mail client integrated):
Tools > Options > Downloads > Uncheck "Show Download Manager window when a download begins
If it is the Mozilla Suite, I would imagine it would be pretty similer:
Tools > Options >
From there, you would look for any tabs or headings pertaining to "Downloads" and uncheck anything that says "Show download manager" or "Have Download Manager Popup on click".
By Disabling, I am not sure if it is possible to get rid of it completely. But I would search the mozilla forums just in case.
http://forums.mozillazine.org/
EDIT: ehhh... I didn't see anything useful over there. If you are using Mozilla Firefox, then you HAVE to use the Download Manager. You can have it not show, but it will always be in use. (Based on Mozilla Firefox 0.8, the latest version). If you don't want the download manager, you would need to return to 0.7 (an earlier version)