Don't want bloated? Go for
DSL!
But honestly though, Gentoo and Debian are probably your best bets. Both have an amazing packaging system, emerge and apt-get respectively. Both come with minimal packages installed on the server and let you then add only what you need. I'd recommend Gentoo over Debian, but that is just based on my personal experiences with Gentoo. If you don't think you are ready for the work involved in a Gentoo install, start with a Stage 3 install for your system architecture, then later on down the road, wipe it and start with a Stage 2, then after you have more experience with it, take that big step and go with a Stage 1 install for Gentoo and bootstrap your own system. Its fun once you've done it a few times and it doesn't overwhelm you any longer.