The problem, as has been pointed out by some people here, is that by *not* voting, you help the candidates you probably dislike most.
The reason US politicians tend to go for "family values" and "christian this-and-that" is simple: religious leaders can organize their people to vote. Politicians care about staying in power, and people that vote keep them there. If a lot of these votes come from extremists, the politician will try to woo these extremists. If the moderates don't vote, nothing will change here.
Your reasoning seems to be based upon the notion that politicians will see that a lot of people aren't voting, and will then suddenly change their ways to woo these people. That notion is wrong, period. You seem to underestimate the power of apathy. People that didn't vote in the last election are probably not going to vote in the next, so there's no reason to listen to their needs. The best way to stay in power is to pander to the needs of those people that *do* vote, and not to upset the rest too much.
If you want to make your country a better place, and hate the politicians that are currently in power, the worst thing you could possibly do is: don't vote.
but go ahead, make no difference.