I've had two women seriously tell me they didn't mind the goatee. We'd joke about its virtue of tickling the thighs.
My fiancee in particular said it "freaked her out" when I would kiss her after first shaving it off... because she wasn't used to feeling the clean-shaven face.
For the record, after over a year, I've just this week started to grow it back.