A fetus' sexual organs start in what we can call a sexually-indifferent stage (up to week 5 or 6). This formation has the ability to develop either into the male or female reproductive organs. Various factors then kick in and then the organs begin to develop - thus they will have anatomical similarity due to the same starting origins.
For a visual representation, check this link:
Sexual Organ Development - internal
Sexual organ development - external