In the early stages of development, male and female embryos are almost indistinguishable. Over the first few weeks of pregnancy, the gonads (what will eventually become the ovaries or testes) start to develop and differentiate (i.e., become different).
After about the seventh week, the male gonads begin to produce male sex hormone (a relation of testosterone), which causes the genital tissues in males to become masculinized. Female development continues without the presence of male sex hormone.
Because the female and male genitals come from the same original embryonic tissue, and differentiation is simply the result of the presence of male sex hormone, much of the male and female genital anatomy can be traced back to a shared origin. The parts of the anatomy that come from the shared original embryonic tissue are called homologous structures. For example, the glans of the penis is homologous to the glans of the clitoris (they come from the same original embryonic tissue).
This video does a great job of explaining the concept: