Many buildings have both male and female qualities. Perhaps the texture is male, and the shape female. Or, the color is female, but the proportions... Well, you get the point.
A building that has both male and female characteristics might be called androgynous.
The Taj Mahal in Agra, India combines rounded shapes with masculine forms.
