I think it's the word "fought" in this context. To me it sounds like there were those that did not want him to be a man. There's no understanding that the singer was fighting himself, there, though you convey that idea well elsewhere...


Fate doesn't hang on a wrong or right choice
Fortune depends on the tone of your voice

-The Divine Comedy (Neil Hannon)
from the song "Songs of Love"
from the album "Casanova" (1996)