Need help with analysis of "Twelth Birthday"

Http://poetryfoundation.org/archive/poem.html?id=29030

I can't seem to get a firm grip on the poem. I know it has to do with growing up but I'm especially confused with the last stanza. The last line says "You knew time was my enemy" but who is it referring to? Time could only be the enemy to youth right?