Annoying

He once said "Do you always have to be annoying?"
I didn't answer
I just kept walking and acted like I didn't hear
I haven't talked to him since
I'm too scared
I don't want to "annoy" him
Every time I see him in the hall I hide
I don't want him to think I'm mad
But I want him to know I heard
I don't know what I did
But it must have been bad
Or he wouldn't have said that
Yet it could have been me talking to him
Or maybe he just hates me
Am I really "annoying"?