Okay, high volume sound has this risk to hurt the ears yes, that's why so many drummers use headphones sometimes to just cover their ears from too much direct voice of drum parts to their ear while playing. But also it depends on the individual and each person's ear system. But it's not so often. As far as i know! For figuring that out you can simply check with a doctor. Not bad to make sure of things anyway. Just for knowing the fact by support of professionals and check ups.
Also, when you listen to high volume sound, you would say "wow it's loud" if you don't be bothered you say nice and will continue to listening, after a while, your ears get use to that rate of voice. So you don't feel it as loud as it used to be, so it's not satisfying you as much as it used to, as the result, you would increase the sound to feel loud again. That's not an ear problem at all. It's just ear's compromise to the volume. And when you lower the voice and keep listening to the same lowered rate constantly, you will get use the lowered volume easily.
Also i just would point out your statement about "btw i don't listen at a really high volume" maybe you do, you just don't know it. I've been pointed out usually as a high volume listener, I admit that, but sometimes some volumes are too normal for me and suddenly that's too loud for someone else.
Due that reason, one of the hardest things in the world for me is to shop headphones, i need to try a thousand of companies to find my satisfaction. I ask for it's promise if it's really good enough with high volume capacity?
And they say yesss! And after i use it, i take a disappointed "sighhh" and ask myself, is that really loud to people ? What i want is to feel my ears blowing up when listening to music. For now Energy Sistem headphones are doing a bit of good for me.