Maybe you just accidentally met more of the people that not liking him!! Well, I love him, he's a great person & actor.
Maybe because he made some movies they didn't like? The one of his that I didn't like was the one about the aliens attacking earth - there was no real ending, it just left you hanging.
In general he really doesn't have the "leading man looks". I think he would do if he kept to taking parts as a wizard.
He does well if the movie fits. There are a few others that have to stay in their lane as it were, He can't play James Bond but if he chooses well he's fine.
Maybe they never met him? My daughter met him and said he was a nice guy. I think it probably has to do with some of the goofy movies he did. He owed money in back taxes a few years ago and took jobs he probably would have passed over otherwise.
I think he's an overall terrible actor. He's made some very good movies unwatchable.