Why do people find arrogance disturbing if it doesn't affect them?

I'm an arrogant little , but I don't go around spouting bullshit about myself. I was asked once if I was arrogant, and I said yes. Then i received a lecture about how arrogance is a terrible quality, even though I didn't say anything vain, and kept it to myself.