Dicks are sort of different than boobs. everyone knows how big your boobs are regardless of how you dress. Unless you walk around in bike shorts no one really knows how big your dick is. Add the whole grower phenomenon to the equation and only intimate partners really know how someone stacks up. So a lot of guys walk around unsure if their dick is big, average or small.
Add to that the idea that things like height, muscles and endowment are really just shorthand for masculinity, virility and generally having worth as a human being and some dudes get really touchy. It's still a stupid concept though. If you have a small dick, just find a small vagina and everyone is happy