Potentially this is because of your unfavorable associations with which the polo shirt is laden. It is commonly considered regarding a uniform — either pretty much, as in the needed get-up of a pizza shipping guy or elementary faculty university student, or metaphorically, since the go-to garb for corporate drones and preppy fraternity brothers.