when I worked at walmart some people would get so offended if I suggested wearing the opposite genders clothes even if the item was basically the same for males and females.
a polo shirt for men is cheaper and you're probably just wearing it because of your workplace why does it matter?