In traditional settings, fashion would not have such a significant weight. However, since individuals tend to detach themselves from “traditional” factors anchors, this is where simmel discusses fashion. Simmel believes that class has a major effect on who wears what according to the class hierarchy. Fashion is visible and an easily symbol of class position. The lower classes set out to imitate those in the upper class that makes the upper class seek out an alternative form of fashion in order to maintain and express their distinctiveness.
I do think it is unjust to judge someone solely on the their appearance, however, humans are very visual and even if you try to block it out, fashion is the first impression of the person. Nowadays, it is difficult to distinguish upper class from lower class in many situations unless you look at the label of the clothing. Many expensive designer clothing look like rags and though it is “upper-class clothing” it does not pass as it. Lower class stores may carry cloths that look resemble high fashion cloths at a lower price. So it may be hard to distinguish whose whom and this illusion can blur the class hierarchy.