If they're slip on style spacers, i would scrap them. The bolt on styles are a different story, depending on who you talk to.
Is it one particular part of your caliper hitting the wheel? if so, could you maybe grind it down a bit so it gives better clearance?
Or maybe you have a wheel bearing showing you some excessive wear? When it's jacked up can you grab the top and bottom of the tire and flex it any back and forth on the wheel bearing?
Sounds like the wheel is definitely too close. But there are a few fixes. It's just a matter of knowing that everything is as it should be before you start curing symptoms w/o addressing the true problem.