How do I get polymeric sand off my pavers?

I laid down my pavers beautifully, gave them some time to settle, and then swept the polymeric sand into the joints. I was very careful to get as much of the sand off the actual pavers as possible, but inevitably there was still a light dusty residue that couldn't be swept off. I misted the pavers as advised (4 times in 20 min increments), and when everything dried up, my pavers looked terrible! There is an obvious residue left on the pavers. We tried scrubbing with a wire brush, but that changed the texture of the pavers. I am hearing that vinegar and a regluar (less abrasive) brush might work? How about a power washer? Has anyone else gone through this before...if so, please help!