I have a 2002 S80 T6, this was the same issue I was experiencing. You need to inspect your front suspension components, it could be one of this components (end links, ball joints, and or your strut upper mount spring seat) or maybe all. I have 120K and mine were all worn out and they all had to be replaced. Physical signs are broken seal bushing, expose ball joints, upper mount seat bearing separated from the pad.
You might be able to save parts cost buying them from the Internet and have a mechanic that does work on Volvo's install them to save labor cost. Volvo dealers wanted more than a mortgage payment to replace them all. I was able to save hundreds doing it this way.
IPDUSA is one of the source I use to buy replacement parts and they are very reasonable but don't take my word, check around and Google Volvo Parts.