Mine used to do the same thing, it gradually got worse, but I drove 2000 miles away from home, thru idaho, Montana, S.D, and Minnesota in the middle of winter, and the car was just fine. So as long as you go slow, you shouldn't have anything to worry about. When I got home I got it fixed, and it turned out to be my rack and pinoin. Cost me $600 and change, but for peace of mind, totally worth it. You might want to try that if you've exhausted every option.