I just had this same problem on my 2002 RX300 at 89000 miles. It was resolved by having the throttle body cleaned. I got my mechanic to do it while he was doing a transmission service for $70. You can do it yourself if you are very handy and clublexus.com has some how to info on how to do it.
The accelerator cable may be failing, make sure the floor mat is not fouling the cable and finally perhaps someone has ill advisedly tried to adjust the idle speed by adjusting the throttle stop screw. It is not to be adjusted, should have a dab of yellow paint on the screw from the factory to show it has not been tampered with.