Idler pulley or engine bearing or ?

Think tensioner or idler pulley. It has to be something in the accessory drive area, since the belt removal silenced the noise. Check every pulley from the water pump to the A/C to the alternator. Check the tensioner and idler as well. Something will be noticable and will stand out. These noises to have a tendency to transfer easily so it sounds like it is from one place when it really originates elsewhere. You're on the right track -- Good luck!