I always hate to replace parts without testing a component and ensuring it is defective. The starter motor is the heaviest electrical draw the battery has to encounter. The lights and windows working show the battery has voltage but the buzzing sound when you try to start the car may indicate the battery voltage drops (defective or under charged battery) or their is an electrical problem (possible voltage drop across a relay or more probable battery connections , ground or power cable.
Ensure your battery connections at the battery are tight, clean and secure. Try to jump start the car after having the car you are jump starting your car from running for a few minutes. If your car starts your battery may be defective. With the key on the battery light should come on, once the engine is running the battery light should extinguish or else the alternator is not charging your battery.