Hand brake refuses to release sometimes

What is your question?

I put the car in park position, engage the hand brake only for me to come back to the car hours after and the hand brake won't release when the button is depressed, sometimes till an hour. I also have a 2006 Passat with the same problem. These models have this button on the dash left of the steering you depress to activate and gepress to deactivate. How do I fix pls