Do I remove the rubber plug on new fuel pump next to fuel intake port?

What is your question?

I'm replacing the fuel pump on my 93 Pontiac GP LE 3.1. During shipping they put a dust cover on the intake port and a cover where the fuel outlet joins the pipe. There is also on the bottom next to the intake port a small black rubber plug. Do I remove this also?