Can I remove an Officer from a poll?

Yes.  Each poll has its own set of Officers.  To remove an Officer, go to the Officers tab of that particular poll, select the officer, and click the delete button.  Note that removing an Officer from a poll that is not yet unlocked could make it difficult, or even impossible, to assemble a quorum of Officers to unlock the results.  If that happens, there will be no way to retrieve those results.