You may choose to vote in as many or as few races as you choose. If your only vote is to remove a judge, that is a legitimate ballot. (It's probably legitimate to send in a blank ballot too)