The voting rights were more of a privilege. Back then, even with the extended voting rights, no back people were allowed to vote. Not only black, but Asians as well. Also, independently of that, women also could not vote..
So in the end, the correct answer, and the only correct answer would be something like: All the white men.
Things were still tough for woman and "non-whites".
