You may be able to check Reconstruction voter rolls. Unless a man had taken the oath of allegiance, he could not be on the voter rolls.