Great post David ! I'm of the opinion there was a fair number of "free men of color", on Confederate muster rolls with nothing mentioned about their race. I recently found this man from Alabama. http://www.archives.alabama.gov/civilwar/soldier.cfm?id=228431

This man began drawing a pension in 1902 at the age of 69. Ten years later the pension board (perhaps new members), on finding out he was a Black Man, attempted to drop him from the pension rolls.Alabama apparently didn't award pensions to Black men. After discovering he was indeed an enlisted soldier, they allowed him to keep his pension. In the 1910 census, box #30 is marked "CA" designating James as a Confederate Veteran.

His pension application says nothing about coming to the army as anyone's servant:

The Pension Board attempts to take away his pension he's drawn for a decade :

Justice prevails after discovering he indeed was a soldier :