Well, that's really interesting! I never would have guessed.
On this site it says: "Company D., E., F. & G. were recruited from Confederate prisoners held at Fort Delaware, who had taken the oath of allegiance to the United States of America."
http://www.2ndmdinfantryus.org/uscav3.html
Maybe he was among them?
Thanks again.