John --

There's a big difference between taking the oath of allegiance and being paroled. If you took the oath of allegiance, rejoining Confederate service could present a problem if recaptured. However, if paroled, a man could rejoin Confederate service after being properly exchanged by the POW cartel.

Ordinarily a man took the oath because he did not intend to fight for the Confederacy again.