Many Vicksburg parolees were later returning; some never returned.

That's one reason.

Another would be lack of firearms and accoutrements. Of course all the exchanged Vicksburg prisoners needed to be rearmed, and this took time. Records of arms lost at MIssionary Ridge show various obsolete smoothbores in the hands of former Vicksburg regiments.

Don't know particulars for Georgia regiments, but some Alabama prisoners in Scott's Brigade were not declared exchanged until the Atlanta Campaign began.