The reputation of the prison caused the Confederate House of Representatives in 1863 to investigate the commandant, Capt. George W. Alexander, who had been accused of "harshness, inhumanity, tyranny, and dishonesty." Alexander was cleared of the charges, partially by his citing the nature of the inmates of the prison. Castle Thunder, located in Richmond, Virginia, was a former tobacco warehouse that had been converted into a prison and was used by the Confederacy to house its civilian prisoners, which included captured Union spies, political prisoners and southerners charged with treason during the War. The early head of the Union Secret Service, Col. Lafayette Baker had several men under his command who were imprisoned there. One was J. Harvey Sherman:

The Washington Star of April 16, 1863 reported that 9 [Mosby] Rangers had captured J. (James) Harvey Sherman, a Baker detective, near his home in Dranesville VA. Sherman investigated, scouted, and provided information on citizens in Northern Virginia to Baker. Although he was reported hung, he had been sent to Castle Thunder in Richmond VA. An attempt was soon made to secure his exchange. A memo sent to Col. William H. Ludlow, Agent for Exchange, in Washington, D. C, May, 1863:

"...The guide or scout, J. Harvey Sherman, who was said to have been captured near Winchester and hung by some of Mosby's Rangers, it appears, is confined in Castle Thunder. Please make some inquiries about him and if you can make some arrangement for his exchange...Very respectfully, your obedient servant, W. HOFFMAN, Colonel Third Infantry, Commissary-General of Prisoners"

They never exchanged him. Sherman later escaped from a North Carolina prison just before he was going to be sent to Andersonville Prison. After the war, Lafayette Baker would be the man that Secretary of War Stanton used to hunt down John Wilkes Booth.