It's funny you mention Singleton's reason for pulling out of the Confederate cause. No doubt the economics involved in the matter weighed upon him. In 1860 his land and personal worth totaled in the neighborhood of $116,000 which would be equivalent to around $4.2 million today. His neutrality was to no avail -- in 1863 his large majestic home near Centralia was burned to the ground by Feds.

By war's end Singleton had lost his fortune, and moved to neighboring Callaway County. In 1879 he traded his much less substantial Callaway County farm for property in Chicago and moved there to practice law. Apparently unsuccessfully -- before long he was living in St. Louis. And the once wealthy Singleton had fallen so far that one of his sons was working as a street car conductor.

But his immediate reason for abandoning the Confederate cause early in the war seems not to have been to protect himself from the Federals, but instead to protect himself from the Confederates. You note he was a scoundrel, of which there is compelling evidence. And it wasn't just being caught inflagrante dilecto in a married woman's back yard. He was also once arrested for attempted murder. And he ran for a number of political offices after the war and lost every election.

Tellingly, in 1878 there was a large barbecue at New Bloomfield in Callaway County. And during the course of the speechifying, a Judge Collier stated "Col. Singleton was in the army with Col. Peacher and Col. Singleton had to leave the army or the men would have hung him for his overbearing disposition." The judge took another potshot at Singleton by saying "Go and look at Col. Singleton's farm. It looks more like a farm a widow with 13 girls lived on."

Ouch.

A Mr. Larimore then spoke, kept up the general theme and parroted: "I saw Col. Peacher in St. Louis and he said that Col. Singleton was so overbearing that he had to leave the army or the men would have hung him." After which the report includes the comment "Very great applause."

It was shortly afterwards Singleton moved to Chicago.