The usual answer to this is: "He was a abolitionist and said this only because he wanted the North to use black troops."

But he doesn't fit the bill- "A life-long Democrat."