Rick --
You're exactly right. Much of this discussion appears in James Oakes, The Ruling Race: A History of American Slaveholders, 1982. Few people have explored the story of the average slaveholder, who was neither planter nor pauper.