Perhaps we didn't. The source link to Slaves at Arlington includes the account described, which I have questioned. Even the questionable account states that Lee had the slaves whipped by either his overseer or the local constable; he did nothing of the kind himself.