I think you're cherry picking. Any documents of that time related to age, I believe, where given by the individual. I see swings of as much of 5 years in census documents.
Then, of course, there is the problem of typographical errors introduced by paper aging, torn corner folds, ink running, faulty memories, etc., and census taker errors.