Yep. . . that's another mystery. I've never found James Nathaniel (my g-grandfather) and Mary F anywhere on any census before 1880, although James was born in 1862, Mary in about 1857; there are even family tales of an older sibling, Dr. Robert Johnson, born in 1851, a year after John Frank was in the Washington county VA home of his parents in the 1850 census. Then again, John Frank is nowehere to found on any 1860 census I've searched. And, there's an irony that James Nathan and Mary are with John Frank in Welch WV in 1880, in that family lore holds that they scorned their "drunken wife-beating turncoat" father.
Nor have I learned what finally became of Julia Ellington. She vanishes altogether after 1870, at least as far as I can tell.