The only way to answer this question would be to order this pension application, I suggest waiting for Mr. Boswell's response.
"Please look at the service records for this Elijah Lee of the 4th Cav to make sure I have interpreted the residence as Covington County correctly. You know how difficult it is to read handwriting sometimes."
There are two surrender rosters available at Ancestry.com, both indicate Elijah Lee's residence as Covington County.
"And the Elijah listed on the service records didn't live in Covington County but Perry County, which isn't close in proximity to Covington County, which is on the service records."
In 1860, Perry County did adjoin Covington County.