There being two Georgia Cavalry Regiments known as the 4th, it's important to distinguish between the two. You asked about a soldier assigned to Avery's 4th Georgia Cavalry. Those records are filed under 12th Georgia Cavalry, the designation given this command by the War Department. All the soldier's records are filed there under the last name 'Stevenson', not Stephenson. The widow's signature on the death claim is also clearly spelled Stevenson.
Isaac w. Avery's regiment organized on Jan. 30, 1863. Duncan L. Clinch's regiment having organized on Jan. 15, 1863, Avery recognized the right of Clinch's regiment to the designation 4th Georgia Cavalry. Avery wrote to the War Department on Jan. 9, 1863 from Fayetteville, TN to request arms for his men. Enough saddles were available to equip his command, but like many other Confederate cavalry units, most of his soldiers were unarmed or only partly armed. This stuation seems to have remedied shortly after Feb. 19, 1863, when Col. Avery requisitioned 15,000 rounds of Colt Navy cartridges from the arsenal at Macon GA. By May 13, 1863, when Avery reported from Chapel Hill, TN, his command had been assigned to Crews Georgia Brigade of Wharton's Cavalry Division, Army of Tennessee.
The combat record of Avery's 4th Georgia Cavalry Regiment began after the death of Thomas J. Stevenson on April 12, 1863.