The 51st Virginia Infantry vacated Fort Donelson, Tennessee, under the cover of darkness across the Cumberland River in February 1862. This was after being surrounded by Union forces Union General Grant who offered unconditional surrender. This men weren't deserters. Any competent commanding officer would choice retreat over surrender.

Many of these soldiers reenlisted as soon as they arrived back in Patrick County. My great grand father, Captain David Lee Ross, reenlisted in 21st Virginia Cavalry on his return and served until the end of the war. The 21st & 23rd Virginia Cavalry are noted for raiding Yankee towns throughout Pennsylvania, Maryland and New Jersey. In short, creating havoc and disrupting Union lines of communication until the end of the war.

I suggest you purchase the book on the 51st Virginia Infantry as I imagine it will shed light on your search.
author James Davis
publisher H. E. Howard, Inc., Lynchburg, Virgina 1984

Best of luck,
Rick Brunson