You're probably on to something ! I see from their itinerary they served there from Oct, 1861, to May, 1862. In fact they list a soldier being captured there :
Philip Tabb Yeatman
Residence was not listed;
Enlisted as a Private (date unknown).
He was listed as:
* POW Gloucester Point, VA (date not stated)
He also had service in:
VA 26th Infantry
Other Information:
born 11/28/1829
died 3/18/1897 in Alexandria, VA
Sources used by Historical Data Systems, Inc.:
- The Virginia Regimental Histories Series
(c) Historical Data Systems, Inc. @ www.civilwardata.com