You can identify the men who deserted on May 10, 1862 from the Regimental Personnel Listing found at the American Civil War Research Database at:

http://www.civilwardata.com/active/hdsquery.dll?Muster?2567&C

Although a subscription service, you might be able to obtain a trial period.

It's possible that these men may have felt that their service in a Harbor Guard unit may have been for local defense only.