It depended on who, what, when and where the person was. Also, by the 1860s vest were considered old fashioned and more of a utility garment than a fashion statement. A coat or jacket was more of a must than a vest. By this period the vest was worn more by clerks, professionals, administrators and managers than the average common working man. My best advice is to study the photographs of the period, not the portraits, but the scenes of daily life, on the streets or in the field and make your own conclusion. Vest were actually hard to make, more so than most articles of clothing and added to the expense of a suit.