Let's not forget the other states.
North Carolina voted 81-39 against secession. After Lincoln called for troops to invade the South they voted 120-0 for.
Much the same for Arkansas, Tennessee and Virginia.