Also, this before picture was about 20 pounds ago.
All I had to do was take up the shoulders a bit. I turned the vest inside out and pinned a few inches down from the original shoulder seams.
Then I used a zig-zag stitch to make sure the knitting doesn't unravel.
I carefully trimmed off the excess as close to the stitching as possible without snipping the threads.
Still plenty loose (it is an XL after all), but I kind of like the blouse-y effect. And now the armholes are much more reasonable.