I've got a 17SE and an 18. A few weeks ago, I took my girlfriend out on the 17 with me (we usually sail the 18 together). Together we weigh about 350lb. I was definitely able to notice the added weight on the 17. It was more sluggish and heavy on the helm. The boat still went pleasantly fast, and my GF liked trapping off the wing but it just had a feeling like it was being overstressed with that much weight on board.
I think for single handing, the 17 is a great boat. For two or three on board (unless you're quite lite) I would say stick with the 18.