The real question is do I leave the fishing gear behind and tool around with the sail before I start fishing heh.
YEP! Get to know your kayak's characteristics under sail before you throw in the whole kitchen sink. At least, that's how I did it and have yet to bring along the fishing gear on a sailing day, and I'm glad I didn't because I would have lost my fishing gear (capsized)! Now that I am more comfortable with the sailing (but no expert by any means), I could probably carry some gear in the back as long as it doesn't snag the sail's mainline.
If you are interested only in sailing to and from the fishing grounds without trolling, ignore the next section...
However, if you are interested in trolling while under sail power, do a test rigging on shore first to see how the mainline from the sail interacts with your rod placement. Here's an example from when I was doing the same:
As you can see from my Revo11, the stock rod holders (as well as my milkcrate triple rod holder) with rods in them would interfere slightly with the mainline's route, and its pretty tight quarters on that kayak anyway so I try to carry as little as possible when its a dedicated "sailing" trip. I plan to bring only a SINGLE rod this weekend to see which of the 3 rod holders works best for trolling, and to see how quickly I can furl the sail once I have a strike on the trolling line.