Fiberglass mat will give you faster buildup but probably pretty difficult to work with in this application because it tends to want to all fall apart. If you use it, I would say lay down a layer of cloth on either side of it. Your best bet is probably to use 2 to 3 inch wide tape and wrap it in a spiral around the existing tube. I find it is generally easier to wet out the glass first and then put it in place. The other option is to paint a coating of resin on the existing tube, then wrap the glass, work the resin into the glass, then paint on another layer of resin, repeat. The difficult part will be working in areas where it's hard to reach/see. This is why wetting out the glass first can be better. Its a good idea to have a mirror and a shop lamp handy. It may also be easier if you take the boat apart.
As for thickness, if you're repairing a fully broken tube, you will at least need to match the existing tube thickness with your repair. 8 oz fiberglass tape is about 0.010" thick. So if the wall thickness of the tube is 1/16" (.0625") then you would be looking at about 6 layers.
For the resin, polyester resin can be a little tricky to dial in. Different manufacturers have different cure rates and the temperature and amount of hardener you use have a big impact. You will need to experiment if you are new to using resin. You may also want to consider using epoxy for this repair. Epoxy is stronger, has better adhesion, and generally has longer and more consistent cure times than polyester.