Too uncomfortable to sail - unstable, too hard on the rigging... 10 knots? 15 knots? A what speed do you decide to leave the sail in the car?
10 knots. Even with a small home-made jib to help the steering track better (yes, and a sailing rudder), the plastic stay clips start to fail at or around 10kts/20kph (and if they fail once, they are more likely to fail next time).
Plus I believe the rig's officially rated for less than that - 10mph.
If you put in metal clips to replace the plastic ones, you can sail in 10kts, maybe a little more, but you are risking too much stress on the rigging, especially the pad-eyes and the mast straightness.
HOWEVER: Don't leave the sail behind if there's any chance of the wind dropping. It's easy to pre-rig and bring along strapped in the boat, and very useful if conditions do improve. Likewise, if you start a multi-hour journey when there's no wind, DO bring the sail.
Where I am, wind conditions frequently change, and it's very frustrating to start with no sail and wish you had one only 15 minutes later.