High speed aircraft have swept wings to reduce the drag caused by flying near the speed of sound. Adventure Islands are fast, but not that fast!
Wiki has a pretty good explanation:
If swept back daggerboards were faster, we would see them on race boats.
I'm a former flight instructor, and sail lots of centerboard boats (we're also Precision and Com-Pac dealers). I think I know what's going on with the observed increase in speed when my daggerboard is swept back.
In the Com-Pacs and Precisions, if the sail is properly set and there is excessive weather helm, the correct response is to pull up the centerboard a bit. This moves its center of effort aft, and makes it a less efficient swept back wing, with more drag. The boat goes faster because you have to use less rudder to go straight, and the rudder is always a brake.
With the balanced rudder on the AI, I'm probably dragging my rudder along sideways in order to go straight when the daggerboard is straight down. I angle it back and it allows me to use less rudder, so I go faster. I'm unaware of the effect because the balanced rudder has little to no "feel" to it, and the little lever is not great for judging the precise angle of the rudder, so I never notice the change.
That's my current guess, anyway...