I have been offline for a while due to illness and have just seen your post. I live about an hour drive from the nearest surf beaches so getting out for a wave more than once a week is a real luxury for me. I am building my fitness again at the moment and find swimming is definitely the activity that makes the most difference to my paddle fitness. Don't just swim endless laps. Mix it up with one lap sprints to simulate chasing a wave. Also swim some laps underwater to build lung capacity. It helps with long paddle outs and when you are lucky enough to get smashed and held down by a decent wave.
Surfing is like any paddle sport and your response time and strength should be built around your core so throw in some strength training like kettebells as well.