Waiting on the intense workout... silly?

So let me just say, I am happy with the weight I am and have been losing. BUT it is not my ultimate goal to just lose weight. Before my wedding I wanted to lose weight so I ate very healthily and worked out like crazy! But I couldn't get below 150 no matter what I did. I was a chunky strong. Now I am working out lightly (walking with the baby, strength twice a week, etc) and losing lots of weight from a good diet. But ultimately I want to be super fit. So is it silly to lose more weight THEN work out like crazy to get fit? For some reason that's what my brain is telling me to do. But it may be the lazy part of me haha