Does weight matter that much?

I think part of it is that a lot of people have a lot more to lose, so this isn't nearly as relevant. If you have 100 pounds to lose before you're in the healthy range? Yeah, your weight matters. If you're hovering around normal BMI range, being higher weight but more in shape makes sense as a goal to strive for.

Everyone's at different points, so everyone has different goals. That doesn't make your goal "good" for everyone else, just like their goal isn't "good" for you.

It depends. In your situation, where you continue to get smaller, no, I'd not worry at all, becoming denser and more compact is very healthy.

But for nearly everyone, it takes a healthy weight to get a healthy percentage of body fat, so these goals are complementary, not competing.

Weight, alone, yes is significant, overweight is a health risk. But if you are already a healthy weight and are getting thinner without getting lighter, that, to me, is the best result of all.

It's entirely possible to be in the overweight bmi range with little negative impact, but once you start pushing the upper range of overweight bmi problems start to arise. Even if you take care of yourself metabolically, excessive weight will still eventually impact your joints. Once the joints start going your tolerance for physical activity will decrease, and in turn make it increasingly difficult to maintain health through diet, particularly as we get older (and for women, there's the issue of breast weight messing with our necks and sleep).

You also have to account for potential situations in the future. Originally my ultimate goal was to reach the 180s, however I hope to have a baby in the next five years, which means I can expect to gain up to 30lbs (accounting for familial history as well as increased likelihood of multiples), so I've had to drop my overall goal to at least the 160s for both my and potential baby's health. For those who intend to pursue serious physical activities, they may have to focus on a higher weight that accounts for necessary muscle mass. Genetics are also important to consider. Someone who is prone to belly fat may need to maintain a lower weight or higher activity level for preventative purposes than someone who distributes fat more evenly.

All that said, health and fitness is a combination of factors, and it sounds like you've reached the point where the importance of weight is balanced more with the other factors. If you're maintaining good dietary and activity habits, there's no harm in shifting your focus away from that last 20lbs (and honestly being able to measure progress via clothing fit is so fun! I discovered last week I've gone down a shirt & dress size at my favorite retailer), just make sure you're checking in from time to time, especially when getting into periods of time where activity levels drop (winter, for example).