Eating above BMR

Hey so I am 84.6kg, male at 6 feet 3 inches. My BMR is approx. 2050 calories. I am aiming to weigh around 78kg.
I recently learned that for a few months I have been undereating (eating around 200-400 calories below my BMR) I know, stupid right? Though at this time I was unaware that i NEEDED to eat above my BMR so that my metabolism doesn't lower. Now i know that I need to eat above my BMR, I am for about 2300 net calories every day. My TDEE is 2800 calories.
My issue is that I am finding it hard to eat this many calories, especially when I have been fixated on eating only healthy foods for months, and healthy foods generally don't have high amounts of calories. And even when I get close to my goal at around 2000 calories, I then do my workout and will knock off another 300-400 calories and then BAM! I have to eat another 1k calories!

Is it alright to consume high calorie foods such as refined foods if I am to achieve my caloric goal? Of course the majority of my foods will be healthy and natural foods but is it ok to implement some more high-calorie refined carb and sugars into my diet to up my caloric intake? Or will this just hurt my fat loss? The past few days I have begun eating a few more high calorie meals like a pizza, and after the pizza I felt so fat and greasy I just felt like I was going backwards, yet every forum I see they are saying "eat more than your BMR" and "it doesn't matter what foods you eat it's only calories in vs calories out that counts"