If you are beyond the first 2 weeks of usage where the Fitbit is trying to decide the HR that exercise starts at for using HR-based calorie burn, and below which step-based calorie burn should be used - then indeed turning it off may be in order.

Several meds, medical conditions, stress, ect - can cause elevated HR that has nothing to do with level of effort that has anything to do with calorie burn.
And you just can't get beyond that.

But - usually those issues are not reflected in exercise level HR, because a max is max, and elevated by 10bpm drops to nothing during exercise.

If your exercise is step-based (walking, running) where calories from distance is better - just leave the HR off unless curious.
Or manually log it after the fact with known distance and replace the HR-based calorie burn. That way you still get to see it, and improvements.

If cardio exercise is not step-based (elliptical, spin, stairs, ect) than HR would be better than database entry that has no level of intensity to it.

If not aerobic cardio (lifting, intervals) than manually log anyway.

Just remember that Fitbit is a replace only system, not addition.
So even if on all above you leave the HR on during the workout just to see improvements over time - you manually logging a workout replaces the calorie burn for what Fitbit came up with.
This can be nice because that Activity Record does give the start/duration times you'll need to manually log it later anyway.