Hi joianett,
I too have done low carb but a ketogenic version in the past before I was pregnant. There are sooo many definitions of low carb out there that you need to be more specific. Are you following something specific "Atkins, PAleo, primal? Are you eating dairy, not eating dairy?
Low Carb in itself is not harmful to the baby the concern seems to be when you are in a ketogenic state because there has not been enough research done on the affect of ketones on the fetus.
However most sites that I have looked at say that as long as you stay out of ketosis while doing "low carb" you should be okay and fir in "healthy" carbs such as veggies, legumes, fruits (to a point).
Now how bad could it be for your baby to stay away from sugar, white refined carbs and white starches!

To me (my own opinion) eating meat, veggies, nuts, yogurt, some fruit here and there sounds pretty heatlhy and great for the baby. The human body does not physically need refined carbs! Sounds like you are doing everything right and if your doctor does not seem concerned I think you are fine.