soy and men okay or stay away?

I've recently started eating a certain high fiber breakfast cereal most mornings , and came to the gross realizization
That it contains soy protein isolate. I am male wondering if I should swap it and drop the soy?
Any guys on here notice any issues when consuming soy? Thanks