I have only used them while running and noticed a difference during and after the runs (*compared to not wearing them). They are not "true" compression socks or sleeves. They are Reebok's athletic compression socks. One of my next purchases will be actual compresison calf sleeves.

Oops! I hate to say this after you have ordered them, since they are rather expensive.
But . . .
I have never understood the rationale behind compression socks for running. Old people wear compression socks because they have venous disease, and their veins are compromised, so blood does not return up the leg the way it should. But runners do not have that problem. In fact, the flexing of their muscles probably aids their venous return. So, there is not reason I can think of that they would help.
Moreover, I was recently on another thread about compression socks. A few people there noted that there has never been any demonstrated advantage to wearing them.
That said, many people like them and think they provide something -- better recovery. And, they might keep your calves warm, which might help reduce the risk of injury.

It sounds like your soleus. I have pulled mine multiple times. A few things I've learned....

First time was prior to a 1/2 marathon. I kept trying to jump back to training after a few days or a week of training only to reinjure it. I ended up staying off of it for about 4-5 weeks right before my race. I bought the sleeve the night before the race at the vendor fair and wore it...and had no problems at all with my soleus.

I pulled it two more times at CF about a year later. Once doing box jumps, once doing double unders. Maybe coincidence, but I had just bought new Nanos. Long story, but I now have inserts that I use after having plantar fasciitis, and can wear my Nanos again, but if I know there is any amount or running, I switch to other shoes.

Every time I injure it, I do wear the sleeve and it helps. There is another guy in my box who tends to pull his a lot too, and he wears one as well. I do think it helps.