Keep working or slow down?

Options
I have what is probably a rather peculiar question. I have never been in shape in my life. I was excused from having to take gym in school because of some health issues and, consequently, have never really learned how to exercise.

I have decided that I want to build some muscle, but I'm not sure how to go about it. I have a tendency to overdo things - my will is stronger than my body and I'm a little worried that I'll hurt myself if I don't know when to stop. I'm being careful, but I don't know how careful I need to be.

My muscles hurt from the exercise I did yesterday. It wasn't too strenuous (a few sets of 10 push-ups throughout the day), but these muscles haven't had to do anything for years. Should I continue to do this exercise while my muscles are hurting or would it be better to wait until they're back to normal? I don't want to stop, but I also don't want to hurt myself. Any advice would be much appreciated, as would suggestions of other exercises I could do that wouldn't kill me.