Weight lose with no workout vs workout

I'm having the hardest time trying to figure out since I haven't worked out in three weeks and I lose the 2 pounds per week that I've aimed for but while working out and eating the exact same way I am now I couldn't lose 2 pounds per week, I would just lose one pound, a pound in a half and so forth. So it's just confusing. Make me not wanna work out. Can anyone give me some suggestions? And by the way I'm doing nothing different other than just not working out.