I’ve experienced this in three ways:
The first, which sounds like what you’re going through now, was in a “You’re not living up to your potential” kind of way. I was a bright kid, and got great grades, but ended up dropping out of high school due to anxiety and never went to college. So, I fell into a career as a secretary, moving up to office management. They were expecting a lawyer or something. I also preferred to work for small companies close to home, rather than opting for more money and better benefits at large corporations with a commute.
The second was exactly as @jonsblond describes. Being a stay-at-home mother is not for everyone, and lots of people don’t understand it. I did it for fourteen years, and occasionally felt I had to defend my choices.
The third is actually happening now. I’m the Community Manager for a Q & A website… nobody understands what in the hell that means. ;) My kids understand what I do, because they’re members here and see it firsthand. I think my ex-husband thinks I just kind of play around on the computer all day. Friends, likewise. The only time it bothers me is when the ex thinks I can be on call for all middle-of-the-day, kid-related things. My job is somewhat flexible, but I can’t just disregard it altogether. I still have to do my work, you know? We both work full-time, and we are both far away from their schools, but he seems to think it should all fall to me, since I don’t have a ‘real’ job. That pisses me off. My job is just as real as his.
Bottom line: If you’re happy, tell them so, and ask them to back off. It’s your life, after all.