It’s really unfortunate that some view lack of eye contact as rude, missing out on many of the sweetest and most sensitive human beings. I usually just figure the person is uncomfortable in (some) social situations. I don’t get offended at all, though I’m very sensitive to being spoken to while the other person’s eyes are on a screen. That, I can’t stand. If you’re just looking at your feet or something, I’m fine with it.
Eye contact is not generally difficult for me (anymore), but it certainly can be uncomfortable with strangers. There’s also not much of a point for just making small talk. With deep (emotional) conversations, much of what is communicated will be nonverbal, so eye contact is not just about manners. Looking at a loved one’s face when they’re telling you about a scary experience, for instance, is a kind thing to do. It will deepen your bond, though touch and tone can be workarounds when eye contact is just too difficult.
I’m sorry your family holds this belief. If it’s any comfort, the amount of eye contact most people enjoy is probably much less than you (or they) think. And plus, people can’t tell if you’re looking in their eyes, or just at their eyes/forehead. If you’re ever in situations where you want to practise eye contact because it’s important to you, try studying the colour of people’s irises. Eyes are beautiful, and having that type of scientific curiosity might make the experience less stressful. I worked at a grocery store for a few months once, and I used the tedious job as eye contact practice by counting the different eye colours I saw through the day. Eye contact with strangers became less stressful after playing this game with myself.