I tend to look back at them, and assess them. But then, I’m a larger guy, so the situation tends to be different than what you’re talking about.
Many women though can shut down most guys by looking back. For example, with a critical eye that also communicates they are not going to put up with misbehavior. Also, many guys are not really up to sustaining eye contact, especially with someone who has no problem maintaining eye contact.
For another example, I once did an eye contact exercise, in a course, with a guy, just giving him kind attention and presence, and he broke down into tears in seconds.
It really depends on the people, and evaluating people and figuring out how to relate to them (even just in the scope of eye contact) is a skill to develop.