Hijab of eyes is a particular way of being decent within Islamic precepts. Both men and women are expected to keep their gazes downcast unless it is allowed. Men are expected to do this to avoid lustful thoughts when looking at any woman who is not their wife. Women are expected to keep their gaze downcast as a way of protecting themselves.
The word hijab stands for screen or concealment. Hijab of eyes can thus be thought of as a way of keeping away from temptation by following principles of decency specified by the Prophet for both men and women as to how to view another. Followers of Islam are expected to follow principles of modesty and decency in their interactions with people of the opposite sex. There are two specific verses in the Quran that guide Muslims as to how to follow hijab of eyes.
Men are expected to earn and provide for the family, so they don't have to hide their bodies. Their work requires them to be around and about. Yet, it isn't all right to stare at a nonmarriageable woman even if she is wearing a scarf or hijab. Decency requires men to cast their eyes downward. The glance is thought to be a poisoned arrow that comes from the arrows of the devil.
Followers of Islam are taught to be wary of the glance because it can allow lust to enter the person's soul. Women are also expected to follow hijab of eyes strictly. The woman's sphere of activity is considered to be her home, so it is relatively easier for her to keep her whole body covered. Women are expected to hide themselves from the view of ghair mahram. Ghair mahram translates as a man with whom marriage is allowed.
It is a little similar to a saying in the Bible, the verse from Matthew that describes how looking at a woman with lust is the same as having committed adultery. Following hijab of eyes is expected to keep both men and women safe from this kind of sin. Muslim men and women who interact with members of different religions may simply be following hijab of eyes if they do not make eye contact while talking to a member of the opposite sex. It is not a mark of disrespect, rudeness, or low self-confidence. Keeping their eyes downcast in a conversation means that they are following Islamic law.