@CharlieGirl I understand now. I’ve experienced this both ways. There is nothing wrong with the attraction, but you really should respect the other persons orientation once it is made clear to you.
(possibly NSFW part) My late wife and I were in a three-way realtionship. Her fem lover was almost entirely oriented toward women. I respected that and sexual contact between the two of us was extremely limited, based entirely on what she tolerated. Following my wifes death, we remain close friends and business partners.
We have many gay friends; on rare occaisions I have been approached. I quietly and gently explain that, although I appreciate the compliment of his interest, my orientation doesn’t go that way. We’re still friends.
You can’t always tell what a persons orientation is when you first meet. The best you can do is handle things with sensitivity and discretion, graciously accepting if your orientations don’t match.