Do you really mean talking to/hanging out with a lot, or are you referring to having romantic feelings for someone who’s not your spouse?
To me, having feelings you don’t want to have—whether they’re romantic feelings for someone you’re not into, or other crappy feelings—is really unpleasant and inconvenient, but there’s not a lot you can do about it. So I would have trouble faulting someone for having an unwanted crush. (That said, if I was the betrayed spouse, I would still be really hurt.)
What you can fault someone for is acting on it. And to me, that includes both spending a lot of time with this hypothetical crush or sleeping with hypothetical crush. In the arbitrary moral system of my life, though, sleeping with unwantedcrush is definitely worse than spending time with unwantedcrush.