There is no way to know. At all. Ever. Period.
There is no physical sign. There is no physiological difference between a male virgin and a male non-virgin. The act of having sex for the first time causes no change to made to a male’s anatomy. Therefore, the only way to know would be to ask – but there is nothing you can do to force him to tell the truth.
For what it’s worth, there is no way to know for sure if a woman is a virgin or not, either. It is possible for the hymen to be broken (ie, the “cherry” to be “popped”) without intercourse. It is possible for pregnancy to occur without penile entry into the vagina. Again, no way to tell, either gender.
If there’s some question in your mind as to whether your partner is being honest with you, YOU ARE NOT READY to engage in sex with this person.