It really doesn’t matter what the relationship.
The plain fact is that NO ONE should ever share razors, according to the CDC. The fact that some people ignore that and do anyway doesn’t change anything.
But simply stating that you’re not comfortable lending it out to ANYONE should be sufficient. If he gets offended, it’s HIS problem. You have every right to decline something you are not comfortable with. Period. End of sentence.
It’s called being assertive. It’s a skill which can be learned with practice. It will come in handy many more times in the future.
The plain fact is that you are never obligated to do anything with which you are uncomfortable. You don’t owe anybody more explanation than that.
If necessary, stand in front of a mirror and practice saying “I’m just not comfortable doing that” until it becomes easy to say. Then say it.