It sounds like you might have a crush on your best friend besides being in love with him and it is called a ‘crush’ because it is difficult when you feel that way but the other person might not have the same feelings.
You mentioned that you can’t avoid it but at the same time you can’t tell him how you feel. These circumstances have the potential to create difficult and painful feelings for you the longer they go without intervention on your part. If you choose to completely forgo talking to him about any of this, you’re going to have to work it out within yourself on how to block out the feelings you have, find distractions to take your mind off of them, or work on diminishing them over time so that they don’t continue to cause you grief.
It might be best for your own piece of mind (and happiness) if you just sit down with him, have a heart to heart talk, and let him know how you feel. There is the possibility it would change your friendship, like you mentioned, but that’s a risk you’d have to take. If he is a true friend and an understanding one, even if his feelings don’t match yours, he won’t let it harm your friendship, hopefully. If your revelations to him do cause him to have a negative opinion of you and the friendship, then he might not be as great of a friend that you’ve always thought.