Ok, so there are two calculations that describe that:
The Roche Limit (the distance from a planet within which a moon would be torn apart by tidal forces).
The Hill sphere (the region where a planet can hold a moon in orbit without it being pulled out of orbit by the star they both orbit).
These do limit the likelihood of moons for Venus and (especially) Mercury, because they are close to each other, or perhaps overlapping for Mercury depending on how big the proposed moon would be.