For the same reason there’s a rough plane of ecliptic for planets in the solar system rather than a randomly-arranged sphere, meteors are going to be concentrated at certain orbits and in certain directions, as those which deviate too far from trajectories which place them in stable orbits, send them into the atmosphere or slingshot them out into space.
For the same reason, you’ll see most of the orbiting meteors travelling in the same direction, since those which oppose that direction are more likely to have massive collisions which send them flying out of orbit or into the atmosphere. In the short term there’s always going to be a fair bit of chaos, but over time Brownian motion of individual meteors will create a rough equilibrium.
This means, while you’re probably more likely to see impacts inside of a gravity well, they will be by objects travelling at roughly similar speeds in roughly the same direction, and thus much less damaging than the more rare but utterly catastrophic random impacts you’re going to see in interplanetary space with objects the speed of which becomes additive to the force of the impact. Something the size of a pea hitting you at a net velocity of a few kilometres a second in orbit may be damaging, but survivable. Getting hit with something the size of a pea travelling at a net velocity of a few thousand kilometres a second in interplanetary space will leave nothing but an expanding cloud of dust and plasma.