The simple explanation is that Buddhism, as it was practiced in India, was a primarily monastic institution. Early Buddhists followed the strict monastic code called the Vinaya, which prohibited monks from working the soil, touching money, and other restrictions which made them entirely dependent upon the support of secular patrons. In addition, they would not bear arms, which made them vulnerable and, again, dependent upon secular authorities for protection.
For the first few centuries of Buddhism in India, the political climate was such that secular benefactors were not wanting. Buddhism flourished under the protection and patronage of several Indian rulers. But in the following centuries, these were succeeded by others less sympathetic to Buddhism and either unwilling to support it or downright hostile to it.
Invasions by the White Huns and the Mongols supplanted Buddhism in some regions, and the spread of Islam under the Moghul empires resulted in widespread destruction of monasteries and shrines.
Under all of this pressure, Buddhists retreated to Himalayan strongholds, or into other surrounding kingdoms where Buddhism had also taken root.
The present small Buddhist population of India actually represents an increase from the virtual absence of Buddhism at the beginning of the modern era.