In most cases, the flavor of herbs and spices comes from an essential oil, and oils do add calories. But it would be a tiny amount given how potent most of these oils are (it only takes 10 parts per million of pepper oil to make a dish “hot”); if there were enough oil present to substantially raise the caloric content, it would be unpalatable..

They are negligible.

Here is a good rule of thumb for you.
Think of the highest calorie content foods around: Oils and fats, and sugar.
Oils are 9 Calories per gram
Sugar is 4 Calories per gram

How much does a couple of shakes of fresh pepper weigh, for example? 0.12 gm. (I just checked it on my Mettler balance)
So if pepper were 100% fat, (it’s not), the absolute most calories it could add to the food is between 0.5 and 1 Calorie. Fuhgeddaboutit

There are so few it would seem silly to count them?

Yes but not a lot.
This is a long list of calories in spices and herbs:
http://caloriecount.about.com/calories-spices-herbs-ic0200