In short, the answer is both. And more. “Boutique” refining also adds to cost. Due to a hodgepodge of formulation requirements by different states, counties, sometimes even cities, gas companies can’t sell the same gas in Chicago that they sell minutes away in Gary. They then have to maintain separate refining, storage, distribution and transportation processes, raising the overall cost.
But if you think gas prices are exorbitant, take a look at bottled water or gourmet “coffee drinks” like Starbucks.