Disclaimer: I am not in New York State, and you’re probably buying at least part of your electricity from my province. :)
I agree, it’s tough to wade through the information about this online. From what I can see here from RG & E, there is a choice between “retail access” (pricing by an ESCO) and “non-retail access” (pricing based on hourly forecasts) pricing.
According to this 2004 annual report from Energy East, the choice between different pricing systems was made by individual customers. It describes a voluntary choice of pricing system, but with restrictions – so I guess accounts that did not qualify were never given a choice (from my first link, “A customer who has not chosen an option and for whom RG&E has not received a retail access enrollment from an ESCO will default to Hourly Pricing.”). In the annual report, EE planned a freeze on all delivery charges, but implemented the retail access surcharge. They don’t explicitly say it is to make up the shortfall, but I guess that must be why they did it. The report covers their plans until 2008, but I couldn’t find a report that begins directly after the end of that freeze. Maybe you can turn one up to find out what happened next.
I notice that in this 2006 report on utility case decisions, RG & E is the only company on the list to add details for such a surcharge (see footnotes) – and that they also add a complementary charge to gas clients (as a “gas merchant function charge”). It may be that utilities in other states simply lump all such charges together. But it sounds to me from skimming articles on recent utilities competition in NY State that they are trying to balance long-term price freezes with smaller-scale rising costs. This kind of surcharge is probably an attempt at that.