Just to put some numbers on it. According to the Consumer Energy Commission the average refrigerator uses 680 kWhr of electricity in year. At 12 cents per kWhr that is $81 per year.
I used a Kill-o-Watt meter to measure my own refrigerator, a relatively new, large GE (without ice maker) and got 470 kWhr /year or $56.40. I need heat for 10 months of the year and do not have air conditioning the refrigerator is basically free for those months since it replaces the heat my normal heating system would need to produce.
Let’s address the other 2 months of the year when it costs me something. That is a total of $9.40 in a year. Even if I managed to do something miraculous so it cost nothing, the time and effort would not be worth the trouble for $9.
Take the other extreme, where you live in a hot climate with A/C needed all the time and electricity costs $0.25 per kWhr. (Maybe @mattbrowne can say how much he pays.) My same refrigerator would cost you 470 kWhr to run the refrigerator + (est.) 1.5×470 kWhr in air conditioning costs to get rid of the heat. 1175 kWhr x 0.25 = $294 per year. Now you have a reason to do something. Maybe.
If you vent all the refrigerator heat out of the house you will save on the a/c cost but you will be drawing in outside air that is presumably hot causing your a/c unit to work harder.
The best energy saving technique you can do is buy a refrigerator that is sized so you keep it full most of the time. Keep water bottles in it if you have the empty space. They will be handy in an emergency and will keep the food cold longer if you lose power.