You pay the full price because you choose to buy the upgraded version. Work and effort goes into making that upgraded version, even if the original film is not worked on (and sometimes they are), there are new features to add in, the DVD needs to be edited and proofed, then manufactured, etc. I’m sure the costs of releasing a Blu-ray movie after a regular DVD exists is less than the costs of producing that original DVD, they’re making a profit off of the Blu-ray release (they make profit off the DVD one as well, but I imagine the margins for Blu-ray are a bit bigger if its been released on DVD before). They’re not in the business so that you have a neat and tidy and easily upgradeable media library. There is no advantage they gain if you have a Blu-ray version of the disc or a simple DVD version. They are in the business to make as much money as they can.
Software is a totally different world. Features are added to releases of software, and it is in the software’s interest to have people use the current version to extend their user base and to collect more feedback for the next version. They can’t give the updates out for free, since it takes many software developers and QA and documentation people to produce it and these people need to be paid. And if people had to pay full retail for each upgrade, there would not be a lot of upgrading happening. So they charge much less for the upgrade than the retail version costs. It’s in their interests as a business to give that discount.
Plus it would simply be a NIGHTMARE to implement what you’re proposing. The biggest reason is due to the way movie rights are sold to other countries, if you moved to another country, that vendor has no liability to give you an upgraded movie because you didn’t buy your DVD from them. Another reason is piracy, serial keys are notoriously easy to crack and not secure one bit.
But the biggest reason they don’t do this, is because it’s not in their interests to do so. They want your money, that’s it, that’s all.