I actually just wrote software to run my blog. I did it for the same reasons you’re talking about, as a programming exercise and to have it work exactly the way I want it to (the big names all go overboard on features).
Here are the major features I included:
Public facing
* Topics: like tags, I assign 1 or more topics to a blog post and it shows up as a tag cloud in the nav
* Archives by year: I don’t blog often enough to warrant breaking this down by month
* Most popular: I track hits to each blog post so I can raise more popular posts up in the nav
* RSS / Atom feeds (as Vincentt said, obligatory)
* Mashup feeds: I pull in photos from Zooomr and links to recently read articles from Ma.gnolia
Admin interface
* No WYSIWYG editor! Hand coding is the only way I can be satisfied (obscure cinematic reference, anyone get it?)
* Live preview: this was really important to me as I code my blog entries by hand, it’s a faithful preview as I include the actual stylesheet used on the public facing pages
* Code snippets: I built a way to click a link and insert code snippets for things like embedded photos and code blocks
That’s pretty much it. What I love about it is that it doesn’t do a million things I don’t need. It does exactly what I want and no more.