There are, of course, detriments to hosting your own photos. Normally I’m a huge fan of rolling one’s own websites for projects and work, but you have to keep in mind that if you’re on Flickr, people will simply stumble upon your photos because that community already exists. How are people going to find your photos if you section them off onto your own site.
Part of the solution to this problem depends on what you’re displaying photos for. Do you want people to view and offer feedback on your photography? I’d go with Flickr. Are you looking for a location to put on a business card, trying to sell prints? I’d host your own.
You could also, perhaps, get the best of both worlds by hosting your photos on Flickr but setting up a shell/content management system on your own server that pulls the photos from your Flickr account and displays them in your own templates. Sort of like http://unexpectedlyquit.com/. Those aren’t photos, but that’s the technical idea I’m getting at. (If you click on any image, you’ll see that it’s actually hosted on his Flickr account.)