Stephen King does this in “The Colorado Kid”. A LOT of movies also never reveal the culprit – Zodiac and Memories of Murders spring to mind. So yeah – it’s absolutely possible. It depends on the writer how good it is though – I didn’t like Memories of Murders for this (and a lot of other) reasons but loved Zodiac simply BECAUSE of this reason.
As always, it lies in the execution – if you’re writing a mystery story where the culprit is never revealed, you must choose to focus on something else. Zodiac focuses on its characters – their reactions and psyche regarding the case. How they ruin their lives trying to solve this thing, and eventually doesn’t solve it at all.
So if you plan on not revealing who the culprit is, you need to have another in. Why should I want to read a mystery novel without an answer? Well, because it portrays excellently the characters’ need to solve it, or to show how bureaucratic police work is, leaving millions of small road-blocks whose consequences are that the murderer gets away with it. Or you could do as in Memories of Murders, where (if I remember right) it is incredibly, hugely vital and important for the Country and the Police Department that the two main detectives solve the case, but its too much of a burden to bear. After going at the case in a completely wrong and stupid way, they’ve essentially burned all bridges and can’t bear it anymore. Blame it on sloppy and idiotic cops, but when the pressure is on and they needed to solve it fast, they made some mistakes. These early mistakes leads to them not being able to solve it, and the murderer gets away with it.
My two cents, is all. (notice how a lot of these stories are based on real-life killings? Such as Zodiac, Jack the Ripper, Memories of Murders, etc.).