It makes more sense to switch careers than to remain in one you don’t want to do.
I would ask you, though – what makes you “attracted” to detective work? Do you know what it realistically entails, or are all of your thoughts about it created by what you’ve seen on TV? Would you be able to choose the kinds of cases you’d like to work? Might you find various aspects of the work depressing?
Do some homework – maybe talk to a couple of people who do this for a living – and find out if it would match your expectations. It would be terrible to throw away a career you could be starting now, in exchange for something you enjoy less.
And in terms of “throwing your hard work down the drain”, remember that there is usually something you can take out of one experience to apply to another, even if they are seemingly very different. One of the things that makes someone a good teacher is the ability to put oneself in other people’s shoes, to be able to see from their eyes. This is a skill that would come in handy in detection. I’m sure you could find many other transferable skills, if you think about it.