Sounds like you will have to bear with him. When he confronts you, delay your response by ten or so seconds longer than you usually would do, give yourself time to think. When he says things like “you will be a horrible father”, respond with “OK”. OK is a neutral response, and if anything it might hit him after a while that what he says is pulling you apart unnecessarily.
Not only does a response of OK not make you anymore angry, but there is no way that it cannot anger him further. Although OK is not an agreement with what he has said, it is not an argument and simply tells him that you accept his opinion and do not wish to discuss the matter further.
I wish you the very best.