I don’t have enough information to offer you a tailored answer, so I will simply present my thoughts.
If you think that you can accomplish this goal by yourself, then go ahead and do it without them. There’s no need to wait….you can get started right away. Don’t wait on other’s permission, because they aren’t going to give it to you. If you’re very lucky, you will find at least one person who unselfishly supports you once they realie how determined you are, but don’t expect it. While you are on that course, practice learning how to politely ignore nay-sayers.
That being said, if you have a wife and/or kids, you CAN’T take off and do what you want, regardless if you are convinced you can achieve it. When you take on a family, you accept that everything you do affects them, as well. That includes everything from ignoring your relationships with them, to tapping into the family’s financial resources.
You asked how we developed that courage. For me (and before I had a family of my own with a husband and two children), it was that their negativity was absolutely killing me. Every contrary comment was like another nail in my coffin lid. I was withering away and dying inside. I couldn’t deal with those people any more, nor did I deserve to live like that. I did everything I could to feel the warmth of the sun on my face, only to be yanked back and caged by who they thought I was. I became extremely determined to escape, and that’s exactly what I did. I left them. I left the first one when I was 9 years old. Every time I escaped another one, my life improved once it was under my control, and so did my confidence levels.