I used to commute into Manhattan and to the airports from Port Chester. It doesn’t make sense to drive into the city unless you are going off hours. Metro North is your only realistic way. It took me about an hour to get to Grand Central on Metro North, then another 20 minutes to take the subway to Soho. Including walk/drive to train station, it was about 1:30 each way. From Danbury, I would guess you are looking at 2:30 each way door-to-door.
To get to the airports, it really depends on the time of day. I had to plan on an hour to drive from Port Chester to LGA and 1:30 to get to JFK. If really rush hour, add an hour to the JFK time to be safe. You can also take various shuttle services (Connecticut Limo Service, etc.) but these take longer because of the multiple stops. If you are going to EWR, you can save yourself some time and traffic by going via Tappan Zee Bridge instead of through the city and George Washington Bridge.
I lived in Brooklyn for two years and Port Chester for two years. Living in the city was much less time on the train, etc. Left much more time to spend with family. But we moved to Port Chester to be near family and that had its advantages. Made it easier on my family when I traveled out of town. I would say that if your commute is once a week, you will just have to plan on very long days and best use of the commute time and it’s doable. If it were a four-five day a week commute, I think you and your family would be miserable.