Are the activities planned something your son is interested in, but your dad knows you aren’t? You said you used to feel kind of “dragged” to the activities your dad wanted to do when you were a kid.
Your dad might genuinely want grandfather grandson time, but I don’t see why you wouldn’t be included for at least a day or two after that long drive.
Next time, if you’re not going to stay, I’d only drive half way or so and meet up, have lunch maybe, your dad pays hopefully, and not put yourself out quite so much. Maybe eat near a gas station. Lol.
When I was young I used to spend three weeks in the summer with my grandparents and I loved it. I think my parents liked the break too.
Where I live now tons of grandparents take in their grandkids for a few weeks over the summer. I live in a retirement community. There are fun activities all day long for grandparents and grandkids to do together. It’s very normal here for the parents of the children not to be here the whole time, or at all, it’s also normal for the parents to be here. It just depends on the particular family.
I would try to talk to your dad a little about doing something together with him, it can also include your son/whole family, that you would all enjoy. That you would like to spend more time with him.