And his character was already pretty old in the last one. The fourth took place about fifteen years later, didn’t it? It’s not hard imaging Indy’s father being dead (which he apparently was).
His character was dead. Indy was supposed to be “too old” for all of that, but if his father was around being even older it would have been harder to take all of the age references from Indy.