I'd buy it with a paranoia wizard, a diplomancer, or certain setups that are amazing at avoiding and mitigating lethal damage, but yeah. In most cases surviving decades of adventuring with average or below-average Con is pretty unbelievable. Even surviving levels of adventuring, retiring early, then coming out of retirement after you're an old fart seems pretty unlikely.
Not that one may not do that, I got no problem with playing to the RAW and telling the RAI and verisimilitude to EAD, but it's pretty obviously all one is doing there.