I've been selfhosting email for years, often behind a residential connection. At some point you have to give in and use an SMTP relay if you actually want it to be useful for communicating with people on big email services (Gmail, Hotmail, etc) or anyone who uses a blacklist.

But, if you're using it to email a couple friends who are also running their own email servers on residential connections, I'm not sure it's a requirement. In fact, thinking of things like the Fediverse, and tilde sites, this would be an interesting movement to start, perhaps with explicit whitelisting to stave off spam. You have a few domains (or subdomains or whatever) that all just agree to exchange mail, without the expectation of being able to scale it up.

Usually, when people talk about how much of a pain hosting email is, it seems like they're thinking of it in terms of having a single address (perhaps with some aliases) that they use for most things. I guess if you don't mind having multiple emails, maybe one that you use as above, and perhaps another even with one of the big providers for stuff you really need to get through anywhere, this could work.