The reason this fear is overstated is because we don't leave it up to the loser to decide that he has lost. We tell him he has lost. He immediately loses all authority, and it doesn't matter if he camps out in the Oval Office, he's no longer president and he no longer has legal authority.
It's not up to Trump to decide whether to leave office or not. If he's not president, he's not president, regardless of what he might think about it.