Newly calculated bounds on the strength of the coupling of an electron to a proton or a neutron by a fifth force are presented. These results are derived from the high precision spectroscopic data currently available for hydrogen, deuterium, helium-3 and helium-4. They do not depend on specific assumptions on how the interaction would couple to a deuteron compared to a proton or would couple to an $α$ particle compared to a helion. They depend on its coupling to a muon, but not in a significant way for carrier masses below 100~keV if one assumes that the strength of the interaction with a muon would be of a similar order of magnitude as the strength of the interaction with an electron in that mass region.