This argument is a mess, and it was rightly dismissed by the lower courts.
In general you can't hold such officials legally accountable for what they don't do. You can hold them politically accountable and refuse to reelect them, but they're not guilty of what they don't do.
It would take some pretty overwhelming evidence to show that the inaction was not at worst a mistake but part of an intentional choice as part of a larger scheme.