I mean yes, any and every American can refuse a request like this. That's always been the case, hence a long history of showboating around empty chairs in hearings, for example.
Congress has always been limited in its authority to haul people in as congresspeople see fit.