Federal lawmakers described the disregard to their invitations by some public officials as an affront on the legislature