International travel is literally a requirement for the job. This is a non-issue.
I would flip this back on the detractors: Which specific trip(s) do you believe the Prime Minister should not have taken? Let them go on the record saying "China was not worth a personal visit", or "The Queen's funeral was not worth a personal visit".
Watch them back down, real fast.