Bookgirl, I feel for you. No one who has not experienced a real anxiety attack can truly understand your dilemma. I do understand. In the midst of my last anxiety attack, my blood pressure (taken by a nurse) was 193/80. My normal blood pressure is 108/68! Anxiety/panic causes physical changes (mostly adverse!) to your body, and is 100% REAL.
Does the thought of giving up on phone shops cause greater anxiety, or less? Start there! Only you can answer that question.
Maybe set a goal? "Today, I will do ONE shop." Maybe find one that is similar to some you've already done?
But meanwhile, communicate with your schedulers, as suggested above. Schedulers, believe it or not, are human, too, and generally understand that health issues can prevent shops.
Best of luck to you.
Meanwhile, you might try some of the many on-line shops. They generally don't require interacting with a live human!