I have read that the IRS NEVER calls, they send mail. I've been contacted several times - errors on tax returns, etc - and it's always via US mail. If you think about it, a lot of people don't answer the phone if they don't recognize the number, so they can't have a reasonable expectation of communicating with you. I NEVER check my messages either, so if they leave a message, I won't get it and they can't prove that I got the message.
As someone above pointed out, there are ways to spoof you with a different and official sounding number that shows up on your caller ID. Also suggested above: Ask them to send their requests in writing via US mail. It is mail fraud to misrepresent yourself saying that you are someone other than who you are. That is a Federal offence that falls under the umbrella of Bank Fraud.