You tried to call back 4 times? It "sounds fishy" because it is a scam. If you want to give your money to a thief that badly, I think the Brooklyn Bridge is still for sale.
The IRS does not need to sue anyone. They send you first class letters delivered by the Post Office. Several of them. Ignore those, and they then send certified letters that you need to sign for. Again delivered by the Post Office. Ignore those and they simply freeze your bank accounts and anything else you own. If they can do that, then why would they bother with a court case that might take them years before they get in front of a judge? They also do not bother arresting people for a few thousand dollars for the same reasons I have already given.
Official IRS Government site detailing this scam and providing links to report it:
http://www.irs.gov/uac/Newsroom/IRS-Warns-of-Pervasive-Telephone-Scam