Do not loose any sleep over it. I'd keep an eye on things...like I am sure you normally do....but for all the hoopla over this "Yes" scam I have yet to hear of any confirmed case of fraud taking place due to just a "yes". It pays to be cautious, but don't go overboard.
You know if you just do not answer unknown calls you won't have to worry about saying yes or no.
This number just called me. No message left. The only reason why I am reporting is that the scammer didn't bother with a legit sounding caller ID. It just listed the number in both fields, a telltale sign of scammers.