I believe the scam you're speaking of is an urban legend even though the news media (biggest spreader of urban legends) is reporting about it. From experience, credit card companies require a lot of identifying information, above and beyond a simple "yes," to approve a suspicious charge/purchase. I doubt much, if anything, can be accomplished with the simple statement you responded with. I wouldn't worry about it.
I forgot to explain. I heard of a scam where they are using recordings of people saying "yes" so they can piece/paste it together with a recording of them offering something or needing your permission for something. I said the word "yes" when he responded hello (to my hello) and I responded back with "yes, who's calling?" From what I've heard all the scammer needs is a recording of your voice saying "yes" and they can attach it to any recording they want and use it as your permission to something, who knows what! Now, I'm worried :-(