I believe the scam you're speaking of is an urban legend even though the news media (biggest spreader of urban legends) is reporting about it. From experience, credit card companies require a lot of identifying information, above and beyond a simple "yes," to approve a suspicious charge/purchase. I doubt much, if anything, can be accomplished with the simple statement you responded with. I wouldn't worry about it.
Their goal is not to get you to say "yes" but rather just to get you to answer. Once they know that they have a person who answers a particular phone number, they sell that information to marketing companies as a "live lead" for potential sales. The best approach is to ignore calls from unknown numbers. If it's important, they'll leave a message.