He was just trollling you for sh*ts & giggles, as the saying goes. He wanted to see how long he could poke you and get you to respond before you got angry and hung up. It's all he has in life to make him feel superior to another human being, so it's pretty sad.
Whenever they ask you a dumb yes or no question like that, consider that they are trying to get a recorded "yes" on the line so they can use it to charge you something to your phone you never agreed to.