If somebody says “Yeah, they would” when you say “Wild horses wouldn’t drag me there”, are they challenging your horsemanship or – more subtly – your professed distaste for ‘there’?
It may also mean they don’t ‘get’ metaphor and are thinking “Of course they would. How could you stop ‘em, they’re wild horses for godssake”.
Do watch how you use metaphor. Though intended to effect more rapid communication via a shared understanding, it can sometimes get in the way and leave the other guy stewing without your even knowing the damage you’ve done.