Being from the Carribean, though not having a patois per se, people always and I mean always say "huh?" , which we've become used to from Americans and the like.
To me, American accents, almost all of them, are too slow and tend to make me feel like "hurry up" lol, cause y'all pronounce your words too much. And British tend to also over-pronounce their words, but just faster.
So, as long as they're speaking English, I can get what they're saying, because all other accents (mostly- except for like Irish lol) to me are slow or pronounced too much.
If I can understand a Jamaican talking to me, Everyone else is a breeze :p
But as long as you are honest and patient with the person, it should work out.
There have been times where I've been giving directions to tourists and some of them are like "HUH?", right in the middle of it, so I just thicken my accent to piss them off further, cause that's rude. Just ask for a more clear response and in the event of unfamilar words/phrases, A more common or universal equivalent.
Just ask Sweets