I always keep them off, as I find them too much of a distraction from what's going on onscreen. I end up reading the game instead of watching what the characters are doing.
There are exceptions, of course. As GreyFuss mentioned, The Lost Crown is one of them, though I had no trouble understanding Nigel, in spite of his bizarre speech patterns.
However, the ambient sounds tend to drown out the voices in that game, so I leave them on. Ditto for Black Mirror 2. And in Runaway 2, there is one scene with some aliens that I am not able to understand at all, so I'll turn them on just for that part and then turn them off again afterward.
For the most part, though, I prefer to leave them off, and it annoys me when the game does not give me the option to do so. Luckily, that doesn't happen too often.