Yeah, same for me when I saw him the first couple of times. The way he spins humorous stuff from thin air is very clever.
However, the last time I saw him, you realise that his 'random' stuff he says tends to follow themes. Although to be fair, that was because audience members kept calling out stuff from his previous shows and he just went with it. But still, he's clearly a very naturally funny guy.