Interesting discussion.
I think I'd have to side with the story.
For example, Final Fantasy 8, which I first played a few months ago, had a story that at times was pretty damn good, but at others completely threw me out of the immersion. In the end, it kinda hurt the experience.
On the other hand, there's the Pokemon Mystery Dungeon series, which I think have marvellous stories. A bit corny at times, but there's something that just draws me in.
Maybe it's the already familiar characters suddenly getting personalities, maybe it's the surprisingly dark route the stories tend to take, but I love it.
So yeah, a decent story definitely helps me enjoy a game. (With obvious exceptions, I'm not expecting much from a Mario game.)
But it most definitely is a double-edged sword.