Sony shouldn't give up the handheld, that is a similar debate as "Should Nintendo give up home consoles?". My answer is simple, no. The PS Vita is a great piece of kit (and one i plan on purchasing next year). Plus it is good competition for Nintendo's 3DS (well, i say that loosely).
But looking at both types of groups with Nintendo and Sony, Nintendo are established in the handheld market, whilst Sony are relative newcomers to the market. And then look at the types of players. Maybe it's the type of player that is responsible for the slow sales of Vita, Sony gamers tend to stick to the console over the Nintendo gamer who stick more to hand-held gaming. That's based on what i've seen.