The potential intrusion vector is, as you've guessed, through the hypervisor. (Or the host OS, where applicable.)
The fact that no-one can even articulate a coherent attack plan hasn't prevented the entire security industry from generating Microsoftish amounts of FUD.
You'll have to evaluate for yourself - how much do you trust your VM vendor to write bug-free code to handle incoming packets and pass them on? This does touch on almost every facet of a hypervisor, so it's not an academic question.
Logically, you aren't exposing any new vulnerabilities. In fact, though, you are opening up a new potential intrusion vector.
As far as I can tell, everyone in the argument seems to derive their authority from one comment by Schneier; if anyone has any sources with actual data (empirical, theoretical or experimental) please let me know.
Personally, I trust VM programmers to get patches out quickly, and I trust the paranoiacs to blatt about news of any new compromise, enough to be willing to do the sort of thing you're talking about.
(Having said that, although I'm *willing* to, I will note that I *don't* do so in real life.)
The one aspect to it, though, is that compromise of the hypervisor essentially means instant, complete, utter, irreversible compromise of *all* the VMs (including non-running disk images!) that server has direct access to. That is a little bit worrisome.
-Adam