Another issue at play is that people often build high expectations of Virgin and their service, and understandably feel deflated when they aren't met - regardless of whether the expectations were unreasonable or not. Basically they probably feel deflated from the moment they join the check-in snake at Heathrow, and feel duty-bound to pick fault from thereon in!
VS could be their own worst enemy here anyway, as their marketing often only makes a noise about their premium cabins, and there is a gaping chasm between Upper and Economy - I'd say it's even more pronounced than most other scheduled airlines.
VS Economy service is somewhere on a par with a charter airline, in terms of gigantic check-in lines, poor leg room, draconian carry-on policy, and only one pass through with the drinks cart (but at least you don't pay extra for anything off it). The charter airlines are threatening to catch up with VS in terms of IFE and gimmicks, which will leave the VS Y service looking a bit jaded.
I'm used to flying United a lot, and only recently switched to Virgin when UA cut their schedules on some routes I use (especially BOS and EWR). I was utterly appalled the first time I travelled VS in Y, compared to the UA Y cabin, but mostly because it wasn't what I was used to. UA come round with the drinks cart several times, they take the foil lid off the food for you, and they have real coffee, rather than bitter-tasting instant muck. They also used to have decent desserts rather than off-the-shelf stuff you can find in the supermarket (e.g. sickly gooey "Crunchie-ish" things), even in Y (UA tend to dish out a lot of ice cream these days).
VS seemed to think they can get away with this stuff by thrusting a big tv in your face. Since then, I've got to know VS' idiosyncracies and I'm better prepared for them!

The good news is that we're told VS will be overhauling their Y service soon, now that they have got the Upper cabin sorted. I live in hope!

I definitely agree with the assertion about things falling apart when there is any irregularity. I'd much rather see a swan (graceful on the surface, despite thrashing below the waterline) during irregular ops, than a load of staff charging around like headless chickens!
In my experience, VS' timekeeping is pretty lousy, almost every flight I've been on in the past 12 months has been at least 30 minutes late at destination, though that could just be Heathrow doing it's worst, and pushing the flight ops guys (Hi Airchabum!) to the max.
There's also the common thing that bad experiences stick out more in your mind than good ones!
All in all, I think VS are one of the better airlines flying, certainly in terms of their premium cabins, and their lounges, which blow almost all of the competition out of the water. (I'm looking forward to ANZ having the Suites and new Y seating, because their food and service is already really good.)
Not so sure about VS' Economy service though, I think it's easily beaten right now - though the VS Y blankets and comfort kits are probably the best out there!
Cheers,
Mike