Vancian's flaw is purely fluff based. It is about as 'unmagical' as you can be for a wizard, since theres very few fictional fantasy(and even less mythological) magic users who actually prepare spells like that(Jack Vance's stuff is much closer to science fiction).
What it does to fluff is indeed horrible, but that's not the only problem.
Vancian has a significant mechanical flaw, i.e., the notion of leveled spell slots. This leaves you with a bunch of spell slots that are either irrelevant if their effects don't scale by caster level, or too scare compared to higher-level slots if they do. It also frequently puts one in the ridiculous situation of being out of Light Candle By Glaring At It for the day, but still having enough juice to Summon DeathStar MK II, or Rend Veil Of SpaceTime.
Also, by forcing spells to be of a particular "level", it forces false parity between spells that aren't necessarily comparable. Meteor Swarm wouldn't be such a bad spell if it didn't have to compete with Shapechange, Gate, and Black Blade Of Disaster.
Not to mention that any other feat or class feature that a caster might have, when it has to access some notion of a mana pool, has to burn spell slots of some level or other, in what I can only describe as a triumph of awkwardness.
The real reason they started with vancian was because D&D used to be a minis wargame. They simply took rocket launcher ammo out and replaced it with fireball slots, and after that it was legacy.
Yeah, that sounds just dumb enough to be true.