Vancian Magic. 1. Magic is in the form of distinct spells that do exactly one predetermined thing. 2. Spells must be prepared well in advance of use. 3. Each prepared spell can be used only once. (Though you may prepare the same spell multiple times.) 4. Mages have a finite capacity of prepared spells.
This is the Vancian system as I understand it. Break even one of these rules and the system is no longer Vancian. When you say you want a Non-Vancian Casting system, you should specify which exactly of these rules we should break.
limitation on amount of magic
-----------------------------
For game balancing reasons, there have to be limitations to how much magic a wizard can use, but the limits need not be exactly Vancian. Here's a brainstorm of limitation ideas from various sources, including this thread. Most of these variants preserve Vancian rule #1 (you probably use seeds for this, which needs its own brainstorm section) but break one or more of the other 3.
[you might want a default basic "eldrich blast" type power, so a wizard isn't completely useless after running out of magic. Variations on each idea are listed with a + mark.]
Slot system: as SRD. like Baulder's gate, SRD slots have levels, however, you may also put a lower level spell in a slot. This is basically like the Vancian system.
+ spontaneous casting (cure/inflict/nature's ally. UA domain/school?) I love this mechanic, especially with the spontaneous domain spell casting for clerics and school spell for specialist wizards variant. Not truly Vancian now, but close.
Point system: recover all mp after 8 hours rest. Psionics are like this, UA had a spell point magic variant that also had slots.
+ psionic augment; spend more points for better effect.
+ fatigue UA variant; become fatigued/exhausted if you spend too much of your pool. Recover partial pool along with stamina.
Point buy uses per day. (my system, playtested at low levels) spells have mp cost, but you must buy uses in advance, makes it similar to, but simpler than slot system. Just add tally marks next to spell in your known spells list.
+ spontaneous casting (cure/inflict/UA variants etc.) still works. expend a use of a spell of required level or higher.
4e system: daily, encounter, at will.
Split point system. [from this thread] Similar flavor to 4e system. Encounter pool is recovered any time you could take 20 on spellcraft check (not in combat.) Daily pool is recovered after 8/h rest. You may draw from either or both pools freely; an MP is an MP. (you want to draw from encounter pool first obviously).
+ psionic-style augment still works here.
Utility roll per round, winds of magic [warhammer?]. GM rolls d20 at beginning of every round You (and other casters) can only cast a spell of the level rolled?
School roll per round, winds of magic. GM rolls d8 at beginning of every round, each # corresponds to one of the 8 schools of magic by a table. spells from rolled school are free. others cost MP.
+ organize table by properties? schools 1 step away from rolled school get a discount.
+ opposite school can't be used?
+ no MP at all, you can only use the roll
Seed roll per round, winds of magic. GM rolls something. Corresponds to spell seed system by a table. This is the "environment seed". You have prepared seeds in advance. Number you can prepare is limited by level. You may combine the current environment seed with a prepared seed on the fly to cast a spell.
+ You may take a feat to cast a spell spontaneously using environment seed alone
+ store the environment seed for later with a full round action?
UA recharge. has tedious temporal bookkeeping
Utility roll variant to UA recharge? My idea, to avoid the bookkeeping issue. GM rolls d20 every round, each spell recharges on certain numbers. Spells with more numbers recharge more quickly. Average number of rounds to recharge = 20/[number of recharge numbers]. So if you wanted something to recharge in 5 rounds on average, give it 4 numbers.
[backlash variants may let you cast higher level spells than you would normally have access to.]
HP Backlash. (Like epic factor?). Magic consumes HP not MP. Spellcraft check may be required. At higher levels, you're more resistant to backlash damage. 5th level wizard might pull off a 8th level horrid wilting, but backlash would kill him. 20th level wizard might only take 5 damage from the same spell? (arbitrary numbers here)
Paradox backlash. (like mage: ascension?) Take a wild surge (d% chart of spell perversions) if you fail spellcraft check? Higher level spells are more difficult, obviously.
Paralysis backlash. (like pokemon hyper beam: 2 round move, but damage is in first round) You can cast any spell in 1 round, but become helpless afterward for 1 round + [1 round per spell level] while you recover from magical shock.
+ get a (will?) save to avoid the backlash. Saves improve with level obviously. Save DC based on spell level.
Sanity backlash. Using UA sanity rules; loss of sanity: casting spells. And read the recovering sanity section too. You don't have to prepare spells, you can cast all of them you know at will, but it's dangerous to your mental health.
Stored casting time. This is nearly Vancian. The more difficult spells simply take longer to cast. Cantrips can be done in 1 round, so are effectively at will. Each level adds 1 minute casting time (arbitrary. maybe 1 min per level squared is better?). You can store a certain number of minutes worth of spells, by preparing them in advance. This number improves with level. When you want to cast it, you simply complete the final round worth of casting. It takes as many minutes to prepare the spell as it would to cast it outright.
+ stored casting time in slots. These Slots don't have level. You get one more slot whenever class says you get access to a new spell level. (and maybe a bonus slot for high int or something). You can store any number of minutes worth of spell per slot, but it takes that long to prepare it. You could then naturally prepare per-day type spells per day, and re-prepare easy spells in freed slots per encounter.
Spell levels per round channeling. Choose a number of rounds to channel. You can't channel again for the same number of rounds afterward (cooldown). 1st level sorcerer maybe generates 1/2 spell level worth of magic per round if doing a "channeling" action. That's enough for a cantrip after 1 round of casting, or level 1 spell after 2 rounds of casting. Get another 1/2 level/round per level up. So an 18th level sorcerer generates 9 spell levels/round channeling, so could cast level drain in 1 round. Or could cast magic missile 9 times in 1 round. A spell probably shouldn't improve with a higher level caster with this variant. So that's like 9 level 1 magic missiles. You could mix-and match, so at 9/round you could cast 2 fireballs, 2 magic missiles and 2 rays of frost in the same round. 2*(3+1+1/2)=9.
+ take a feat to halve cooldown time?
Pseudo-logarithmic MP. Like a perfect20 wealth check. Spells have a base cast DC of 20 + [2 x spell level]. (arbitrary numbers here, may need tweaking). You have a reserve of MP, which you add as a bonus to your d20 "casting check" to cast a spell. If you pass the check but the DC exceeded your MP reserve number you lose 1 MP per 5 DC over the reserve (round up, 1 over still costs 1 MP). Also, 7th level spells automatically drain 1 MP, 8th drain 2 and 9th drain 3. (that's in addition to MP loss if you exceed reserves.). You don't expend MP at all on a failed check or if MP reserve was at DC or better. So a high level mage with full MP can spam magic missiles all day without expending any MP, but that same mage will use up MP to cast magic missile if reserves are low enough. Also the lower your reserve, the less likely you'll pass the casting check. Magic becomes progressively harder and harder the lower your reserves. (So you can see a magic point is more valuable the more MP you have, that's why MP score is like a logarithmic scale. 1 MP may be worth a magic missile, or a finger of death, depending.)
Para-magic draw (like FF VIII). Magic isn't prepared, it is collected. From monsters. Mages have the "draw magic" ability, which they can use at will. As a full round action at close range, make an opposed charisma check to one target monster. Gain 1 magic bead for passing the check, plus 1 for each point in excess of the monster's roll (after modifiers). Magic beads come in 10 levels (0 to 9) and 8 flavors (the schools of magic.) You can expend a bead to cast a spell of the appropriate school of equal level or lower. For example you could use a level 9 necromancy bead to cast energy drain or chill touch. What kind of beads you get depends on what kind of monster. The monster type determines the flavor (like aberration/transmutation, undead/necromancy, etc. from a lookup table) and the CR (or hit dice?) determines the level. I call them beads because you can use colored beads instead of pencil marks to keep track of your magical stock, but in the game world the beads are just magical energy stored in the mage's body, it has no real substance. Higher level mages have a higher capacity. Normal monsters have a limited supply of only 5 beads, you can't draw more than that. (they recover after resting, not that it matters usually, since adventurers tend to kill them first.) You can steal beads by drawing from a magic user, so magic-using monsters may have more than 5.
+ willing mages can trade beads like psionic tattoos
+ "drawcast" feat lets you draw one bead from a monster and expend it as a spell the same round.
+ "improved draw" feat extends the range you can draw to medium and adds a +2 bonus to your draw checks.
+ some types of monsters may have more than one flavor of bead. You could draw them (it) randomly from a bag of beads appropriate for that monster.
+ beads may be an ingredient for crafting magical items.
+ beads evaporate over time. Mages "leak" a certain number of beads per week. Randomly. Once a week, put your bead collection in a bag and pull out and discard the required number.
+ magical energy springs. Certain areas or natural features may accumulate magical energies. You may be able to draw level 3 necromancy beads from a certain gravestone, for example. They form a small pool of beads like a spring accumulates water. Bead evaporation keeps such pools fairly small, they contain only 5-10 beads at most, almost always of fairly low level. If you draw from a pool it will take a few days to fill up again. Springs are nearly invisible, so you usually have to use detect magic or the like to find them. The location of known springs is usually a closely guarded secret, but some may be willing to sell such information...
+ you can add an expensive gem as a material component to increase the effective level of a bead, letting you cast a higher level spell than otherwise possible. (use the price of a scroll for a reasonable value?)