One of these days I'd like to try out the Tome material, I've liked it since I first took a look at it.
It is not that much of a "allright I counter you", it's "now you're fucked and I gave basically nothing up to do this"
This describes many spells and general caster advantages.
Most powerful effects, such as AMF, should come with a hefty cost. In case of AMF, the cost won't be enough if it is just given to fighters.
Since it's a 6th level spell effect and therefore requires 11th level unoptimized, I would say 11 levels of fighter is a much heftier cost than 11 levels of wizard.
I prefer a Paper-Rock-Scissors system for party roles. Beatstick should be > caster when beatstick can melee caster, caster should be > everything when (insert ideal circumstances). Casters who aren't built for melee should dread it, and beatsticks who can't get to the casters should dread them. The idea is that your party has Paper, Rock, and Scissors, but typically you're facing enemies who are only Paper, Rock, or Scissors.
There are other ways to do it, but they require more original thinking than was used in 3.5's inception.
That's not RPS, it's basically odds and evens. And it doesn't deal with certain situations well. First, because it only lists two possible outcomes for each and the third is a tossup (rock on rock is supposed to be a draw, and draws don't happen in D&D). Second, because we're talking about a game with A LOT more variants.
What about ranged fighters? Skillmonkeys? Monsters, which don't always fit well in any category? What happens when you add things like maritime combat to the mix?
Obviously I assumed that the concept would be expanded.
For example:
Ranged fighters are actually just direct-damage beatsticks who don't have to run to their target. I usually see this balanced by lower total damage output and features that take advantage of fighting at range (Attack multiple targets since you don't have to move to them, inflict status conditions, attack difficult-to-reach targets, force concentration checks, etc.). To avoid counterattack, many ranged attackers include enhanced mobility. Vice Versa, many highly-mobile characters are ranged attackers. The basic concept is "hard to hit, but keeps hitting you."
Skillmonkeys are usually doing their thing outside of combat. Either you're in combat or you're out of combat, utility in one neither includes nor excludes utility in the other. Casters can be skillmonkeys, melee characters can be skillmonkeys, ranged attackers can be skillmonkeys. Basically I see skillmonkeys as separate from a lot of balancing, because you only need to balance them against level-appropriate DCs and other characters.
Monsters do actually fit into categories, but frequently more than one. Many of the higher level monsters threaten in every category, and usually can threaten in two in a round (Although to threaten in all three they usually need more than one round). After that there are puzzle monsters, which tend to threaten in only one or two categories but can only be threatened by specific things. They're a different balancing act.
The point is: Everyone must be threatened by something. Everyone must fulfill a useful party role (Often being the aforementioned threat). Everyone must be capable of something that only they can offer to the party, so that their role cannot be easily ignored.
All the sentences that started with "everyone" just now are the players' jobs to make happen, but the game shouldn't fail at offering them means of doing those things.
So if you want to have a party of casters that's fine, but you'll be heavily threatened in one area where you'll have to compensate. Casters don't HAVE to be squishy, but traditional casters are so a party of traditional casters would have to account for direct-damage threats in their spell selection.
Also highly-flexible "Role-switching" classes like the Binder (and most Tier 1s) CAN switch their role constantly but in an actual campaign they end up focusing on one or two, depending on what the party needs. Just because they CAN fill any job when they wake up in the morning does not mean they DO fill EVERY job.
Throughout the length of this thread I keep thinking "Dragon Age" the video game, because the three classes in that game balance each other well. While you can use a party that doesn't have one specific class, you will have to focus your energy on compensating for that loss.
That game has Warriors, Mages, and Rogues. Any one of those can be optimized for a tank role (In fact the mage tank is the most resilient), but what neither the Rogue nor Mage can do as well as the Warrior is gather threat from enemy A.I. The warriors have abilities that specifically make A.I. target them.
D&D is much larger and more complex than Dragon Age, but the basic concept DOES expand to a more complex usage. Everyone brings something to the table, and if you do without a certain party role you will have fill that hole or you'll be weakened.
That's what I meant mentioning Rock Paper Scissors, and I stand by it. The fact that WotC failed to do this doesn't weaken my argument. Fighters still suck, but having someone who can melee is still helpful to the party.
As for maritime combat, I'd love to see it work, but have yet to see someone spend the time to really make it happen. I played a MUD once that had it, and I didn't do anything else in that game except kill sea monsters because naval combat is so rarely available.