I don't see why you can't toss a single high level caster at them and rawk their faces right off, if you wanted to.
Yes, I understand action economy and 6 actions per round is 6x as effective as 1 action per round...but casters break the action economy and are super efficient. You should be able to throw casters with at least 7th level spells at their faces, how isn't a buffed solo-Wizard encounter not kicking their asses with Spell Turning, Project Image, Teleport shenanigans? (Obviously, there are MANY different kinds of shenanigans, I just pulled some core spells at the appropriate level - "Coward" Wizard should be able to wear that party down.) That's without turning to Celerity and Familiar abuse and so on and so on. Your high level casters should have multiple buffs that make them nigh immune to your party (for a few rounds at least) then you "divide and conquer" or "retreat and recoup."
I am wondering how skilled you are as a DM versus how skilled your players are at min/max, because I think that is the real heart of the matter. If your players are very good, with great and complementary builds, then ok - you need to come up with *situations* that take them down a peg. You don't even have to use heavy handed shit like AMFs and Wild Magic or Dead Magic zones. The lava caldera was a neat idea by Weenog, for example. However, I think that gets to the root of the matter. If you're just picking random monsters from the MMs and putting them up against your group of super bad-ass optimizers, then I would see the situation you are describing happening.
Honestly, I need more information to give you more specific advice. I have DM'd for groups of 6 without issues, and I am in a play group with 6 right now; our DM also has no issues. I can describe a bunch of stuff he's doing right now to keep it challenging at 8th level (which is close to your group's 11th). Still, I'd like to know a little bit more about your groups tolerance towards Min/Max and your preparations vs your player's optimizations.