To me it seems like you've tried to counter brute force and brute toughness with more brute force and brute toughness. The problems you describe revolve around dealing HP damage. This is not the way to approach your problem. I expect where you're really struggling is tactics and encounter composition. Your PC party is composed almost entirely of warrior types, and they lack a dedicated full caster. If you want to challenge them, make sure they cannot benefit too much from what they're good at: a) hitting things, and b) not being hit by things. Also make sure that c) you challenge them in more miscellaneous ways than outright combat.
a) Stop giving them opportunities to approach their foes in melee, at least without having some trouble getting there. Highly mobile, ranged attackers are the simplest solution: a large part of the PCs can't really catch up to them, so they must rely on the ranged element of their own party to take down the threat. Flying, teleporting, burrowing etc. enemies will also mix things up quite a bit. Remember: what the party can't reach they can't destroy.
Even better is throwing in some battlefield control abilities. A Druid with Entangle or Sorcerer with Web can make sure around half the party is unable to move, or forced to move very slowly. Solid Fog, Black Tentacles, Wall of Ice etc. are spells they should be encountering at the levels they're at. Other abilities that provide nice BC either mimic such spells (Assassin Vine's Entangle, Bone Devil's Ice Wall), are available as feats (Earth Devotion, Entangling Exhalation, Improved Trip), or are miscellaneous monstrous abilities (Air Elemental's Whirlwind, spiderwebs).
One resource for not being hit are available to a wide range of enemies: miss chances. Blacklight, Blink, Blur, Displacement, Mirror Image, Pyrotechnics, and the fog spells are spell solutions for this. Incorporeality is a good defense, as well. Also, there's environmental conditions (namely, fog or dust in the air, dense foliage, a hailstorm etc.) that even darkvision doesn't negate.
Finally, some monsters are outright immune to a wide range of attacks. A Hellwasp Swarm may be just the thing to scare them. A Black Pudding can also take away excess equipment...
You might want to start with a bunch of Arrowhawks (or Blink Dogs with a Warlock level each), accompanying a Sorcerer-Druid-couple, and work your way up from there.
Another route you might want to go are debuffing tactics. A bunch of Rays of Enfeeblement plus Rays of Exhaustion will cut the melee department down to size. A Dragonne can accomplish similar things with its Roar. A horde of undead Shadows (or a Greater Shadow) will put the fear of god into even mid- to high-level parties. I expect their Fort saves to be pretty good, but at their level, poisonous creatures might still be an option that scares them. Other debuffs, like the Glitterdust, Slow, Unluck and Enervation spells, provide fantastic options as well. Rust Monsters? Good... debuffers, to say the least.
b) You want to get away from attacking their strong point (AC) and target some weak points occasionally (weak saves, touch AC) or rely more on attacks that outright do some kind of damage (area attacks, damaging auras, no-save-no-attack-roll spells, even environmental conditions). I don't need to go into too much detail, but consider any or all of the following attack forms:
breath weapons
(dragons, cryo-/pyrohydra, chimera)
area spells
(monsters with SLAs, casters)
other area damage abilities
(Medusa, Destrachan, swarms)
trip/disarm/sunder
(warrior types; even better with very large creatures such as giants; best done at huge reach)
grapple (plus constrict/rake/swallow whole)
(Tendriculouses and Behirs are tough)
touch attacks/incorporeal touch attacks
(Lamia, Spectre; touch spells)
ranged touch attacks
(Efreeti, Gauth beholder; Warlocks; ranged touch spells)
mind-affecting nasties
(Formian Taskmaster, Mindflayer, Succubus)
c) Finally, you need to make sure they understand how much they've pigeonholed their party by staying away from problem-solving in favor of combat prowess. I can't tell you how to do this in a few short paragraphs; just make sure that there'll be some puzzle locks to pick, disgruntled nobles to placate, ships to get safely through a storm, businessmen to swindle, underwater caves to navigate, magical traps to unravel, mountains to climb, Lord High Priests to impress, city defenses to plan, women to woo etc.