A like CR challenge for character level X party of Y members is a group of Y members at Character level X. So, simply put, X*1/y = their CR on an individual basis. Put them together, get a EL of the party's standard level. If I put even a strongly optimized NPC against a party, they'll slaughter him. On his own, he's just 1/y their power.
If you must make them fight 1 powerful dude, ad hoc is the best way to go. Experience goes a long way in figuring out what level of adversary is suitable for your party. As an example, a group of 2nd level, with 4 members (psion, factotum, sorcerer, Draegloth) was able to match up with a 5th level warblade. The fight ended with nearly a tpk, but they got the best of the warblade in the end. Yet they were also, at level 5, able to fight four level 5 class-leveled enemies and prevail with little difficulty. Had I given them the treasure from that encounter and xp as the rules are laid out, it would have been a idiotic move on my part.
In short, making an iron clad rule on this would be difficult. Use common sense and slap on appropriate rewards.