The Magus get a few nice abilities, but he under performs compared to other fighter/casters. At level 12 an well built Eldritch Knight, Oracle, Cleric or Druid will all be better. That being said, if you want to give the Magus enough levels he can definitely challenge your party.
You want to do two things with the Magus, you have to make him look like a threat to the party, and you have to actually be a threat to the party. This isn't the same thing, if your Magus takes Pathfinder's cleave/great cleave he can attack multiple PCs with one attack, which makes the PCs feel they are in danger but you probably won't be doing enough damage to any single character to get a kill. Picking up Power Attack might also make your Magus seem like more of a threat if the PCs AC is low. If you want to be an actual threat then you should load your Magus up with spells that give bad conditions without a save, like Black Tentacles.
I'd also recommend that your Antagonist have a good AC, so that only a few of the PCs attacks will land in a round, but not so high they get frustrated. You should pump up your Hitpoints as well, so that the PCs feel like they are doing damage but not enough to legitimately drop him. Your Magus should also have a way to disappear when things are going poorly for him, that can be something as simple as a contingent Dimension Door.
