As for Armored Mage ACF note the: This benefit applies only to spells of a level equal to or
lower than your fighter class level + 1.
So at most you'd be able to cast 5th levels spells without penalty.
That's why it's important to lose a bunch of caster levels - so you don't accidentally outpace your armored casting! 
That's like becoming a eunuch as a form of birth control. It works, but there are a lot better (and less painful) ways to achieve the same result.
