the only major disadvantage of ghost levels (assuming that one is using the savage progressions leveled template) is that they grant no HD, BAB, saves, or skill point and feats progressions. however, if one can manage to use a canny level adjustment buy-off method, this becomes less difficult from a mechanical perspective, though the dm is unlikely to allow something like this....
I've mentioned that before for ghost. The biggest question, really, is whether you have to buy off your LA at exactly the levels given(IE: you must buyoff your first LA at level 3, and if you don't you miss the boat), or whether the levels given are offsets from when you acquire the LA (IE: if you take a level of ghost at 4 you buy it off at 7)
in that link i gave i quote the exact rule. basically, it says that once you reach 3x LA in class levels (not counting monstrous HD), then you must immediately buy off the first level of your LA. so if your LA is 5, once you reach 15 class levels, you must buy off the first LA immediately upon obtaining enough xp to reach the 15th class level.
however, with the rules for gaining templates mid campaign, the templates listed in that link are broken down into levels, which count as levels for the purpose of leveling up. so you explicitly don't have to take all the levels of any given template, period. and you can still buy off the LA of what you have taken, in fact you must, once you hit 3x the LA in class levels.
so, to answer your question specifically, no you don't have to buy off the level at "3rd" level only. that table assumes that you have the LA from the start of your character, as in, you are starting off the game at that level. the formula for buy off = 3 times the number of class levels as you have level adjustment... and ECL and HD (class or prestige granted HD, that is) don't count for the purpose of this calculation.
to further illustrate, i'll use your example. you mentioned taking a ghost template level at 4th. this would make you an ECL of 4 consisting of 3 class levels, and 1 LA. according to the rules as stated in the text, you are now at a ratio of 3:1 class levels to LA, therefore you must immediately spend (ECL-1)*1000 or (4-1)*1000 = 3000xp. this will drop you to 3rd level at 3000xp with 0 LA and the abilities of one level of ghost template.
since i was working on a ghost telekinetic when i came up with the idea for the level adjustment mini handbook, that is what i'm using for an example.