You'd need to create your own invocation to mimic haste for the swiftblade to work. The problem is, most warlock buffs are of the 24 hour type (ie: all the +skills, fell flight, etc). The warlock is designed along a totally different line than the typical "cast something and it eventually wears off" archetype. So if I was your DM, I'm make your haste invocation at least a Greater, and maybe a Dark, since it's really the equivalent of a level 9 spell (persistent haste).
But if you could somehow convince your DM to make it a lesser, GO FOR IT! Swiftblade goodies all the time! Oh, the fun you could have.