What causes the iPhone app switcher to lag?

I just bought an iPhone 5 and compared to my 2.5 year old 4 it's crazy fast. One thing I noticed was that double clicking the home and bring up the app switcher was instantaneous, and on the 4 it took 2-3 seconds. Although now my 5 is starting to do it. What causes it and is it fixable?

Background processes can slow phones down eg downloading files in the background or running heavy apps eg games, photo editing etc.
Filling up your internal storage slows it down as well.