Where does the old iOS go after you upgrade your iPhone?

Currently my iPhone is running on iOS 8 and i'm planning to iOS 9 since it has be officially revealed by Apple.

I want to know do they save the old files of the previous version of iOS (iOS 8.4 in my case) in phone's memory?
If not do the old files get deleted?

Good programs delete the unecessary files and tidy up after themselves. Not all programs do this

It is replaced with the new one.

IOS 9 would overwrite iOS 8.4.

Of course data used in apps & whatnot would be saved & restored in iOS 9 but the actual OS would overwrite the older version. IOS 9 continues the latest trend for iOS updates of being smaller over all so you would regain space after the update