Can songs be deleted off the iPhone 5?

I slide the the song to the left and press delete, it stayed in my songs but just put a redownload simble next to it and it still plays with out me having to press redownload.

It could be because of one of two reasons:
1. ICloud is turned on for Music. Got to Settings, Music, then check to see that the 'Show all Music' is switched off. This means that it will show all the music you have ever downloaded on iTunes, even though you may have deleted it off iTunes long ago.
2. After deleting the song, Sync your iPhone with iTunes and it should be removed then.