Why doesn't my FaceTime work on my iPhone 4S?

I have an iPhone 4S, and my boyfriend has my iPod 4th gen. We've FaceTimed before, but we usually have a problem where neither of us will receive the calls we send to each other. I'll call him, and it will tell me it's ringing, but he never gets the call. It's the same when he tries to call me. Very rarely will we get this to work. I'm using wifi, and so is he. What could be causing this problem, and what can I do to stop it from doing this?

Make sure you are both connected to wifi,
Make sure you both are calling the correct email address or phone number, and make sure in settings you are set up to receive FaceTime calls from a specific number or email address