When God takes something from your grasp, He's not punishing you, but merely opening your hands to receive something better.
'The will of God will never take you where the Grace of God will not protect you.'
There comes a point in your life when you realize:
Who matters,
Who never did,
Who won't anymore...
And who always will.
So, don't worry about people from your past, there's a reason why they didn't make it to your future.