You've made the best decisions for your family and owe no explanations to anyone. Your son will have benefited from having had both of you as primary caregivers at different times in his formative years. Sounds like a great childhood to me.