Which matters most- residency or custody?

My son wanted to live with his father, so last August I allowed him to go to NC to stay with him. I am getting absolutely no feedback from his dad and had to contact the school to get his report card. There has always been open communication between us until he married a person that makes him turn on the speaker every time he gets on the phone. She refuses to let me speak to him and he doesn’t answer my calls, nor returns my texts. I just found out he failed, and is into some bad things. When I finally got his father on the phone and told him I want my son home, he said for me just to remember that Scottie is a resident of NC now. I have sole custody given to me last July by my local family court in SC. Am I still able to go get him? I did write a letter stating I’d like for me and dad to share joint custody so he could enroll him in school. Should I rescind that? Please help as my son is walking in his father’s bad footsteps. I thought he had changed, but it’s worse.