Custody issues

Hi, I am in s situation where my soon to be ex has moved out of state and wishes to have our children from time to time. We have a separation agreement that states I have sole custody and there is no set visitation because he did not want any. He now decides after 7 months that he wants to get our children out of state where he is living with his girlfriend. Do I have to let them go out of state or can do I have a say in where he takes our children if I have sole custody. Could someone please explain the custody laws in North Carolina. There is no court ordered visitation nor have we worked out visitation, he left after telling me that I I’ll have sole custody. Now he wants joint custody and wants to take me to court if I don’t let them go to Virginia. I am not trying to keep the children away from him, they need a father, but do I neccessarily have to send my children with him if there is no set visitation, and with us still being married he is living with another woman? My older children do not want to go because they don’t like him living with this woman and hearing her call him “baby”. We are a christian household and even though we have this separation agreement, aren’t we still legally married? Is this considered adultery (He was with this woman before we even separated). I am so confused, could someone help me out. There has to be a reason why my children kick scream and cry because they don’t want to go to this woman’s house. I am trying to keep this out of court, but now this man wants to play hardball. I am trying to do right by my children, I have tried and tried to get him to visit our children and come see them and spend time with them, He never had the time. He now decided that he wanted to get them two days after Christmas until New Year’s Day. I was kind enough to let them go, but my two daughters did not want to go only my 4 year old son. They haven’t seen him in 3 months. Now because they don’t want to go be with him, I am now the bad guy because I wouldn’t force them to go with him this week. I literally have to talk to my daughter for 30 minutes to calm her down because she is afraid to go. It scares me because I wonder what happened the first time I let them go with him. Should I be concerned or could it be that she is around me so much that she doesn’t want to be away from me. Please help, I don’t know what to do.