There’s really no one that can answer that question for you, and ultimately you and your children will be the ones to live with the consequences of either decision. My only advice would be this: First, decide for yourself and your children if you want to stay in this marriage, then talk to your husband (if you can) about whether or not he wants your marriage to work. If the decision on both sides is yes then you will need to discuss what you both expect out of it. If either of you decide you don’t want to stay married, I would suggest that you first file for separation. That means that when he comes to visit the children, he no longer stays with you overnight. If he’s visiting only once a month, and not providing for his family or making it possible for you to provide then that may be considered abandonment. I’m not sure on what abandonment entails. You could possibly be entitled to alimony, since you do not work, and child support. If moving closer to your family is a feasable option for you, it may help you to have some emotional support there and someone to be “on your side”.
This is only my opinion, but it sounds as though there may be a 3rd party involved if he doesn’t want you to move to where he is working and living now. I know that sometimes it’s necessary for a husband and wife not to live together for short periods of time, like in the instance of finding new jobs, but it should never be a choice unless the marriage is over.