If custody and visitation were set by court order (or via a separation agreement incorporated into the divorce decree), the moving party will need to show a material change of circumstance affecting the welfare of the children before custody/visitation may be changed. The court will then apply a “best interest” standard when setting the new regimen. Keep in mind that a change in a child’s residence is not, in and of itself, a substantial change of circumstance. Other factors must also be present.
If custody and visitation were set by unincorporated separation agreement, the situation is a little murkier. You’re not required to do anything. You could take the kids to Timbuktu if you wanted. The father’s only recourse is to seek involvement of the court. A court is not bound by separation agreement provisions pertaining to child custody/visitation (and child support). An agreement of this type has value only as long as both parties abide by it. Both parties need to trust each other, and be of like mind about what’s best for the children, for it to work.
My advice to you is to attempt to secure the father’s “blessing” prior to the move. If the distance between you and the kids’ father is greatly increased, you’ll need to negotiate a new visitation schedule. Flexibility is the key here, as is your acknowledgement of the importance of the father in the childrens’ lives. A major benefit to this approach is that the parents are able apply their own standards in determining what’s best for their children. Court is the absolute worst place to “settle” custody and visitation issues.
Not everything can be settled by out-of-court agreement. If you go to court for an initial custody ruling, or the court finds changed circumstances (relative to the then-existing order) affecting the welfare of the children, the judge will hear what you all have to say and then make a “best interest of the children” determination. Almost anything is possible in this case. Judges use their broad discretion to do pretty much whatever they want with custody and visitation. So yes, you could lose primary custody of the kids. Relocation cases are quite fact-specific. The custody order will probably also specify the transportation and travel arrangements necessary to implement the new court-ordered visitation schedule.