Visatation Threats

I am non custodial parent, live in another State, children in North Carolina. We meet halfway for the exchange with the following set by the Judge: I get my children Spring break, every other Christmas and Thanksgiving, 10 days after school is out and return 2 weeks before school resumes, one three day weekend a month (my choice) when they have day off of school.
Now the issues: my ex wanted children for weekend of birthday and I stated I had plans but was willing to let them go if only I could have extra make up days, ex refused. I had birthday party on another day because I had to work (on the acutual birthday day but had days off prior to this day) and now Ex is threatening to take me back to court. Ex states I should have requested this day off. Ex is complaining no time with children during summer (ex has 24 summer days plus the other months) Ex also states the three day visit is too much for our children and I should go to NC to see them. My children never complain about this, but have heard Ex and others complain about this, they told me. My children were in counseling prior to this summer because they told Ex they want to live with me. I pushed for this to get my children help dealing with this divorce and wanting to live with me, not the EX.
My question does my EX have grounds for change of visitation just because I work on certain days when my children are with me? This has been an issue before and I spoke then with my attorney and he stated that when the children are with me they are in my care and Ex can’t dictate our life, so to speak, on anything unless emergency issues. My EX doesn’t work weekends and I am a nurse and my work schedule obviously isn’t like the EX’S. My children look forward to our visits and with the issues they have wanting to live with me, if the EX does accomplish this would devastate them. My children understand that I have to work and have no issue with this at all.
Please advise how to handle this, wait and see if he files, and not respond back to the email I recieved about this threat?

Thank You

No, your work schedule, unless it has changed in such a way as to affect the children’s best interests, is not a reason for your ex to modify custody.

Thank you Mr. Clarely for your quick response. My EX has an issue with the 3 day weekend visitation this is what the issue is truly about and will throw in everything else. Ex stated all complaints and wants me to give up visitation time and travel all the way in to NC. In the next line EX states wants my suggestions. I am fine with the court orders but should I respond back to EX? If I don’t and EX does file will it look like I am not willing to work with EX and best interest of our children? Side note, when our children couldn’t come for 3 day weekend (school make up days) I asked for more time, EX refused my request. Again, should I reply back or just ignore this email?

Thank You

Settlement negotiations are not admissible in court, you cannot be penalized for not engaging in the same. If you are happy with the current schedule you should not feel pressure to change it.