In this situation you have the option of going back to court to enforce the original court order since that is what the courts will use. Your ex would probably then argue that you and he came to a verbal agreement changing this. If you made a verbal agreement with him then it shows that you can work together and if he files to modify custody, it may show the courts that joint custody would benefit the child and that you and the ex could make it work.
You should consider your reasoning behind this though. You have no control over what your ex tells the daughter, regardless of whether or not you have that child 24/7. The courts are not going to care about the new wife’s actions towards you or whether or not you question the ex’s judgements. If you question his judgements, maybe you shouldn’t have agreed to let him have more time.
The court will look at this as that you have altered the agreement verbally and though it has no legal standing in reality they would likely want to know why you no longer agree to this. The courts will see that a father wants more time with his daughter and that you agreed but only until he remarried. The best situation for any child is to have both parents as involved in their lives as much as possible. You and his new wife do not have to have a relationship, but you should give your daughter a chance to have a relationship to someone who is now related to her.
Just my opinion though…