Can dad threaten to take daughter?

I used Rosen to get help with my divorce about three years ago. At that time, my ex-husband and I went through the Wake Co court system and he gave me complete and total sole custody. It specifies in the contract that he sees her only at my discretion. This has worked fine so far but I am always concerned that he may come back and try to play her as his only trump card. Once he gave her up to me, wasn’t that it (unless I’m unfit, I mean)? Could he now come back and get partial custody? He sees her for about a couple of hours every other month. She is now six-years-old so what kind of say could she have in the matter? She doesn’t care to see him at all but I’ve been able to talk her into seeing him a little. We did not file any child support through the court system. I’m just so frustrated because I thought that filing everything through the court system eliminated him threatening to get her but I’m always scared that he could find some way around it.