Can I move out of State?

Long story short, I have been divorced for almost 2 years. I have sole legal/physical custody of our child, and ex hasn’t seen our child in over 2 years (by his own doing). I live in one state, he lives 3 states away. IF the opportunity should arise for us (me and child) to move to another state, is there anything that the ex could do to stop it? Like I said, I have sole legal and physical custody, and ex hasn’t attempted to see the child in over 2 years (he’s a drug addict who refuses to do what the court ordered to get a visit), so I’m thinking that a move to a different state would not make his visitation any less likely or possible since he has to travel by plane to see the child now as it is, and has not attempted to see the child in over 2 years anyway.

I currently have no plans to move, but it is something that could be on the horizon in the next couple of years, so I want to make sure I have my ducks in a row before it becomes reality. Also, would the answer be any different if ex does start traveling to our current state to see the child in the next 2 years (I don’t foresee that happening, but hypothetically)?

If you have a custody order that gives you sole legal and physical custody, there should be nothing that could stop you from moving other than the father filing a motion to modify custody.