Change of Custody

If you try for change of custody and lose, can the other party change visitation rights? Was told that this could happen and reduced time may occur. Why would this be the case when it’s not a major issue, been following for 4 years and would affect the children the most?
Don’t understand atty logic on this, can you explain? I understand things change over the years regarding children and visitation may have to be adapted in ways for them. Don’t understand why it can change if one loses. Seems non custodial can’t even try and still get slammed.

Thank You

Regardless of who brings the action, or why, the judge may make a new award of custody based on the best interest of the children.