Is this contempt?

My ex and I have a custody order that outlines a very specific visitation schedule. For at least 5 years, the ex has not exercised visitation as outlined in the order. He actually exercises far less visitation than is indicated in the order.

Is he in contempt?

No, the order gives your ex the right to exercise visitation, not the obligation. He would be in contempt if he refused to return the children to you for your time.