Custody/support- no legal orders

I have had physical custody of my son for 15 month, my husband was taking him 1 weekend a month… (but not consistently) now he is threatening to take custody of my son even though he has not paid any support for the past 15 month. Can he do this? How worried should I be? And do I have to make my son go with his dad on his up-coming weekend? My son is 13 years old.
I realize support and visitation are not tied together legally, but isn’t common sense and moral ethics considered?

If there is a custody order in place then he could file for a modification, and if there is no custody order in place then he could file for custody. If he filed he’d likely receive some visitation.