If your custody arrangment is by agreement and not a court order you would need to file for primary physical custody.
I’m not certain that a move for a better job would be the sole deciding factor for granting primary custody. It would have to be in conjunction with other factors that are in the child’s best interest.
EDIT: There may have been a detail that you left out. Your subject says job loss but you did not specify that to be the case.:EDIT
You can still retain joint physical custody even if one parent has the majority of the time. The custody definitions are determined by whatever the party decide they mean. You could have joint physical custody with equal time, joint physical custody with one parent having the majority of the time. You can have primary physical custody with the other parent having visitations twice a week and every other weekend (almost equal)…
Would your ex consider revising the current custody arrangement? Is your son old enough to have an input and would your ex take his feelings about it into consideration?