Dealing with Exs husband re: my children

I am the custodial father and ex has every other weekend visitation. Ex and I had communicated about her picking the kids up at a later time Friday as my daughter had a special religious event until 8pm ( normal pickup is 6).We agreed on 8:30 pm as the time.
I got a voice message ( I dont keep phone on in the chapel) not from my Ex, but from Exs husband ( whom I have never had a single conversation with) telling me they wanted to get kids early an that I should call my Ex to tell her it was ok.
After the service which ran about ten minutes past 8, the kids used the restroom and we left. I called Ex and told her we would be about 9 or 10 minutes late, explaining why. She was very nasty and said I was supposed to be there ,etc.

At the house as I watched the kids get in her car , Exs husband gets out of her car and in front of the children tells me in the future if I am late I need to call , and then asked if I have a problem with that ( in a very confrontational manner).
All the while Ex is smiling and never says a single word to me.

I held my tongue and did not respond to him at all, not a single word. I addressed my kids and told them to have a safe trip , Exs husband got in the car and muttered “whatever”.

In my opinion he has no right to address me at all regarding pick up times or anything else regarding my children, especially when my Ex is right there to speak.
Secondly I had clearly communicated with my ex about the times and just 9 minutes prior had spoken with her which he was in her car so he was aware of it, again though it’s none of his affair, not with me at least .

Question: I felt very uneasy about his overbearing manner and want to know how to best handle it again should he try to insert himself or intimidate me again ? I do not wish to have any interaction with him again.

I’m not a lawyer but I don’t believe there is anything you can do about him speaking with you. However, if his behavior escalates to harassment then you may be able to take some kind of legal action.

End of the Line is correct, unless he makes a true threat there is not anything you can do about his presence at the exchanges, or his talking to you.