If you don’t have a custody agreement or court order, then you both have equal rights to the children. He can take them and you can take them. Do you want to play tug-of-war?
I understand it’s upsetting and given your situation, money is non-existent unless you borrow from family. BUT the only way to remedy this is to file for child custody and quickly. If the courts know your ex took the children without your knowledge, AFTER you’ve had them for 2-4 years, then that will not look good on the ex. You need to tell them ALL your circumstances and history with keeping the children with you.
I do not know what the lack of housing and job/money will do with the short-term situation concerning custody.
I’m sorry you’re going through this. [:(]