First of all, the threats to kick you out need to stop. I know for a fact that the military–regardless of which service–will not tolerate their soldiers exhibiting such behavior. You can contact his CO if the threats continue. If you live on base, you might be able to work with the legal department on base about housing. He can’t just KICK you and the kids out. Look into your options as a military spouse.
And no, you’re not being evil…it’s called surviving. Do what you have to do. That, my dear, is up to you. Good luck [:I]