You’d be surprised. It’s not all that unique. There are simply some people who believe that their need for revenge outweighs the needs of themselves, their family, and their children. You are going through very similar to what I’m going through, except you are divorced.
It sounds as though your ex needs to control people…and she knows the one way to still get to you is through the kids. Now that you have a girlfriend, she is feeling replaced not only in your mind, but possibly even more importantly, in the minds of her children. This is what is triggering her to behave in this manner. When you offered to give up your rights, she saw that possibility of control going away and found a way to reinstate it by threatening you with “making you pay forever.”
Do not give up on your kids, for their sakes. Eventually, if you stay calm and professional when dealing with the mom, they will grow to see the situation as it is. Insist upon seeing your children. If the ex refuses, document that in writing and date it every single time it happens. Keep evidence that will later demonstrate to your children and to the courts that you made every attempt to maintain contact with them. (Even if you have to send letters to them certified mail, return receipt requested.)
Do not allow your ex to emotionally draw you in to fight. Document everything, all threats, all broken promises, all conversations. Document when the kids demonstrate behaviour that is worrisome. Develop a calendar, record conversations, put the majority of your communications in emails (save them with their headers, make backups). Treat every discussion as evidence.
Set boundaries with your ex and do not let her cross them. When she finds that a certain series of her actions are ineffective, she will try another route. Don’t give her the satisfaction of reaction, unless it is something you want. Reward your ex for good behaviour, punish her with non-reaction for poor behaviour. This will also help you with your GF as she probably feels like her life is being controlled by a 3rd party. (Keep the lines of discussion open between you two.)
As far as legal advice goes, if you can’t afford a lawyer, I’m not sure what to do, but I can tell you that if she draws you into court, have as much evidence as you can muster on your side.
As an aside…and this is just a hunch since it so closely parallels my situation, read everything you can on borderline personality disorder. Mediation rarely works with those who have this disorder. They will agree on things only to change their mind when their mood shifts or when a new trigger causes them to feel inadequate.
Good luck and God Bless.