Short and simple, your ex- is messing with your emotions, he knows he has the upper hand and you are weak for/to him.
Pity you broke up with a pretty decent guy; your ex- was doing his "alpha male" thing which is why he was glad the other guy wasn't around. I'd bet you that the other guy would've seen through him and your ex- didn't need your eyes to be opened to his mess.
You need to emotionally let go of him and move on with your life, for your sake as much as your children's. If you keep going the way you are the children will lose their father again and the blame will fall on you (even if it isn't wholly true).
Your children might need a father but you don't need him.