Personally, after Reimu and then Remillia, the game kind of balances itself out, with a couple of odd situations (not really broken fights, just unique situations). I honestly didn't have much trouble past SDM, as from there you start getting access to alot of really strong late game options, such as Sakuya, Remillia, Flandre, Aya, Hatate, Momiji, to name some right at or after SDM. You can also pick up a Mokou in Bamboo Forest, a Youmu in Myouren, or even some strong 80 costs even earlier (namely the three fairies of light). Don't forget that cost determines level rate, as well as general stat allotment, so if you're having trouble honestly, switching your team up might be the better option. The E4 have good teams and good builds/stating, so you may want to make sure to change earlier than later if you feel you need to.