Once you complete an entire level of Mod Battles, then the next Mod Challenges will open up. After the first set of Mod Challenges, you will be limited to the type of characters you can bring in. Mod Challenges are the only way to get 4 and 5 pip mods, but they give out mods for random slots.
If you have unlocked a Mod Challenge, it is usually a waste of time to spend money on the 3 and 4 pip mods. Wait until you have a team that can 3 star the final Mod Challenge, then start using sim tickets to farm a whole bunch of those mods until you find the ones that you want.
There is a science to finding mods with the highest possible secondary stats, but that is beyond the scope of this document.