Sterility is possible. I had it happen with some of mine... I had two females who seemed to have very low fertility... One never produced a litter, and the other only had very small litters every other month or so. I figured it was because they were older (at least 6 months when they were introduced to the male), so I kept back two females from the one female's first litter... One of these NEVER produced, and the other gave small litters about once a month. Weirdly enough, their brothers (who went on to create colonies with female offspring of an unrelated colony) produced perfectly well. My point is, sterility is possible... However, if the "sterile" mouse's parents were both unquestionably fertile, I have serious doubts that this is the case.
Aside from that, illness seems to cease breeding. Found this one out the hard way, too, with a few colonies that stopped breeding completely shortly after developing colds (which developed into very smelly RIs before I gave up and gassed them).
Also, as has been mentioned, it could easily be the fault of your "changing it up". Once a colony is established, you should generally leave it alone... Most people recommend that the only new females you add be from the current females' own litters, and that a replacement male be, again, raised up from the females' litters (and the father retired once the replacement comes of age). It's likely that once you stop making changes to the colony, they'll get back on a regular breeding schedule for you.