“Why is the new version slower?” is a common, yet dreaded question in software development. To avoid it, you need to know if your software is getting slower, and ideally, when (in development time) your software’s performance changes. The earlier you can detect and address performance changes, the easier (and less costly) it is to address the performance changes.

Performance testing is part of the development process at MongoDB, integrated into our continuous integration system, specifically to detect and address performance changes as early as possible.

Our performance testing system has evolved over time:

• First, we focused on complete test automation, then we focused on test noise mitigation, next we shifted our focus to improving our analysis code.
• We reimplemented our entire analysis system to a fundamentally different (and better) one based on change-point detection. With the addition of the change-point detection, our performance test infrastructure became significantly more useful to us.
• Since that time we have added a number of incremental improvements to our system.

These improvements increased coverage, provided faster and more accurate signaling for performance changes, and helped us to better understand the state of performance. In addition to each component performing better, we believe that we have created and exploited a virtuous cycle: performance test improvements drive impact, which drives more use, which drives further impact and investment in improvements. Overall, MongoDB is getting faster, and we avoid shipping major performance regressions to our customers because of this infrastructure.