Versioning (storing, signing, and promoting) tests alongside code is an approach to maintaining parity and cadence of change between project assets.
Tests are meant to verify and validate app code, and when things change, modifications of either app or tests should be versioned together. Source Code Management methods such as trunk and branch-based development support this methodology, but only if teams retain BOTH app and test assets in the same repository or with strict controls between sibling repositories that enforce parity in versioning strategy.
Versioning (storing, signing, and promoting) tests alongside code is an approach to maintaining parity and cadence of change between project assets. When developers change certain parts of an app, either due to introducing a new feature, fixing an existing bug, refactoring the codebase, or changing infrastructure or configuration elements, these are not the only assets that are necessary to delivering the thing to users. There are also tests, configuration, metadata, telemetry, notifications, and work-tracking status that all need to be considered because of that change.
It is far too often that, when changes are made, existing tests break. Why? Well, tests usually verify outcomes of code, and as such, often mirror assumptions and assertions of the very thing they are meant to test. Not all changes result in the need to change tests, so it’s not like a developer or test engineer knows exactly which tests need to be re-run and what is a superfluous activity. Worse, there is no such thing as “perfect coverage” or often even reliable association methods between which tests cover which code, and in modern distributed systems, this only gets worse. Distributed dependencies are often managed by other teams or even external 3rd parties, so when you make a change, you can’t always re-run their tests to prove that your stuff didn’t break a downstream dependency of theirs.
At the end of the day, the best you can do is to maintain your own level of excellence over your own test suite, balancing whatever level of coverage your team has agreed to using unit, functional, integration, performance, security, and other testing approaches to ensure that any issues that might arise from your change are caught as early as possible.
One practice that teams espouse to accomplish this is to have all their tests for a particular project, app, service, or component in the same Source Code Management (SCM) repository.
A code repository (repo) is like a bucket of files with version history on everything, the most popular of which includes Git, the underlying technology that enables teams to have local versions of a common bucket and make changes to that repository in a manner that is easy to synchronize between each other. There are many platforms that use Git as the paradigm and technology under the hood, such as GitHub, Gitlab, Bitbucket, and AWS CodeCommit, and provide added features and capabilities to increase collaboration and efficiency while teams contribute to shared repositories.
Tests versioned alongside code is a way to apply all the benefits of proper source code management (such as versioning, auditability, promotion paths, branching, etc.) to testing assets and practices. Another benefit of keeping all tests in the same place as project code is that anyone, at any time, can as the question, “where is the test for this functionality over here?”, and not have excuses like “it’s somewhere else that we don’t have access to, so I assume it’s covered”.
Test assets in the same repo as project code don’t even have to use the same technologies as the app or service itself. You can write your API or app in Java or React Native, but author your tests in another technology. Test authoring and modification doesn’t have to even happen in the same commit or pull request as the project code change, though the closer related to the change, the better from diagnostic troubleshooting and auditing perspective.
Historically, certain types of testing (such as UI functional, API, performance, and security) have been treated as separate practices than the testing done in preliminary stages of development. Most teams already espouse storing their unit tests in the same repo as the test code, in part because the target of unit testing is the code-level functionality and interfaces, not necessarily what the code manifests as UI or service implementation surface area (API endpoints). They also often store other project assets, such as small images, stylesheets, reference or static data, changelogs, and other resources that may or may not ultimately appear in the final packaged version of the project.
The larger the asset is, the more unlikely you are to see it in the repository. Why? Because synchronizing large files is time and bandwidth consuming. Though Git has support for Large File System (LFS) objects, teams often shun the inclusion of big binaries such as compiled dependencies that can be regenerated by IDEs and compilers, or source images that take up tens or sometimes hundreds of megabytes. Code repositories should be just that…code…not everything and the kitchen sink too. This mindset leaves some types of assets to need their own types of repositories, such as built container images, audio/video files, and other large proprietary resources (this is where traditional test tools fail us).
There are also cultural reasons, some for real reasons but most for not. Often conversations with development teams about working with external Quality Assurance (QA) teams goes something like this:
They use their own testing tools that aren’t compatible with our SCM platform; since it’s not our job to do that and we can’t dictate they use ‘good’ tools, they should just keep their testing suites out of our bread and butter (codebase).
They have their own testing and asset modification cadence, and all their test-related changes are noise to us (developers); plus, our build system gets confused about what changes need to trigger which tests, so we just keep the code and unit tests together and keep everything else out of the mix for simplicity’s sake
They don’t follow our [developer] “best practices”, such as few files per commit or well documented pull request (also, we don’t document our processes very well, but that’s another matter); since they don’t code, they don’t understand us who do code
We don’t trust them with the source code, and if the tests and source code are both in the same repository, there’s no way to exclude the code from the test assets in Git
…so, let’s just keep to our own repos, okay?”
But it’s often not okay to have many repos for the same logical thing. Why are assets, code or tests changing independently of each other? Why have two buckets for things that are directly related to each other? The fact is, if it is worth developing something, it is equally worth making sure it works as expected. Project code (the things that make up the app or service) and tests (code-based, code-less, or otherwise) are two sides of the quality engineering coin. Neither can result in a working, safe, and reliable outcome without each other.
There are very few legitimate reasons for having project code and tests separate, though some might try to put up straw-man arguments for it simply because it is currently the “way things work.”
One of the most prominent excuses encountered is of access and permission control, often the case between app/service development teams and a separate “DevOps” or infrastructure-as-code team. In this anti-pattern laden story, you often see deployment scripts in a separate repo from the app/source code being deployed because “developers don’t know what it takes to properly deploy an app”. Then version numbers get out-of-sync because the app code changes its output, but the infrastructure-as-code packaging still references the previous version number and does not deploy the updated version of the code.
At the core of this issue is trust between teams, such as dev to QA or DevOps to dev, but at the end of the day, trust issues aren’t resolved by roommates splitting up groceries and storing them in separate refrigerators (okay, sometimes that works), but by exercising self-control, establishing safety guardrails to prevent accidents from happening, and building rapport about cultural and procedural norms.
All testing activity should be visible to development teams, and similarly, all changes to app/service code should be considered as context during all types of testing. Even if the person doing the testing does not know exactly what the code-under-change is doing, it is often easier to fault isolate (e.g., “root cause”, “big contributing factors” analysis) with a real-time change log at hand than simply tests that just fail. This is why you see many testing platforms and tools starting to incorporate ‘signals’ into their failure analysis and reports. Change logs, service and system telemetry, traceability data, historical failure trends…all these things improve the speed at which software teams (developers, testers, or otherwise) can diagnose the core of issues, fix them, and deliver to production users rapidly.
At first, testers who are unfamiliar with the rest of the repo may ignore simply what’s going on in the project codebase. You will likely need to include people in your SCM, set appropriate permissions, and possibly come up with special segmentation tags or conventions to retain efficiency swim lanes between development work and testing-specific work. Over time though, the context of activity as well as read-only view of changes (sometimes just the dev change log is enough), will elevate the team’s testing approach and ability to improve the right feedback loops in the right places.
One final benefit of tests versioned alongside code, and this must include infrastructure-as-code or anything related to reproducing the working software (usually in its own environment), is that the final deployed (not released to users) result is verifiable via automation…you know, your tests! If you have good coverage across multiple aspects of software quality, and all the tests have passed, it’s easier to have confidence that the final results are ready for production users.
Though there are many benefits of versioning tests alongside project code, there are a few challenges that must be addressed for this approach to truly shine:
When teams are aligned in a mission and in planning cycles, the results are tremendous. Sharing the same SCM repository for project and testing assets ensures that changes in one area are visible to everyone, that test requirements and coverage is considered as part of development work, that testing activities inherit benefits such as asset versioning, traceability, and promotion, and that test failures are resolved as quickly as possible to keep delivery on track. Tests versioned alongside code is modern, widespread practice in many teams for these and many other reasons.