Javascript for Testing

Using a general-purpose, ubiquitous language for testing has its limits. It may work for specific teams, but does it scale to your whole organization and enterprise portfolio?

Having developers work with test automation frameworks through a shift-left initiative or as part of a team collaboration has spurred the adoption of Javascript as a language for scripting automation frameworks. While Javascript (ECMAScript, ES6/7, etc.) is a versatile and ubiquitous language, its flexibility especially in domain-specific situations, is often a double-edged sword. 

Using a general-purpose, ubiquitous language for testing has its limits. It may work for specific teams, but does it scale to your whole organization and enterprise portfolio?

Reason for Topic

Many tools and platforms have been designed to allow users to write Javascript to implement domain-specific logic, especially in testing. While there’s nothing inherently wrong with using a general-purpose programming language to implement test scripts as automated checks, this impedance mismatch between the problem domain and the technical approach often runs the risk of the train going off the rails unless there are clear standards and skilled implementation and coding practices.  

Code-only testing may be fine if most of the tests (not just unit tests) are written and troubleshot by developers, but in larger teams and organizations it’s more common for testing subject-matter experts to take on the more elaborate (i.e. user experience, performance, security, etc.) testing activities, and this is where code-only testing approaches can often contribute to brittle and flakey test suites.

Introduction / Definition

A Very Brief History of Javascript

Javascript has been around for almost as long as the World Wide Web, the first version of ECMAScript officially released in 1997. Not only is it useful in browser-based apps to implement a wide variety of dynamic interactions with data and user interfaces, but it’s also become a mainstay in back-end services, particularly since Node.js rose in prominence over the past decade.  

Given its popularity, there are both many skilled and unskilled practitioners who author parts of systems and processes in Javascript. Many tools implement some form of a Javascript interpreter, even if the tool itself is often written in another formal programming language such as C, Java, C#, Python, or Ruby, in order to provide users a simple alternative to customization or rule definition authoring.

There are many reasons to love and hate Javascript. It was originally meant to be an interpreted language, meaning an engine receives a script, processes and then executes it. Modern and more formal Javascript packaging and execution runtimes provide some level of pre-execution validation, but much is left to the moments where the code simply doesn’t do what you expect it to. And while it complies with certain levels of standardization, it lacks the rigor of other more strictly defined languages, causing the amount of work an Integrated Development Environment (IDE) can do for a developer to be limited comparatively. 

This is in contrast to compiled languages, which often check and validate the code before it is executed by a runtime/engine. Not only are there performance tradeoffs to these models, but also security and stylistic implications as codebase grows. Javascript is also a prototyping language, meaning that many structures don’t have to be defined up-front; similar to dynamic language runtimes, this flexibility is both a blessing and a curse at times.  

Using a general-purpose programming language for domain-specific problems ultimately: 

  • Provides flexibility, but forces inherent complexity on users (which is like a leaky abstraction) 
  • Saves the tool/framework developers lots of time, since they can’t account for every future need, but not necessarily the users (which in this case is also developers and testers) 
  • Overshoots the skills requirement for grey-to-black box level testing; at some point, you shouldn’t have to be a (or the) dev to test what was developed, certainly by the time we’re testing the UI and business validity of features and apps. 
  • Forces future implementation options, typically perpetuating the same language/framework even when it grows stale or out-of-favor with teams

How Javascript for Testing Impacts Software Teams

So. Many. People. Know. It.

Almost every software practitioner knows or has been exposed to Javascript, either from school or through on-the-job experiences, at least enough to read it during diagnostic sessions. So long as it is incredibly straight-forward code, this can help multiple teams (such as development and testing engineers) collaborate to reduce the time to resolution when issues are identified by testing suites. 

Though some level of Javascript knowledge is now ubiquitous in the industry, it is often not what many team members write their enterprise apps and services in (unless we’re strictly talking about web front-ends and Node.js back-ends), so there’s always a bit of thrash bouncing back-and-forth between languages. In part, this is to be expected in any software delivery process, especially in distributed complex systems and cloud-native technologies where practitioners must know a dozen technologies just to be effective. But especially when everyone is affected by tests that sometimes stop working, break the build, aren’t themselves verified, and now anyone on the team has to debug something someone else wrote…it can get tricky, quickly. 

The choice of language for code-based tests really must depend on the technological ecosystem and appetite of a team, as well as the likelihood of the test assets being handled by other less-skilled practitioners. Also, how mature the organization’s testing practices are impact the velocity at which all-code testing approaches produce useful feedback to developers and ultimately decision-makers. 

But just as a thought experiment, would you do the same thing with C? With Java? Some have. Does all testing activity benefit from being manifested as code? 

Many proprietary testing tools use a mix of standards-based technologies such as Git, YAML, XML, OpenAPI, W3C WebDriver, and Javascript within a framework that enables code-based snippets where necessary…not everything-is-code by default. This results in proprietary project formats for testing assets, and though often open in nature, are not perfectly portable…just like tests written in Javascript on a particular testing framework. So whether your tests are all-code or on some proprietary format, it’s a zero sum game from a portability perspective. 

Even though almost every software practitioner knows Javascript, that alone doesn’t make it the best option in all (or even many cases). Teams need to think carefully about how much code they’re writing that doesn’t need to exist. When unnecessary code is produced, it often becomes a maintenance mess later on.

So. Many. Options.

In the universe of Javascript web testing (frameworks and languages), there are a plethora of options. Here are just a few of the most common: 

A pseudo-equation to help understand how complex Javascript for testing can become in an enterprise context is as follows: 

[likelihood of complexity] = [all the framework options] + [all the plugins] + [all the styles]
                                     + [all the teams in an organization] + [all the variations of skill over time] 

At the core, a great testing framework accelerates the busy work of constructing, proofing, executing, and maintaining tests. Frameworks often include additional supportive capabilities such as mocking, data connectivity, element selector or object repository mapping, extensibility for customization, and thorough data export and reporting options. A great testing framework should also come with thorough documentation and tutorials, a community (or at least forum) of support, and examples 

It’s unfair to judge a automated testing framework on cultural and people issues, yes? Maybe a little, but without thinking about the impact on the people and process components of your software delivery strategy, blindly introducing or perpetuating a technological choice can lead to unexpected outcomes. 

Drawbacks / Gotchas

A key problem with using a general-purpose programming language in domain-specific contexts such as in functional, performance, or security testing is that it’s very difficult for a framework author to provide the proper amount of ‘safety guardrails’ that might otherwise be easier to implement with Domain Specific Languages (DSLs). With great power comes great responsibility, and the immense flexibility of general-purpose programming languages when applied to specialized problem domains often allows new or novice practitioners to end up either incorrectly addressing their domain,… 

Security is also a critical consideration when accepting an everything-is-code approach to testing. Inside malicious acting (unless thoroughly sandboxed) as well as external (library vulnerabilities that are not mitigated continuously) can represent dangerous channels and vectors of attack via Javascript. Exploits to Javascript are being revealed almost daily, everything from embedding malicious Javascript in SVGs (images) to arbitrary code execution in VM2 (used by many core engines, including Node.js). Imagine a library your developer used in a bunch of system tests sending user account data outside the organization. The more we use blunt instruments (such as general-purpose programming languages) for every problem simply because it’s known and available, we increase this possible surface area of attack. 

The biggest drawback of Javascript for testing is that there is no standard, no common way to implement it, no consolidated learning path, no inherent support system…unless an organization (not just a single team) seeks maturity about their overall quality engineering approach and testing practices enough to put in the time, money, and people to actively develop these things. 

Summary

Though there’s a ton of examples of it out there, Javascript for testing is ultimately a choice. With every line of code a developer or tester writes, does it contribute lasting value to the organization or simply get a job done and add technical maintenance debt to the pile? Should these choices be made in a practitioner or leadership vacuum, or should organizations collaboratively decide how and what technologies are used to implement business goals? You might find that challenging assumptions about how testing is implemented also give rise to even more important questions about timeliness and usefulness of feedback provided by these tests to the overall software delivery process as well.