In this article, we share how and why the Nintex engineering team migrated our performance testing projects from Visual Studio (VS) Enterprise to StressStimulus.
Testing with Visual Studio
Here on the Nintex K2 development team, we use a testing environment known internally as “K2 Labs”. In this environment, we have been using Visual Studio since 2006 as our de facto load and performance testing kit throughout development life cycles, performance regression tests, and performance investigations.
Visual Studio’s performance testing features and capabilities have matured well over the years, even after the updated release versions of Visual Studio to the point where Visual Studio Enterprise was comparing well against other commercial Load & Performance testing tools available on the market.
However, in late 2018, Microsoft announced that the performance testing features will be deprecated in versions after Visual Studio Enterprise 2019, leaving us no choice but to look for an alternative load and performance testing tool going forward and so the search journey began.
How our research led us to StressStimulus
As we had more than a decade of test collateral build on Visual Studio web test scripts the first requirement for a replacement tool was one that we could use on top of what we already had, thereby avoiding having to redo hundreds of man-hours recreating test scenarios. Other key requirements were ease of use, same or more performance testing capabilities than Visual Studio, fair pricing, good compatibility with our code, and good product support.
Since Nintex acquired K2 software in October of 2020, we had immediate access to Jmeter and BlazeMeter for performance testing however as Jmeter and Visual Studio performance test scripts are built on entirely different test script types, it would have required a total rework of all existing collateral – which was an issue we were trying to avoid in the first place.
We evaluated many tools similar to Visual Studio including well-known products in the field and finally settled on a lesser-known tool called StressStimulus.
9 reasons why StressStimulus matched our requirements:
- Learning curve:
Being so similar in use and behavior the learning curve on the new tool is near flat so any new skill coming over with Visual Studio, Load Runner, WebLoad, Neoload, and the likes will be up to speed using this tool in no time.
- Converting existing test collateral:
Visual Studio and StresStimulus test scripts are fully interchangeable, thus there is no need to recreate existing test collateral and we continue as is.
- Client-side testing:
- Data-driven testing:
Using Jmeter with Selenium webdriver instances is known to be resource-heavy, which is a standard for cloud test resources. Data-driven tests in StressStimulus work exactly like Visual Studio in that the source parameter files are light flat files as part of the test project and used by the test controller, not the test agents injecting the load, be it cloud test agents or on-premises test agents or a combination thereof.
- Integration into APM solutions and pipelines:
Just like Jmeter StressStimulus can integrate with Dynatrace and all popular CI/CD pipelines.
Build-in rich full-featured automated reporting far superior to Jmeter and Visual Studio Enterprise.
- Server-side metrics monitoring:
Just like Visual Studio and Micro Focus Loadrunner StressStimulus collects standard Windows Perfmon counters on the target servers and correlates the metrics in real-time with the load test execution making it easy to track server resource usage inline with the test scenarios being executed (i.e. no need for a third-party tool to monitor the application during testing).
With commercial tooling comes full on support as part of the package and based on our experience we get exceptional customer service.
Compared to well-known competitive commercial offerings like Loadrunner, WebLoad, Neoload, LoadView & Neotys the license costing on StressStimulus is highly competitive and more flexible.
We recognize that adopting StressStimulus and veering away from Visual Studio Enterprise was a breeze and we will eventually phase out Visual Studio Enterprise for performance and load testing as we move performance testing into the DevOps team at Nintex.
Interested in exploring more engineering posts from our team here at Nintex? Click here to discover more.