What’s so important about a pull request environment?
For web applications—where things change a lot—it might be difficult to pinpoint every change that causes an issue. It is also challenging to ensure the quality of a pull request before merging into an environment without both unit tests and automation tests.
We are a workflow automation company. We give users the capability to drag-and-drop actions in a workflow designer to automate complex business processes – without the need for complex code.
All the action happens in our workflow designer
It is also a place where many developers from different teams (and time zones) are busy programming. With so many contributors and teams working on the workflow designer, we have some legacy code which can make testing complicated.
Since so many of us depend on the workflow designer, any major breaking change affects a lot of engineers. All of this made it hard for our quality assurance team to pinpoint the specific commit or pull request that caused an issue. So, we needed a developer who could triage these issues and then pass it on to the appropriate developer who is responsible for it. This created a bottleneck and excess turn-around time to address blocking and high-priority bugs.
Even though we try our best to create small pull requests, but sometimes there are a lot of changes in one single pull request – which makes triaging a nightmare. It can be summarized with this meme I found on the internet (I do not own the rights to this).
So, we embarked on a journey to make testing more efficient before deployment. And it’s paying dividends in benefits that we share below.
Architectural overview of our workflow designer
Before we start discussing the pull request environment in detail, let us give you a high-level overview of the architecture of our workflow designer. If we remove all the security elements, proxies, virtual network, WAFs, etc. at a high-level our application is simple – with a react app deployed to a blob storage, fronted by a CDN which gets all the data from the APIs hosted on Azure Kubernetes. We authenticate our users using Auth0 and the APIs rely on the OAuth token from Auth0.
As you can see the basic infrastructure we use for our app is easy to replicate. For authentication, we need to generate OAuth token from Auth0 (i.e. any of our existing hosted tenants in test environment).
This inspired our team to spin up a secondary environment to manage every pull request which depends on the APIs deployed in our test environment (i.e. we didn’t need all the other teams to deploy their APIs to this new environment)
How we created a pull request environment
Our team configured another environment that has a similar structure but with a few tweaks and security features.
The idea was that our engineers will get a Auth0 token from an existing test environment, put it in a query string and hit a URL with the relevant pull request number. e.g. <CDN URL>/designer/pr/1234
As you can see above the infrastructure setup looks very similar to the one we have for our regular environments (again, for simplicity I have not included WAF and virtual network configurations here) but we have added a function proxy with Azure Active Directory (AAD) authentication so that only Nintex employees can access this environment.
After we had the basic infra setup ready, we had to tweak the pipelines to deploy every pull request to PR/<PR number> folder which was done using Azure pipelines and bash scripts using Azure CLI commands.
You can find the pull request number as a predefined variable (System.PullRequest.PullRequestId)
Once we had the basic infra ready, we had to tweak our application to receive the auth token from query string and for security, we made sure that this feature is enabled only for this new environment.
So, then we had everything ready to start testing:
- Changes in the app to accept auth token from query string.
- Changes in pull request validation pipeline to upload the built artifacts to Azure blob storage.
- Changes in infra to support hosting.
Our testers and quality assurance team could now generate a token from Auth0 (most identity providers support user level tokens) and simply use the pull request environment URL to access relevant PR (i.e. <pull-req-environment-url>/workflow_designer/pr/<pr number>?token=<generated JWT token>).
The benefits of this solution
We have been using this method for more than a year and we have seen quite a few benefits such as:
- Reduced cycle time of bugs – As I have mentioned before, triaging bugs in this part of the product has been a challenge, but now our test engineers can directly go back to the pull request responsible for the bug and work with the relevant developer.
- Preview demos – For more complex features, we can do a demo with our test engineers/ product owners/UX designers to get early feedback – even before deployment to our test environment.
- Automation runs – More complex features may cause regression issues and until now we did not have any way to test these features in an automated way unless we deployed them in our test environment. Now, we can run our automation suite and QA these features before deploying to test environment.
- Cost – One of the major drawbacks of creating a new environment is the associated infra and maintenance costs. But since this is just a new deployment of our react app which depends on the test environment for the API calls the infra and maintenance cost is minimal.
Since we have found this environment useful, we have extended the same concept to test the integration of unpublished internal node packages. The basic infra is still the same but we have enhanced the build pipelines to build the dependent package first (from its feature branch which is unpublished), copy the build output to the app which is consuming it (e.g. our workflow designer) and deploy this app to a similar environment.
This way we can now test the integration between our custom node packages and our consuming web apps without publishing a new version of the package.
Find more technology and engineering focused posts from me and my peers on the Nintex blog.