Setting up pull-request review environments

Say you’re a developer working on a feature, and you want to get early feedback from other teammates. How would you go about it? Most companies don’t have a good process for this, and it ends up crippling their ability to ship features and iterate.

The process at companies without review app environments:

An engineer works on a UI feature that she wants to get some early feedback on how it looks and feels.

She would have to ask people to come and look at her changes on her development machine.

Or she would have to ask others to pull down her code to see it, which would require other people to have a development environment setup. Of course, this would only be a minor nuisance if the other person was a developer. What if they aren’t a developer? What if they’re the product owner, QA, or another team? After going through all this setup, you still end up with a development environment. You lose all the benefits of having a staging environment that closely resembles your production one.

Or she would have to deploy to your company’s staging environment just to get feedback on her work in progress.

Or she would get feedback on just the code portion of her branch while in PR review. Once approved, she would merge her branch into master. Then deploy it to your team’s staging environment to verify her changes live. Found bugs and visual defects require her to make adjustments and go through the code review and PR process again. You have buggy code that should never be deployed on your master branch.

Not having review app environments is expensive:

• Master can be in a suboptimal and undeployable state, which violates continuous delivery.
• The later in the development cycle a defect is found, the more expensive it is to fix. You will miss bugs and errors that you could have caught much earlier.
• Your team can iterate faster with shorter cycle times. A product owner can see a developer’s work far sooner and without requiring a code review.
• Catching and isolating bugs is more complicated when a staging environment contains multiple sets of changes. For example, let’s say you only had a single staging environment, and there are numerous changes since the last deploy. Your QA team finds defects. How do you know which commit introduced the error?

Setting Up Review Apps

There are a lot of different ways to go about setting up review apps for your team. I’m going to show you a high-level, simple setup that you can choose to adapt to your workflow. I’ll start by assuming you already have some sort of continuous integration (CI) service and are using Github along with docker.

First, you’ll need to set up a server that would run the review apps.

Pick a reasonably sized machine to run your review apps and install docker and nginx on it. You’ll also need to point a wildcard subdomain to your server. Nginx will be responsible for doing the reverse proxying of requests from the wildcard subdomain to the correct container.

Next, you need to configure your CI service. Your CI service should be hooked up to Github’s Pull Request events so that when a developer creates a PR, your CI service should be triggered to test and build your application into a docker image after your tests pass. Push the image to a docker registry.

During the build, you will need to populate a file with some information about the build that your review app will later need. For example, if you want your review app subdomain to have the branch name, you need to include that at build time. I tend to put in the SHA of the artifact, the build date, and the CI build number. The file that you end up writing should be easily parsable. I recommend using YAML over JSON purely for ease of writing and editing. You can then trivially write shell scripts that emit the key-value pairs you need. The build information you include will be used for the subdomain scheme you want for each environment. For example, https://<branch>-<ci-build-number>.yourdomain.com.

After building the artifact, you’ll want another step in your CI process that deploys your review app. This step would pull the docker image from your registry onto the staging server, and run a script that starts the docker container and hooks it up with nginx.

I’m a fan of using ansible because it’s relatively simple and is a step up from using shell scripts. If you’re using ansible, what you want to do after getting the artifact on the staging server is to start the image. Then you’ll need to configure nginx. Render a template for this review app’s nginx server block. You’ll want to add a block to reverse proxy the subdomain to your container’s randomized port (to avoid conflicts with the other containers).

Render your template with the variables substituted into nginx’s site conf directory. On Ubuntu-based machines, they should be in /etc/nginx/sites-available/. Next, symlink that template you just rendered to /etc/nginx/sites-enabled/ and reload nginx.

If all goes well, you should be all set to be able to hit your review app at https://<branch>-<ci-build-number>.yourdomain.com.

Enhancements

Depending on your needs, you may choose to do a few enhancements and customize your team’s review app workflow.

Github Deployments

Github has a deployment API that lets you manage the lifecycle of your deployments on a pull request page.

The deployments API makes it easier for you and your team to visit your review environments.

Destroy Inactive Deploys

Teams that want to save resources will often opt to delete review environments after a period of idleness automatically.

You may want to reap inactive review apps that haven’t been updated for a few days and re-create them when the author updates their PR branch.

After Review Apps

Review apps would automatically give you a separate, disposable, staging-like environment whenever you open a PR. When you make changes and update the branch associated with the PR, the associated review app is updated, and the old one is destroyed.

Your team can easily share their work in progress with other developers, product owners, QA, or stakeholders by merely opening a pull request. Everybody can then visit the review app’s URL to see it. Your engineers can start getting feedback on the feature they’re working on from product owners while at the same time getting their code reviewed. If it turns out the feature you’re building wasn’t what you imagined, you can quickly demo and scrap it before going through a formal code review process. If there’s a bug or something feels off, you can address it before the code is merged into master.

Smaller deploys allow you to verify and test features independently. You can have multiple product teams working in tandem with each other since you can now verify features independently of each other. If you deploy various features from your master branch to staging at once, you can’t quickly determine which feature or commit introduced a bug. Multiple review app environments help you avoid getting into awkward situations where:

• You have to wait for a deploy of new code to the staging environment to see it.

• Others are testing the staging environment so you can’t deploy and verify your changes, delaying progress on other new features.

• You have to coordinate database migrations on your staging environment. Review apps let you test your database migrations however you want. You can have a copy of your staging database and run your migrations and new code against it. At the same time, somebody else may have different migrations in another environment. There’s no need to do any coordination or do awkward rollbacks.

Automating and making staging deploys a regular part of your process reduces overhead while leading your team closer to a continuous deployment/delivery process.

Since each review environment would have its subdomain, you can now run acceptance tests against each branch before merging. Get insights on how the new feature affects performance, run SEO audits, do user testing, etc. Catch exceptions and errors before they make it to production.

If you want to chat more about review apps or need help setting up review apps at your company, send me an email.

---