What is visual regression testing

Don’t let the fancy name fool you, visual regression testing is fairly simple. The aim is to find any unwanted visual difference that might have occurred in a web page during its development. A simple example of regression might happen when someone changes the style of a button in the ‘contact us’ page, and this changes styles in the home page. This unintended change is called regression. To avoid such scenarios, most website owners apply various strategies. There are many ways to detect such issues.

Until recently, people widely used an approach called functional testing. This means developers or QA Engineers write tests that validate the logic on the page and search for missing or unwanted elements. However, this approach ignores the visual part entirely, since it is not possible to test what users see programmatically. This means that software engineers ignore all issues related to styles (CSS).

In a web page, the most important thing is what the user sees. Correspondingly, this inspired websites to change their testing strategy to visual regression testing. In this approach, a baseline snapshot (screen shot) of the page is captured. Next, the test recaptures the same screen shot and compares the two images pixel by pixel. As a result, this type of testing has proven itself to be more accurate and close to the end user experience. Best of all it is entirely independent from the technologies used to implement the website.

Benefits of visual testing

Switching to visual testing can have various benefits, some of which not apparent at a first glance.

Test what the user sees not what the developer sees

Usually, people write tests against code bases, visual tests however are written against screen shots. This eliminates potential issues with wrong or missing assertions in the test code, incorrect test data, or simply bugs in the test.

Moreover, with functional tests you trust that QA engineers have written their tests properly, with visual regression testing you test what is actually rendered in the browser.

Lower cost for test maintenance

Visual regression tests are very simple in most cases, making them very easy to maintain. With other tests you need to understand the programming language in which the page is written, but with visual tests you only need to integrate with some visual regression testing tool/framework like Snapics.

Programming language independence

There are plenty of frameworks, languages and tools for implementing websites. It is simply impossible to know even half of them. Here are a few of the most popular ones: WordPress, Joomla, Django, Angular, ReactJS, Thymeleaf. On top of that you can implement the website from scratch using your favorite language, for example using JavaScript, Python, Java. With this in mind, the possibilities to be able to create a functional test that works with all sites are just too many, which is why there are still many website owners that stick to manual testing. Automated visual testing solves all of these problems.

Easy to see what is wrong

When a functional test fails you get a log saying something like “Unable to locate element with ‘id’ ‘some element id’” followed by code stack trace, which is hard to read even for the developer that wrote the code. As an illustration, with visual testing you see pictures like this:

Example result of visual regression testing.

you can immediately tell what is wrong and exactly which element has visual differences compared to the baseline image, in this case the font of one of the headers has changed.

No more manual testing

The main benefit of manual over the functional testing is that an actual person goes to the web page and examines it for visual errors. As a result, with visual testing you get the best of both worlds. Additionally, you have automated way of verifying the page functionality and you capture the visual differences.

Visual regression testing with Snapics

The goal of our service is to allow users with or without technical knowledge to test a website.
Following is a short guide on how to visually test your website with Snapics.

Create project

After we have logged in to the dashboard, we need to create our project. Projects are just a logical organization of tests.
Upon clicking on ‘Add New Project button’ you will see the edit project dialog:

Snapics visual regression testing - Create new project dialog
  • Project Name – should be unique for your account
  • Host – Base url for your website, in our case that is https://snapics.co
  • Desktop, Table, Mobile – controls the resolution for all tests under the project
  • Email notification for scheduled runs – if enabled you will receive emails with summary for each scheduled run that executed
  • Basic access authentication – provide these if your site is protected with basic auth
Create project video

Create test

Now that our project is setup we can add some tests to it, to do so navigate to the ‘tests‘ section and click the ‘Add Test ‘ button.
You will see dialog like this:

Snapics visual regression testing - Create new test dialog
  • Test Name – should be unique per project
  • Page Url – in our case ‘/blog’, meaning we are going to be testing https://snapics.co/blog/, where https://snapics.co comes from the project
  • Full page, Selector, Region – specify what portion of the page to be tested, full page for the whole page, css selector for one particular html element and region defined by x, y, width, height parameters, for our tests we will be using full page
  • Snapshot difference tolerance – how strict the image pixel comparison will be, 0% means the test will always fail even if the images are identical and 100% test will always pass even if the images are completely different. We will set this to 60%, because our page is not very dynamic and there are no animation or effects. For pages with more effects you should lower this number.
  • Add steps – steps are actions that is performed before the test snapshot is taken, example for such action would be close a pop up, or delete header with add in it. We don’t have such use case here, so we will skip this part.

Update baseline

In order to perform any testing we need to take our very first baseline snapshot.

To do so we can click on the update baseline snapshot on the test:

Snapics visual regression testing - Test without base snapshot

or update all baseline snapshots on the side menu with the projects

Snapics visual regression testing -  Side menu with projects, update all baseline snapshots button

Here is a short video on how to update baseline snapshots

After the update completes we can click on the test to view a dialog with magnifying glass to inspect the image, clicking in the dialog will open the image in new tab if that is more convenient to you.

Baseline snapshot for snapiscs home page

Test runs

After the run finishes we can view the results in ‘last run ‘ section.
In our case one of the tests failed the other one has passed, for every failed test we can see image that consists of

We have changed parts of the text in the blog post titles and you can see in the middle picture the highlighted pixels that differ, due to this change.
Again, we have the option to click on the image and observe it in any way we like, but in addition we have the ‘download all’ button, which will download all of the test images

View last run results video

Going back to the ‘tests ’, we have the option to configure scheduled runs, for every 2, 6, 12, 24 hours.

Start using Snapics now

Snapics.co is currently in beta and free, so you can create a free account here and try out our service. Also, feel free to contact us if you have any further questions or something is unclear.