forgejo/tests/e2e/README.md

End to end tests

Thank you for your effort to provide good software tests for Forgejo. Please also read the general testing instructions in the Forgejo contributor documentation and make sure to also check the Playwright documentation for further information.

This file is meant to provide specific information for the integration tests as well as some tips and tricks you should know.

Feel free to extend this file with more instructions if you feel like you have something to share!

How to run the tests?

Before running any tests, please ensure you perform a clean frontend build:

make clean frontend

Whenever you modify frontend code (i.e. JavaScript and CSS files), you need to create a new frontend build.

For tests that require interactive Git repos, you also need to ensure a Forgejo binary is ready to be used by Git hooks. For this, you additionally need to run

make TAGS="sqlite sqlite_unlock_notify" backend

Install dependencies

Browsertesting is performed by playwright. You need certain system libraries and playwright will download required browsers. Playwright takes care of this when you run:

npx playwright install-deps

Note

On some operating systems, the installation of missing libraries can complicate testing certain browsers. It is often not necessary to test with all browsers locally. Choosing either Firefox or Chromium is fine.

Run all tests

If you want to run the full test suite, you can use

make test-e2e-sqlite

Interactive testing

We recommend that you use interactive testing for the development. After you performed the required builds, you should use one shell to start the debugserver (and leave it running):

make test-e2e-debugserver

It allows you to explore the test data in your local browser, and playwright to perform tests on it.

Note

The modifications persist while the debugserver is running. If you modified things, it might be useful to restart it to get back to a fresh state. While writing playwright tests, you either need to ensure they are resilient against repeated runs (e.g. when only creating new content), or that they restore the initial state for the next browser run.

With the playwright UI:

Playwright ships with an integrated UI mode which allows you to run individual tests and to debug them by seeing detailed traces of what playwright does. Launch it with:

npx playwright test --ui

Running individual tests

npx playwright test actions.test.e2e.js:9

First, specify the complete test filename, and after the colon you can put the linenumber where the test is defined.

With VSCodium or VSCode

To debug a test, you can also use "Playwright Test" for VScodium or VSCode.

Run all tests via local act_runner

If you have a forgejo runner, you can use it to run the test jobs:

forgejo-runner exec -W .forgejo/workflows/e2e.yml --event=pull_request

Run e2e tests with another database

This approach is not currently used, neither in the CI/CD nor by core contributors on their lcoal machines. It is still documented for the sake of completeness: You can also perform e2e tests using MariaDB/MySQL or PostgreSQL if you want.

Setup a MySQL database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql

Running individual tests

Example command to run example.test.e2e.js test file:

Note

Unlike integration tests, this filtering is at the file level, not function

For SQLite:

make test-e2e-sqlite#example

Visual testing

Warning

This is not currently used by most Forgejo contributors. Your help to improve the situation and allow for visual testing is appreciated.

Although the main goal of e2e is assertion testing, we have added a framework for visual regress testing. If you are working on front-end features, please use the following:

• Check out main, make clean frontend, and run e2e tests with VISUAL_TEST=1 to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert it passes.
• Check out your branch, make clean frontend, and run e2e tests with VISUAL_TEST=1. You should be able to assert you front-end changes don't break any other tests unintentionally.

VISUAL_TEST=1 will create screenshots in tests/e2e/test-snapshots. The test will fail the first time this is enabled (until we get visual test image persistence figured out), because it will be testing against an empty screenshot folder.

ACCEPT_VISUAL=1 will overwrite the snapshot images with new images.

Tips and tricks

If you know noteworthy tests that can act as an inspiration for new tests, please add some details here.

Run tests very selectively

Browser testing can take some time. If you want to iterate fast, save your time and only run very selected tests. Use only one browser.

Skip Safari if it doesn't work

Many contributors have issues getting Safari (webkit) and especially Safari Mobile to work.

At the top of your test function, you can use:

test.skip(workerInfo.project.name === 'Mobile Safari', 'Unable to get tests working on Safari Mobile.');

Don't forget the formatting.

When writing tests without modifying other frontend code, it is easy to forget that the JavaScript test files also need formatting.

Run make lint-frontend-fix.

Define new repos

Take a look at declare_repos_test.go to see how to add your repositories. Feel free to improve the logic used there if you need more advanced functionality (it is a simplified version of the code used in the integration tests).

Accessibility testing

If you can, perform automated accessibility testing using AxeCore.

Take a look at shared/forms.js and some other places for inspiration.

List related files coverage

If you think your playwright tests covers an important aspect of some template, CSS or backend files, consider adding the paths to .forgejo/workflows/e2e.yml in the path filter.

It ensures that future modifications to this file will be tested as well.

Currently, we do not run the e2e tests on all changes.