trezor-suite/suite/e2e/docs/e2e-playwright-suite.md

@trezor/suite-desktop and @trezor/suite-web e2e tests

@suite/e2e uses Playwright to run e2e tests. It also uses trezor-user-env which is daily built into a docker image providing all the necessary instrumentation required to run tests (bridge and emulators).

Run it locally

Note: All paths below are relative to the root of trezor-suite repository, if not specified otherwise.

Common

• Docker
• macOS only: XQuartz (to share your screen with Docker)
• Trezor user env
• No other instance of Suite or trezord service is running

Full steps:
(in case of Linux with X11 support, skip to step 6.)

1. Run XQuartz. Wait till it is launched. Leave it running in the background.
2. In XQuartz settings go to Preferences -> Security and enable "Allow connections from network clients".
3. Open a new terminal window (not in XQuartz) and add yourself to the X access control list:
   • xhost +127.0.0.1
   • You will probably need to logout/login after XQuartz installation to have xhost command available.
4. Run Docker and go to Preferences -> Resources -> Advanced and increase RAM to at least 4GB. Otherwise, the app during tests does not even load.
5. In the terminal window, set two environment variables:
   • export HOSTNAME=`hostname`
   • export DISPLAY=:0
6. In terminal window, navigate to trezor-user-env repo root and run ./run.sh.
7. In workspace packages/suite create a .env file according to the .example.env

Web

1. In another window, run web Suite with yarn suite:dev.
2. In a third window, run yarn workspace @trezor/suite-e2e test:e2e:web.

Desktop

1. TEST_BUILD=true yarn workspace @trezor/suite-desktop build:ui

   Produces suite-desktop/build directory with javascript bundles & assets in production mode for the electron-renderer process. TEST_BUILD env variable serves to mock bundled message-system config .

   Note: This step needs to be repeated on each change in suite or suite-desktop-ui package.

2. yarn workspace @trezor/suite-desktop build:app

   Produces suite-desktop/dist directory with javascript bundles & assets in production mode for the electron-main process.

   Note: This step needs to be repeated on each change in connect or suite-desktop-core package.

3. yarn workspace @trezor/suite-e2e test:e2e:desktop

Troubleshooting

1. To run tests headed (showing UI) you can add: --headed.

2. To run just one test file you can do: yarn workspace @trezor/suite-e2e test:e2e:web general/wallet-discovery.test.ts or yarn workspace @trezor/suite-e2e test:e2e:desktop general/wallet-discovery.test.ts

3. To run just one test you can add: -g "Basic cardano walkthrough"

4. To run on a specific device model you can only run the device specific project i.e.: yarn workspace @trezor/suite-e2e test:e2e:web --project=T3T1 general/wallet-discovery.test.ts

5. To run tests with canary FW you need to use canary yarn script i.e. yarn workspace @trezor/suite-e2e test:e2e:web:canary general/wallet-discovery.test.ts

6. To open advance playwright runner/debugger ui you can add: --ui

7. To check for flakiness you can specify test/suite and how many time it should run: --repeat-each=10

8. To check for flakiness on CI you can edit in packages/suite/package.json script "test:orchestrated:e2e:desktop": "NODE_OPTIONS='--no-warnings=DEP0040' yarn xvfb-maybe -- pwc-p --config=./playwright-config/playwright-desktop-nightly.config.ts --repeat-each=30 --grep=<test-file-name>",, commit, push and run this CI against your branch. This will run this one test 30 times. Please, always specify limited number of tests, never run full suite 30 times.

9. To debug test add await page.pause(); to place where you want test to stop. Debugger window will open. This works only in --headed run.

10. To enable Debug Tools in the browser press Ctrl+Shift+I

11. To enable Electron verbose logging add env variable LOGLEVEL=debug or any other level

12. To increase test timeouts when your local run exceed 1m limit, you can specify test timeout override in packages/suite/.env. (UI runner --ui needs to be restarted to reflect the change in .env)

13. To find a breaking commit in develop you can checkout latest develop and run yarn workspace @trezor/suite-desktop git:bisect <last_good_commit> <desktop|web> <test_file>

Contribution

Please follow our general Playwright contribution guide

Tags

Each test must be assigned a tag according to what device model it is supported by the test i.e.

• @T1B1
• @T2T1
• @T3B1
• @T3T1
• @T3W1

At the moment, there are these additional tags:

• @smoke
• @desktopOnly
• @webOnly
• @nightlyOnly

@smoke

Tests belonging to a smoke set are executed on all supported devices in PR pipelines. Otherwise only T3W1 is used.

@desktopOnly or @webOnly

Some tests are only applicable for Desktop app or Web and you can use this tag to notify the runner, that the test should be ignored when running against opposite Suite. This negative filtering is done on playwright-config level.

Currently, we are also applying @webOnly as a positive filter on Web PR runs. This is done in Web PR workflow definition. Meaning, Web PR runs execute only tests with @WebOnly tag to reduce amount of test run daily and save quota on Currents, where we are paying extra for any test runs over 100 000.

@nightlyOnly

Some tests should run on nighty runs only for specific reasons. This tags is used for reversed filtering in PR and release workflows definitions. This means tests with this tags will run only on nightly and canary runs.

Results

Results contains traces, metadata, logs, screenshots, videos and various useful information for debugging. Traces contain electron logs of our desktop suit app. Both in CI, currents and local env (suite/e2e/test-results). CI runs have artifact with logs from trezor-user-env. They exist per group. Log are separated by Log entry - - - STARTING TEST trading/swap-tokens.test.ts and - - - FINISHING TEST trading/swap-tokens.test.ts

• electrum-regtest.txt
• trezor-user-env-debugging.log
• tenv-emulator-bridge-debugging.log

Currents.dev

Test reports are uploaded to currents.dev