Playwright Day 5 - Grouping Tests

Day 4 was supposed to be about grouping tests, but I got sidetracked debugging a resource issue. Still very valid experience, but now that it's done I can wrap up test grouping, then move on to CI integration and IDE integration.

I've created a series of tests using the organizational structures of folders, files, test names, and tags. Today I'd like to play around with running different combinations of tests by calling for different permutations of these organizational structures.

Here's the table of tests:

In addition to these tests, I have a "test template" file with 10 tests in it.

For some reason, running all tests with this file alongside the others causes 84 tests to be run

Running 84 tests using 4 workers
              84 passed (1.7m)
            

But without it, only 18 tests.

Running 18 tests using 4 workers
            

No problem, I can check the report and see where the extra tests are coming from.

Huh. I moved the 10-test template file back into the tests directory and now I'm getting the correct number of tests

Running 28 tests using 4 workers
            

After moving the test around and eventually using this command

$ npx playwright test --list
            npm notice run 003@1.0.0 npx
            npm notice run 'playwright' test --list
            Listing tests:
            
            

it lists 84 tests, across 3 browsers. So the clear difference is that sometimes it's running across all browsers, sometimes just one.

After a little investigation, I found that when I run the test list command from here:

playwright_training/100_plays/003

I get 3x as many times as when I run it from here

playwright_training/100_plays/003/tests

That's because the "playwright.config.ts" file where "projects" is defined, is in the first directory

$ ls
            node_modules  package-lock.json     playwright-report  template.spec.ts  tests
            package.json  playwright.config.ts  README.md          test-results
            $ grep -A15 "Configure projects" playwright.config.ts 
              /* Configure projects for major browsers */
              projects: [
                {
                  name: 'chromium',
                  use: { ...devices['Desktop Chrome'] },
                },
            
                {
                  name: 'firefox',
                  use: { ...devices['Desktop Firefox'] },
                },
            
                {
                  name: 'webkit',
                  use: { ...devices['Desktop Safari'] },
                },
            
            

So it would run 3x as many tests when it can access that file.

Alright, back to grouping different permutations of tests

Specifying Tests

Run everything:

npx playwright test

Run everything in a directory:

npx playwright test red_tests

Run one specific file:

npx playwright test red_tests/group_tiger.spec.ts

Run multiple specific files:

npx playwright test red_tests/group_tiger.spec.ts blue_tests/group_giraffe.spec.ts

Using grep to specify tags and test names

Select a test by name:

npx playwright test -g circle

Select by tag:

npx playwright test -g @one

Combinations

Directory and test name:

npx playwright test red_tests -g circle

Directory and tag:

npx playwright test red_tests -g @one

File and test name:

npx playwright test red_tests/group_tiger.spec.ts -g circle

File and tag:

npx playwright test red_tests/group_tiger.spec.ts -g @one

And you can add projects (which by default are browsers) to any run command like so:

npx playwright test red_tests --project=firefox

Boolean operators with grep

OR operator "|"

npx playwright test -g "@smoke|@critical"

Excluding tags:

npx playwright test --grep-invert @slow

You can combine this with the normal grep flag like a NOT operator, like so:

npx playwright test --grep @one --grep-invert @slow

There isn't an "AND" operator, you have to hack it together with grep lookaheads

npx playwright test --grep "(?=.*@one)(?=.*@slow)"