Guidance for combining benchmark job results into one report

[dyc3]

We use github action's native feature to only run benchmarks on relevant files changed. In order to do this, we have to silo each benchmark group that we care about into it's own GHA workflow, as you can see here: https://github.com/biomejs/biome/tree/main/.github/workflows

Each workflow is conditionally run with triggers that look like this:

pull_request:
    types: [ opened, synchronize ]
    branches:
      - main
      - next
    paths:
      - 'Cargo.lock'
      - 'crates/biome_configuration/**/*.rs'
      - 'crates/biome_rowan/**/*.rs'

I noticed on biomejs/biome#9945 that the codspeed bot comment says

❌ 7 regressed benchmarks
✅ 22 untouched benchmarks
⏩ 199 skipped benchmarks

Codspeed link: https://codspeed.io/biomejs/biome/branches/dyc3%2Fmain-into-next

I know for a fact that 199 benchmarks were not skipped, because the jobs for those "skipped" benchmarks ran. Presumably the css benchmarks finished last, and that resulted in the bot updating the comment only considering the css benchmark results.

Essentially, it appears that all of our benchmark jobs generate different reports. I'm not sure if this is intended behavior which is why I'm reaching out. Ideally, all workflows that ran for a commit would be combined into the same codspeed report.

[adriencaccia]

Indeed, we do not support having runs on different workflows, all runs must be part of the same workflow, either in different jobs or running serially in a job.

Given your requirement, I would recommend having a single workflow, and leverage the dorny/paths-filter GH action that allows for conditional workflow execution.
With something like this, you would be able to have all CodSpeed runs in a single report, while still leveraging sharded benchmarks.

Hope that helps, let me know if it does not!

[dyc3]

That seems to work great. Thank you!