I recently came across a problem with running BenchmarkCI with a package that has binary dependencies. Turns out you need to run the julia-actions/julia-buildpkg@latest
step in the action, or dependencies don't get built.
I updated the action listed in the README to reflect this, so other people don't run into this problem.
I also changed the Julia version from 1.4 to 1, so it doesn't go out of date.