task: refactor ASV benchmarks - #2996
Conversation
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/2996/index.html |
|
Array API standard conformance tests for dpnp=0.21.0dev6=py314ha0e2e8e_9 ran successfully. |
| `asv.conf.json` sets `branches` to `HEAD` rather than to named branches, so that | ||
| results recorded on a feature branch are picked up. With named branches | ||
| `asv publish` reports `Couldn't find <hash> in branches (...)` and silently | ||
| drops them. |
There was a problem hiding this comment.
That says the config sets branches to HEAD so feature-branch results are picked up, and warns that named branches make asv publish silently drop results. But benchmarks/asv.conf.json:7-10 actually sets ["master", "dev-milestone"] — and the diff shows this PR changed it from ["HEAD"].
By the README's own reasoning the documented workflow will silently drop results on any feature branch. Either the config regressed or the README is stale.
There was a problem hiding this comment.
README is stale, thanks for picking up on this. Gireesh informed me I needed to change the asv.conf.json to include those branches but I didn't review the documentation here afterwards. Will do so now
| """Black-Scholes formula workload. | ||
|
|
||
| The dpnp implementation, the NumPy reference and the data initialization are | ||
| copied verbatim from dpBench (https://github.com/IntelPython/dpbench), and the |
There was a problem hiding this comment.
Verbatim dpBench code relicensed with no provenance in the header — the workloads and runner state in their docstrings they're "copied verbatim from dpBench" (Apache-2.0), yet each carries only dpnp's standard BSD-3-Clause header _dpbench_runner.py:1-27 with no SPDX tag or original-license note.
Both projects are Intel-copyright so relicensing is likely fine legally, and prose-level attribution exists (README + docstrings) — but the file-header license lineage is silent and worth an explicit maintainer sign-off.
There was a problem hiding this comment.
Do we need to add something like?
# Portions of this file are derived from IntelPython/dpBench
# (https://github.com/IntelPython/dpbench), originally licensed
# Apache-2.0, Copyright (c) 2022-2023 Intel Corporation, and
# relicensed here under BSD-3-Clause by the copyright holder.
Or since it's another IntelPython project there is no need to any provenance?
There was a problem hiding this comment.
I had to assume that because we own both repos, I can copy IP/code from either without having to worry about licensing. Open to adding the clause though if we really think it's necessary.
I believe one goal is to eventually decommission dpbench anyways and start moving our benchmarks to be in-repo for everything like this
| f"{self.WORKLOAD.NAME} has no {preset} preset." | ||
| ) | ||
|
|
||
| if not runner.preset_fits(self.WORKLOAD, preset, queue.sycl_device): |
There was a problem hiding this comment.
The fit verdict is computed with itemsize=8 (f8) for both the single and double param cells:
| if not runner.preset_fits(self.WORKLOAD, preset, queue.sycl_device): | |
| if not runner.preset_fits(self.WORKLOAD, preset, queue.sycl_device, precision): |
| # ***************************************************************************** | ||
|
|
||
| import numpy | ||
| """Benchmarks for unary elementwise math functions, dpnp against NumPy.""" |
There was a problem hiding this comment.
Elementwise covers 26 unary transcendental ufuncs — a subset of NumPy's bench_ufunc.py, which benchmarks all 95 ufuncs via one parametrized method.
Two notes:
degreesandrad2degare aliases of the same ufunc, so they benchmark the identical kernel twice- the notable gaps vs. NumPy are unary rounding/sign (floor, ceil, trunc, rint, sign, abs) and binary arithmetic (add, multiply, divide, power, hypot, arctan2)
There was a problem hiding this comment.
Cleaned up the gaps and double kernel benchmarking. Super good catch on that one!
| # ***************************************************************************** | ||
|
|
||
| import numpy | ||
| """Benchmarks for unary elementwise math functions, dpnp against NumPy.""" |
There was a problem hiding this comment.
It runs 26 unary transcendental ufuncs over _DTYPES = [float64, float32, int64, int32].
Integer input from arange promotes to float and mostly lands out of domain (nan/inf), and mixes native-FP timings with int→float-promotion timings. For reference, NumPy's own ASV suite restricts its transcendental/strided benchmarks to float dtypes ('e','f','d') and filters others by ufunc.types.
_DTYPES is shared with bench_linalg.py (where ints are valid), so suggest a separate float-only axis (_FLOAT_DTYPES = [float64, float32]) used only in the elementwise class. This also removes the risk that a dpnp ufunc rejecting int input where NumPy promotes it shows up as an asymmetric failure.
There was a problem hiding this comment.
implemented, thanks for the suggestion
| self.b = np.arange(size * size, dtype=dt).reshape((size, size)) | ||
| # self.c = np.arange(600) | ||
| # self.d = np.arange(400) | ||
| class MatMul: |
There was a problem hiding this comment.
MatMul covers dot/matmul/inner/einsum over a clean square-size sweep (16→1024) and both executors — the size sweep is actually more systematic than NumPy's Eindot.
Two suggestions vs. NumPy's bench_linalg.py:
- add a transposed/non-contiguous variant (e.g. dot(a, a.T)) — NumPy's Eindot benches these deliberately because they hit different BLAS/stride kernels, and dpnp/oneMKL is likely no different;
- if broadening later, NumPy also covers decompositions (svd/det/solve) and norm, which dpnp supports via oneMKL LAPACK. Including integer dtypes here is fine (unlike the transcendental case, integer matmul is well-defined).
Fixes and refactors ASV benchmarking to mirror dpbench implementations.