Sitelet https://github.com/IntelPython/dpnp/pull/2996
Skip to content

task: refactor ASV benchmarks - #2996

Open
jharlow-intel wants to merge 6 commits into
masterfrom
task/SAT-8518
Open

task: refactor ASV benchmarks#2996
jharlow-intel wants to merge 6 commits into
masterfrom
task/SAT-8518

Conversation

@jharlow-intel

Copy link
Copy Markdown
Collaborator

Fixes and refactors ASV benchmarking to mirror dpbench implementations.

@jharlow-intel jharlow-intel self-assigned this Jul 9, 2026
@jharlow-intel jharlow-intel added the enhancement New feature or request label Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/2996/index.html

@coveralls

coveralls commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 78.477%. remained the same — task/SAT-8518 into master

@antonwolfy antonwolfy added this to the 0.21.0 release milestone Jul 10, 2026
Comment thread benchmarks/benchmarks/dpbench/workloads/__init__.py Outdated
Comment thread benchmarks/benchmarks/common.py Outdated
Comment thread benchmarks/benchmarks/dpbench/workloads/black_scholes.py Outdated
Comment thread benchmarks/benchmarks/dpbench/workloads/black_scholes.py
Comment thread benchmarks/benchmarks/dpbench/workloads/black_scholes.py Outdated
Comment thread pyproject.toml Outdated
Comment thread benchmarks/benchmarks/dpbench/workloads/black_scholes.py
Comment thread benchmarks/benchmarks/dpbench/_dpbench_runner.py
Comment thread benchmarks/README.md Outdated
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.21.0dev6=py314ha0e2e8e_9 ran successfully.
Passed: 1375
Failed: 0
Skipped: 7

@jharlow-intel
jharlow-intel marked this pull request as draft July 10, 2026 15:47
@jharlow-intel
jharlow-intel marked this pull request as ready for review August 24, 2026 19:45
Comment thread benchmarks/asv.conf.json Outdated
Comment thread benchmarks/README.md
Comment on lines +218 to +221
`asv.conf.json` sets `branches` to `HEAD` rather than to named branches, so that
results recorded on a feature branch are picked up. With named branches
`asv publish` reports `Couldn't find <hash> in branches (...)` and silently
drops them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That says the config sets branches to HEAD so feature-branch results are picked up, and warns that named branches make asv publish silently drop results. But benchmarks/asv.conf.json:7-10 actually sets ["master", "dev-milestone"] — and the diff shows this PR changed it from ["HEAD"].

By the README's own reasoning the documented workflow will silently drop results on any feature branch. Either the config regressed or the README is stale.

@jharlow-intel jharlow-intel Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README is stale, thanks for picking up on this. Gireesh informed me I needed to change the asv.conf.json to include those branches but I didn't review the documentation here afterwards. Will do so now

"""Black-Scholes formula workload.

The dpnp implementation, the NumPy reference and the data initialization are
copied verbatim from dpBench (https://github.com/IntelPython/dpbench), and the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verbatim dpBench code relicensed with no provenance in the header — the workloads and runner state in their docstrings they're "copied verbatim from dpBench" (Apache-2.0), yet each carries only dpnp's standard BSD-3-Clause header _dpbench_runner.py:1-27 with no SPDX tag or original-license note.

Both projects are Intel-copyright so relicensing is likely fine legally, and prose-level attribution exists (README + docstrings) — but the file-header license lineage is silent and worth an explicit maintainer sign-off.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add something like?

# Portions of this file are derived from IntelPython/dpBench
# (https://github.com/IntelPython/dpbench), originally licensed
# Apache-2.0, Copyright (c) 2022-2023 Intel Corporation, and
# relicensed here under BSD-3-Clause by the copyright holder.

Or since it's another IntelPython project there is no need to any provenance?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to assume that because we own both repos, I can copy IP/code from either without having to worry about licensing. Open to adding the clause though if we really think it's necessary.

I believe one goal is to eventually decommission dpbench anyways and start moving our benchmarks to be in-repo for everything like this

Comment thread benchmarks/benchmarks/dpbench/_dpbench_runner.py
Comment thread benchmarks/benchmarks/dpbench/_dpbench_runner.py
f"{self.WORKLOAD.NAME} has no {preset} preset."
)

if not runner.preset_fits(self.WORKLOAD, preset, queue.sycl_device):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fit verdict is computed with itemsize=8 (f8) for both the single and double param cells:

Suggested change
if not runner.preset_fits(self.WORKLOAD, preset, queue.sycl_device):
if not runner.preset_fits(self.WORKLOAD, preset, queue.sycl_device, precision):

# *****************************************************************************

import numpy
"""Benchmarks for unary elementwise math functions, dpnp against NumPy."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elementwise covers 26 unary transcendental ufuncs — a subset of NumPy's bench_ufunc.py, which benchmarks all 95 ufuncs via one parametrized method.
Two notes:

  • degrees and rad2deg are aliases of the same ufunc, so they benchmark the identical kernel twice
  • the notable gaps vs. NumPy are unary rounding/sign (floor, ceil, trunc, rint, sign, abs) and binary arithmetic (add, multiply, divide, power, hypot, arctan2)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleaned up the gaps and double kernel benchmarking. Super good catch on that one!

# *****************************************************************************

import numpy
"""Benchmarks for unary elementwise math functions, dpnp against NumPy."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It runs 26 unary transcendental ufuncs over _DTYPES = [float64, float32, int64, int32].

Integer input from arange promotes to float and mostly lands out of domain (nan/inf), and mixes native-FP timings with int→float-promotion timings. For reference, NumPy's own ASV suite restricts its transcendental/strided benchmarks to float dtypes ('e','f','d') and filters others by ufunc.types.

_DTYPES is shared with bench_linalg.py (where ints are valid), so suggest a separate float-only axis (_FLOAT_DTYPES = [float64, float32]) used only in the elementwise class. This also removes the risk that a dpnp ufunc rejecting int input where NumPy promotes it shows up as an asymmetric failure.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

implemented, thanks for the suggestion

self.b = np.arange(size * size, dtype=dt).reshape((size, size))
# self.c = np.arange(600)
# self.d = np.arange(400)
class MatMul:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MatMul covers dot/matmul/inner/einsum over a clean square-size sweep (16→1024) and both executors — the size sweep is actually more systematic than NumPy's Eindot.

Two suggestions vs. NumPy's bench_linalg.py:

  • add a transposed/non-contiguous variant (e.g. dot(a, a.T)) — NumPy's Eindot benches these deliberately because they hit different BLAS/stride kernels, and dpnp/oneMKL is likely no different;
  • if broadening later, NumPy also covers decompositions (svd/det/solve) and norm, which dpnp supports via oneMKL LAPACK. Including integer dtypes here is fine (unlike the transcendental case, integer matmul is well-defined).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants