Skip to content

Add support for pathfinder.load_nvidia_dynamic_lib("cupti")#1693

Merged
rwgk merged 11 commits intoNVIDIA:mainfrom
rwgk:pathfinder_cupti
Mar 6, 2026
Merged

Add support for pathfinder.load_nvidia_dynamic_lib("cupti")#1693
rwgk merged 11 commits intoNVIDIA:mainfrom
rwgk:pathfinder_cupti

Conversation

@rwgk
Copy link
Collaborator

@rwgk rwgk commented Feb 26, 2026

Closes #1572

Complete support for (Linux, Windows) x (site-packages, Conda, CTK)

The CTK and site-packages support is fully tested in the cuda-python CI (all_must_work).

The Conda support is based on inspecting local Conda installations: (Linux, Windows) x (CTK 12, CTK 13)

In the cuda-python CI, Conda support is tested via mocks only. Additional manual testing:

smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ conda activate py314_cupti12

(py314_cupti12) smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ pip install cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Processing ./cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.4.1.dev31+g40f60225d

(py314_cupti12) smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ python -c 'from cuda import pathfinder; print(pathfinder.__version__); print(pathfinder.load_nvidia_dynamic_lib("cupti"))'
1.4.1.dev31+g40f60225d
LoadedDL(abs_path='/wrk/miniforge3/envs/py314_cupti12/lib/libcupti.so.12', was_already_loaded_from_elsewhere=False, _handle_uint=94773688522640, found_via='conda')
smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ conda activate py314_cupti13

(py314_cupti13) smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ pip install cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Processing ./cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.4.1.dev31+g40f60225d

(py314_cupti13) smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ python -c 'from cuda import pathfinder; print(pathfinder.__version__); print(pathfinder.load_nvidia_dynamic_lib("cupti"))'
1.4.1.dev31+g40f60225d
LoadedDL(abs_path='/wrk/miniforge3/envs/py314_cupti13/lib/libcupti.so.13', was_already_loaded_from_elsewhere=False, _handle_uint=104439350888336, found_via='conda')
PS C:\Users\rgrossekunst> conda activate py314_ctk129
(py314_ctk129) PS C:\Users\rgrossekunst> pip install H:\cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Processing H:\cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.4.1.dev31+g40f60225d
(py314_ctk129) PS C:\Users\rgrossekunst> python -c "from cuda import pathfinder; print(pathfinder.__version__); print(pathfinder.load_nvidia_dynamic_lib('cupti'))"
1.4.1.dev31+g40f60225d
LoadedDL(abs_path='C:\\Users\\rgrossekunst\\AppData\\Local\\miniforge3\\envs\\py314_ctk129\\Library\\bin\\cupti64_2025.2.1.dll', was_already_loaded_from_elsewhere=False, _handle_uint=140716902645760, found_via='conda')
PS C:\Users\rgrossekunst> conda activate py314_ctk131
(py314_ctk131) PS C:\Users\rgrossekunst> pip install H:\cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Processing H:\cuda_pathfinder-1.4.1.dev31+g40f60225d-py3-none-any.whl
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.4.1.dev31+g40f60225d
(py314_ctk131) PS C:\Users\rgrossekunst> python -c "from cuda import pathfinder; print(pathfinder.__version__); print(pathfinder.load_nvidia_dynamic_lib('cupti'))"
1.4.1.dev31+g40f60225d
LoadedDL(abs_path='C:\\Users\\rgrossekunst\\AppData\\Local\\miniforge3\\envs\\py314_ctk131\\Library\\bin\\cupti64_2025.4.1.dll', was_already_loaded_from_elsewhere=False, _handle_uint=140716917850112, found_via='conda')

abs_path found in CI logs for https://github.com/NVIDIA/cuda-python/actions/runs/22734923321/

$ grep -a 'Z INFO test_load_nvidia_dynamic_lib\[cupti\]: abs_path=' Test*.txt | sed 's/.*Z INFO /INFO /' | sort | uniq -c
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.10.11\x64\lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.4.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.10.11\x64\lib\site-packages\nvidia/cuda_cupti/bin\cupti64_2025.2.1.dll
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.11.9\x64\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.4.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.11.9\x64\Lib\site-packages\nvidia/cuda_cupti/bin\cupti64_2025.2.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.12.10\x64\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.3.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.12.10\x64\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.4.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.12.10\x64\Lib\site-packages\nvidia/cuda_cupti/bin\cupti64_2025.2.1.dll
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.13.12\x64\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.4.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.13.12\x64\Lib\site-packages\nvidia/cuda_cupti/bin\cupti64_2025.2.1.dll
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.14.3\x64-freethreaded\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.4.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.14.3\x64-freethreaded\Lib\site-packages\nvidia/cuda_cupti/bin\cupti64_2025.2.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.14.3\x64\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.3.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.14.3\x64\Lib\site-packages\nvidia/cu13/bin/x86_64\cupti64_2025.4.1.dll
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=C:\actions-runner\_work\_tool\Python\3.14.3\x64\Lib\site-packages\nvidia/cuda_cupti/bin\cupti64_2025.2.1.dll
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.10.20/arm64/lib/python3.10/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.10.20/arm64/lib/python3.10/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.11.15/arm64/lib/python3.11/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.11.15/arm64/lib/python3.11/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.12.13/arm64/lib/python3.12/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.12.13/arm64/lib/python3.12/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.13.12/arm64/lib/python3.13/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.13.12/arm64/lib/python3.13/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      4 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.13.12/x64/lib/python3.13/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.13.12/x64/lib/python3.13/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/arm64-freethreaded/lib/python3.14t/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/arm64-freethreaded/lib/python3.14t/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      2 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/arm64/lib/python3.14/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/arm64/lib/python3.14/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      3 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/x64-freethreaded/lib/python3.14t/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/x64-freethreaded/lib/python3.14t/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      3 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/x64/lib/python3.14/site-packages/nvidia/cu13/lib/libcupti.so.13
      1 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/opt/hostedtoolcache/Python/3.14.3/x64/lib/python3.14/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12
      6 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/__w/cuda-python/cuda-python/cuda_toolkit/lib64/libcupti.so.12
     19 INFO test_load_nvidia_dynamic_lib[cupti]: abs_path=/__w/cuda-python/cuda-python/cuda_toolkit/lib64/libcupti.so.13

This commit adds support for finding and loading CUPTI libraries on Linux
through cuda.pathfinder. It implements support for all enumerated installation
methods:

- Site-packages: nvidia/cuda_cupti/lib (CUDA 12) and nvidia/cu13/lib (CUDA 13)
- Conda: $CONDA_PREFIX/lib (colocated with other CUDA libraries)
- CTK via CUDA_HOME: $CUDA_HOME/extras/CUPTI/lib64
- CTK via canary probe: system CTK root discovery (similar to nvvm)

Changes:
- Add 'cupti' to supported library names and SONAMEs
- Add site-packages paths for CUDA 12 and 13
- Add cupti to CTK root canary discoverable libraries
- Update find_nvidia_dynamic_lib to handle extras/CUPTI/lib64 path
- Add logic to distinguish CTK (extras/CUPTI/lib64) vs conda (lib) paths
- Update _find_so_using_lib_dir to support versioned libraries via glob
- Add comprehensive mock tests covering all installation methods

Fixes NVIDIA#1572 (Linux support)

Made-with: Cursor
@copy-pr-bot
Copy link
Contributor

copy-pr-bot bot commented Feb 26, 2026

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk
Copy link
Collaborator Author

rwgk commented Feb 26, 2026

/ok to test

@github-actions

This comment has been minimized.

@rwgk rwgk self-assigned this Mar 3, 2026
@rwgk rwgk added the cuda.pathfinder Everything related to the cuda.pathfinder module label Mar 3, 2026
rwgk added 3 commits March 4, 2026 20:10
Resolved conflicts by:
- Adding cupti to descriptor_catalog.py with proper configuration
- Updating search_platform.py to handle versioned libraries via glob
- Removing old find_nvidia_dynamic_lib.py (refactored into new structure)
- Resolving supported_nvidia_libs.py conflicts (now derives from catalog)

The cupti support is now integrated into the new descriptor-based architecture.

Made-with: Cursor
Migrated test_load_nvidia_dynamic_lib_using_mocker.py from the old
_FindNvidiaDynamicLib API to the new descriptor-based SearchContext API.

Changes:
- Replace _FindNvidiaDynamicLib imports with search_steps and load_nvidia_dynamic_lib modules
- Update mocks to use run_find_steps, LOADER, and SearchContext
- Use LIB_DESCRIPTORS to get cupti descriptor
- Update all test functions to work with the new search step architecture

Made-with: Cursor
These variables (_CTK_ROOT_CANARY_ANCHOR_LIBNAMES and
_CTK_ROOT_CANARY_DISCOVERABLE_LIBNAMES) were added in the cupti PR but
are not used in the new descriptor-based architecture. The new code
uses desc.ctk_root_canary_anchor_libnames directly from descriptors.

Made-with: Cursor
@rwgk rwgk force-pushed the pathfinder_cupti branch from 33dfbb6 to b819eee Compare March 5, 2026 05:38
@rwgk
Copy link
Collaborator Author

rwgk commented Mar 5, 2026

/ok to test

rwgk added 2 commits March 5, 2026 11:51
Remove tests covered by real CI:
- Site-packages tests (CUDA 12 and 13) - covered by real CI
- CTK tests (CUDA_HOME and canary probe) - covered by real CI
- Search order tests involving site-packages/CTK - covered by real CI

Keep tests not covered by real CI:
- Conda discovery test - Conda not covered by real CI
- Error path test (not found) - error path not covered
- Conda vs CTK search order test - Conda not covered by real CI

Also remove unused imports and helper functions.

Made-with: Cursor
@rwgk
Copy link
Collaborator Author

rwgk commented Mar 5, 2026

/ok to test

@rwgk rwgk changed the title Add Linux support for loading libcupti.so.12 and libcupti.so.13 Add support for pathfinder.load_nvidia_dynamic_lib("cupti") Mar 5, 2026
@rwgk rwgk marked this pull request as ready for review March 5, 2026 21:27
@copy-pr-bot
Copy link
Contributor

copy-pr-bot bot commented Mar 5, 2026

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Copy link
Contributor

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Follow-up for Linux fallback glob ordering is tracked in #1732.

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 6, 2026

We want your header PR in the release, too, so timing works for having the reversed here for full internal consistency. I started on that already before seeing your comment, and before triggering the CI for your PR.

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 6, 2026

/ok to test

@rwgk rwgk enabled auto-merge (squash) March 6, 2026 17:49
@rwgk rwgk merged commit 3a7222b into NVIDIA:main Mar 6, 2026
86 checks passed
@rwgk rwgk deleted the pathfinder_cupti branch March 6, 2026 18:54
@github-actions
Copy link

github-actions bot commented Mar 6, 2026

Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.pathfinder Everything related to the cuda.pathfinder module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA]: Support loading CUPTI

2 participants