🐛 [Bug] flux Examples are broken in thor platform

[lanluo-nvidia]

Bug Description

File "/workspace/TensorRT/tools/perf/Flux/flux_perf.py", line 111, in <module>████████████████████████████████████████████████████| 9.95G/9.95G [04:10<00:00, 87.5MB/s]
    main(args)
  File "/workspace/TensorRT/tools/perf/Flux/flux_perf.py", line 53, in main
    pipe, backbone, trt_gm = compile_model(args)
  File "/workspace/TensorRT/tools/perf/Flux/../../../examples/apps/flux_demo.py", line 144, in compile_model
    image = pipe(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 828, in __call__
    ) = self.encode_prompt(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 363, in encode_prompt
    pooled_prompt_embeds = self._get_clip_prompt_embeds(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 299, in _get_clip_prompt_embeds
    prompt_embeds = self.text_encoder(text_input_ids.to(device), output_hidden_states=False)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/utils/generic.py", line 943, in wrapper
    output = func(self, *args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 717, in forward
    return self.text_model(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/utils/generic.py", line 943, in wrapper
    output = func(self, *args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 628, in forward
    encoder_outputs: BaseModelOutput = self.encoder(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/utils/generic.py", line 943, in wrapper
    output = func(self, *args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 557, in forward
    layer_outputs = encoder_layer(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/modeling_layers.py", line 83, in __call__
    return super().__call__(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 404, in forward
    hidden_states, attn_weights = self.self_attn(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 313, in forward
    queries = self.q_proj(hidden_states)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 134, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasLtMatmulAlgoGetHeuristic( ltHandle, computeDesc.descriptor(), Adesc.descriptor(), Bdesc.descriptor(), Cdesc.descriptor(), Cdesc.descriptor(), preference.descriptor(), 1, &heuristicResult, &returnedResult)`

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• Torch-TensorRT Version (e.g. 1.0.0):
• PyTorch Version (e.g. 1.0):
• CPU Architecture:
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, libtorch, source):
• Build command you used (if compiling from source):
• Are you using local sources or building from archives:
• Python version:
• CUDA version:
• GPU models and configuration:
• Any other relevant information:

Additional context

[lanluo-nvidia]

after setting
export LD_LIBRARY_PATH=/opt/conda/envs/py310/lib/python3.10/site-packages/nvidia/cu13/lib:$LD_LIBRARY_PATH

the cublas error has gone,
but got the Killed error.
It might be OOM, might not.
thor has 120G GPU memory

[lanluo-nvidia]

(py310) root@dl20g11-2012:/workspace/TensorRT# python tools/perf/Flux/flux_perf.py --dtype fp16
[02/05/2026-16:10:56] [TRT] [W] Functionality provided through tensorrt.plugin module is experimental.
Running flux_perfwith args: Namespace(model='black-forest-labs/FLUX.1-dev', use_sdpa=False, dtype='fp16', fp4_mha=False, low_vram_mode=False, dynamic_shapes=False, pytorch=False, max_batch_size=1)

Using fp16
Loading pipeline components...:   0%|                                                                                                                     | 0/7 [00:00<?, ?it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:04<00:00,  1.41s/it]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.16s/it]
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:09<00:00,  1.34s/it]
  0%|                                                                                                                                                    | 0/30 [00:00<?, ?it/s]WARNING:py.warnings:/opt/conda/envs/py310/lib/python3.10/copyreg.py:101: FutureWarning: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
  return cls.__new__(cls, *args)

WARNING:torch_tensorrt.dynamo._compiler:Remaining GPU memory may not be enough to compile the TensorRT engine for this model resulting in an OOM error, Consider setting offload_module_to_cpu=True
WARNING:torch_tensorrt [TensorRT Conversion Context]:pos_embed/mul_14_rhs [CONSTANT] contains out-of-range weights when cast to Half, clipping to +/- 65504.
WARNING:torch_tensorrt [TensorRT Conversion Context]:Detected layernorm nodes in FP16.
WARNING:torch_tensorrt [TensorRT Conversion Context]:Running layernorm after self-attention with FP16 Reduce or Pow may cause overflow. Forcing Reduce or Pow Layers in FP32 precision, or exporting the model to use INormalizationLayer (available with ONNX opset >= 17) can help preserving accuracy.
Killed

[lanluo-nvidia]

after killed error, the thor is no longer working

lanl@dl20g11-2012:~/TensorRT-Incubator$ nvidia-smi
Unable to determine the device handle for GPU0: 0000:01:00.0: Unknown Error
No devices were found