Error message "Bad memory unallocation" when cores exceed NUM_THREADS

[sitsofe]

When using PyTorch 2.10.0 on an Aarch64 system with more than 128 cores, running the Hugging Face THUDM/CogVideoX-2b model causes PyTorch's built in OpenBLAS (apparently v0.3.30) to print out "Bad memory unallocation!" warnings:

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
[...]
BLAS : Bad memory unallocation! :  768  0xf52c94000000
BLAS : Bad memory unallocation! :  768  0xf52c78000000
BLAS : Bad memory unallocation! :  768  0xf52c80000000
BLAS : Bad memory unallocation! :  768  0xf52c8e000000
BLAS : Bad memory unallocation! :  768  0xf52c8a000000
BLAS : Bad memory unallocation! :  768  0xf52c88000000
[...]

The problem has also been reproduced with OpenBLAS v0.2.20-7494-g986ba2949. All warnings go away if the threads are restricted with OMP_NUM_THREADS=128 or OpenBLAS is recompiled with NUM_THREADS=256.

The following is a script that reproduces the problem (but note it downloads over 14GiB of data before it runs):

#!/usr/bin/env python3
#
# apt update
# apt install python3-pip python3-venv
# python3 -m venv venv
# . venv/bin/activate
# pip install accelerate diffusers protobuf sentencepiece tiktoken torch transformers
# # WARNING: This script needs to download over 14GiB of model data!
# OMP_NUM_THREADS=140 python3 openblas-bad-memory-unallocation.py
#
# If you want to use a custom OpenBLAS:
# git clone https://github.com/OpenMathLib/OpenBLAS.git
# cd OpenBLAS
# git describe
# v0.2.20-7494-g986ba2949
# make -j 8 NUM_THREADS=128 USE_OPENMP=1 NO_SHARED=0 DYNAMIC_ARCH=1 TARGET=ARMV8 CFLAGS=-O3 BUILD_BFLOAT16=1
# cd ../
# OMP_NUM_THREADS=140 LD_PRELOAD=OpenBLAS/libopenblas.so python3 openblas-bad-memory-unallocation.py

import torch
from diffusers import CogVideoXPipeline

def main():
    # From https://huggingface.co/zai-org/CogVideoX-2b
    model_name = 'THUDM/CogVideoX-2b'
    
    # Create main processing pipeline
    print("Creating pipeline...")
    pipe = (CogVideoXPipeline.from_pretrained(
        model_name,
        torch_dtype=torch.float32,
    ).to('cpu'))
    
    # Configure pipeline
    print("Configuring pipeline...")
    generator = torch.Generator().manual_seed(42)
    
    # Warmup
    print("Warmup...")
    # The default model (CogVideoX-2b) specifically says it only works with 720x480
    pipe(prompt="warmup", generator=generator, num_inference_steps=1, num_frames=1)

    print("Finished!")

if __name__ == '__main__':
    main()

[rgommers]

Thanks for the report @sitsofe

The problem has also been reproduced with OpenBLAS v0.2.20-7494-g986ba2949.

That's from 2017, seems unlikely. Can you reproduce this with the most recent released version or a build from develop? Or is this a problem with how that version string was saved perhaps?

OMP_NUM_THREADS=140 python3 openblas-bad-memory-unallocation.py

Just to make sure: can you reproduce this with python3 openblas-bad-memory-unallocation.py, or only if you explicitly set OMP_NUM_THREADS to a value higher than the NUM_THREADS used at build time?

[martin-frbg]

The code in question was added in 0.3.18 of October 2021, earlier versions would simply have exited with an error message telling to recompile the library. Perhaps you meant 0.3.20 (but the hash number does not match anything in the OpenBLAS history) ?
I haven't gotten around to trying to reproduce this yet - the warning is probably harmless, especially if it only occurs at the end of the run, but it suggests there is a race condition in the code that frees the additional buffer on thread termination.

[sitsofe]

Hi @rgommers:

That's from 2017, seems unlikely. Can you reproduce this with the most recent released version or a build from develop? Or is this a problem with how that version string was saved perhaps?

The leading part of the git describe output is misleading (I fell for it too). If you break it down it "I am a commit that is 7494 commits away from v0.2.20 and my hash is 986ba29". Commit 986ba29 is from (at the time of writing) 2 days ago. Why git describe chose to base itself off a tag so old I don't know (UPDATE: it was because it was only using annotated tags) but I promise the commit that was checked out and built was 986ba29 .

Just to make sure: can you reproduce this with python3 openblas-bad-memory-unallocation.py, or only if you explicitly set OMP_NUM_THREADS to a value higher than the NUM_THREADS used at build time?

On a machines with 140 or 144 cores I can reproduce the problem with just python3 openblas-bad-memory-unallocation.py (and OpenBLAS reports that the NUM_THREADS it was compiled with is 128). On a machine with 35 cores I can only reproduce the problem on the same NUM_THREADS=128 OpenBLAS library if I explicitly set OMP_NUM_THREADS to a value higher than the NUM_THREADS. The problem does NOT happen if I compile OpenBLAS with NUM_THREADS=8 and I run with 35 cores or set OMP_NUM_THREADS=32.

@martin-frbg:

The code in question was added in 0.3.18 of October 2021, earlier versions would simply have exited with an error message telling to recompile the library. Perhaps you meant 0.3.20 (but the hash number does not match anything in the OpenBLAS history) ?

The hash number (the bit after the g of g986ba2949) matches 986ba29 which is part of the develop branch.

I haven't gotten around to trying to reproduce this yet - the warning is probably harmless, especially if it only occurs at the end of the run, but it suggests there is a race condition in the code that frees the additional buffer on thread termination.

It may not be as harmless as it first looks. The short example here always runs to completion but the longer program the example was extracted from seems to go on to hang indefinitely when a second set of warnings are produced BUT only when it is run on the high core count machine. In other words:

• Longer program runs to completion in 40 minutes on 35 core ARM machine.
• Longer program prints warnings and hangs forever (over 12 hours) when run on 140 core ARM machine and last output on the console is BLAS : Bad memory unallocation! : 768 0xe96926800000. This behaviour has happened on multiple different machines so seems reproducible.

[sitsofe]

(It turns out the reason that git describe picked such an old tag is because by default it searches for the most recent annotated tag and it looks like v0.2.20 is the last annotated tag. When you use git show v0.2.20 you can see extra tag meta data above the usual commit metadata:

> git show v0.2.20
tag v0.2.20
Tagger: Zhang Xianyi <traits.zhang@...>
Date:   Mon Jul 24 12:04:14 2017 +0800

OpenBLAS 0.2.20 version

commit 5dde4e65d321076582a2fafe16949d2160551e81 (tag: v0.2.20, origin/release-0.2.21)
Merge: bcfc298c3 27a9df647
Author: Zhang Xianyi <traits.zhang@...>
Date:   Mon Jul 24 12:03:35 2017 +0800

    Merge branch 'develop'
    
    0.2.20 version

Whereas if I compare that to git show v0.3.31:

> git show v0.3.31
commit 76f1be470c9b9f80dc6e27407e13b975df436489 (tag: v0.3.31, origin/release-0.3.0)
Merge: ca1aefc5f 0e7b11fbc
Author: Martin Kroeker <martin@...>
Date:   Thu Jan 15 23:57:26 2026 +0100

    Merge pull request #5605 from OpenMathLib/develop
    
    Merge from develop for  0.3.31 release

If I run git describe --tags (allowing git describe to use lightweight tags) then I get

> git describe --tags
v0.3.31-53-g986ba2949

which is much more reasonable).

[martin-frbg]

Ah right, did not notice the missing "tags" argument. Releases are created from the "0.3.0" branch that was created as a copy of 0.2.20 at the time I took over the maintainer role, maybe I've been tagging differently than he did (I'm using the tag option
on the release screen in the gh web frontend).
Haven't had a chance to reproduce the problem yet (tried on the biggest machine I have shell access to, a POWER system at OSUOSL, but it was shut down soon after - IIRC they're moving shop :/ )