cavity3d , run-time error ‘invalid device symbol’, CUDA 11.7

[hungryfish]

The default example ‘olb-1.5r0/examples/laminar/cavity3d’ brings this runtime error under CUDA 11.7 on a V100, do you have any advice for me? (downgrading complex)

[prepareGeometry] Prepare Geometry … OK
terminate called after throwing an instance of ‘std::runtime_error’
what(): invalid device symbol
Aborted

Environment:
used ‘gpu_only.mk’

mpic++ –version
g++ (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvcc –version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_21:27:02_PDT_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0

gcc –version
gcc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

cavity3d]$ nvidia-smi
Tue May 31 14:18:28 2022
+—————————————————————————–+
| NVIDIA-SMI 515.43.04 Driver Version: 515.43.04 CUDA Version: 11.7 |
|——————————-+———————-+———————-+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE… Off | 00000000:21:01.0 Off | 0 |
| N/A 32C P0 34W / 250W | 0MiB / 32768MiB | 1% Default |
| | | N/A |
+——————————-+———————-+———————-+

+—————————————————————————–+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+—————————————————————————–+

cavity3d]$ make clean
rm -f tmp/*.* tmp/vtkData/*.* tmp/vtkData/data/*.* tmp/imageData/*.* tmp/imageData/data/*.* tmp/gnuplotData/*.* tmp/gnuplotData/data/*.*
rm -f cavity3d.o cavity3d.d cavity3d
[demo1706@nodus-p2s-2xlarge-8-3307-0 cavity3d]$ make
make -C ../../../external
make[1]: Entering directory `/mnt/nfs/olb-1.5r0/external’
make -C zlib
make[2]: Entering directory `/mnt/nfs/olb-1.5r0/external/zlib’
make[2]: Nothing to be done for `all’.
make[2]: Leaving directory `/mnt/nfs/olb-1.5r0/external/zlib’
cp zlib/build/libz.a lib/
make -C tinyxml
make[2]: Entering directory `/mnt/nfs/olb-1.5r0/external/tinyxml’
make[2]: Nothing to be done for `all’.
make[2]: Leaving directory `/mnt/nfs/olb-1.5r0/external/tinyxml’
cp tinyxml/build/libtinyxml.a lib/
make[1]: Leaving directory `/mnt/nfs/olb-1.5r0/external’
nvcc -O3 -std=c++17 –generate-code=arch=compute_86,code=[compute_86,sm_86] –extended-lambda –expt-relaxed-constexpr -x cu -Xcudafe “–diag_suppress=implicit_return_from_non_void_function –display_error_number –diag_suppress=20014 –diag_suppress=20011” -DPLATFORM_CPU_SISD -DPLATFORM_GPU_CUDA -I../../../src -I../../../external/zlib -I../../../external/tinyxml -c -o cavity3d.o cavity3d.cpp
nvcc cavity3d.o -o cavity3d -L../../../external/lib -lz -ltinyxml -lcuda -lcudadevrt -lcudart
[demo1706@nodus-p2s-2xlarge-8-3307-0 cavity3d]$ ls -ltr
total 53744
-rw-r–r– 1 demo1706 demo1706 71 Apr 14 12:00 Makefile
-rw-r–r– 1 demo1706 demo1706 303836 Apr 14 12:00 cavity3d.pvsm
-rw-r–r– 1 demo1706 demo1706 9416 Apr 14 12:00 cavity3d.cpp
-rw-rw-r– 1 demo1706 demo1706 102325 May 31 14:00 cavity3d.d
-rw-rw-r– 1 demo1706 demo1706 29892392 May 31 14:07 cavity3d.o
-rwxrwxr-x 1 demo1706 demo1706 24713096 May 31 14:07 cavity3d
[demo1706@nodus-p2s-2xlarge-8-3307-0 cavity3d]$ ./cavity3d
[Directories] Directory ./tmp/ created.
[Directories] Directory ./tmp/imageData/ created.
[Directories] Directory ./tmp/imageData/data/ created.
[Directories] Directory ./tmp/vtkData/ created.
[Directories] Directory ./tmp/vtkData/data/ created.
[Directories] Directory ./tmp/gnuplotData/ created.
[Directories] Directory ./tmp/gnuplotData/data/ created.
[UnitConverter] —————– UnitConverter information —————–
[UnitConverter] — Parameters:
[UnitConverter] Resolution: N= 30
[UnitConverter] Lattice velocity: latticeU= 0.1
[UnitConverter] Lattice relaxation frequency: omega= 1.96464
[UnitConverter] Lattice relaxation time: tau= 0.509
[UnitConverter] Characteristical length(m): charL= 1
[UnitConverter] Characteristical speed(m/s): charU= 1
[UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 0.001
[UnitConverter] Phys. density(kg/m^d): charRho= 1
[UnitConverter] Characteristical pressure(N/m^2): charPressure= 0
[UnitConverter] Mach number: machNumber= 0.173205
[UnitConverter] Reynolds number: reynoldsNumber= 1000
[UnitConverter] Knudsen number: knudsenNumber= 0.000173205
[UnitConverter]
[UnitConverter] — Conversion factors:
[UnitConverter] Voxel length(m): physDeltaX= 0.0333333
[UnitConverter] Time step(s): physDeltaT= 0.00333333
[UnitConverter] Velocity factor(m/s): physVelocity= 10
[UnitConverter] Density factor(kg/m^3): physDensity= 1
[UnitConverter] Mass factor(kg): physMass= 3.7037e-05
[UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.333333
[UnitConverter] Force factor(N): physForce= 0.111111
[UnitConverter] Pressure factor(N/m^2): physPressure= 100
[UnitConverter] ————————————————————-
[prepareGeometry] Prepare Geometry …
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 inner boundary voxel(s)
[SuperGeometry3D] the model is correct!
[CuboidGeometry3D] —Cuboid Stucture Statistics—
[CuboidGeometry3D] Number of Cuboids: 1
[CuboidGeometry3D] Delta (min): 0.0333333
[CuboidGeometry3D] (max): 0.0333333
[CuboidGeometry3D] Ratio (min): 1
[CuboidGeometry3D] (max): 1
[CuboidGeometry3D] Nodes (min): 29791
[CuboidGeometry3D] (max): 29791
[CuboidGeometry3D] Weight (min): 29791
[CuboidGeometry3D] (max): 29791
[CuboidGeometry3D] ——————————–
[SuperGeometryStatistics3D] materialNumber=1; count=24389; minPhysR=(0.0333333,0.0333333,0.0333333); maxPhysR=(0.966667,0.966667,0.966667)
[SuperGeometryStatistics3D] materialNumber=2; count=4561; minPhysR=(0,0,0); maxPhysR=(1,1,1)
[SuperGeometryStatistics3D] materialNumber=3; count=841; minPhysR=(0.0333333,1,0.0333333); maxPhysR=(0.966667,1,0.966667)
[prepareGeometry] Prepare Geometry … OK
terminate called after throwing an instance of ‘std::runtime_error’
what(): invalid device symbol
Aborted

***********
The same runtime error occurs in ‘cavity3d’

[Adrian]

The CUDA_ARCH value needs to match the targeted GPU. i.e. for a V100 (Volta) 70 or 72 should be good (lower is also possible but will lead to additional startup time for bytecode translation).

See also the reference table in rules.mk mentioned in the comment for CUDA_ARCH in config/gpu_only.mk.

Note that further adaptation (e.g. following config/gpu_openmpi.mk) will be needed for MPI-based multi-GPU execution (regarding the inclusion of mpic++ output).

Especially for the laminar/cavity3d example you will want to significantly increase the spatial resolution from the default value to take advantage of the GPU.

See also e.g. the forum/gpu-examples thread.

• This reply was modified 2 years, 3 months ago by Adrian.
• This reply was modified 2 years, 3 months ago by Adrian.

[Author]

Posts