## 🛠️ NVBandwidth Troubleshooting Guide (FAQ)

This guide helps diagnose and resolve common issues encountered when using NVBandwidth. Each question provides error messages, root causes, verification steps, and solutions.

<details>
<summary><strong>📋 What are the system requirements for NVBandwidth?</strong></summary>

**Multi-node requirements:**
- CUDA Toolkit 12.3 & driver and above
- IMEX setup
- Proper MPI configuration

**Single-node requirements:**
- CUDA Toolkit 11.x or above

**CUDA Version Notes:**
- Multi-node features were introduced in CUDA 12.3
- Using earlier versions will limit functionality to single-node tests
- When building for multi-node:

```bash
# Ensure CUDA 12.3+ is in use
$ nvcc --version

# Build with multi-node support
$ cmake -DMULTINODE=1
```

</details>

<details>
<summary><strong>❌ I'm getting "CUDA_ERROR_NO_DEVICE" - no CUDA device detected. How do I fix this?</strong></summary>

**Error Message:**
```bash
[CUDA_ERROR_NO_DEVICE] no CUDA-capable device is detected in expression cuInit(0)
```

**Root Cause:**
This error occurs when:
- NVIDIA driver is not installed
- GPU(s) are not properly detected
- CUDA driver mismatch with the installed CUDA toolkit
- Hardware issues

**Verification Steps:**

1. Check if GPU is detected:
```bash
$ nvidia-smi
```

2. Verify driver installation:
```bash
# Check driver version
$ cat /proc/driver/nvidia/version

# Check loaded modules
$ lsmod | grep nvidia

# Check driver installation
$ dpkg -l | grep nvidia-driver
```

3. Check CUDA setup:
```bash
# Check CUDA driver version
$ nvidia-smi | grep "CUDA Version"

# Check CUDA runtime version
$ nvcc --version
```

You can consult this [page](https://docs.nvidia.com/deploy/cuda-compatibility) for CUDA driver and toolkit version compatibility.

**Solution:**
1. Install/reinstall NVIDIA driver/CUDA toolkit

**Notes:**
- Consider running nvidia-bug-report for detailed diagnostics:
```bash
$ nvidia-bug-report.sh
```

</details>

<details>
<summary><strong>⚠️ I'm getting "CUDA_ERROR_SYSTEM_NOT_READY" error. What does this mean?</strong></summary>

**Error Message:**
```bash
[CUDA_ERROR_SYSTEM_NOT_READY] system not yet initialized
```

**Root Cause:**
On multi-GPU or multi-node systems this error generally indicates your system requires the NVLink/NVSwitch fabric manager, and you have not installed it.

**Solution:**
Install and start nvidia-fabricmanager: [Instructions](https://docs.nvidia.com/vss/latest/content/quickstart.html#install-the-nvidia-fabric-manager)

</details>

<details>
<summary><strong>🔧 I'm getting "cudaErrorUnsupportedPtxVersion" when building. How do I fix this?</strong></summary>

**Error Message:**
```bash
[cudaErrorUnsupportedPtxVersion] the provided PTX was compiled with an unsupported toolchain
```

**Root Cause:**
This error means that the PTX code you are attempting to run was generated by a CUDA toolkit version that is newer than what your currently installed NVIDIA driver supports. The driver's JIT compiler cannot understand or process the PTX generated by a more recent toolchain.

**Solution:**
Ensure compatibility between your CUDA driver and the CUDA toolkit used for compilation.

1. Clean build:
```bash
rm -rf CMakeCache.txt CMakeFiles
make clean
```

2. Set CUDA version:
```bash
export PATH=/usr/local/cuda-12.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH
```

3. Check driver compatibility:
```bash
# Check NVIDIA driver version
nvidia-smi

# Check CUDA driver compatibility
nvidia-smi | grep "CUDA Version"

# Verify CUDA runtime vs driver compatibility
nvcc --version
```

4. Rebuild with explicit CUDA version

</details>

<details>
<summary><strong>🏗️ I'm getting "Unsupported gpu architecture" error during compilation. What should I do?</strong></summary>

**Error Message:**
```bash
nvcc fatal   : Unsupported gpu architecture 'compute_52'
```

**Solution:**
Use CUDA < 13.0 for older architectures (Volta, Maxwell, or Pascal):
```bash
# For Hopper
cmake -DCMAKE_CUDA_ARCHITECTURES=sm_90
```

</details>

<details>
<summary><strong>📦 I'm getting "No rule to make target libboost_program_options.a" error. How do I fix this?</strong></summary>

**Error Message:**
```bash
No rule to make target '/usr/lib64/libboost_program_options.a'
```

**Solution:**
```bash
sudo apt-get install libboost-program-options-dev
```

</details>

<details>
<summary><strong>🔗 I'm getting "CUDA_ERROR_NVLINK_UNCORRECTABLE" error. How do I diagnose NVLink issues?</strong></summary>

**Error Message:**
```bash
[CUDA_ERROR_NVLINK_UNCORRECTABLE] uncorrectable NVLink error
```

**Verification:**
```bash
# Check NVLink status
nvidia-smi nvlink -s
```
Output should show active for each NVLink, inactive or missing links may indicate a problem.

```bash
# Check topology
nvidia-smi topo -m
```

</details>

<details>
<summary><strong>🚫 I'm getting "CUDA_ERROR_NOT_PERMITTED" during multi-node runs. How do I fix IMEX issues?</strong></summary>

**Error Message:**
```bash
$ mpirun -np 2 -N 2 ./nvbandwidth -t multinode_device_to_device_bidirectional_memcpy_write_sm
[CUDA_ERROR_NOT_PERMITTED] operation not permitted in expression cuMemCreate(&handle, roundedUpAllocationSize, &prop, 0 )
```

**Root Cause:**
Missing or improper IMEX channel setup, required for multi-node runs.

**Solution:**
1. Start imex daemon and set up IMEX channels. The creation of the /dev/ nodes /dev/nvidia-caps-imex-channels/channelN must be handled by the administrator, where N is the minor number.

```bash
$ sudo systemctl start nvidia-imex
# Query the major number
$ /proc/devices | grep nvidia-caps-imex-channels
# Create the IMEX channels
$ sudo mkdir /dev/nvidia-caps-imex-channels/
$ sudo mknod /dev/nvidia-caps-imex-channels/channel0 c <major number> 0 

# Check IMEX service status
$ systemctl status nvidia-imex
```

Also refer to: [IMEX Setup Guide](https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/imexchannels.html)

</details>

<details>
<summary><strong>📚 I'm getting "libmpi_cxx.so.40 => not found" error. How do I install MPI?</strong></summary>

**Error Message:**
```bash
$ ldd nvbandwidth
libmpi_cxx.so.40 => not found
```

**Solution:**
Install OpenMPI development packages (package names vary by distribution):

**Ubuntu/Debian:**
```bash
sudo apt-get install openmpi-bin libopenmpi-dev

**Red Hat/CentOS/Fedora:**
```bash
# RHEL/CentOS 7/8
sudo yum install openmpi-devel

# Fedora
sudo dnf install openmpi-devel
```

</details>

<details>
<summary><strong>📖 Where can I find additional resources and documentation?</strong></summary>

**Additional Resources:**
- [NVIDIA Documentation](https://docs.nvidia.com)
- [CUDA Documentation](https://docs.nvidia.com/cuda)
- [CUDA Installations](https://docs.nvidia.com/cuda/cuda-installation-guide-linux)

</details>
