Skip to content

Home

Jump to bottom
likelovewant edited this page Feb 16, 2025 · 59 revisions

Welcome to the ollama-for-amd wiki!

This wiki aims to extend support for AMD GPUs that Ollama Official doesn't currently cover due to limitations in official ROCm on Windows. The goal is to remove these GPU limitations and include support for more AMD graphics card models. Remember, this help is provided voluntarily by community.

Important

"This repository is essentially a fork of ollama, primarily modified the gpu lists to work with more AMD GPUs. It doesn't introduce any new features beyond that."

(For those who prefer another language, right-click and use the auto-translate feature in your browser or device. Most platforms have this built-in!.)

Need Help?

issues? Please share your server logs in the issue section .also check out the official ollama issue tracker: (ollama/issues) for more in-depth assistance or get help from similiar issue.If you suspect the issue originates from the Ollama program itself, please hold off on further troubleshooting for now. ollama teams are working on updates to address this problem. Keep an eye out for a new release!"

Tip

Compilation Not Required? Try the Demo!

If you're not comfortable compiling or build the software yourself, no problem! You can skip the development steps and jump straight to using the ** Step 7 ,Demo Release Version**.

To use the demo, simply replace the rocblas files in the Rocblas Support Section with those provided in the demo release.

**Development Guide **

（"This guide outlines how the repository builds the program. You don't need to follow these steps unless you're interested in customizing the build process. One benefit of following these steps is that the compiled program may take up less space, as it will only be optimized for your specific GPU architecture instead of a wide range of architectures.").

1, Prepare Development Enviroment

Follow the guide on development.md for Windows development setup.

Downloading & Editing

2, Prepare Scripts

  1. Clone the Ollama Repository: 
    git clone https://github.com/ollama/ollama.git

or simpley clone this repo.

  1. Edit the CMakePresets.json File:

  • Locate this file:CMakePresets.json

    • Add your GPU model to the existing list at line 59. Here's an example of the official supported list:
    "AMDGPU_TARGETS": "gfx900;gfx940;gfx941;gfx942;gfx1010;gfx1012;gfx1030;gfx1100;gfx1101;gfx1102;gfx906:xnack-;gfx908:xnack-;gfx90a:xnack+;gfx90a:xnack-"

    "gfx803"(rocm5),"gfx900:xnack-"(rocm5), "gfx902" (rocm5),"gfx90c:xnack-"(rocm5,only available at self-build),//"gfx906:xnack-","gfx1010:xnack-", "gfx1011", "gfx1012:xnack-", "gfx1031", "gfx1032", "gfx1034", "gfx1035", "gfx1036","gfx1103", "gfx1150"

while also need to edit CMakeLists.txt in line 99

official support lists .

list(FILTER AMDGPU_TARGETS INCLUDE REGEX "^gfx(900|94[012]|101[02]|1030|110[012])$")

meaning : gfx900 gfx940、gfx941、gfx942 gfx1010、gfx1012 gfx1030 gfx1100、gfx1101、gfx1102

update list on this repo

list(FILTER AMDGPU_TARGETS INCLUDE REGEX "^gfx(803|900(:xnack-)|902|90c(:xnack-)|1010(:xnack-)|1011|1012(:xnack-)|103[0-6]|110[0-3]|1150)$")

meaning ：gfx803 ；gfx900:xnack-；gfx902；gfx906:xnack-；gfx90c:xnack-；gfx1010:xnack-； gfx1011；gfx1012:xnack-；gfx1030 to gfx1036； gfx1100 to gfx1103； 1150：gfx1150；

ROCm Support

3, Prepare ROCm

  • Ensure you have a compatible Hip SDK installed. Install HIP SDK 6.1.2/HIP SDK 5.7.1 from hip-sdk.

  • Important( necessary when you build with rocm5.7( eg, gfx803 ,gfx90c,...): Currently, HIP SDK 5.7 is out of supported. can be enabled by change the amdhip64_6.dll to amdhip64.dll in line 38 amd_hip_windows.go and line 27,change the C:\\Program Files\\AMD\\ROCm\\6.1\\bin" to C:\\Program Files\\AMD\\ROCm\\5.7\\bin" in line 26 /discover/amd_windows.go

  • Enable HSA_OVERRIDE_GFX_VERSION edits，optional for some undetected device in self-build, eg, gfx90c or gfx90c:xnack-: , More refer to Previous-Ollama-Build-Steps

Official AMD ROCm Supported GPUs: You can find a complete list of officially supported GPUs here: rocm.docs.amd.com

For supported GPUs, please use the official ROCm version from AMD also the Ollama Official. You don't need to build anything unless you want to learn how. (Examples:gfx900 gfx940 gfx941 gfx942 gfx1010 gfx1012 gfx1030 gfx1100 gfx1101 gfx1102)

If your GPU is not on the list:

  • Try pre-built ROCm libraries:
    • Some pre-built rocblas libraries are available for specific GPUs at:

ROCmLibs for HIP SDK 6.1.2

ROCmLibs for HIP SDK 5.7

Note

  • ROCm 5.7 Compatibility Issue: The latest llama.cpp upstream version has broken support for ROCm 5.7 during local testing.
  • Temporary Workaround: There is a workaround to fix this issue. by copying the following files from ROCm 6.1 (or any other llvm with clang) (located at C:\Program Files\AMD\ROCm\6.1\bin) to ROCm 5.7 (located at C:\Program Files\AMD\ROCm\5.7\bin):
* `clang.exe` `clang++.exe` `clang-cl.exe`  `clang-format.exe` `clang-offload-bundler.exe`

Additionally, copy the entire directory " 19" from C:\Program Files\AMD\ROCm\6.1\lib\clang to C:\Program Files\AMD\ROCm\5.7\lib\clang.

  • Choose the appropriate library based on your GPU model (e.g., gfx902, gfx1032).

Building ROCm libraries:

Installing and Replacing Libraries:

  1. Install HIP SDK: This will create a C:\Program Files\AMD\ROCm\6.1\bin folder.
  2. Place rocblas.dll in the C:\Program Files\AMD\ROCm\6.1\bin folder. Replace any existing rocblas.dll.
  3. Replace library within rocblas\library: This ensures your code uses the correct library files.
  4. Add bin in the PATH: eg , C:\Program Files\AMD\ROCm\6.1\bin\ to Path( systeme envrimental , PATH , most people forgot this)

(change the path to 5.7 if you want to work on HIP 5.7 )

Community Support:

  • If you encounter issues or need assistance in rocblas and library, please leave a message on the project's issues page for community support.

4 , Building the ollama.exe:**

cmake -B build -GNinja -DAMDGPU_TARGETS="gfx803;gfx900:xnack-;gfx902;gfx906:xnack-;gfx90c:xnack-;gfx1010:xnack-;gfx1011;gfx1012:xnack-;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150"

Then run

cmake --build build --config Release

or

ninja -C build

Throughout the examples DAMDGPU_TARGETS is fully arches,to speed up the build. You can build with your need arches to reduce build times. eg, DAMDGPU_TARGETS="gfx1031;"

ROCm 5.7 (HIPSDK 5.7) Build Instructions:

  • Build with ROCm 5.7 and HIPSDK 5.7 only if HipSDK 6 does not support your GPU architecture.
  • For building instructions, refer to the official llama.cpp documentation: llama.cpp/build.md#hip 
set PATH=%HIP_PATH%\bin;%PATH%
  
cmake -S . -B build -G Ninja -DAMDGPU_TARGETS="gfx803;gfx900:xnack-" -DGGML_HIP=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release
  
cmake --build build --config Release -j 8

After build.

5, Start Server

you can start the server in your Ollama directory:

go run . serve

Finally, in a separate shell, run a model(eg):

./ollama run llama3

6, Building the Installer:**

  • Install Inno Setup: To build an installer for Ollama, you'll need to install Inno Setup: https://jrsoftware.org/isinfo.php.
  • Run PowerShell Script: In the top directory of the Ollama repository, run this PowerShell script: 
    powershell -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1
  • The Installer: After the build is complete, you'll find the OllamaSetup.exe installer in the dist folder. This will work exactly like the official release.

7, Demo Release Version:

  • Test Drive Ollama: If you want to test before building from source, download and install a demo release from ollama-for-amd/releases.

  1. Install the Base Ollama Setup: Start by running OllamaSetup.exe. This sets up the basic framework for Ollama.(optional,If you skip these steps, make sure to remove any existing Ollama clients to avoid version conflicts.)

  2. Unzip and Replace Libraries: Download ollama-windows-amd64.7z and unzip it. Then, replace the files within C:\Users\usrname\AppData\Local\Programs\Ollama\lib\ollama with those from the extracted archive.

  3. Match ROCm to Your GPU: Download the appropriate ROCm libraries for your GPU architecture from ROCmlibs for 6.1.2. or ROCmLibs for HIP SDK 5.7 (based on each release package)

    • Important: Ensure the rocblas.dll and rocblas/library folder you choose match your GPU architecture.
    • Replace the existing files in your Ollama program folder (ie,C:\Users\UsrName\AppData\Local\Programs\Ollama\lib\ollama\rocm) with these downloaded libraries .

  4. Start Serving: Once everything is installed, run ollama run llama3.1 in CMD OR Powershell to start your model or Skip step1 , navigate to your ollama-windows-amd64 program directory and replace with your target rocmlibs and run ./ollama serve.

This repository serves as an example, and won't be regularly updated.

Try pre-built ROCm libraries:

  • Some pre-built rocblas libraries are available for specific GPUs at:

ROCmLibs for HIP SDK 6.1.2

ROCmLibs for HIP SDK 5.7

  • Choose the appropriate library based on your GPU model (e.g., gfx902, gfx1032).

Multi gpu Notes: if your has multi gpu and the arches not on the support lists . please consider build your rocmlibs and olloma as the guide in development-guide .

Note

gfx1103 (AMD 780M) is natively supported by this repository, so no file replacements are needed. For v0.5.4 or before usage refer to previous-emo-release-version

gfx803 hip memory error fixed by this rocm 5.7 libsrocm.gfx803.optic.test.version.7z for rocm5.7 in rocm 5.7 ,received feedback for gfx900 series ( gfx803;gfx900:xnack,gfx90c:xnack-...) can not detacted in Hip SDK 6.1.2 and forward . please consider build ollama with rocm5.7 yourself if there is no ollama-windows-amd64-rocm5.7.7z package available at release page .the ROCmlibs for 5.7 available at HIP SDK 5.7

Note

"The HSA_OVERRIDE_GFX_VERSION approach no longer works.Optional by force enable it in self-build gfx90c:xnack-, gfx1010:xnack-, gfx1012：xnack- and Similar Architectures:

  • Start by trying the pre-built rocblas and library files for your GPU. *If you encounter issues, try setting the environment variable HSA_OVERRIDE_GFX_VERSION=9.0.12（for gfx90c），HSA_OVERRIDE_GFX_VERSION=10.1.2（ gfx1012） . Refer to the Windows documentation for guidance on setting environment variables: https://www.computerhope.com/issues/ch000549.htm.

  • Running Your Models: After setting up ROCm and Ollama, run your models in a separate shell using the command: ./ollama run [model name], replacing "[model name]" with the actual name of your model.

and start the server in C:\Users\usrname\AppData\Local\Programs\Ollama\:

./ollama serve

Finally, in a separate shell, run a model(eg):

./ollama run llama3

  • Update Carefully: If you are using a demo release, DO NOT click the "Update" button if Ollama prompts you. Download updates manually from this release page: ollama-for-amd/releases

  • or update via Ollama-For-AMD-Installer created by ByronLeeeee, a simply click will do the update and replace libs process .

Trouble shootings on Windows:

error 0xc0000005

  1. make sure get the latest drivers.and replace the files match your gpu.
  2. some proven solutions as shows in 8#,14#,18#,by mannuly move the all the .dll files in Ollama\lib\ollama and rocblas folder into Ollama\lib\ollama\runners\rocm_v6.1.

No GPU detected

evel=INFO source=gpu.go:386 msg="no compatible GPUs were discovered

install HIPSDK , add hipinfo in path , ie, rocm/bin in the path,refer to hipinfo#, that's usually happens in ROCm5.7 package or Release.

Hide Nvida GPU detected

CUDA_VISIBLE_DEVICES=-1,0,1,2... can be get info by run nvidia-smi -L

Hide AMD GPU detected

HIP_VISIBLE_DEVICES=-1,0,1,2... can be get info by run hipinfo

Multiple AMD GPU OVERRIDE Have mulitiple OVERRIDE_GFX_VERSION can add extra tag 0,1 behidn version

HSA_OVERRIDE_GFX_VERSION_0=10.3.0 HSA_OVERRIDE_GFX_VERSION_1=11.0.0

Troubleshooting AMD GPU Support in Linux:

For users who prefer a self-build approach or use ollama official release on unsupported arches, setting environment variables like HSA_OVERRIDE_GFX_VERSION=9.0.0 or 9.0.6, 10.1.0, 10.3.0, 11.0.0 can enable GPU support on Linux systems. Configuring Environment Variables

Setting on Linux For Linux, set variables using systemd: 1, Edit your systemd service:

sudo systemctl edit ollama.service

2, Add the following line under the [Service]

Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

Have mulitiple GPUs ,can add extra tag 0,1 behidn version

HSA_OVERRIDE_GFX_VERSION_0=10.3.0 HSA_OVERRIDE_GFX_VERSION_1=11.0.0

3, Reload the service and restart:

$ sudo systemctl daemon-reload
$ sudo systemctl restart ollama.service

However, some users may encounter issues such as:

  • "amdgpu too old gfx%d%x%x"
  • "unsupported Radeon iGPU detected skipping"
  • "unable to verify rocm library: %w", err)

These errors due to incompatibility between the installed AMDGPU drivers and rocm supports issue. To potentially resolve these issues, consider modifying the discover/amd_linux.go file within your ollama directory if the HSA_OVERRIDE_GFX_VERSIONnot works for you.

Caution: Editing this file can have unintended consequences. It's recommended to back up the original file before making any changes.

Specifically, focus on lines 295 to 320 and either delete or edit the relevant content. The exact modifications required will depend on your specific hardware and driver version. Refer to online forums and documentation for guidance on tailoring these edits for your situation.

Clone this wiki locally