Best backend & model for discrete GPU on Raspberry Pi?

[pepijndevos]

I have a Raspberry Pi with a discrete AMD GPU: https://github.com/pepijndevos/rpi-cm5io-mini-itx

AFAIK LM Studio doesn't support ARM Linux and Ollama doesn't support Vulkan.

It seems like you can directly interface with ollama.cpp, which does support Vulkan on ARM, although I'll have to compile it myself with GPU support I guess.

What is less clear to me is how tool use or function calling works in home-llm and if I need to use your specific models for that.

[pepijndevos]

llama-cpp-python doesn't seem to actually use my GPU: abetlen/llama-cpp-python#1826

I tried using the upstream llama.cpp server with the generic openai API.
At first I get KeyError("choices") but when I select the chat endpoint it works.
Except neither llama3.1 8b nor your 3b model seem to be able to call any functions

[pepijndevos]

With Home 3B I get the following logs:

2025-01-09 18:05:41.635 DEBUG (MainThread) [custom_components.llama_conversation.conversation] temperature = 1.5
2025-01-09 18:05:41.635 DEBUG (MainThread) [custom_components.llama_conversation.conversation] humidity = 92
2025-01-09 18:05:41.635 DEBUG (MainThread) [custom_components.llama_conversation.conversation] wind_speed = 15.1
2025-01-09 18:05:41.639 DEBUG (MainThread) [custom_components.llama_conversation.conversation] [{'role': 'system', 'message': "You are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.\nThe current time and date is 05:05 PM on Thursday January 09, 2025\nServices: weather.get_forecasts(), input_boolean.reload(), input_boolean.turn_on(), input_boolean.turn_off(), input_boolean.toggle(), todo.add_item(item), todo.update_item(item), todo.remove_item(item), todo.get_items(), todo.remove_completed_items()\nDevices:\ninput_boolean.heater 'heater' = off\nsun.sun 'Sun' = below_horizon\nsensor.system_monitor_memory_usage 'System Monitor Memory usage' = 36.5 %\nsensor.system_monitor_processor_use 'System Monitor Processor use' = 5 %\nsensor.system_monitor_processor_temperature 'System Monitor Processor temperature' = 50.7 °C\ntodo.shopping_list 'Shopping List' = 2\nweather.forecast_home 'Forecast Home' = partlycloudy;1.5 °C;92%;15.1 km/h"}, {'role': 'user', 'message': 'please turn the heater on'}]
2025-01-09 18:05:42.889 DEBUG (MainThread) [custom_components.llama_conversation.conversation] {'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'content': 'i\'ll get the heater going for you.\n```homeassistant\n{"service": "input_boolean.turn_on", "target_device": "input_boolean.heater"}\n```', 'role': 'assistant'}}], 'created': 1736442342, 'model': 'acon96/Home-3B-v3-GGUF', 'system_fingerprint': 'b4447-f7cd1330', 'object': 'chat.completion', 'usage': {'completion_tokens': 43, 'prompt_tokens': 310, 'total_tokens': 353}, 'id': 'chatcmpl-tSNfBKYJiW41Td40pLOQzuHvB0btHNuN', '__verbose': {'index': 0, 'content': 'i\'ll get the heater going for you.\n```homeassistant\n{"service": "input_boolean.turn_on", "target_device": "input_boolean.heater"}\n```', 'tokens': [], 'id_slot': 0, 'stop': True, 'model': 'acon96/Home-3B-v3-GGUF', 'tokens_predicted': 43, 'tokens_evaluated': 310, 'generation_settings': {'n_predict': 128, 'seed': 4294967295, 'temperature': 0.10000000149011612, 'dynatemp_range': 0.0, 'dynatemp_exponent': 1.0, 'top_k': 40, 'top_p': 1.0, 'min_p': 0.05000000074505806, 'xtc_probability': 0.0, 'xtc_threshold': 0.10000000149011612, 'typical_p': 1.0, 'repeat_last_n': 64, 'repeat_penalty': 1.0, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'dry_multiplier': 0.0, 'dry_base': 1.75, 'dry_allowed_length': 2, 'dry_penalty_last_n': 4096, 'dry_sequence_breakers': ['\n', ':', '"', '*'], 'mirostat': 0, 'mirostat_tau': 5.0, 'mirostat_eta': 0.10000000149011612, 'stop': [], 'max_tokens': 128, 'n_keep': 0, 'n_discard': 0, 'ignore_eos': False, 'stream': False, 'logit_bias': [], 'n_probs': 0, 'min_keep': 0, 'grammar': '', 'samplers': ['penalties', 'dry', 'top_k', 'typ_p', 'top_p', 'min_p', 'xtc', 'temperature'], 'speculative.n_max': 16, 'speculative.n_min': 5, 'speculative.p_min': 0.8999999761581421, 'timings_per_token': False, 'post_sampling_probs': False, 'lora': []}, 'prompt': "<|system|>\nYou are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.\nThe current time and date is 05:05 PM on Thursday January 09, 2025\nServices: weather.get_forecasts(), input_boolean.reload(), input_boolean.turn_on(), input_boolean.turn_off(), input_boolean.toggle(), todo.add_item(item), todo.update_item(item), todo.remove_item(item), todo.get_items(), todo.remove_completed_items()\nDevices:\ninput_boolean.heater 'heater' = off\nsun.sun 'Sun' = below_horizon\nsensor.system_monitor_memory_usage 'System Monitor Memory usage' = 36.5 %\nsensor.system_monitor_processor_use 'System Monitor Processor use' = 5 %\nsensor.system_monitor_processor_temperature 'System Monitor Processor temperature' = 50.7 °C\ntodo.shopping_list 'Shopping List' = 2\nweather.forecast_home 'Forecast Home' = partlycloudy;1.5 °C;92%;15.1 km/h\n<|user|>\nplease turn the heater on\n<|assistant|>\n", 'has_new_line': True, 'truncated': False, 'stop_type': 'eos', 'stopping_word': '', 'tokens_cached': 352, 'timings': {'prompt_n': 265, 'prompt_ms': 440.784, 'prompt_per_token_ms': 1.6633358490566037, 'prompt_per_second': 601.2014955170787, 'predicted_n': 43, 'predicted_ms': 798.566, 'predicted_per_token_ms': 18.571302325581396, 'predicted_per_second': 53.84651988689726}}, 'timings': {'prompt_n': 265, 'prompt_ms': 440.784, 'prompt_per_token_ms': 1.6633358490566037, 'prompt_per_second': 601.2014955170787, 'predicted_n': 43, 'predicted_ms': 798.566, 'predicted_per_token_ms': 18.571302325581396, 'predicted_per_second': 53.84651988689726}}
2025-01-09 18:05:42.889 DEBUG (MainThread) [custom_components.llama_conversation.conversation] i'll get the heater going for you.
` ` `homeassistant
{"service": "input_boolean.turn_on", "target_device": "input_boolean.heater"}
` ` `
2025-01-09 18:05:42.890 INFO (MainThread) [custom_components.llama_conversation.conversation] calling tool: {"service": "input_boolean.turn_on", "target_device": "input_boolean.heater"}

2025-01-09 18:05:42.890 DEBUG (MainThread) [custom_components.llama_conversation.conversation] Tool response: {'result': 'unknown service'}

Actually it seems like it doesn't support this type of device? I added an actual switch device and it works!