Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

satellite gets stuck after only 1 wake word detection #154

Open
jerome83136 opened this issue Apr 16, 2024 · 18 comments
Open

satellite gets stuck after only 1 wake word detection #154

jerome83136 opened this issue Apr 16, 2024 · 18 comments

Comments

@jerome83136
Copy link

jerome83136 commented Apr 16, 2024

Hello,

I built wyoming-satellite and I run it this way:

/opt/wyoming-satellite# script/run --name "my satellite" --uri tcp://0.0.0.0:10700 --mic-command "arecord -D plughw:CARD=seeed8micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw" --snd-command "aplay -D plughw:CARD=seeed8micvoicec,DEV=1 -r 22050 -c 1 -f S16_LE -t raw" --awake-wav sounds/awake.wav --detection-command examples/commands/detection.sh --wake-refractory-seconds 0 --debug

I'm using a custom wakeword (Google colab) and it works fine; but it only works "once"
I mean:

  1. I start the satellite process with the above command line
  2. I wait for it to start
  3. Then I say the wakeword "hé chou pee"
  4. The satellite activates, plays the sound (-awake-wav) and wait for my command
  5. The command is recognized (STT --> ASR --> Intent) and gets executed
  6. The audio feedback (TTS) is sent back to the satellite and I can hear it

And that's it. I am unable to get the wakeword activating again without killing and restarting the satellite process.
After restarting; it works (for 1 time) etc...

NB: I had the same result when using standard wakewords (hey_rhasspy, hey_jarvis, etc...)

Here are DEBUG logs of my satellite:

DEBUG:root:Namespace(mic_uri=None, mic_command='arecord -D plughw:CARD=seeed8micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw', mic_command_rate=16000, mic_command_width=2, mic_command_channels=1, mic_command_samples_per_chunk=1024, mic_volume_multiplier=1.0, mic_noise_suppression=0, mic_auto_gain=0, mic_seconds_to_mute_after_awake_wav=0.5, mic_no_mute_during_awake_wav=False, mic_channel_index=None, snd_uri=None, snd_command='aplay -D plughw:CARD=seeed8micvoicec,DEV=1 -r 22050 -c 1 -f S16_LE -t raw', snd_command_rate=22050, snd_command_width=2, snd_command_channels=1, snd_volume_multiplier=1.0, wake_uri=None, wake_word_name=[], wake_command=None, wake_command_rate=16000, wake_command_width=2, wake_command_channels=1, wake_refractory_seconds=0.0, vad=False, vad_threshold=0.5, vad_trigger_level=1, vad_buffer_seconds=2, vad_wake_word_timeout=5.0, event_uri=None, startup_command=None, detect_command=None, detection_command='examples/commands/detection.sh', transcript_command=None, stt_start_command=None, stt_stop_command=None, synthesize_command=None, tts_start_command=None, tts_stop_command=None, tts_played_command=None, streaming_start_command=None, streaming_stop_command=None, error_command=None, connected_command=None, disconnected_command=None, awake_wav='sounds/awake.wav', done_wav=None, uri='tcp://0.0.0.0:10700', name='my satellite', area=None, no_zeroconf=False, zeroconf_name=None, zeroconf_host=None, debug_recording_dir=None, debug=True, log_format='%(levelname)s:%(name)s:%(message)s')
INFO:root:Ready
DEBUG:root:Detected IP: 192.168.1.244
DEBUG:root:Zeroconf discovery enabled (name=6c21a2437eae, host=None)
DEBUG:root:Connecting to mic service: ['arecord', '-D', 'plughw:CARD=seeed8micvoicec,DEV=0', '-r', '16000', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
DEBUG:root:Connecting to snd service: ['aplay', '-D', 'plughw:CARD=seeed8micvoicec,DEV=1', '-r', '22050', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
INFO:root:Connected to services
DEBUG:root:Connected to mic service
Recording raw data 'stdin' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono
DEBUG:root:Server set: 34436877217185
INFO:root:Connected to server
INFO:root:Streaming audio
DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'wake', 'end_stage': 'tts', 'restart_on_end': True, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
DEBUG:root:Ping enabled



DEBUG:root:Wake word detected
DEBUG:root:Running ['examples/commands/detection.sh']
Wake word detected: éh_chou_pee
DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
DEBUG:root:Connected to snd service
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
DEBUG:root:Unmuted microphone
DEBUG:root:Event(type='transcript', data={'text': ' Allume la cuisine !'}, payload=None)
INFO:root:Streaming audio
DEBUG:root:Event(type='synthesize', data={'text': 'Allumé', 'voice': {'name': 'fr_FR-siwis-low'}}, payload=None)
DEBUG:root:Connected to snd service
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono

What did I do wrong please ?

Hardware:

Thank you for your help :)

Best regards

@jerome83136
Copy link
Author

Hi all,

After compiling required libs for activating VAD; I noticed that my voice is well detected

Here is the command I'm using with VAD argument:
/opt/wyoming-satellite# script/run --name "my satellite" --uri tcp://0.0.0.0:10700 --mic-command "arecord -D plughw:CARD=seeed8micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw" --snd-command "aplay -D plughw:CARD=seeed8micvoicec,DEV=1 -r 22050 -c 1 -f S16_LE -t raw" --awake-wav sounds/awake.wav --detection-command examples/commands/detection.sh --wake-refractory-seconds 0 --vad --debug

And here are the logs whowing a successful wakeword recognition:

DEBUG:root:Ping enabled
INFO:root:Waiting for speech
INFO:root:Streaming audio
DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'wake', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
DEBUG:root:Wake word detected
DEBUG:root:Running ['examples/commands/detection.sh']
Wake word detected: éh_chou_pee
DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
DEBUG:root:Connected to snd service
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
DEBUG:root:Unmuted microphone
DEBUG:root:Event(type='transcript', data={'text': ' Éteins la cuisine.'}, payload=None)
DEBUG:root:Event(type='synthesize', data={'text': 'Éteint', 'voice': {'name': 'fr_FR-siwis-low'}}, payload=None)
DEBUG:root:Connected to snd service
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
INFO:root:Streaming audio
DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'wake', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
INFO:root:Waiting for speech

And here; same logs showing my attempts to trigger wakeword again with no luck: (we can see the audio stream is detected when I speak; so it seems it is detected by the satellite, but the wakeword never gets triggered)

INFO:root:Waiting for speech
INFO:root:Streaming audio
DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'wake', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
INFO:root:Waiting for speech
INFO:root:Streaming audio
DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'wake', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
INFO:root:Waiting for speech
INFO:root:Streaming audio
DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'wake', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)

If I want to detect a wakeword again; I still need to kill and restart the satellite process. Then it works only once and I have to restart the process again.

Do anyone has some idea to share on this so that I can get it fixed maybe ? :)

NB: I'm also wondering what is "restart_on_end" that shows up in the logs ? Is it supposed to be "True" to somehow "allow" the wakeword going in "detecting" state again ?

Thank you for your feedback :)
Best regards

@d1egoaz
Copy link

d1egoaz commented Apr 17, 2024

I have the same issue, I installed everything yesterday.

Apr 17 14:28:21 pizero sudo[3807]: pam_unix(sudo:session): session opened for user root(uid=0) by diego(uid=1000)
Apr 17 14:28:24 pizero sudo[3807]: pam_unix(sudo:session): session closed for user root
Apr 17 14:28:29 pizero run[3794]: DEBUG:root:Detection(name='hey_jarvis_v0.1', timestamp=20400732344365)
Apr 17 14:28:29 pizero run[3794]: DEBUG:root:Streaming audio
Apr 17 14:28:29 pizero run[3794]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
Apr 17 14:28:29 pizero run[3794]: DEBUG:root:Running ['examples/commands/detection.sh']
Apr 17 14:28:29 pizero run[3810]: Wake word detected: hey_jarvis_v0.1
Apr 17 14:28:29 pizero run[3794]: DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
Apr 17 14:28:29 pizero run[3794]: DEBUG:root:Connected to snd service
Apr 17 14:28:29 pizero run[3813]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Apr 17 14:28:30 pizero run[3794]: DEBUG:root:Unmuted microphone

@d1egoaz
Copy link

d1egoaz commented Apr 17, 2024

I've tried previous versions, but the result was the same.

I've tried:

Apr 17 15:02:33 pizero run[5230]: Wake word detected: hey_jarvis_v0.1
Apr 17 15:02:33 pizero run[5206]: DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
Apr 17 15:02:33 pizero run[5206]: DEBUG:root:Connected to snd service
Apr 17 15:02:33 pizero run[5233]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Apr 17 15:02:34 pizero run[5206]: DEBUG:root:Unmuted microphone

@jerome83136
Copy link
Author

Hi all,
I did some more tests with ESP32-S3-BOX-3 (tutorial link)

So the device is using the same Home Assistant setup than the one I used with wyoming-satellite: wyoming openwakeword (still with my custom wakeword)

Only difference is that I replaced wyoming-satellite by an SP32-S3-BOX-3

And with this setup; I'm able to trigger the wakeword several times without having to restart anything.
So I think Home Assistant setup + wyoming-openwakeword are working correctly and I think there is a problem with the configuration of my wyoming-satellite. But what can be the problem ? Really need some feedback on this please :)

NB: In case it can help: I'm using docker version of Home Assistant and wyoming-openwakeword

Thank you for your help
Best regards

@d1egoaz
Copy link

d1egoaz commented Apr 18, 2024

Additional context:

I'm following https://www.youtube.com/watch?v=eTKgc0YDCwE&list=WL&index=54 with the same hardware:

- Raspberry Pi Zero W 2
- SeeedStudio: 2-Mic Pi Hat 

Additional logs:

I added >>> to the examples/commands to make then easier to be viewed:

Apr 18 12:41:47 pizero systemd[1]: Started Wyoming Satellite.
Apr 18 12:41:47 pizero python3[4091]: INFO:root:Ready
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Namespace(mic_uri=None, mic_command='arecord -D plughw:CARD=seeed2micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw', mic_command_rate=16000, mic_command_width=2, mic_command_channels=1, mic_command_samples_per_chunk=1024, mic_volume_multiplier=1.0, mic_noise_suppression=2, mic_auto_gain=0, mic_seconds_to_mute_after_awake_wav=0.5, mic_no_mute_during_awake_wav=False, mic_channel_index=None, snd_uri=None, snd_command='aplay -D plughw:CARD=seeed2micvoicec,DEV=0 -r 22050 -c 1 -f S16_LE -t raw', snd_command_rate=22050, snd_command_width=2, snd_command_channels=1, snd_volume_multiplier=1.0, wake_uri='tcp://127.0.0.1:10400', wake_word_name=[['hey_jarvis']], wake_command=None, wake_command_rate=16000, wake_command_width=2, wake_command_channels=1, wake_refractory_seconds=5.0, vad=False, vad_threshold=0.5, vad_trigger_level=1, vad_buffer_seconds=2, vad_wake_word_timeout=5.0, event_uri='tcp://127.0.0.1:10500', startup_command='examples/commands/startup.sh', detect_command=None, detection_command='examples/commands/detection.sh', transcript_command=None, stt_start_command='examples/commands/stt_start.sh', stt_stop_command='examples/commands/stt_stop.sh', synthesize_command='examples/commands/synthesize.sh', tts_start_command='examples/commands/tts_start.sh', tts_stop_command='examples/commands/tts_stop.sh', tts_played_command=None, streaming_start_command='examples/commands/streaming_start.sh', streaming_stop_command='examples/commands/streaming_stop.sh', error_command=None, connected_command=None, disconnected_command=None, awake_wav='sounds/awake.wav', done_wav='sounds/done.wav', uri='tcp://0.0.0.0:10700', name='Alfred Satellite', area=None, no_zeroconf=False, zeroconf_name=None, zeroconf_host=None, debug_recording_dir=None, debug=True, log_format='%(levelname)s:%(name)s:%(message)s')
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Running ['examples/commands/startup.sh']
Apr 18 12:41:48 pizero run[4099]: >>> Satellite has started.
Apr 18 12:41:48 pizero run[4095]: INFO:root:Ready
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Detected IP: 192.168.42.228
Apr 18 12:41:48 pizero run[4094]: DEBUG:root:Namespace(uri='tcp://127.0.0.1:10400', models_dir=PosixPath('/home/diego/wyoming-openwakeword/wyoming_openwakeword/models'), custom_model_dir=[], preload_model=['hey_jarvis'], threshold=0.5, trigger_level=1, output_dir=None, debug=True, log_format='%(levelname)s:%(name)s:%(message)s', debug_probability=False, version=False, model=[])
Apr 18 12:41:48 pizero run[4094]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/diego/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
Apr 18 12:41:48 pizero run[4094]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
Apr 18 12:41:48 pizero run[4094]: DEBUG:root:Loading /home/diego/wyoming-openwakeword/wyoming_openwakeword/models/melspectrogram.tflite
Apr 18 12:41:48 pizero run[4094]: DEBUG:root:Loading /home/diego/wyoming-openwakeword/wyoming_openwakeword/models/embedding_model.tflite
Apr 18 12:41:48 pizero run[4094]: INFO:root:Ready
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Zeroconf discovery enabled (name=2ccf6704eefa, host=None)
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Connecting to mic service: ['arecord', '-D', 'plughw:CARD=seeed2micvoicec,DEV=0', '-r', '16000', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Connecting to snd service: ['aplay', '-D', 'plughw:CARD=seeed2micvoicec,DEV=0', '-r', '22050', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Connecting to wake service: tcp://127.0.0.1:10400
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Connecting to event service: tcp://127.0.0.1:10500
Apr 18 12:41:48 pizero run[4095]: INFO:root:Connected to services
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Using webrtc audio enhancements
Apr 18 12:41:48 pizero run[4094]: DEBUG:wyoming_openwakeword.handler:Client connected: 14883546340095
Apr 18 12:41:48 pizero run[4105]: Recording raw data 'stdin' : Signed 16 bit Little Endian,
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Connected to mic service
Apr 18 12:41:48 pizero run[4095]: DEBUG:root:Connected to wake service
Apr 18 12:41:48 pizero run[4105]: Rate 16000 Hz, Mono
Apr 18 12:41:48 pizero run[4094]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/diego/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
Apr 18 12:41:48 pizero run[4094]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
Apr 18 12:41:49 pizero run[4094]: INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Apr 18 12:41:49 pizero run[4095]: DEBUG:root:Server set: 14884328193371
Apr 18 12:41:49 pizero run[4095]: INFO:root:Connected to server
Apr 18 12:41:49 pizero run[4095]: DEBUG:root:Running ['examples/commands/streaming_stop.sh']
Apr 18 12:41:49 pizero run[4108]: >>> Audio streaming to server has stopped
Apr 18 12:41:49 pizero run[4095]: INFO:root:Waiting for wake word
Apr 18 12:41:49 pizero run[4095]: DEBUG:root:Connected to event service
Apr 18 12:41:49 pizero run[4094]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/diego/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
Apr 18 12:41:49 pizero run[4094]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
Apr 18 12:41:49 pizero run[4094]: DEBUG:wyoming_openwakeword.handler:Sent info to client: 14883546340095
Apr 18 12:41:51 pizero run[4095]: DEBUG:root:Ping enabled
Apr 18 12:42:06 pizero run[4094]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=14883546340095)
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Detection(name='hey_jarvis_v0.1', timestamp=14900721294606)
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Streaming audio
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Running ['examples/commands/detection.sh']
Apr 18 12:42:06 pizero run[4094]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=14883546340095)
Apr 18 12:42:06 pizero run[4111]: >>> Wake word detected: hey_jarvis_v0.1
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Running ['examples/commands/streaming_start.sh']
Apr 18 12:42:06 pizero run[4094]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=14883546340095)
Apr 18 12:42:06 pizero run[4114]: >>> Audio streaming to server has started
Apr 18 12:42:06 pizero run[4095]: DEBUG:root:Connected to snd service
Apr 18 12:42:06 pizero run[4116]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Apr 18 12:42:07 pizero run[4095]: DEBUG:root:Unmuted microphone

Only the called commands logs:

Apr 18 12:41:48 pizero run[4099]: >>> Satellite has started.
Apr 18 12:41:49 pizero run[4108]: >>> Audio streaming to server has stopped
Apr 18 12:42:06 pizero run[4111]: >>> Wake word detected: hey_jarvis_v0.1
Apr 18 12:42:06 pizero run[4114]: >>> Audio streaming to server has started

And this is my service definition:

[Unit]
Description=Wyoming Satellite
Wants=network-online.target
After=network-online.target
Requires=wyoming-openwakeword.service
Requires=2mic_leds.service

[Service]
Type=simple
ExecStart=/home/diego/wyoming-satellite/script/run \
        --name 'Alfred Satellite' \
        --uri 'tcp://0.0.0.0:10700' \
        --mic-command 'arecord -D plughw:CARD=seeed2micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw' \
        --snd-command 'aplay -D plughw:CARD=seeed2micvoicec,DEV=0 -r 22050 -c 1 -f S16_LE -t raw' \
        --mic-auto-gain 0 \
        --mic-noise-suppression 2 \
        --mic-volume-multiplier 1.0 \
        --wake-uri 'tcp://127.0.0.1:10400' \
        --wake-word-name 'hey_jarvis' \
        --event-uri 'tcp://127.0.0.1:10500' \
        --awake-wav sounds/awake.wav \
        --done-wav sounds/done.wav \
        --tts-stop-command 'true' \
        --detection-command examples/commands/detection.sh \
        --startup-command examples/commands/startup.sh \
        --streaming-start-command examples/commands/streaming_start.sh \
        --streaming-stop-command examples/commands/streaming_stop.sh \
        --synthesize-command examples/commands/synthesize.sh \
        --stt-start-command examples/commands/stt_start.sh \
        --stt-stop-command examples/commands/stt_stop.sh \
        --tts-start-command examples/commands/tts_start.sh \
        --tts-stop-command examples/commands/tts_stop.sh \
        --debug
WorkingDirectory=/home/diego/wyoming-satellite
Restart=always

Hopefully @synesthesiam can take a look at this 🙏

@radunemerenco
Copy link

I have the same setup, and I get into the same issue.

  1. I say the wake word
  2. I get the audio feedback
  3. I see it in the logs: Wake word detected: hey_jarvis_v0.1
  4. The last log is DEBUG:root:Unmuted microphone

The LED service doesn't even work. I was never able to turn the LED on

@jerome83136
Copy link
Author

Hi everyone
Anyone able to help here maybe? 🙏
Thank you
Best regards

@richardoswald
Copy link

Similar issue here, in debug mode it only responds to wake word one time and gets stuck. The satellite is configured in Home Assistant. I'm using most of the defaults from the tutorial since I have the same hardware
wyoming-satellite v1.2.0
wyoming-openwakeword v1.10.0
Home Assistant v2024.5.4 (docker)

richard@wysat:~ $ journalctl -u wyoming-openwakeword.service -f
-- Journal begins at Mon 2024-03-11 20:13:58 CDT. --
May 18 00:47:51 wysat run[469]: DEBUG:root:Loading /home/richard/wyoming-openwakeword/wyoming_openwakeword/models/embedding_model.tflite
May 18 00:47:51 wysat run[469]: INFO:root:Ready
May 18 00:47:56 wysat run[469]: DEBUG:wyoming_openwakeword.handler:Client connected: 20644978897
May 18 00:47:56 wysat run[469]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/richard/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
May 18 00:47:56 wysat run[469]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
May 18 00:47:57 wysat run[469]: INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
May 18 00:47:58 wysat run[469]: DEBUG:wyoming_openwakeword.handler:Sent info to client: 20644978897
May 18 00:48:03 wysat run[469]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/richard/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
May 18 00:48:03 wysat run[469]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
May 18 00:48:03 wysat run[469]: DEBUG:wyoming_openwakeword.handler:Sent info to client: 20644978897
May 18 00:49:31 wysat run[469]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=20644978897)
May 18 00:49:31 wysat run[469]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=20644978897)
May 18 00:49:31 wysat run[469]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=20644978897)
^C
richard@wysat:~ $ journalctl -u wyoming-satellite.service -f
-- Journal begins at Mon 2024-03-11 20:13:58 CDT. --
May 18 00:47:56 wysat run[644]: Recording raw data 'stdin' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono
May 18 00:47:56 wysat run[642]: DEBUG:root:Connected to mic service
May 18 00:47:56 wysat run[642]: DEBUG:root:Connected to wake service
May 18 00:48:03 wysat run[642]: DEBUG:root:Server set: 27291338790
May 18 00:48:03 wysat run[642]: INFO:root:Connected to server
May 18 00:48:03 wysat run[642]: INFO:root:Waiting for wake word
May 18 00:48:05 wysat run[642]: DEBUG:root:Ping enabled
May 18 00:49:31 wysat run[642]: DEBUG:root:Detection(name='hey_jarvis_v0.1', timestamp=102095540476)
May 18 00:49:31 wysat run[642]: DEBUG:root:Streaming audio
May 18 00:49:31 wysat run[642]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)

@betodaviola
Copy link

Same problem here, trying to fix for a couple of days now with no success. Similarly to some people on this thread, I am using this page from the docs since that is the equipment I have (raspberry pi 0 2w and respeaker 2-mic pihat). In the logs bellow you can see the wake word triggering the first time I use it, but then it stops working:

$ sudo journalctl -u wyoming-satellite.service -f:
-- Journal begins at Thu 2024-07-04 01:14:43 BST. --
Jul 09 05:12:06 spoodnik1 run[1139]: INFO:root:Connected to services
Jul 09 05:12:06 spoodnik1 run[1139]: DEBUG:root:Using webrtc audio enhancements
Jul 09 05:12:06 spoodnik1 run[1141]: Recording raw data 'stdin' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono
Jul 09 05:12:06 spoodnik1 run[1139]: DEBUG:root:Connected to mic service
Jul 09 05:12:06 spoodnik1 run[1139]: DEBUG:root:Connected to wake service
Jul 09 05:12:07 spoodnik1 run[1139]: DEBUG:root:Server set: 1431054424930
Jul 09 05:12:07 spoodnik1 run[1139]: INFO:root:Connected to server
Jul 09 05:12:07 spoodnik1 run[1139]: INFO:root:Waiting for wake word
Jul 09 05:12:07 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1431193952790
Jul 09 05:12:09 spoodnik1 run[1139]: DEBUG:root:Ping enabled
Jul 09 05:12:15 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1439199064354
Jul 09 05:12:23 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1447203942010
Jul 09 05:12:31 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1455209925458
Jul 09 05:12:35 spoodnik1 run[1139]: DEBUG:root:Detection(name='ok_nabu_v0.1', timestamp=1458797918622, speaker=None)
Jul 09 05:12:35 spoodnik1 run[1139]: DEBUG:root:Streaming audio
Jul 09 05:12:35 spoodnik1 run[1139]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 16000, 'width': 2, 'channels': 1}}, payload=None)
Jul 09 05:12:39 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1463215630900
Jul 09 05:12:47 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1471219517763
Jul 09 05:12:55 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1479229214886
Jul 09 05:13:03 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1487227850808
Jul 09 05:13:11 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1495233622580
Jul 09 05:13:19 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1503238541588
Jul 09 05:13:27 spoodnik1 run[1139]: DEBUG:root:Connection cancelled: 1511243289506

My wyoming-satellite.service file:

[Unit]
Description=Wyoming Satellite
Wants=network-online.target
After=network-online.target
Requires=wyoming-openwakeword.service

[Service]
Type=simple
ExecStart=/home/spoodnik1/wyoming-satellite/script/run \
        --debug \
        --name 'spoodnik1' \
        --uri 'tcp://0.0.0.0:10700' \
        --mic-command 'arecord -D plughw:CARD=seeed2micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw' \
        --snd-command 'aplay -D plughw:CARD=seeed2micvoicec,DEV=0 -r 16000 -c 1 -f S16_LE -t raw' \
        --snd-command-rate 16000 \
        --snd-volume-multiplier 0.5 \
        --mic-auto-gain 5 \
        --mic-noise-suppression 2 \
        --wake-uri 'tcp://127.0.0.1:10400' \
        --wake-refractory-seconds 5 \
        --wake-word-name 'ok_nabu'
WorkingDirectory=/home/spoodnik1/wyoming-satellite
Restart=always
RestartSec=1

[Install]
WantedBy=default.target

@jerome83136
Copy link
Author

Hi @synesthesiam,
It seems there are several people facing the "keyword spotting only work once" issue

Can we kindly ask you for some support on this please? 🙏
Best regards

@TomasKliner
Copy link

Hello,
I had a similliar issue, but just getting detected once, without ever executing any commands.
I managed to locate the problem with Speech To Text in homeassistant logs.

Steps I took to fix it:

  1. installing addons Piper, Whisper and openWakeWord
  2. Adding them in integrations.
  3. Changing Speech To Text option to faster-whisper (inside settings -> voice assistant)

Not sure if all of the above is needed, but the core of the issue for me was not having a Speech to text agent configured, which made the service stuck in listening mode.

#87 I suppose this could be the same issue as I had.

@betodaviola
Copy link

Hello, I had a similliar issue, but just getting detected once, without ever executing any commands. I managed to locate the problem with Speech To Text in homeassistant logs.

Steps I took to fix it:

1. installing addons **Piper**, **Whisper** and **openWakeWord**

2. Adding them in **integrations**.

3. Changing Speech To Text option to **faster-whisper** (inside settings -> voice assistant)

Not sure if all of the above is needed, but the core of the issue for me was not having a Speech to text agent configured, which made the service stuck in listening mode.

#87 I suppose this could be the same issue as I had.

Thank you for sharing this. I will research about it and possibly try your solution today or tomorrow and report back. The logs you refer too are just the one in settings >> system >> logs? I don't seem to have any error regarding to speech-to-text in mine. DO you have any idea why these integrations need to be installed on the HA side of things? I thought that the satellite should take care of these things on their end, which is why I was trying to get it ready to go before working on the HA side of things. I am trying to figure out if we had indeed the same error

@TomasKliner
Copy link

First, I enabled logging on the Wyoming integration and then recreated the bug. Afterward, I went through the 'home-assistant.log' file, but it seems to be the same thing as you suggested. (Im fairly new to HA)

To be honest, I'm not entirely sure how it is intended to work in Home Assistant. My guess is you only need to have any speech-to-text service set up in the voice assistant settings.

ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved (None)
Traceback (most recent call last):

 File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/__init__.py", line 123, in 
 async_pipeline_from_audio_stream
    await pipeline_input.validate()

  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 1448, in validate
    raise PipelineRunValidationError(

homeassistant.components.assist_pipeline.pipeline.PipelineRunValidationError: the pipeline does not support speech-to-text

WARNING (MainThread) [homeassistant.components.wyoming.satellite] Satellite has been disconnected. Reconnecting in 10 second(s)

@betodaviola
Copy link

betodaviola commented Jul 10, 2024

It was not as straight forward for me, but it finally worked, thanks to @TomasKliner ! Unfortunately, I could not install the add-ons since there is no possibility for me to get a dedicated machine for HA, and add-ons are not supported in the docker container of HA installation. I understand all the arguments of why it is like this and although I agree with most of them, this problem proves that the argument of "but if you install on docker you are good enough to have any add-on on docker as well", which I read around github and reddit posts, is just wrong, but I digress.
I installed Piper, Whisper, and openwakeword on docker compose, and then created the pipeline on the assist (the pipeline is an easy process). Finding the compose files that half-worked for my setup, and then trying to find what was wrong and piece something that worked together was pretty difficult, since these compose files are nowhere I can find in the documentation. I don't know why they are not easily available since the container images are already there and documented.
If anyone find this thread in a similar situation as me, make sure to use the rhasspy image for the container since there are others, and if anyone wants to see my compose files I will happily share them, just bear in mind that the exact same thing might not work for you.
Edited for grammar.

@s-heppner
Copy link

s-heppner commented Oct 23, 2024

Similar issue here, in debug mode it only responds to wake word one time and gets stuck.

I've had the same problem and interestingly enough, as @richardoswald implies, when I removed the --debug flag from my startup command, it started working as intended, reacting to multiple wake words after each other.

Edit: Cancel that, it sometimes works and sometimes doesn't without a clear pattern to me.

Edit 2: After some more searching, I've got it to work via #217. I had to copy the custom wake word model to the default model location: <dir>/wyoming_openwakeword/models (See this comment).
In case of the docker container, this was located at: /usr/src/.venv/lib/python3.11/site-packages/wyoming_openwakeword/models/.
Then, I had to rename my custom wake word by adding _v0.1, so from example.tflite to example_v0.1.tflite (see this comment).
I have not the slightest idea of why this would work, but it does.

Having to copy a file into a Python package installed in a virtual environment is obviously an ugly workaround, though.

From my testing, I think there's also a second bug, that makes the wyoming-speaker docker container crash, if the --done-wav is still playing when the satellite wants to play the TTS response, that may or may not have anything to do with this problem in the broader sense.

Here's a docker compose file that works on my Raspberry Pi:

services:
  openwakeword:
    container_name: openwakeword
    image: rhasspy/wyoming-openwakeword
    ports:
      - "10400:10400"
    volumes:
      # (2024-10-24, s-heppner)
      # I've heard that putting the custom models in the same folder as the main ones
      # might fix the problems I have with the wake word randomly stopping working:
      # https://github.com/rhasspy/wyoming-satellite/issues/217#issuecomment-2345074481
      # (2024-10-24, s-heppner)
      # Don't ask me how I found out the default model directory...
      # (2024-10-24, s-heppner)
      # Apparently, you also have to add `_v0.1` to the filename: 
      # https://github.com/rhasspy/wyoming-satellite/issues/217#issuecomment-2349154398
      # Makes total sense now. 
      - ./custom_wakeword_models/example.tflite:/usr/src/.venv/lib/python3.11/site-packages/wyoming_openwakeword/models/example_v0.1.tflite
    command: >
      --preload-model 'example'
      --custom-model-dir '/usr/src/.venv/lib/python3.11/site-packages/wyoming_openwakeword/models/'

  microphone:
    container_name: wyoming-microphone
    build:
      context: https://github.com/rhasspy/wyoming-mic-external.git
    ports:
      - "10600:10600"
    devices:
      - /dev/snd:/dev/snd
    group_add:
      - audio
    command:
      - "--device"
      - "<your device here>"
      - "--debug"
  playback:
    container_name: wyoming-speaker
    build:
      context: https://github.com/rhasspy/wyoming-snd-external.git
    ports:
      - "10601:10601"
    devices:
      - /dev/snd:/dev/snd
    group_add:
      - audio
    command:
      - "--device"
      - "<your device here>"
      - "--debug"
  satellite:
    container_name: wyoming-satellite
    build:
      context: https://github.com/rhasspy/wyoming-satellite.git
    ports:
      - "10700:10700"
    volumes:
      # Mount the sound effects
      - ./sounds/on.wav:/etc/on.wav
      - ./sounds/off.wav:/etc/off.wav
    command:
      - "--name"
      - "Office Satellite"
      - "--mic-uri"
      - "tcp://microphone:10600"
      # (2024-10-24, s-heppner)
      # Maybe we want to make the `mic_seconds_to_mute_after_awake_wav=0.5`
      # default shorter here? Sometimes I feel it cuts the beginning of my
      # sentence.
      - "--snd-uri"
      - "tcp://playback:10601"
      - "--wake-uri"
      - "tcp://openwakeword:10400"
      - "--wake-word-name"
      - 'example'
      - "--awake-wav"
      - "/etc/on.wav"
      # (2024-10-24, s-heppner)
      # We cannot use the `done` sound, as it itself is not done playing
      # when satellite tries to answer, resulting in an error. Maybe this can
      # be fixed by making the sound shorter? 
      # - "--done-wav"
      # - "/etc/off.wav"
      - "--debug"

@SofaSurfer76
Copy link

SofaSurfer76 commented Oct 26, 2024

@jerome83136 did you find a solution? I'm stuck here as well:

hass@assistant:~ $ journalctl -u wyoming-satellite.service -f
Oct 26 22:18:42 assistant run[1923]: DEBUG:root:Connecting to wake service: tcp://127.0.0.1:10400
Oct 26 22:18:42 assistant run[1923]: INFO:root:Connected to services
Oct 26 22:18:42 assistant run[1923]: DEBUG:root:Using webrtc audio enhancements
Oct 26 22:18:42 assistant run[1923]: DEBUG:root:Connected to mic service
Oct 26 22:18:42 assistant run[1923]: DEBUG:root:Connected to wake service
Oct 26 22:18:44 assistant run[1923]: DEBUG:root:Server set: 392895522564
Oct 26 22:18:44 assistant run[1923]: INFO:root:Connected to server
Oct 26 22:18:44 assistant run[1923]: INFO:root:Waiting for wake word
Oct 26 22:18:44 assistant run[1923]: DEBUG:root:Started recording to /home/hass/wyoming-satellite/local/debug-recording/392896863474-wake.wav
Oct 26 22:18:46 assistant run[1923]: DEBUG:root:Ping enabled
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Stopped recording to /home/hass/wyoming-satellite/local/debug-recording/392896863474-wake.wav
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Started recording to /home/hass/wyoming-satellite/local/debug-recording/392896863474-stt.wav
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Detection(name='hey_jarvis_v0.1', timestamp=606123017352, speaker=None)
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Streaming audio
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
Oct 26 22:22:17 assistant run[1923]: DEBUG:root:Connected to snd service
Oct 26 22:22:18 assistant run[1923]: DEBUG:root:Unmuted microphone

hass@assistant:~/wyoming-satellite $ journalctl -u wyoming-wakeword.service -f
Oct 26 21:26:24 assistant run[3191]: DEBUG:wyoming_openwakeword.handler:Client connected: 6316746992816
Oct 26 21:26:24 assistant run[3191]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/hass/wyoming-satellite/local/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
Oct 26 21:26:24 assistant run[3191]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
Oct 26 21:26:24 assistant run[3191]: INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Oct 26 21:26:26 assistant run[3191]: DEBUG:root:Loading hey_jarvis_v0.1 from /home/hass/wyoming-satellite/local/wyoming-openwakeword/wyoming_openwakeword/models/hey_jarvis_v0.1.tflite
Oct 26 21:26:26 assistant run[3191]: DEBUG:wyoming_openwakeword.handler:Started thread for hey_jarvis_v0.1
Oct 26 21:26:26 assistant run[3191]: DEBUG:wyoming_openwakeword.handler:Sent info to client: 6316746992816
Oct 26 21:26:38 assistant run[3191]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=6316746992816)
Oct 26 21:26:38 assistant run[3191]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=6316746992816)
Oct 26 21:26:38 assistant run[3191]: DEBUG:root:Triggered hey_jarvis_v0.1 (client=6316746992816)
[Service]
Type=simple
User=hass
Environment=XDG_RUNTIME_DIR=/run/user/1000
ExecStart=/home/hass/wyoming-satellite/script/run \
    --name Assist \
    --uri tcp://0.0.0.0:10700 \
    --mic-command 'arecord -D plughw:CARD=wm8960soundcard,DEV=0 -q -r 16000 -c 1 -f S16_LE -t raw' \
    --snd-command 'aplay -D plughw:CARD=wm8960soundcard,DEV=0 -q -r 22050 -c 1 -f S16_LE -t raw' \
    --awake-wav /home/hass/wyoming-satellite/sounds/awake.wav \
    --done-wav /home/hass/wyoming-satellite/sounds/done.wav \
    --mic-noise-suppression 2 \
    --mic-auto-gain 5 \
    --wake-uri tcp://127.0.0.1:10400 \
    --wake-word-name hey_jarvis \
    --debug \
    --debug-recording-dir /home/hass/wyoming-satellite/local/debug-recording \
    --wake-refractory-seconds 10 \

I have no Idea how to debug it and understand where the pipeline stop. after restarting the service I can trigger the wake word again, sometimes rarely it (kind of) auto-reset.

@synesthesiam :-)

edit: added complete DEBUG journals

@SofaSurfer76
Copy link

SofaSurfer76 commented Nov 7, 2024

After several attempt I went a bit forward:

hass@assistant:~ $ journalctl -u wyoming-satellite.service -f
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Detection(name='ok_nabu_v0.1', timestamp=19785407726927, speaker=None)
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Streaming audio
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Running ['examples/commands/detection.sh']
Nov 07 18:49:24 assistant run[2771]: Wake word detected: ok_nabu_v0.1
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Running ['examples/commands/streaming_start.sh']
Nov 07 18:49:24 assistant run[2774]: Audio streaming to server has started
Nov 07 18:49:24 assistant run[2753]: DEBUG:root:Connected to snd service
Nov 07 18:49:25 assistant run[2753]: DEBUG:root:Unmuted microphone

my config for wyoming satellite is

[Service]
Type=simple
User=hass
Environment=XDG_RUNTIME_DIR=/run/user/1000
ExecStart=/home/hass/wyoming-satellite/script/run \
    --name 'Assist' \
    --uri 'tcp://0.0.0.0:10700' \
    --mic-command 'arecord -D plughw:CARD=seeed2micvoicec,DEV=0 -q -r 16000 -c 1 -f S16_LE -t raw' \
    --snd-command 'aplay -D plughw:CARD=seeed2micvoicec,DEV=0 -q -r 22050 -c 1 -f S16_LE -t raw' \
    --awake-wav '/home/hass/wyoming-satellite/sounds/awake.wav' \
    --done-wav '/home/hass/wyoming-satellite/sounds/done.wav' \
    --timer-finished-wav '/home/hass/wyoming-satellite/sounds/done.wav' \
    --mic-noise-suppression 2 \
    --mic-auto-gain 5 \
    --wake-uri 'tcp://127.0.0.1:10400' \
    --wake-word-name 'ok_nabu' \
    --debug \
    --debug-recording-dir '/home/hass/wyoming-satellite/local/debug-recording' \
#    --mic-seconds-to-mute-after-awake-wav 0.1 \
    --wake-refractory-seconds 10 \
    --tts-stop-command 'true' \
    --detection-command 'examples/commands/detection.sh' \
    --startup-command 'examples/commands/startup.sh' \
    --streaming-start-command 'examples/commands/streaming_start.sh' \
    --streaming-stop-command 'examples/commands/streaming_stop.sh' \
    --synthesize-command 'examples/commands/synthesize.sh' \
    --stt-start-command 'examples/commands/stt_start.sh' \
    --stt-stop-command 'examples/commands/stt_stop.sh' \
    --tts-start-command 'examples/commands/tts_start.sh' \
    --tts-stop-command 'examples/commands/tts_stop.sh'
#    --event-uri tcp://127.0.0.1:10500
WorkingDirectory=/home/hass/wyoming-satellite
Restart=always
RestartSec=1

But on Home Assistant side I still don't see "Assist in progress" binary sensor activating..
The mute button on HA works and pause the satellite.

Someone can help me to understand?

EDIT: Solved, the TTS and STT integrations were not configured.

@d1egoaz
Copy link

d1egoaz commented Nov 18, 2024

For reference, when I test this directly on the device, it doesn't work and it gets stuck (see my previous comments).

However, when I use it through Home Assistant, it magically works as expected 🤷, which is how I plan to use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants