🤖Have you ever tried Chat.M5Stack.com before asking??😎
    M5Stack Community
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    How to debug Module LLM Kit when it silently dies?

    SOFTWARE
    llm
    1
    4
    147
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      riywo
      last edited by

      Hi, I've just got Module LLM Kit and got started with CoreS3 Lite by running some examples.

      TextAssistant works well:
      https://github.com/m5stack/M5Module-LLM/blob/main/examples/TextAssistant/TextAssistant.ino

      However, TTS never works. It keeps sending the text to TTS module but no audio sounds:
      https://github.com/m5stack/M5Module-LLM/blob/main/examples/TTS/TTS.ino

      I installed many apt packages but nothing changed. So, I tried strace because I don't know where the log output is and realized that llm-sys restarted once it receives a message. I couldn't spot the exact message yet but the last message was something like this:

      [pid  3285] write(1, "W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\nllm_sys will be exit!\n", 1526) = 1526

      Q: Is there anyway to debug more on the LLM Kit side?

      Update:

      I realized they are systemd services. So, these are the logs:

      llm-tts.service

      root@m5stack-LLM:~# journalctl -u llm-tts -f
Oct 01 03:34:11 m5stack-LLM systemd[1]: Started llm-tts Service.
Oct 01 03:34:45 m5stack-LLM llm_tts-1.6[10412]: [I][operator()][ 189]: serial_zmq_url:ipc:///tmp/llm/5556.sock
Oct 01 03:34:45 m5stack-LLM llm_tts-1.6[10412]: [W][__sigint][  24]: llm_sys will be exit!
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopping llm-tts Service...
Oct 01 03:34:45 m5stack-LLM systemd[1]: llm-tts.service: Deactivated successfully.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopped llm-tts Service.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Started llm-tts Service.

      llm-sys.service

      Oct 01 03:34:10 m5stack-LLM systemd[1]: Started llm-sys Service.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopping llm-sys Service...
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: server listen on port 10001, listenfd=18 ...
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: llm_sys will be exit!
Oct 01 03:34:45 m5stack-LLM systemd[1]: llm-sys.service: Deactivated successfully.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopped llm-sys Service.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Started llm-sys Service.

      My environment:

      • M5Stack CoreS3 Lite
        • PlatformIO
          • pioarduino stable
          • lib_deps = M5Unified, M5Module-LLM
      • Module LLM Kit
        • apt update and apt upgrade
        • apt list | grep llm | grep installed is below:
      lib-llm/stable,now 1.8 arm64 [installed]
llm-asr/stable,now 1.7 arm64 [installed]
llm-audio-en-us/now 0.2 arm64 [installed,local]
llm-audio-zh-cn/now 0.2 arm64 [installed,local]
llm-audio/stable,now 1.6 arm64 [installed]
llm-camera/stable,now 1.9 arm64 [installed]
llm-kws/stable,now 1.9 arm64 [installed]
llm-llm/stable,now 1.9 arm64 [installed]
llm-melotts-zh-cn/now 0.2 arm64 [installed,local]
llm-melotts/stable,now 1.9 arm64 [installed]
llm-model-melotts-en-default/stable,now 0.6 arm64 [installed]
llm-model-melotts-en-us/stable,now 0.6 arm64 [installed]
llm-model-melotts-ja-jp/stable,now 0.6 arm64 [installed]
llm-model-qwen2.5-0.5b-prefill-20e/stable,now 0.2 arm64 [installed]
llm-model-qwen3-0.6b-ax630c/stable,now 0.4 arm64 [installed]
llm-model-sherpa-onnx-kws-zipformer-gigaspeech-3.3m-2024-01-01/stable,now 0.3 arm64 [installed]
llm-model-silero-vad/stable,now 0.4 arm64 [installed]
llm-model-smolvlm-500m-ax630c/stable,now 0.4 arm64 [installed]
llm-model-whisper-tiny/stable,now 0.4 arm64 [installed]
llm-openai-api/stable,now 1.7 arm64 [installed]
llm-sherpa-ncnn-streaming-zipformer-20m-2023-02-17/now 0.2 arm64 [installed,local]
llm-sherpa-ncnn-streaming-zipformer-zh-14m-2023-02-23/now 0.2 arm64 [installed,local]
llm-sherpa-onnx-kws-zipformer-wenetspeech-3.3m-2024-01-01/now 0.2 arm64 [installed,local]
llm-single-speaker-english-fast/now 0.2 arm64 [installed,local]
llm-single-speaker-fast/now 0.2 arm64 [installed,local]
llm-skel/stable,now 1.5 arm64 [installed]
llm-sys/stable,now 1.6 arm64 [installed]
llm-tts/stable,now 1.6 arm64 [installed]
llm-vad/stable,now 1.8 arm64 [installed]
llm-vlm/stable,now 1.9 arm64 [installed]
llm-whisper/stable,now 1.8 arm64 [installed]
llm-yolo11n-pose/now 0.2 arm64 [installed,local]
llm-yolo11n-seg/now 0.2 arm64 [installed,local]
llm-yolo11n/now 0.2 arm64 [installed,local]
llm-yolo/stable,now 1.9 arm64 [installed]
      1 Reply Last reply Reply Quote 0
      • R
        riywo
        last edited by

        Update 2:

        I confirmed TTS itself is working if it is used through OpenAI API server: https://docs.m5stack.com/en/stackflow/openai_api/text_to_speech

        1 Reply Last reply Reply Quote 0
        • R
          riywo
          last edited by

          Update 3:

          Captured this log sometime:

          Oct 01 11:18:51 m5stack-LLM systemd[1]: llm-tts.service: Main process exited, code=killed, status=11/SEGV
Oct 01 11:18:51 m5stack-LLM systemd[1]: llm-tts.service: Failed with result 'signal'.
Oct 01 11:18:52 m5stack-LLM systemd[1]: llm-tts.service: Scheduled restart job, restart counter is at 1.
Oct 01 11:18:52 m5stack-LLM systemd[1]: Stopped llm-tts Service.
Oct 01 11:18:52 m5stack-LLM systemd[1]: Started llm-tts Service.
          1 Reply Last reply Reply Quote 0
          • R
            riywo
            last edited by

            Ok, it worked finally. At least, I needed to use module_llm.melotts instead of module_llm.tts.

            The model specification in tts_config doesn't matter for my case. It picks melotts-en-default somehow automatically even though the default value is melotts_zh-cn.

            The module is very unstable when the controller is powered on. It requires several clicks of the reset button until it finally speaks. (The return value of melotts.inference is -97 when it's successful.)

            1 Reply Last reply Reply Quote 0
            • First post
              Last post