[PATCH 0/9] ASoC: SOF: ipc4/Intel: Support for firmware exception handling
Hi,
When a firmware crashes it creats a panic information into a telemetry slot. The panic format is defined by Zephyr, includes stack and additional information to help to identify the reason for the crash. Part of the firmware exception handling the firmware also sends an EXCEPTION_CAUGHT notification.
This series implements the kernel side handling of the exception: print information into the kernel log export the whole telemetry slot to user space for tools extract additional information from the panic dump.
Regards, Peter --- Rander Wang (9): ASoC: SOF: Xtensa: dump ar registers to restore call stack ASoC: SOF: ipc4-mtrace: move debug slot related definitions to header.h ASoC: SOF: ipc4: add a helper function to search debug slot ASoC: SOF: ipc4: add definition of telemetry slot for exception handling ASoC: SOF: ipc4: add exception node in sof debugfs directory ASoC: SOF: Intel: add telemetry retrieval support on Intel platforms ASoC: SOF: Intel: mtl: dump dsp stack ASoC: SOF: Intel: hda: add ipc4 FW panic support on CAVS 2.5+ platforms ASoC: SOF: ipc4: handle EXCEPTION_CAUGHT notification from firmware
include/sound/sof/ipc4/header.h | 17 ++++++ sound/soc/sof/Makefile | 2 +- sound/soc/sof/intel/Makefile | 3 +- sound/soc/sof/intel/hda.c | 14 +++++ sound/soc/sof/intel/hda.h | 1 + sound/soc/sof/intel/mtl.c | 3 ++ sound/soc/sof/intel/telemetry.c | 95 +++++++++++++++++++++++++++++++++ sound/soc/sof/intel/telemetry.h | 35 ++++++++++++ sound/soc/sof/intel/tgl.c | 1 + sound/soc/sof/ipc4-mtrace.c | 34 ++++-------- sound/soc/sof/ipc4-priv.h | 3 ++ sound/soc/sof/ipc4-telemetry.c | 95 +++++++++++++++++++++++++++++++++ sound/soc/sof/ipc4-telemetry.h | 73 +++++++++++++++++++++++++ sound/soc/sof/ipc4.c | 29 ++++++++++ sound/soc/sof/xtensa/core.c | 11 ++++ 15 files changed, 391 insertions(+), 25 deletions(-) create mode 100644 sound/soc/sof/intel/telemetry.c create mode 100644 sound/soc/sof/intel/telemetry.h create mode 100644 sound/soc/sof/ipc4-telemetry.c create mode 100644 sound/soc/sof/ipc4-telemetry.h
From: Rander Wang rander.wang@intel.com
On Xtensa platform ar0 is for caller address and ar1 is for stack address. The ar register dump can be used to rebuild call stack with FW elf file by debug tools.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/xtensa/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
diff --git a/sound/soc/sof/xtensa/core.c b/sound/soc/sof/xtensa/core.c index bebbe3a2865c..7c91a919eadc 100644 --- a/sound/soc/sof/xtensa/core.c +++ b/sound/soc/sof/xtensa/core.c @@ -132,6 +132,17 @@ static void xtensa_stack(struct snd_sof_dev *sdev, const char *level, void *oops buf, sizeof(buf), false); dev_printk(level, sdev->dev, "0x%08x: %s\n", stack_ptr + i * 4, buf); } + + if (!xoops->plat_hdr.numaregs) + return; + + dev_printk(level, sdev->dev, "AR registers:\n"); + /* the number of ar registers is a multiple of 4 */ + for (i = 0; i < xoops->plat_hdr.numaregs; i += 4) { + hex_dump_to_buffer(xoops->ar + i, 16, 16, 4, + buf, sizeof(buf), false); + dev_printk(level, sdev->dev, "%#x: %s\n", i * 4, buf); + } }
const struct dsp_arch_ops sof_xtensa_arch_ops = {
From: Rander Wang rander.wang@intel.com
The macro definitions of debug slot can be used by gdb, telemetry and mtrace log, so move these definitions to header.h from mtrace. Then these macro definitions can be shared
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- include/sound/sof/ipc4/header.h | 17 +++++++++++++++++ sound/soc/sof/ipc4-mtrace.c | 34 +++++++++++---------------------- 2 files changed, 28 insertions(+), 23 deletions(-)
diff --git a/include/sound/sof/ipc4/header.h b/include/sound/sof/ipc4/header.h index c58f00ef054a..574a9d581f88 100644 --- a/include/sound/sof/ipc4/header.h +++ b/include/sound/sof/ipc4/header.h @@ -515,6 +515,23 @@ struct sof_ipc4_notify_resource_data { uint32_t data[6]; } __packed __aligned(4);
+#define SOF_IPC4_DEBUG_DESCRIPTOR_SIZE 12 /* 3 x u32 */ + +/* + * The debug memory window is divided into 16 slots, and the + * first slot is used as a recorder for the other 15 slots. + */ +#define SOF_IPC4_MAX_DEBUG_SLOTS 15 +#define SOF_IPC4_DEBUG_SLOT_SIZE 0x1000 + +/* debug log slot types */ +#define SOF_IPC4_DEBUG_SLOT_UNUSED 0x00000000 +#define SOF_IPC4_DEBUG_SLOT_CRITICAL_LOG 0x54524300 /* byte 0: core ID */ +#define SOF_IPC4_DEBUG_SLOT_DEBUG_LOG 0x474f4c00 /* byte 0: core ID */ +#define SOF_IPC4_DEBUG_SLOT_GDB_STUB 0x42444700 +#define SOF_IPC4_DEBUG_SLOT_TELEMETRY 0x4c455400 +#define SOF_IPC4_DEBUG_SLOT_BROKEN 0x44414544 + /** @}*/
#endif diff --git a/sound/soc/sof/ipc4-mtrace.c b/sound/soc/sof/ipc4-mtrace.c index 2b4659a1768e..9f1e33ee8826 100644 --- a/sound/soc/sof/ipc4-mtrace.c +++ b/sound/soc/sof/ipc4-mtrace.c @@ -41,24 +41,12 @@ * The two pointers are offsets within the buffer. */
-#define SOF_MTRACE_DESCRIPTOR_SIZE 12 /* 3 x u32 */ - #define FW_EPOCH_DELTA 11644473600LL
-#define INVALID_SLOT_OFFSET 0xffffffff #define MAX_ALLOWED_LIBRARIES 16 -#define MAX_MTRACE_SLOTS 15 - -#define SOF_MTRACE_PAGE_SIZE 0x1000 -#define SOF_MTRACE_SLOT_SIZE SOF_MTRACE_PAGE_SIZE - -/* debug log slot types */ -#define SOF_MTRACE_SLOT_UNUSED 0x00000000 -#define SOF_MTRACE_SLOT_CRITICAL_LOG 0x54524300 /* byte 0: core ID */ -#define SOF_MTRACE_SLOT_DEBUG_LOG 0x474f4c00 /* byte 0: core ID */ -#define SOF_MTRACE_SLOT_GDB_STUB 0x42444700 -#define SOF_MTRACE_SLOT_TELEMETRY 0x4c455400 -#define SOF_MTRACE_SLOT_BROKEN 0x44414544 + +#define SOF_IPC4_INVALID_SLOT_OFFSET 0xffffffff + /* for debug and critical types */ #define SOF_MTRACE_SLOT_CORE_MASK GENMASK(7, 0) #define SOF_MTRACE_SLOT_TYPE_MASK GENMASK(31, 8) @@ -140,7 +128,7 @@ static int sof_ipc4_mtrace_dfs_open(struct inode *inode, struct file *file) if (unlikely(ret)) goto out;
- core_data->log_buffer = kmalloc(SOF_MTRACE_SLOT_SIZE, GFP_KERNEL); + core_data->log_buffer = kmalloc(SOF_IPC4_DEBUG_SLOT_SIZE, GFP_KERNEL); if (!core_data->log_buffer) { debugfs_file_put(file->f_path.dentry); ret = -ENOMEM; @@ -212,13 +200,13 @@ static ssize_t sof_ipc4_mtrace_dfs_read(struct file *file, char __user *buffer, return 0; }
- if (core_data->slot_offset == INVALID_SLOT_OFFSET) + if (core_data->slot_offset == SOF_IPC4_INVALID_SLOT_OFFSET) return 0;
/* The log data buffer starts after the two pointer in the slot */ log_buffer_offset = core_data->slot_offset + (sizeof(u32) * 2); /* The log data size excludes the pointers */ - log_buffer_size = SOF_MTRACE_SLOT_SIZE - (sizeof(u32) * 2); + log_buffer_size = SOF_IPC4_DEBUG_SLOT_SIZE - (sizeof(u32) * 2);
read_ptr = core_data->host_read_ptr; write_ptr = core_data->dsp_write_ptr; @@ -510,13 +498,13 @@ static void sof_mtrace_find_core_slots(struct snd_sof_dev *sdev) u32 slot_desc_type_offset, type, core; int i;
- for (i = 0; i < MAX_MTRACE_SLOTS; i++) { + for (i = 0; i < SOF_IPC4_MAX_DEBUG_SLOTS; i++) { /* The type is the second u32 in the slot descriptor */ slot_desc_type_offset = sdev->debug_box.offset; - slot_desc_type_offset += SOF_MTRACE_DESCRIPTOR_SIZE * i + sizeof(u32); + slot_desc_type_offset += SOF_IPC4_DEBUG_DESCRIPTOR_SIZE * i + sizeof(u32); sof_mailbox_read(sdev, slot_desc_type_offset, &type, sizeof(type));
- if ((type & SOF_MTRACE_SLOT_TYPE_MASK) == SOF_MTRACE_SLOT_DEBUG_LOG) { + if ((type & SOF_MTRACE_SLOT_TYPE_MASK) == SOF_IPC4_DEBUG_SLOT_DEBUG_LOG) { core = type & SOF_MTRACE_SLOT_CORE_MASK;
if (core >= sdev->num_cores) { @@ -533,7 +521,7 @@ static void sof_mtrace_find_core_slots(struct snd_sof_dev *sdev) * debug_box + SOF_MTRACE_SLOT_SIZE offset */ core_data->slot_offset = sdev->debug_box.offset; - core_data->slot_offset += SOF_MTRACE_SLOT_SIZE * (i + 1); + core_data->slot_offset += SOF_IPC4_DEBUG_SLOT_SIZE * (i + 1); dev_dbg(sdev->dev, "slot%d is used for core%u\n", i, core); if (core_data->delayed_pos_update) { sof_ipc4_mtrace_update_pos(sdev, core); @@ -633,7 +621,7 @@ int sof_ipc4_mtrace_update_pos(struct snd_sof_dev *sdev, int core)
core_data = &priv->cores[core];
- if (core_data->slot_offset == INVALID_SLOT_OFFSET) { + if (core_data->slot_offset == SOF_IPC4_INVALID_SLOT_OFFSET) { core_data->delayed_pos_update = true; return 0; }
From: Rander Wang rander.wang@intel.com
Currently IPC4 supports GDB slot, telemetry slot and debug slot. This helper function will be used to get the slot offset in debug windows for further processing.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/ipc4-priv.h | 3 +++ sound/soc/sof/ipc4.c | 23 +++++++++++++++++++++++ 2 files changed, 26 insertions(+)
diff --git a/sound/soc/sof/ipc4-priv.h b/sound/soc/sof/ipc4-priv.h index a5d0b2eae464..9e69b7c29117 100644 --- a/sound/soc/sof/ipc4-priv.h +++ b/sound/soc/sof/ipc4-priv.h @@ -120,4 +120,7 @@ void sof_ipc4_update_cpc_from_manifest(struct snd_sof_dev *sdev, struct sof_ipc4_fw_module *fw_module, struct sof_ipc4_base_module_cfg *basecfg);
+size_t sof_ipc4_find_debug_slot_offset_by_type(struct snd_sof_dev *sdev, + u32 slot_type); + #endif diff --git a/sound/soc/sof/ipc4.c b/sound/soc/sof/ipc4.c index 6c33ee5af32b..4673c53f6c03 100644 --- a/sound/soc/sof/ipc4.c +++ b/sound/soc/sof/ipc4.c @@ -547,6 +547,29 @@ static int sof_ipc4_init_msg_memory(struct snd_sof_dev *sdev) return 0; }
+size_t sof_ipc4_find_debug_slot_offset_by_type(struct snd_sof_dev *sdev, + u32 slot_type) +{ + size_t slot_desc_type_offset; + u32 type; + int i; + + /* The type is the second u32 in the slot descriptor */ + slot_desc_type_offset = sdev->debug_box.offset + sizeof(u32); + for (i = 0; i < SOF_IPC4_MAX_DEBUG_SLOTS; i++) { + sof_mailbox_read(sdev, slot_desc_type_offset, &type, sizeof(type)); + + if (type == slot_type) + return sdev->debug_box.offset + (i + 1) * SOF_IPC4_DEBUG_SLOT_SIZE; + + slot_desc_type_offset += SOF_IPC4_DEBUG_DESCRIPTOR_SIZE; + } + + dev_dbg(sdev->dev, "Slot type %#x is not available in debug window\n", slot_type); + return 0; +} +EXPORT_SYMBOL(sof_ipc4_find_debug_slot_offset_by_type); + static int ipc4_fw_ready(struct snd_sof_dev *sdev, struct sof_ipc4_msg *ipc4_msg) { int inbox_offset, inbox_size, outbox_offset, outbox_size;
From: Rander Wang rander.wang@intel.com
Core dump includes hardware platform information, cpu registers and exception call stack. FW saves core dump to telemetry slot in shared memory window for host in the event of FW exception. This patch creates exception node in debugfs for user to dump telemetry data.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/Makefile | 2 +- sound/soc/sof/ipc4-telemetry.c | 95 ++++++++++++++++++++++++++++++++++ sound/soc/sof/ipc4-telemetry.h | 73 ++++++++++++++++++++++++++ 3 files changed, 169 insertions(+), 1 deletion(-) create mode 100644 sound/soc/sof/ipc4-telemetry.c create mode 100644 sound/soc/sof/ipc4-telemetry.h
diff --git a/sound/soc/sof/Makefile b/sound/soc/sof/Makefile index 744d40bd8c8b..42dc48e53964 100644 --- a/sound/soc/sof/Makefile +++ b/sound/soc/sof/Makefile @@ -10,7 +10,7 @@ snd-sof-objs += ipc3.o ipc3-loader.o ipc3-topology.o ipc3-control.o ipc3-pcm.o\ endif ifneq ($(CONFIG_SND_SOC_SOF_INTEL_IPC4),) snd-sof-objs += ipc4.o ipc4-loader.o ipc4-topology.o ipc4-control.o ipc4-pcm.o\ - ipc4-mtrace.o + ipc4-mtrace.o ipc4-telemetry.o endif
# SOF client support diff --git a/sound/soc/sof/ipc4-telemetry.c b/sound/soc/sof/ipc4-telemetry.c new file mode 100644 index 000000000000..ec4ae9674364 --- /dev/null +++ b/sound/soc/sof/ipc4-telemetry.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) +// +// This file is provided under a dual BSD/GPLv2 license. When using or +// redistributing this file, you may do so under either license. +// +// Copyright(c) 2018-2023 Intel Corporation. All rights reserved. +// + +#include <linux/debugfs.h> +#include <linux/io.h> +#include <linux/pm_runtime.h> +#include <sound/sof/debug.h> +#include <sound/sof/ipc4/header.h> +#include "sof-priv.h" +#include "ops.h" +#include "ipc4-telemetry.h" +#include "ipc4-priv.h" + +static void __iomem *sof_ipc4_query_exception_address(struct snd_sof_dev *sdev) +{ + u32 type = SOF_IPC4_DEBUG_SLOT_TELEMETRY; + size_t telemetry_slot_offset; + u32 offset; + + telemetry_slot_offset = sof_ipc4_find_debug_slot_offset_by_type(sdev, type); + if (!telemetry_slot_offset) + return NULL; + + /* skip the first separator magic number */ + offset = telemetry_slot_offset + sizeof(u32); + + return sdev->bar[sdev->mailbox_bar] + offset; +} + +static ssize_t sof_telemetry_entry_read(struct file *file, char __user *buffer, + size_t count, loff_t *ppos) +{ + struct snd_sof_dfsentry *dfse = file->private_data; + struct snd_sof_dev *sdev = dfse->sdev; + void __iomem *io_addr; + loff_t pos = *ppos; + size_t size_ret; + u8 *buf; + + if (pos < 0) + return -EINVAL; + /* skip the first separator magic number */ + if (pos >= SOF_IPC4_DEBUG_SLOT_SIZE - 4 || !count) + return 0; + if (count > SOF_IPC4_DEBUG_SLOT_SIZE - 4 - pos) + count = SOF_IPC4_DEBUG_SLOT_SIZE - 4 - pos; + + io_addr = sof_ipc4_query_exception_address(sdev); + if (!io_addr) + return -EFAULT; + + buf = kzalloc(SOF_IPC4_DEBUG_SLOT_SIZE - 4, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + memcpy_fromio(buf, io_addr, SOF_IPC4_DEBUG_SLOT_SIZE - 4); + size_ret = copy_to_user(buffer, buf + pos, count); + if (size_ret) { + kfree(buf); + return -EFAULT; + } + + *ppos = pos + count; + kfree(buf); + + return count; +} + +static const struct file_operations sof_telemetry_fops = { + .open = simple_open, + .read = sof_telemetry_entry_read, +}; + +void sof_ipc4_create_exception_debugfs_node(struct snd_sof_dev *sdev) +{ + struct snd_sof_dfsentry *dfse; + + dfse = devm_kzalloc(sdev->dev, sizeof(*dfse), GFP_KERNEL); + if (!dfse) + return; + + dfse->type = SOF_DFSENTRY_TYPE_IOMEM; + dfse->size = SOF_IPC4_DEBUG_SLOT_SIZE - 4; + dfse->access_type = SOF_DEBUGFS_ACCESS_ALWAYS; + dfse->sdev = sdev; + + list_add(&dfse->list, &sdev->dfsentry_list); + + debugfs_create_file("exception", 0444, sdev->debugfs_root, dfse, &sof_telemetry_fops); +} diff --git a/sound/soc/sof/ipc4-telemetry.h b/sound/soc/sof/ipc4-telemetry.h new file mode 100644 index 000000000000..ab3599e3d87d --- /dev/null +++ b/sound/soc/sof/ipc4-telemetry.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */ +/* + * This file is provided under a dual BSD/GPLv2 license. When using or + * redistributing this file, you may do so under either license. + * + * Copyright(c) 2023 Intel Corporation. All rights reserved. + */ + +#ifndef __SOUND_SOC_SOF_IPC4_TELEMETRY_H +#define __SOUND_SOC_SOF_IPC4_TELEMETRY_H + +/* Target code */ +enum sof_ipc4_coredump_tgt_code { + COREDUMP_TGT_UNKNOWN = 0, + COREDUMP_TGT_X86, + COREDUMP_TGT_X86_64, + COREDUMP_TGT_ARM_CORTEX_M, + COREDUMP_TGT_RISC_V, + COREDUMP_TGT_XTENSA, +}; + +#define COREDUMP_ARCH_HDR_ID 'A' +#define COREDUMP_HDR_ID0 'Z' +#define COREDUMP_HDR_ID1 'E' + +#define XTENSA_BLOCK_HDR_VER 2 +#define XTENSA_CORE_DUMP_SEPARATOR 0x0DEC0DEB +#define XTENSA_CORE_AR_REGS_COUNT 16 +#define XTENSA_SOC_INTEL_ADSP 3 +#define XTENSA_TOOL_CHAIN_ZEPHYR 1 +#define XTENSA_TOOL_CHAIN_XCC 2 + +/* Coredump header */ +struct sof_ipc4_coredump_hdr { + /* 'Z', 'E' as identifier of file */ + char id[2]; + + /* Identify the version of the header */ + u16 hdr_version; + + /* Indicate which target (e.g. architecture or SoC) */ + u16 tgt_code; + + /* Size of uintptr_t in power of 2. (e.g. 5 for 32-bit, 6 for 64-bit) */ + u8 ptr_size_bits; + + u8 flag; + + /* Reason for the fatal error */ + u32 reason; +} __packed; + +/* Architecture-specific block header */ +struct sof_ipc4_coredump_arch_hdr { + /* COREDUMP_ARCH_HDR_ID to indicate this is a architecture-specific block */ + char id; + + /* Identify the version of this block */ + u16 hdr_version; + + /* Number of bytes following the header */ + u16 num_bytes; +} __packed; + +struct sof_ipc4_telemetry_slot_data { + u32 separator; + struct sof_ipc4_coredump_hdr hdr; + struct sof_ipc4_coredump_arch_hdr arch_hdr; + u32 arch_data[]; +} __packed; + +void sof_ipc4_create_exception_debugfs_node(struct snd_sof_dev *sdev); +#endif
From: Rander Wang rander.wang@intel.com
The exception node is created when FW is ready and clear to zero when FW post boot.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/ipc4.c | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/sound/soc/sof/ipc4.c b/sound/soc/sof/ipc4.c index 4673c53f6c03..82f2f196c9c2 100644 --- a/sound/soc/sof/ipc4.c +++ b/sound/soc/sof/ipc4.c @@ -15,6 +15,7 @@ #include "sof-audio.h" #include "ipc4-fw-reg.h" #include "ipc4-priv.h" +#include "ipc4-telemetry.h" #include "ops.h"
static const struct sof_ipc4_fw_status { @@ -598,6 +599,8 @@ static int ipc4_fw_ready(struct snd_sof_dev *sdev, struct sof_ipc4_msg *ipc4_msg sdev->debug_box.offset = snd_sof_dsp_get_window_offset(sdev, SOF_IPC4_DEBUG_WINDOW_IDX);
+ sof_ipc4_create_exception_debugfs_node(sdev); + dev_dbg(sdev->dev, "mailbox upstream 0x%x - size 0x%x\n", inbox_offset, inbox_size); dev_dbg(sdev->dev, "mailbox downstream 0x%x - size 0x%x\n",
From: Rander Wang rander.wang@intel.com
Telemetry data is decoded based on intel xtensa design and printed in kernel log by sof debug framework.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/intel/Makefile | 3 +- sound/soc/sof/intel/telemetry.c | 95 +++++++++++++++++++++++++++++++++ sound/soc/sof/intel/telemetry.h | 35 ++++++++++++ 3 files changed, 132 insertions(+), 1 deletion(-) create mode 100644 sound/soc/sof/intel/telemetry.c create mode 100644 sound/soc/sof/intel/telemetry.h
diff --git a/sound/soc/sof/intel/Makefile b/sound/soc/sof/intel/Makefile index 030574dbc998..6489d0660d58 100644 --- a/sound/soc/sof/intel/Makefile +++ b/sound/soc/sof/intel/Makefile @@ -7,7 +7,8 @@ snd-sof-intel-hda-common-objs := hda.o hda-loader.o hda-stream.o hda-trace.o \ hda-dsp.o hda-ipc.o hda-ctrl.o hda-pcm.o \ hda-dai.o hda-dai-ops.o hda-bus.o \ skl.o hda-loader-skl.o \ - apl.o cnl.o tgl.o icl.o mtl.o lnl.o hda-common-ops.o + apl.o cnl.o tgl.o icl.o mtl.o lnl.o hda-common-ops.o \ + telemetry.o
snd-sof-intel-hda-mlink-objs := hda-mlink.o
diff --git a/sound/soc/sof/intel/telemetry.c b/sound/soc/sof/intel/telemetry.c new file mode 100644 index 000000000000..1a3b5c28a6f0 --- /dev/null +++ b/sound/soc/sof/intel/telemetry.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) +// +// This file is provided under a dual BSD/GPLv2 license. When using or +// redistributing this file, you may do so under either license. +// +// Copyright(c) 2023 Intel Corporation. All rights reserved. + +/* telemetry data queried from debug window */ + +#include <sound/sof/ipc4/header.h> +#include <sound/sof/xtensa.h> +#include "../ipc4-priv.h" +#include "../sof-priv.h" +#include "hda.h" +#include "telemetry.h" + +void sof_ipc4_intel_dump_telemetry_state(struct snd_sof_dev *sdev, u32 flags) +{ + static const char invalid_slot_msg[] = "Core dump is not available due to"; + struct sof_ipc4_telemetry_slot_data *telemetry_data; + struct sof_ipc_dsp_oops_xtensa *xoops; + struct xtensa_arch_block *block; + u32 slot_offset; + char *level; + + level = (flags & SOF_DBG_DUMP_OPTIONAL) ? KERN_DEBUG : KERN_ERR; + + slot_offset = sof_ipc4_find_debug_slot_offset_by_type(sdev, SOF_IPC4_DEBUG_SLOT_TELEMETRY); + if (!slot_offset) + return; + + telemetry_data = kmalloc(sizeof(*telemetry_data), GFP_KERNEL); + if (!telemetry_data) + return; + sof_mailbox_read(sdev, slot_offset, telemetry_data, sizeof(*telemetry_data)); + if (telemetry_data->separator != XTENSA_CORE_DUMP_SEPARATOR) { + dev_err(sdev->dev, "%s invalid separator %#x\n", invalid_slot_msg, + telemetry_data->separator); + goto free_telemetry_data; + } + + block = kmalloc(sizeof(*block), GFP_KERNEL); + if (!block) + goto free_telemetry_data; + + sof_mailbox_read(sdev, slot_offset + sizeof(*telemetry_data), block, sizeof(*block)); + if (block->soc != XTENSA_SOC_INTEL_ADSP) { + dev_err(sdev->dev, "%s invalid SOC %d\n", invalid_slot_msg, block->soc); + goto free_block; + } + + if (telemetry_data->hdr.id[0] != COREDUMP_HDR_ID0 || + telemetry_data->hdr.id[1] != COREDUMP_HDR_ID1 || + telemetry_data->arch_hdr.id != COREDUMP_ARCH_HDR_ID) { + dev_err(sdev->dev, "%s invalid coredump header %c%c, arch hdr %c\n", + invalid_slot_msg, telemetry_data->hdr.id[0], + telemetry_data->hdr.id[1], + telemetry_data->arch_hdr.id); + goto free_block; + } + + switch (block->toolchain) { + case XTENSA_TOOL_CHAIN_ZEPHYR: + dev_printk(level, sdev->dev, "FW is built with Zephyr toolchain\n"); + break; + case XTENSA_TOOL_CHAIN_XCC: + dev_printk(level, sdev->dev, "FW is built with XCC toolchain\n"); + break; + default: + dev_printk(level, sdev->dev, "Unknown toolchain is used\n"); + break; + } + + xoops = kzalloc(struct_size(xoops, ar, XTENSA_CORE_AR_REGS_COUNT), GFP_KERNEL); + if (!xoops) + goto free_block; + + xoops->exccause = block->exccause; + xoops->excvaddr = block->excvaddr; + xoops->epc1 = block->pc; + xoops->ps = block->ps; + xoops->sar = block->sar; + + xoops->plat_hdr.numaregs = XTENSA_CORE_AR_REGS_COUNT; + memcpy((void *)xoops->ar, block->ar, XTENSA_CORE_AR_REGS_COUNT * sizeof(u32)); + + sof_oops(sdev, level, xoops); + sof_stack(sdev, level, xoops, NULL, 0); + + kfree(xoops); +free_block: + kfree(block); +free_telemetry_data: + kfree(telemetry_data); +} diff --git a/sound/soc/sof/intel/telemetry.h b/sound/soc/sof/intel/telemetry.h new file mode 100644 index 000000000000..3c2b23c75f5d --- /dev/null +++ b/sound/soc/sof/intel/telemetry.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */ +/* + * This file is provided under a dual BSD/GPLv2 license. When using or + * redistributing this file, you may do so under either license. + * + * Copyright(c) 2023 Intel Corporation. All rights reserved. + * + * telemetry data in debug windows + */ + +#ifndef _SOF_INTEL_TELEMETRY_H +#define _SOF_INTEL_TELEMETRY_H + +#include "../ipc4-telemetry.h" + +struct xtensa_arch_block { + u8 soc; /* should be equal to XTENSA_SOC_INTEL_ADSP */ + u16 version; + u8 toolchain; /* ZEPHYR or XCC */ + + u32 pc; + u32 exccause; + u32 excvaddr; + u32 sar; + u32 ps; + u32 scompare1; + u32 ar[XTENSA_CORE_AR_REGS_COUNT]; + u32 lbeg; + u32 lend; + u32 lcount; +} __packed; + +void sof_ipc4_intel_dump_telemetry_state(struct snd_sof_dev *sdev, u32 flags); + +#endif /* _SOF_INTEL_TELEMETRY_H */
From: Rander Wang rander.wang@intel.com
Dump dsp stack with sof_ipc4_intel_dump_telemetry_state since dsp stack information is included by telemetry data. This also supports lnl since the mtl code is reused.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/intel/mtl.c | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/sound/soc/sof/intel/mtl.c b/sound/soc/sof/intel/mtl.c index f9412517eaf2..254dbbeac1d0 100644 --- a/sound/soc/sof/intel/mtl.c +++ b/sound/soc/sof/intel/mtl.c @@ -18,6 +18,7 @@ #include "hda-ipc.h" #include "../sof-audio.h" #include "mtl.h" +#include "telemetry.h"
static const struct snd_sof_debugfs_map mtl_dsp_debugfs[] = { {"hda", HDA_DSP_HDA_BAR, 0, 0x4000, SOF_DEBUGFS_ACCESS_ALWAYS}, @@ -320,6 +321,8 @@ void mtl_dsp_dump(struct snd_sof_dev *sdev, u32 flags) romdbgsts = snd_sof_dsp_read(sdev, HDA_DSP_BAR, MTL_DSP_REG_HFFLGPXQWY + 0x8 * 3); dev_printk(level, sdev->dev, "ROM feature bit%s enabled\n", romdbgsts & BIT(24) ? "" : " not"); + + sof_ipc4_intel_dump_telemetry_state(sdev, flags); }
static bool mtl_dsp_primary_core_is_enabled(struct snd_sof_dev *sdev)
From: Rander Wang rander.wang@intel.com
Get the FW panic information from telemetry data in memory window and dump it to kernel log. The old platforms before CAVS 2.5+ don't support it since there is no support in FW for them.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/intel/hda.c | 14 ++++++++++++++ sound/soc/sof/intel/hda.h | 1 + sound/soc/sof/intel/tgl.c | 1 + 3 files changed, 16 insertions(+)
diff --git a/sound/soc/sof/intel/hda.c b/sound/soc/sof/intel/hda.c index 15e6779efaa3..02c82ccb9f66 100644 --- a/sound/soc/sof/intel/hda.c +++ b/sound/soc/sof/intel/hda.c @@ -31,6 +31,7 @@ #include "../sof-pci-dev.h" #include "../ops.h" #include "hda.h" +#include "telemetry.h"
#define CREATE_TRACE_POINTS #include <trace/events/sof_intel.h> @@ -731,6 +732,19 @@ void hda_dsp_dump(struct snd_sof_dev *sdev, u32 flags) } }
+void hda_ipc4_dsp_dump(struct snd_sof_dev *sdev, u32 flags) +{ + char *level = (flags & SOF_DBG_DUMP_OPTIONAL) ? KERN_DEBUG : KERN_ERR; + + /* print ROM/FW status */ + hda_dsp_get_state(sdev, level); + + if (flags & SOF_DBG_DUMP_REGS) + sof_ipc4_intel_dump_telemetry_state(sdev, flags); + else + hda_dsp_dump_ext_rom_status(sdev, level, flags); +} + static bool hda_check_ipc_irq(struct snd_sof_dev *sdev) { const struct sof_intel_dsp_desc *chip; diff --git a/sound/soc/sof/intel/hda.h b/sound/soc/sof/intel/hda.h index 2b228c63905b..7c575ba9462c 100644 --- a/sound/soc/sof/intel/hda.h +++ b/sound/soc/sof/intel/hda.h @@ -603,6 +603,7 @@ int hda_dsp_shutdown_dma_flush(struct snd_sof_dev *sdev); int hda_dsp_shutdown(struct snd_sof_dev *sdev); int hda_dsp_set_hw_params_upon_resume(struct snd_sof_dev *sdev); void hda_dsp_dump(struct snd_sof_dev *sdev, u32 flags); +void hda_ipc4_dsp_dump(struct snd_sof_dev *sdev, u32 flags); void hda_ipc_dump(struct snd_sof_dev *sdev); void hda_ipc_irq_dump(struct snd_sof_dev *sdev); void hda_dsp_d0i3_work(struct work_struct *work); diff --git a/sound/soc/sof/intel/tgl.c b/sound/soc/sof/intel/tgl.c index bb9f20253c99..4a61f6d28ae5 100644 --- a/sound/soc/sof/intel/tgl.c +++ b/sound/soc/sof/intel/tgl.c @@ -102,6 +102,7 @@ int sof_tgl_ops_init(struct snd_sof_dev *sdev)
/* debug */ sof_tgl_ops.ipc_dump = cnl_ipc4_dump; + sof_tgl_ops.dbg_dump = hda_ipc4_dsp_dump;
sof_tgl_ops.set_power_state = hda_dsp_set_power_state_ipc4; }
From: Rander Wang rander.wang@intel.com
Driver will receive exception IPC message and process it by snd_sof_dsp_panic.
Signed-off-by: Rander Wang rander.wang@intel.com Reviewed-by: Péter Ujfalusi peter.ujfalusi@linux.intel.com Reviewed-by: Kai Vehmanen kai.vehmanen@linux.intel.com Reviewed-by: Pierre-Louis Bossart pierre-louis.bossart@linux.intel.com Reviewed-by: Guennadi Liakhovetski guennadi.liakhovetski@linux.intel.com Signed-off-by: Peter Ujfalusi peter.ujfalusi@linux.intel.com --- sound/soc/sof/ipc4.c | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/sound/soc/sof/ipc4.c b/sound/soc/sof/ipc4.c index 82f2f196c9c2..3f4d57dba972 100644 --- a/sound/soc/sof/ipc4.c +++ b/sound/soc/sof/ipc4.c @@ -645,6 +645,9 @@ static void sof_ipc4_rx_msg(struct snd_sof_dev *sdev) case SOF_IPC4_NOTIFY_LOG_BUFFER_STATUS: sof_ipc4_mtrace_update_pos(sdev, SOF_IPC4_LOG_CORE_GET(ipc4_msg->primary)); break; + case SOF_IPC4_NOTIFY_EXCEPTION_CAUGHT: + snd_sof_dsp_panic(sdev, 0, true); + break; default: dev_dbg(sdev->dev, "Unhandled DSP message: %#x|%#x\n", ipc4_msg->primary, ipc4_msg->extension);
On Tue, 19 Sep 2023 12:24:07 +0300, Peter Ujfalusi wrote:
When a firmware crashes it creats a panic information into a telemetry slot. The panic format is defined by Zephyr, includes stack and additional information to help to identify the reason for the crash. Part of the firmware exception handling the firmware also sends an EXCEPTION_CAUGHT notification.
This series implements the kernel side handling of the exception: print information into the kernel log export the whole telemetry slot to user space for tools extract additional information from the panic dump.
[...]
Applied to
https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git for-next
Thanks!
[1/9] ASoC: SOF: Xtensa: dump ar registers to restore call stack commit: 58bb5081cba130f12c26d8e4d5e9416a0272f07e [2/9] ASoC: SOF: ipc4-mtrace: move debug slot related definitions to header.h commit: 4287205065f244f4d40ae6aa7875b3ebffedcf8d [3/9] ASoC: SOF: ipc4: add a helper function to search debug slot commit: a397899f81d52202265d4977a99085f53e426826 [4/9] ASoC: SOF: ipc4: add definition of telemetry slot for exception handling commit: ab05061d25806515358d184eb4d305f7f12befdc [5/9] ASoC: SOF: ipc4: add exception node in sof debugfs directory commit: 80b567f8995757d36008f835853cea8d2f7c34c0 [6/9] ASoC: SOF: Intel: add telemetry retrieval support on Intel platforms commit: c8b54a2f7af41740b5faad2f6846d927b14369ca [7/9] ASoC: SOF: Intel: mtl: dump dsp stack commit: e449b18ff03c2f90430d00486fd713854b28c077 [8/9] ASoC: SOF: Intel: hda: add ipc4 FW panic support on CAVS 2.5+ platforms commit: eb6e5dab11401c64f5d5576c71e5fc0a4c7b321a [9/9] ASoC: SOF: ipc4: handle EXCEPTION_CAUGHT notification from firmware commit: c1c48fd6bbe788458e3685fea74bdb3cb148ff93
All being well this means that it will be integrated into the linux-next tree (usually sometime in the next 24 hours) and sent to Linus during the next merge window (or sooner if it is a bug fix), however if problems are discovered then the patch may be dropped or reverted.
You may get further e-mails resulting from automated or manual testing and review of the tree, please engage with people reporting problems and send followup patches addressing any issues that are reported if needed.
If any updates are required or you are submitting further changes they should be sent as incremental updates against current git, existing patches will not be replaced.
Please add any relevant lists and maintainers to the CCs when replying to this mail.
Thanks, Mark
participants (2)
-
Mark Brown -
Peter Ujfalusi