[Sound-open-firmware] Graceful exit during DSP exception
Hi, While trying to send an IPC message to configure the DMIC params, I forgot to add the message size in the in the IPC header. The FW ran into an exception and seemed like it was stuck in an infinite loop. Is there way to exit gracefully in such cases? The dmesg log (partial)is shown below. Thanks, Ranjani [ 124.894619] sof-audio sof-audio: pcm: hw params stream 0 dir 1 [ 124.894625] sof-audio sof-audio: generating page table for ffffabae00dd1000 size 0xfc00 pages 16 [ 124.894630] sof-audio sof-audio: pfn i 0 idx 0 pfn 177730 [ 124.894633] sof-audio sof-audio: pfn i 1 idx 2 pfn 177731 [ 124.894637] sof-audio sof-audio: pfn i 2 idx 5 pfn 177732 [ 124.894640] sof-audio sof-audio: pfn i 3 idx 7 pfn 177733 [ 124.894643] sof-audio sof-audio: pfn i 4 idx 10 pfn 177734 [ 124.894646] sof-audio sof-audio: pfn i 5 idx 12 pfn 177735 [ 124.894649] sof-audio sof-audio: pfn i 6 idx 15 pfn 177736 [ 124.894653] sof-audio sof-audio: pfn i 7 idx 17 pfn 177737 [ 124.894656] sof-audio sof-audio: pfn i 8 idx 20 pfn 177738 [ 124.894659] sof-audio sof-audio: pfn i 9 idx 22 pfn 177739 [ 124.894662] sof-audio sof-audio: pfn i 10 idx 25 pfn 17773a [ 124.894665] sof-audio sof-audio: pfn i 11 idx 27 pfn 17773b [ 124.894669] sof-audio sof-audio: pfn i 12 idx 30 pfn 17773c [ 124.894672] sof-audio sof-audio: pfn i 13 idx 32 pfn 17773d [ 124.894675] sof-audio sof-audio: pfn i 14 idx 35 pfn 17773e [ 124.894678] sof-audio sof-audio: pfn i 15 idx 37 pfn 17773f [ 124.894701] sof-audio sof-audio: period_bytes:0x3c00 [ 124.894713] sof-audio sof-audio: stream_tag 2 [ 125.198613] sof-audio sof-audio: error: ipc timed out for 0x60010000 size 0x60 [ 125.206695] sof-audio sof-audio: status: fw entered - code 00000005 [ 125.213713] sof-audio sof-audio: error: runtime exception [ 125.219744] sof-audio sof-audio: error: trace point 00004000 [ 125.226063] sof-audio sof-audio: error: DSP Firmware Oops [ 125.232093] sof-audio sof-audio: EXCCAUSE 0x0000000d EXCVADDR 0xbe020000 PS 0x00070025 SAR 0x00000020 [ 125.243365] sof-audio sof-audio: EPC1 0xa001b7bc EPC2 0xa000b882 EPC3 0x00000000 EPC4 0x00000000 [ 125.254639] sof-audio sof-audio: EPC5 0x00000000 EPC6 0x00000000 EPC7 0x00000000 DEPC 0x00000000 [ 125.265926] sof-audio sof-audio: EPS2 0x00060520 EPS3 0x00000000 EPS4 0x00000000 EPS5 0x00000000 [ 125.277203] sof-audio sof-audio: EPS6 0x00000000 EPS7 0x00000000 INTENABL 0x000101d8 INTERRU 0x00000222 [ 125.288479] sof-audio sof-audio: stack dump from 0xa005fe50 [ 125.294706] sof-audio sof-audio: 0xa005fe50: 0xa001b7bc 0x00000000 0x00000020 0x00000000 [ 125.303748] sof-audio sof-audio: 0xa005fe54: 0xa0041874 0x00000000 0xfffffff8 0x00000000 [ 125.312791] sof-audio sof-audio: 0xa005fe58: 0x0000000d 0x00000000 0xa001b7b6 0xa001b7cc [ 125.321833] sof-audio sof-audio: 0xa005fe5c: 0xa000ebd1 0x00000000 0xa0040acc 0x00000000 [ 125.330875] sof-audio sof-audio: 0xa005fe60: 0xa0015c8c 0x00000000 0xa00408bc 0x00000000 [ 125.339918] sof-audio sof-audio: 0xa005fe64: 0xa000f5ce 0x00000000 0xa004186c 0x00000000 [ 125.348960] sof-audio sof-audio: 0xa005fe68: 0x00000000 0x00000000 0x00000001 0x00000000 [ 125.357995] sof-audio sof-audio: 0xa005fe6c: 0x019175d6 0x00000000 0x024e6d73 0x00000000 [ 125.367036] sof-audio sof-audio: error: waking up any trace sleepers [ 125.374283] sof-audio sof-audio: error: got wrong posn offset 0xffff8d1b for PCM -110 [ 125.383046] sof-audio sof-audio: ASoC: sof-audio hw params failed: -110 [ 125.396414] sof-audio sof-audio: pcm: free stream 0 dir 1 [ 125.710883] sof-audio sof-audio: error: ipc timed out for 0x60030000 size 0xc [ 125.718873] sof-audio sof-audio: status: fw entered - code 00000005 [ 125.725897] sof-audio sof-audio: error: runtime exception [ 125.731929] sof-audio sof-audio: error: trace point 00004000 [ 125.738253] sof-audio sof-audio: error: DSP Firmware Oops [ 125.744287] sof-audio sof-audio: EXCCAUSE 0x0000000d EXCVADDR 0xbe020000 PS 0x00070025 SAR 0x00000020 [ 125.755568] sof-audio sof-audio: EPC1 0xa001b7bc EPC2 0xa000b882 EPC3 0x00000000 EPC4 0x00000000 [ 125.766847] sof-audio sof-audio: EPC5 0x00000000 EPC6 0x00000000 EPC7 0x00000000 DEPC 0x00000000 [ 125.778127] sof-audio sof-audio: EPS2 0x00060520 EPS3 0x00000000 EPS4 0x00000000 EPS5 0x00000000 [ 125.789408] sof-audio sof-audio: EPS6 0x00000000 EPS7 0x00000000 INTENABL 0x000101d8 INTERRU 0x00000222 [ 125.800688] sof-audio sof-audio: stack dump from 0xa005fe50 [ 125.806917] sof-audio sof-audio: 0xa005fe50: 0xa001b7bc 0x00000000 0x00000020 0x00000000 [ 125.815964] sof-audio sof-audio: 0xa005fe54: 0xa0041874 0x00000000 0xfffffff8 0x00000000 [ 125.825010] sof-audio sof-audio: 0xa005fe58: 0x0000000d 0x00000000 0xa001b7b6 0xa001b7cc [ 125.834057] sof-audio sof-audio: 0xa005fe5c: 0xa000ebd1 0x00000000 0xa0040acc 0x00000000 [ 125.843102] sof-audio sof-audio: 0xa005fe60: 0xa0015c8c 0x00000000 0xa00408bc 0x00000000 [ 125.852148] sof-audio sof-audio: 0xa005fe64: 0xa000f5ce 0x00000000 0xa004186c 0x00000000 [ 125.861193] sof-audio sof-audio: 0xa005fe68: 0x00000000 0x00000000 0x00000001 0x00000000 [ 125.870238] sof-audio sof-audio: 0xa005fe6c: 0x019175d6 0x00000000 0x024e6d73 0x00000000 [ 125.879284] sof-audio sof-audio: error: waking up any trace sleepers [ 125.886887] sof-audio sof-audio: pcm: hw params stream 0 dir 1 [ 125.886890] sof-audio sof-audio: generating page table for ffffabae00dd1000 size 0xfc00 pages 16 [ 125.886892] sof-audio sof-audio: pfn i 0 idx 0 pfn 177730 [ 125.886894] sof-audio sof-audio: pfn i 1 idx 2 pfn 177731 [ 125.886895] sof-audio sof-audio: pfn i 2 idx 5 pfn 177732 [ 125.886897] sof-audio sof-audio: pfn i 3 idx 7 pfn 177733 [ 125.886898] sof-audio sof-audio: pfn i 4 idx 10 pfn 177734 [ 125.886900] sof-audio sof-audio: pfn i 5 idx 12 pfn 177735 [ 125.886901] sof-audio sof-audio: pfn i 6 idx 15 pfn 177736 [ 125.886903] sof-audio sof-audio: pfn i 7 idx 17 pfn 177737 [ 125.886904] sof-audio sof-audio: pfn i 8 idx 20 pfn 177738 [ 125.886906] sof-audio sof-audio: pfn i 9 idx 22 pfn 177739 [ 125.886907] sof-audio sof-audio: pfn i 10 idx 25 pfn 17773a [ 125.886908] sof-audio sof-audio: pfn i 11 idx 27 pfn 17773b [ 125.886910] sof-audio sof-audio: pfn i 12 idx 30 pfn 17773c [ 125.886911] sof-audio sof-audio: pfn i 13 idx 32 pfn 17773d [ 125.886913] sof-audio sof-audio: pfn i 14 idx 35 pfn 17773e [ 125.886914] sof-audio sof-audio: pfn i 15 idx 37 pfn 17773f [ 125.886933] sof-audio sof-audio: period_bytes:0x1800 [ 125.886940] sof-audio sof-audio: stream_tag 2
On Fri, 2018-05-25 at 21:02 -0700, Ranjani Sridharan wrote:
While trying to send an IPC message to configure the DMIC params, I forgot to add the message size in the in the IPC header. The FW ran into an exception and seemed like it was stuck in an infinite loop.
Is there way to exit gracefully in such cases? The dmesg log (partial)is shown below.
The driver will eventually reboot the FW when a FW exception occurs. Liam
On Sat, 2018-05-26 at 10:16 +0100, Liam Girdwood wrote:
On Fri, 2018-05-25 at 21:02 -0700, Ranjani Sridharan wrote:
While trying to send an IPC message to configure the DMIC params, I forgot to add the message size in the in the IPC header. The FW ran into an exception and seemed like it was stuck in an infinite loop.
Is there way to exit gracefully in such cases? The dmesg log (partial)is shown below.
The driver will eventually reboot the FW when a FW exception occurs. But it doesnt seem to be happening in this case? The dmesg log overflows with the same FW exception message over and over again.
Liam _______________________________________________ Sound-open-firmware mailing list Sound-open-firmware@alsa-project.org http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmware
On 5/27/2018 00:26, Ranjani Sridharan wrote:
On Sat, 2018-05-26 at 10:16 +0100, Liam Girdwood wrote:
On Fri, 2018-05-25 at 21:02 -0700, Ranjani Sridharan wrote:
While trying to send an IPC message to configure the DMIC params, I forgot to add the message size in the in the IPC header. The FW ran into an exception and seemed like it was stuck in an infinite loop.
Is there way to exit gracefully in such cases? The dmesg log (partial)is shown below. The driver will eventually reboot the FW when a FW exception occurs. But it doesnt seem to be happening in this case? The dmesg log overflows with the same FW exception message over and over again.
The driver will not reset the DSP when PANIC happen now. Should we add this to reset the DSP? Actually the PANIC info will dump in two situation: 1. Panic happen will send a IPC message to driver and we will handle that 2. When timeout happen, we will also try to dump PANIC info So for your situation, the panic happen for the first time, but the driver had no idea that the DSP is panic and continue send IPCs to DSP. These IPCs all timeout and had these panic infos. I think you get a very complex topology? Thanks Xiuli
Liam _______________________________________________ Sound-open-firmware mailing list Sound-open-firmware@alsa-project.org http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmware
Sound-open-firmware mailing list Sound-open-firmware@alsa-project.org http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmware
On Mon, 2018-05-28 at 10:54 +0800, Pan, Xiuli wrote:
On 5/27/2018 00:26, Ranjani Sridharan wrote:
On Sat, 2018-05-26 at 10:16 +0100, Liam Girdwood wrote:
On Fri, 2018-05-25 at 21:02 -0700, Ranjani Sridharan wrote:
While trying to send an IPC message to configure the DMIC params, I forgot to add the message size in the in the IPC header. The FW ran into an exception and seemed like it was stuck in an infinite loop.
Is there way to exit gracefully in such cases? The dmesg log (partial)is shown below.
The driver will eventually reboot the FW when a FW exception occurs.
But it doesnt seem to be happening in this case? The dmesg log overflows with the same FW exception message over and over again.
The driver will not reset the DSP when PANIC happen now. Should we add this to reset the DSP? Actually the PANIC info will dump in two situation: 1. Panic happen will send a IPC message to driver and we will handle that 2. When timeout happen, we will also try to dump PANIC info
So for your situation, the panic happen for the first time, but the driver had no idea that the DSP is panic and continue send IPCs to DSP. These IPCs all timeout and had these panic infos. I think you get a very complex topology?
The topology is quite simple in fact. I've got a passthrough topology with a DMIC DAI instead of SSP. I'm guessing we should be able to reproduce my problem with any ipc message if you remove the line where we set the size of the ipc message. I'll look into debugging this.
Thanks Xiuli
Liam _______________________________________________ Sound-open-firmware mailing list Sound-open-firmware@alsa-project.org http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmw are
_______________________________________________ Sound-open-firmware mailing list Sound-open-firmware@alsa-project.org http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmwar e
On 5/28/2018 11:21, Ranjani Sridharan wrote: > On Mon, 2018-05-28 at 10:54 +0800, Pan, Xiuli wrote: >> On 5/27/2018 00:26, Ranjani Sridharan wrote: >>> On Sat, 2018-05-26 at 10:16 +0100, Liam Girdwood wrote: >>>> On Fri, 2018-05-25 at 21:02 -0700, Ranjani Sridharan wrote: >>>>> While trying to send an IPC message to configure the DMIC >>>>> params, I >>>>> forgot to add the message size in the in the IPC header. The >>>>> FW ran into an exception and seemed like it was stuck in an >>>>> infinite >>>>> loop. >>>>> >>>>> Is there way to exit gracefully in such cases? The dmesg log >>>>> (partial)is shown >>>>> below. >>>> The driver will eventually reboot the FW when a FW exception >>>> occurs. >>> But it doesnt seem to be happening in this case? The dmesg log >>> overflows with the same FW exception message over and over again. >> The driver will not reset the DSP when PANIC happen now. >> Should we add this to reset the DSP? >> Actually the PANIC info will dump in two situation: >> 1. Panic happen will send a IPC message to driver and we will handle >> that >> 2. When timeout happen, we will also try to dump PANIC info >> >> So for your situation, the panic happen for the first time, but the >> driver had no idea that the DSP is panic and continue send IPCs to >> DSP. >> These IPCs all timeout and had these panic infos. I think you get a >> very >> complex topology? > The topology is quite simple in fact. I've got a passthrough topology > with a DMIC DAI instead of SSP. > > I'm guessing we should be able to reproduce my problem with any ipc > message if you remove the line where we set the size of the ipc > message. I'll look into debugging this. Then we may have a bug here. If some of the IPCs is failed due to timeout or something else. We should not try to send more IPCs to DSP. Thanks Xiuli >> Thanks >> Xiuli >>>> Liam >>>> _______________________________________________ >>>> Sound-open-firmware mailing list >>>> Sound-open-firmware@alsa-project.org >>>> http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmw >>>> are >>> _______________________________________________ >>> Sound-open-firmware mailing list >>> Sound-open-firmware@alsa-project.org >>> http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmwar >>> e >> > _______________________________________________ > Sound-open-firmware mailing list > Sound-open-firmware@alsa-project.org > http://mailman.alsa-project.org/mailman/listinfo/sound-open-firmware
On Sat, 2018-05-26 at 09:26 -0700, Ranjani Sridharan wrote:
On Sat, 2018-05-26 at 10:16 +0100, Liam Girdwood wrote:
On Fri, 2018-05-25 at 21:02 -0700, Ranjani Sridharan wrote:
While trying to send an IPC message to configure the DMIC params, I forgot to add the message size in the in the IPC header. The FW ran into an exception and seemed like it was stuck in an infinite loop.
Is there way to exit gracefully in such cases? The dmesg log (partial)is shown below.
The driver will eventually reboot the FW when a FW exception occurs.
But it doesnt seem to be happening in this case? The dmesg log overflows with the same FW exception message over and over again.
Ok, I should have mentioned the reboot part is not implemented yet. Please add a patch though that stops after 1 exception. Thanks Liam
participants (3)
-
Liam Girdwood -
Pan, Xiuli -
Ranjani Sridharan