[alsa-devel] [BUG] bdw-rt5650 DSP boot timeout

[Jon Flatley]

On Mon, Jul 29, 2019 at 7:23 PM Pierre-Louis Bossart
<pierre-louis.bossart at linux.intel.com> wrote:
>
>
>
> On 7/29/19 7:53 PM, Ranjani Sridharan wrote:
> > On Mon, 2019-07-29 at 18:02 -0500, Pierre-Louis Bossart wrote:
> >>
> >> On 7/29/19 4:53 PM, Jon Flatley wrote:
> >>> I've been working on upstreaming the bdw-rt5650 machine driver for
> >>> the
> >>> Acer Chromebase 24 (buddy). There seems to be an issue when first
> >>> setting the hardware controls that appears to be crashing the DSP:
> >>>
> >>> [   51.424554] haswell-pcm-audio haswell-pcm-audio: FW loaded,
> >>> mailbox
> >>> readback FW info: type 01, - version: 00.00, build 77, source
> >>> commit
> >>> id: 876ac6906f31a43b6772b23c7c983ce9dcb18a19
> >>> ...
> >>> [   84.924666] haswell-pcm-audio haswell-pcm-audio: error: audio
> >>> DSP
> >>> boot timeout IPCD 0x0 IPCX 0x0
> >>> [   85.260655] haswell-pcm-audio haswell-pcm-audio: ipc: --message
> >>> timeout-- ipcx 0x83000000 isr 0x00000000 ipcd 0x00000000 imrx
> >>> 0x7fff0000
> >>> [   85.273609] haswell-pcm-audio haswell-pcm-audio: error: stream
> >>> commit failed
> >>> [   85.279746]  System PCM: error: failed to commit stream -110
> >>> [   85.285388] haswell-pcm-audio haswell-pcm-audio: ASoC:
> >>> haswell-pcm-audio hw params failed: -110
> >>> [   85.293963]  System PCM: ASoC: hw_params FE failed -110
> >>>
> >>> This happens roughly 50% of the time when first setting hardware
> >>> controls after a reboot. The other 50% of the time the DSP comes up
> >>> just fine and audio works fine thereafter. Adding "#define DEBUG 1"
> >>> to
> >>> sound/soc/intel/haswell/sst-haswell-ipc.c makes the issue occur
> >>> much
> >>> less frequently in my testing. Seems like a subtle timing issue.
> >>>
> >>> There were timing issues encountered during the bringup of the 2015
> >>> chromebook pixel (samus) which uses the bdw-rt5677 machine driver.
> >>> Those were slightly different, and manifested during repeated
> >>> arecords. Both devices use the same revision of the sst2 firmware.
> >>>
> >>> Any ideas for how to debug this?
> >>
> >> this could be trying to send an IPC while you are already waiting
> >> for
> >> one to complete. we've seen this before with SOF, if the IPCs are
> >> not
> >> strictly serialized then things go in the weeds and timeout.
> > Pierre/Jon
> >
> > In this case it looks like the DSP boot failed leading to the IPC
> > timeout? WOndering if increasing the boot timeout would help?

I did actually try this without success.

>
> Yes, that too. The boot timeout is typically experimentally defined, and
> never decreasing due to platform variations...
> I am still leaning more on the side of an side effect between two IPCs,
> the added DEBUG points to the printk which solves timing issues. The
> boot timeout would typically not be impacted by such changes.

I think the real struggle I'm having is finding a good debugging
method that doesn't impact the timing of the IPCs significantly (as
adding DEBUG seems to). This could maybe be overcome with using a
stress test to reproduce. The crash only seems to occur when first
booting the DSP, and so far I've been testing this by completely power
cycling the machine on every test, which is very slow and tedious. So
maybe the issue with DEBUG defined occurs 1 in 20 reboots rather than
1 in 2, I wouldn't know. If there's a way to reboot the DSP and
reproduce this crash without rebooting the entire device that would be
very helpful to me.

Thanks,
Jon