[Sound-open-firmware] How does one debug IPC errors?

[Lauda, Tomasz]

If your pipeline processing takes almost 1 ms to run there is something 
fundamentally wrong in your system.

Even on BYT with cache disabled and gcc compiled binary we can 
successfully run at least couple of pipelines simultaneously.

In the future we intend to support EDF scheduling for modules, which 
requires larger frames to process, but as of right now low latency 
scheduling should be enough.

You should verify:

1. What is your DSP max clock and are you sure it's set correctly. Maybe 
you're running on low power settings.

2. Your core-isa header suggests you support L1 cache. Have you enabled it?

3. Use xcc built binary.

On 21.10.2019 13:00, Paul Olaru wrote:
> I think I figured out the source of the IPC issues I got here. Apparently,
> pipeline_copy() is being run in IRQ context if I read the code correctly.
> But if this function takes around 0.95+ milliseconds, there is the
> possibility that the next EDMA interrupt will arrive before a complete
> loop of the main task, which means that at least for that millisecond
> the DSP both becomes unresponsive to IPCs and fails to send any IPC
> message it has created on its own (in particular the "host position change"
> ones).
>
> For the moment I have a workaround which effectively disables the volume
> component so I can compile using the generic compiler without hitting
> this issue. Another good workaround might have been to compile using
> the Xtensa toolchain which allows the HIFI4 optimized variant of the
> volume component to run instead.
>
> I believe however that running pipeline_copy() within the IRQ context
> (directly as the task registered in the DMA domain) rather than an
> indirect scheduling in the EDF scheduler (as before, which may mean
> two tasks instead of one possibly) doesn't sound good. Changing to
> that indirect scheduling may make the code more complex but will
> also allow such components which take too long to cause Xruns rather
> than lost IPCs and the risk of getting the DSP completely unresponsive.
>
> Tomasz, I believe you know quite a bit about scheduling (from you having
> introduced the DMA domain, of course). Do you have any ideas on how
> this should be approached?
>
> -----Original Message-----
> From: Sound-open-firmware <sound-open-firmware-bounces at alsa-project.org> On Behalf Of Paul Olaru
> Sent: Friday, October 18, 2019 1:14 PM
> To: Liam Girdwood <liam.r.girdwood at linux.intel.com>; sound-open-firmware at alsa-project.org
> Cc: Daniel Baluta <daniel.baluta at nxp.com>; Jerome Laclavere <jerome.laclavere at nxp.com>; Guido Roncarolo <guido.roncarolo at nxp.com>
> Subject: Re: [Sound-open-firmware] How does one debug IPC errors?
>
> Found that the loop only occurs when the SAI driver is also in, and also
> Daniel just told me as I got to work today that tweaking the various
> memory region sizes has an effect on whether the bug manifests or not.
>
> I will investigate today.
>
> Right now, the only thing I'm getting in etrace is the failure to enable DMA
> trace (which is benign, I didn't push any implementation for that with my
> Dummy DMA driver) but no other issues.
>
> Don't pointer updates happen via IPC (which on our platform is putting the
> message in the mailbox and giving an interrupt to the ARM core via the MU)
> anyway?
>
> Also on the buggy version whenever I do Ctrl-C on the aplay I get continuous
> ipc_period_elapsed messages on a null runtime (of course, the DSP did not
> receive the STOP and RESET triggers for some reason and it keeps trying to
> play -- and I'm getting silence in my headphones).
>
> I think I'll try to look into the MU driver (i.MX IPC implementation) on both
> sides and see why I'm getting issues. This is really fishy.
>
> -----Original Message-----
> From: Liam Girdwood <liam.r.girdwood at linux.intel.com>
> Sent: Thursday, October 17, 2019 8:26 PM
> To: Paul Olaru <paul.olaru at nxp.com>; sound-open-firmware at alsa-project.org
> Cc: Jerome Laclavere <jerome.laclavere at nxp.com>; Daniel Baluta <daniel.baluta at nxp.com>; Guido Roncarolo <guido.roncarolo at nxp.com>
> Subject: Re: [Sound-open-firmware] How does one debug IPC errors?
>
> On Wed, 2019-10-16 at 15:57 +0000, Paul Olaru wrote:
>> Right now I'm still working on the ESAI SOF driver. Based on the
>> current branch I'm getting
>> an out of memory issue. Solved by increasing the system runtime heap,
>> not a real problem
>> even if a surprising one to me, will probably send the fix together
>> with the ESAI driver.
>>
>> However, I'm getting a second issue: Whenever I'm trying to play
>> anything I see the etrace
>> log filled with "ipc: msg hdr for 0x600a0000 not found replace 0" and
>> playback doesn't
>> progress. Also in another weirder circumstance (invalid aplay
>> followed by a valid one)
>> I heard it play but for some reason the host buffer was copied only
>> at odd times (and I heard
>> the same 64k or 16384 samples many times over instead of once).
> Seems like the pointer is not being updated. ALSA wont copy more
> periods and the buffer will repeat.
>
> You can either update the pointer via IPC or update via mailbox
> (preferred).
>
>> How should I continue investigating this issue?
>>
>> (also due to a failed stop - kernel reported IPC timeout for
>> 0x60050000 and 0x60030000,
>> I'm still getting the issues for 0x600a0000 in a continuous loop on
>> the firmware side, about
>> the time it would take to normally play those samples; until reboot I
>> cannot send any other
>> commands to the DSP either).
>>
> Looks like your DMA or I2S driver may be looping forever ? Possible in
> IRQ context if no other work can be scheduled ?
>
>> Also, should I send the current state of the ESAI driver, even if
>> there is a little more cleanup
>> that needs to be done on it? Now that I cannot reliably test it
>> anymore.
> We can wait until you have fixed this issue.
>
> Liam
>
>> _______________________________________________
>> Sound-open-firmware mailing list
>> Sound-open-firmware at alsa-project.org
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.alsa-project.org%2Fmailman%2Flistinfo%2Fsound-open-firmware&data=02%7C01%7Cpaul.olaru%40nxp.com%7C8b8f12445a8e4dd337a408d753b487ff%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637069907149449410&sdata=aJWS7uPljDQpT0D2EjZzQN10NSBCor4k6qsWfjtvgkg%3D&reserved=0
> _______________________________________________
> Sound-open-firmware mailing list
> Sound-open-firmware at alsa-project.org
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.alsa-project.org%2Fmailman%2Flistinfo%2Fsound-open-firmware&data=02%7C01%7Cpaul.olaru%40nxp.com%7C8b8f12445a8e4dd337a408d753b487ff%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637069907149459409&sdata=wsXr0O%2Fo002AmDQuaMLW%2Bzobcw6%2Bwlm7xHLEFiNv5kU%3D&reserved=0
> _______________________________________________
> Sound-open-firmware mailing list
> Sound-open-firmware at alsa-project.org
> https://mailman.alsa-project.org/mailman/listinfo/sound-open-firmware