[Sound-open-firmware] Zephyr memory allocation from heap
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGN sizeof(void *) +#define PLATFORM_DCACHE_ALIGN 32
everything seems to work.
But I cannot find any logical explanation.
+ Andy, as he also worked on the initial zephyr with SOF.
On Mon, Aug 22, 2022 at 5:04 PM Daniel Baluta daniel.baluta@gmail.com wrote:
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGN sizeof(void *) +#define PLATFORM_DCACHE_ALIGN 32
everything seems to work.
But I cannot find any logical explanation.
+ andyross@google.com
Subject: Re: [Sound-open-firmware] Zephyr memory allocation from heap
+ Andy, as he also worked on the initial zephyr with SOF.
On Mon, Aug 22, 2022 at 5:04 PM Daniel Baluta daniel.baluta@gmail.com wrote:
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGN sizeof(void *) +#define PLATFORM_DCACHE_ALIGN 32
everything seems to work.
But I cannot find any logical explanation.
On Mon, 2022-08-22 at 16:44 +0000, Daniel Baluta wrote:
- andyross@google.com
Subject: Re: [Sound-open-firmware] Zephyr memory allocation from heap
- Andy, as he also worked on the initial zephyr with SOF.
On Mon, Aug 22, 2022 at 5:04 PM Daniel Baluta daniel.baluta@gmail.com wrote:
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGNÂ sizeof(void *) +#define PLATFORM_DCACHE_ALIGNÂ 32
everything seems to work.
But I cannot find any logical explanation.
The cache line size should align to the value in core-isa.h.
IIRC, I think the sizeof(void*) is a catch all when its not defined.
Liam
I see. We've done this in the past.
But with https://github.com/thesofproject/sof/pull/3602
but for i.MX8 PLATFORM_DCACHE_ALIGN is 128 and with XTOS memory management we got some -ENOMEM.
This seems to be because all blocks < 128 bytes could not be used with such a big alignment size.
Any idea, why SOF with XTOS requires that all allocations to be aligned to cache line size?
Thanks, Daniel.
On Mon, Aug 22, 2022 at 10:50 PM Girdwood, Liam R liam.r.girdwood@intel.com wrote:
On Mon, 2022-08-22 at 16:44 +0000, Daniel Baluta wrote:
- andyross@google.com
Subject: Re: [Sound-open-firmware] Zephyr memory allocation from heap
- Andy, as he also worked on the initial zephyr with SOF.
On Mon, Aug 22, 2022 at 5:04 PM Daniel Baluta daniel.baluta@gmail.com wrote:
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGN sizeof(void *) +#define PLATFORM_DCACHE_ALIGN 32
everything seems to work.
But I cannot find any logical explanation.
The cache line size should align to the value in core-isa.h.
IIRC, I think the sizeof(void*) is a catch all when its not defined.
Liam
It depends on what SOF is going to do with that memory, obviously. :)
The two big limitations in the past have been cache and hardware. The Xtensa architectural cache line size is 64 bytes, not 32, so that's probably not what's tripping you up (though it might be: you'd be 50/50 to get a working pointer!). IIRC the SOF heap actually returns uncached memory these days by default anyway? You need to use the uncached_to_cached() API (or the Zephyr equivalent that it's wrapping) to end up polluting the cache. There were multiple bugs fixed in this space last year.
The other issue on Intel was that one of the DMA controllers (I think it was the Synopsys one, but it might have been HDA) had an (undocumented?) alignment requirement of (I think?) 32 bytes. I'm fuzzier on the details here, but this seems more likely to be the kind of thing you're looking at. The resolution there was to align the requirement at the driver level, I think? So if you're using different drivers you might need to do the same thing. Guennadi or +Kai would be likely to know more about this.
Andy
On Mon, Aug 22, 2022 at 9:44 AM Daniel Baluta daniel.baluta@nxp.com wrote:
- andyross@google.com
Subject: Re: [Sound-open-firmware] Zephyr memory allocation from heap
- Andy, as he also worked on the initial zephyr with SOF.
On Mon, Aug 22, 2022 at 5:04 PM Daniel Baluta daniel.baluta@gmail.com wrote:
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGN sizeof(void *) +#define PLATFORM_DCACHE_ALIGN 32
everything seems to work.
But I cannot find any logical explanation.
Concerning allocations:
On Tue, 23 Aug 2022, Andy Ross wrote:
It depends on what SOF is going to do with that memory, obviously. :)
The two big limitations in the past have been cache and hardware. The Xtensa architectural cache line size is 64 bytes, not 32, so that's probably not what's tripping you up (though it might be: you'd be 50/50 to get a working pointer!). IIRC the SOF heap actually returns uncached memory these days by default anyway?
I don't think this is the case any more. We're now allocating memory according to the flags: only when _SHARED is used, then uncached memory pointer is returned, otherwise a cached one.
Thanks Guennadi
You need to use the uncached_to_cached() API (or the Zephyr equivalent that it's wrapping) to end up polluting the cache. There were multiple bugs fixed in this space last year.
The other issue on Intel was that one of the DMA controllers (I think it was the Synopsys one, but it might have been HDA) had an (undocumented?) alignment requirement of (I think?) 32 bytes. I'm fuzzier on the details here, but this seems more likely to be the kind of thing you're looking at. The resolution there was to align the requirement at the driver level, I think? So if you're using different drivers you might need to do the same thing. Guennadi or +Kai would be likely to know more about this.
Andy
On Mon, Aug 22, 2022 at 9:44 AM Daniel Baluta daniel.baluta@nxp.com wrote:
- andyross@google.com
Subject: Re: [Sound-open-firmware] Zephyr memory allocation from heap
- Andy, as he also worked on the initial zephyr with SOF.
On Mon, Aug 22, 2022 at 5:04 PM Daniel Baluta daniel.baluta@gmail.com wrote:
Hi guys,
Did you encountered any problems with memory allocations when switching to Zephyr?
I'm facing a very strange issue with a playback pipeline with module adapter (MP3) where it looks like the DAI hits underruns as soon as I start the pipeline.
With normal PCM playback everything works fine.
I think it might be a problem in the sense Zephyr allocates memory and/or Cadence memory alignment requirements?
Looks like as soon as I change this:
-#define PLATFORM_DCACHE_ALIGN sizeof(void *) +#define PLATFORM_DCACHE_ALIGN 32
everything seems to work.
But I cannot find any logical explanation.
Intel Deutschland GmbH Registered Address: Am Campeon 10, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de http://www.intel.de Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928
Thanks a lot Andy / Guennadi for your answers.
I wonder why SOF rmalloc & friends always force the allocations to be aligned with PLATFORM_DCACHE_ALIGN.
On Tue, Aug 23, 2022 at 5:53 PM Andy Ross andyross@google.com wrote:
It depends on what SOF is going to do with that memory, obviously. :)
The two big limitations in the past have been cache and hardware. The Xtensa architectural cache line size is 64 bytes, not 32, so that's probably not what's tripping you up (though it might be: you'd be 50/50 to get a working pointer!). IIRC the SOF heap actually returns uncached memory these days by default anyway? You need to use the uncached_to_cached() API (or the Zephyr equivalent that it's wrapping) to end up polluting the cache. There were multiple bugs fixed in this space last year.
Actually the cache line size depends on processor configuration. On i.MX8 is 128.
The other issue on Intel was that one of the DMA controllers (I think it was the Synopsys one, but it might have been HDA) had an (undocumented?) alignment requirement of (I think?) 32 bytes. I'm fuzzier on the details here, but this seems more likely to be the kind of thing you're looking at. The resolution there was to align the requirement at the driver level, I think? So if you're using different drivers you might need to do the same thing. Guennadi or +Kai would be likely to know more about this.
Thanks this is a good point. Will look into it.
Hi,
On Tue, 23 Aug 2022, Daniel Baluta wrote:
Thanks a lot Andy / Guennadi for your answers.
I wonder why SOF rmalloc & friends always force the allocations to be aligned with PLATFORM_DCACHE_ALIGN.
this is probably due to sof/zephyr/wrapper.c:rmalloc() and way it is currently implemented.
The problem in short: 1) we don't want a fixed partition of a cached and uncached heap - split between cached/uncached usage may vary depending on use-cases 2) we've implemented rmalloc on top of zephyr/sys_heap that stores metadata at start of allocated blocks
Now this has the distinct disadvantage (on noncoherent cache arch like xtensa) that we need to ensure that the metadata (maintained by zephyr sys_heap) of a block allocated for cached use, does not live on the same cacheline as blocks allocated noncached, or cached but used by other cores. IOW, to ensure validity of the heap metadata, we need to align all allocations to cacheline granularity.
This is not a fundamental limitation, just a property of the current mapping. There are a number of ways to allow for more effient approach for small allocations, but each have their own cons: - separate heaps for cached and noncached - use something else than sys_heap to implement the heap (e.g. keep metadata separate from the allocated blocks) - use per-core heaps
... discussion (and patches :)) are welcome. Thanks Daniel for kicking this off.
Br, Kai
participants (6)
-
Andy Ross -
Daniel Baluta -
Daniel Baluta -
Girdwood, Liam R -
Guennadi Liakhovetski -
Kai Vehmanen