More Generic Audio Graph Sound Card idea
Hi Mark Cc Pierre-Louis, Sameer
Current audio-graph-card driver has DPCM support, but it is limited (= Mix/Mux/TDM-split/rate-convert/channel-convert). It was expanded forcibly expanded.
Because of it, the connection judgement for normal vs DPCM is tricky. I know Pierre-Louis want to use it for SOF, but something is missing, thus, can't use (?). I know now Sameer is currently trying to expand it now. It is OK for me, but, more expansion in the future will be difficult/buggy. For example supporting many *generic* DPCM and normal connection in the same time will be difficult.
I'm thinking below style as new audio-graph-card2. It is not tricky, thus connection judgement for normal vs DSP is easy. Then, code can be more readable (= no guaranty :P) ?
I guss "audio-graph-card" and "audio-graph-card2" can have some (= not full) compatible for normal connection. But has no compatible for DPCM connection.
I think I need to keep as-is for current audio-graph-card, and create new audio-graph-card2 driver. Because there is users, and card2 implementation will have huge different. Keeping compatible and update will be very difficult or buggy.
But what do you think ?
Note 1 is that this is just idea for now. I'm very happy to implement it, but it will be after that my all remaining patch was posted/accepted.
Note 2 is that I'm not full DPCM analyst. Thus, maybe audio-graph-card2 can solve current connection issue, but might can't for unknown connection case. In such came we need ver3 (?).
------------ image ---------------------------
| Front End PCMs | SoC DSP | Back End DAIs | Audio devices |
************* PCM0 <------------> * * <----DAI0-----> Codec Headset * * PCM1 <------------> * * <----DAI1-----> Codec Speakers * DSP * PCM2 <------------> * * <----DAI2-----> MODEM * * PCM3 <------------> * * <----DAI3-----> BT * * * * <----DAI4-----> DMIC * * * * <----DAI5-----> FM ************* ------------ DT sample ---------------------------
sound { compatible = "audio-graph-card2";
dais = <&pcm0, &pcm1, &pcm2, &pcm3>; };
front-end { ports { pcm0: port@0 { pcm0_endpoint: endpoint { remote-endpoint = <&dsp_f0>; }; }; pcm1: port@1 { pcm1_endpoint: endpoint { remote-endpoint = <&dsp_f1>; }; }; pcm2: port@2 { pcm2_endpoint: endpoint { remote-endpoint = <&dsp_f2>; }; }; pcm3: port@3 { pcm3_endpoint: endpoint { remote-endpoint = <&dsp_f3>; }; }; }; };
dsp { compatible = "audio-graph-card2-dsp";
ports { /* Front End side */ port@0 { dsp_f0: endpoint { remote-endpoint = <&pcm0>; }; }; port@1 { dsp_f1: endpoint { remote-endpoint = <&pcm1>; }; }; port@2 { dsp_f2: endpoint { remote-endpoint = <&pcm2>; }; }; port@3 { dsp_f3: endpoint { remote-endpoint = <&pcm3>; }; };
/* Back End side */ port@4 { dsp_b0: endpoint { remote-endpoint = <&dai0>; }; }; port@5 { dsp_b1: endpoint { remote-endpoint = <&dai1>; }; }; port@6 { dsp_b2: endpoint { remote-endpoint = <&dai2>; }; }; port@7 { dsp_b3: endpoint { remote-endpoint = <&dai3>; }; }; port@8 { dsp_b4: endpoint { remote-endpoint = <&dai4>; }; }; port@9 { dsp_b5: endpoint { remote-endpoint = <&dai5>; }; }; }; };
back-end { ports { port@0 { dai0: endpoint { remote-endpoint = <&dsp_b0>; }; }; port@1 { dai1: endpoint { remote-endpoint = <&dsp_b1>; }; }; port@2 { dai2: endpoint { remote-endpoint = <&dsp_b2>; }; }; port@3 { dai3: endpoint { remote-endpoint = <&dsp_b3>; }; }; port@4 { dai4: endpoint { remote-endpoint = <&dsp_b4>; }; }; port@5 { dai5: endpoint { remote-endpoint = <&dsp_b5>; }; }; }; };
Hi Kuninori,
CC Jon
Some of my thoughts on below.
On 8/21/2020 9:45 AM, Kuninori Morimoto wrote:
External email: Use caution opening links or attachments
Hi Mark Cc Pierre-Louis, Sameer
Current audio-graph-card driver has DPCM support, but it is limited (= Mix/Mux/TDM-split/rate-convert/channel-convert). It was expanded forcibly expanded.
Yes it appears to be so. Right now it does not seem to be supporting 'demux' kind of use case.
Because of it, the connection judgement for normal vs DPCM is tricky.
Please note that asoc_simple_init_dai_link_params() makes the link codec-to-codec if any component involved has 'non_legacy_dai_naming' flag enabled. It is used by both audio-graph and simple-card. So it is a combination of three (DPCM, normal and codec-to-codec). To avoid all this complication, I am treating all links as DPCM in the series [0] I sent.
I know Pierre-Louis want to use it for SOF, but something is missing, thus, can't use (?).
I know now Sameer is currently trying to expand it now. It is OK for me, but, more expansion in the future will be difficult/buggy. For example supporting many *generic* DPCM and normal connection in the same time will be difficult.
If we encode use case information in DT, it would become regid and is not flexible when the HW is giving us the all flexibility (at least in my case). Thus may lead to complications. If there is a way to configure PCM parameters from the user space it would help to simplify things. Then DT can just describe the HW links.
I'm thinking below style as new audio-graph-card2. It is not tricky, thus connection judgement for normal vs DSP is easy. Then, code can be more readable (= no guaranty :P) ?
I guss "audio-graph-card" and "audio-graph-card2" can have some (= not full) compatible for normal connection. But has no compatible for DPCM connection.
I think I need to keep as-is for current audio-graph-card, and create new audio-graph-card2 driver. Because there is users, and card2 implementation will have huge different. Keeping compatible and update will be very difficult or buggy.
But what do you think ?
Note 1 is that this is just idea for now. I'm very happy to implement it, but it will be after that my all remaining patch was posted/accepted.
Note 2 is that I'm not full DPCM analyst. Thus, maybe audio-graph-card2 can solve current connection issue, but might can't for unknown connection case. In such came we need ver3 (?).
------------ image ---------------------------
| Front End PCMs | SoC DSP | Back End DAIs | Audio devices |
*************PCM0 <------------> * * <----DAI0-----> Codec Headset * * PCM1 <------------> * * <----DAI1-----> Codec Speakers * DSP * PCM2 <------------> * * <----DAI2-----> MODEM * * PCM3 <------------> * * <----DAI3-----> BT * * * * <----DAI4-----> DMIC * * * * <----DAI5-----> FM ************* ------------ DT sample ---------------------------
sound { compatible = "audio-graph-card2"; dais = <&pcm0, &pcm1, &pcm2, &pcm3>; }; front-end { ports { pcm0: port@0 { pcm0_endpoint: endpoint { remote-endpoint = <&dsp_f0>; }; }; pcm1: port@1 { pcm1_endpoint: endpoint { remote-endpoint = <&dsp_f1>; }; }; pcm2: port@2 { pcm2_endpoint: endpoint { remote-endpoint = <&dsp_f2>; }; }; pcm3: port@3 { pcm3_endpoint: endpoint { remote-endpoint = <&dsp_f3>; }; }; }; }; dsp { compatible = "audio-graph-card2-dsp"; ports { /* Front End side */ port@0 { dsp_f0: endpoint { remote-endpoint = <&pcm0>; }; }; port@1 { dsp_f1: endpoint { remote-endpoint = <&pcm1>; }; }; port@2 { dsp_f2: endpoint { remote-endpoint = <&pcm2>; }; }; port@3 { dsp_f3: endpoint { remote-endpoint = <&pcm3>; }; }; /* Back End side */ port@4 { dsp_b0: endpoint { remote-endpoint = <&dai0>; }; }; port@5 { dsp_b1: endpoint { remote-endpoint = <&dai1>; }; }; port@6 { dsp_b2: endpoint { remote-endpoint = <&dai2>; }; }; port@7 { dsp_b3: endpoint { remote-endpoint = <&dai3>; }; }; port@8 { dsp_b4: endpoint { remote-endpoint = <&dai4>; }; }; port@9 { dsp_b5: endpoint { remote-endpoint = <&dai5>; }; }; }; }; back-end { ports { port@0 { dai0: endpoint { remote-endpoint = <&dsp_b0>; }; }; port@1 { dai1: endpoint { remote-endpoint = <&dsp_b1>; }; }; port@2 { dai2: endpoint { remote-endpoint = <&dsp_b2>; }; }; port@3 { dai3: endpoint { remote-endpoint = <&dsp_b3>; }; }; port@4 { dai4: endpoint { remote-endpoint = <&dsp_b4>; }; }; port@5 { dai5: endpoint { remote-endpoint = <&dsp_b5>; }; }; }; };
In my series [0], I am trying to achieve the similar as above though the description may look little different in my case, but that is what HW describes I believe.
Hi Sameer Cc Mark
Thank you for your feedback
Current audio-graph-card driver has DPCM support, but it is limited (= Mix/Mux/TDM-split/rate-convert/channel-convert). It was expanded forcibly expanded.
Yes it appears to be so. Right now it does not seem to be supporting 'demux' kind of use case.
I'm not sure how to do demux. By DPCM ? by multi-Codec ? I'm happy to consider it if there was sample code.
Please note that asoc_simple_init_dai_link_params() makes the link codec-to-codec if any component involved has 'non_legacy_dai_naming' flag enabled. It is used by both audio-graph and simple-card. So it is a combination of three (DPCM, normal and codec-to-codec). To avoid all this complication, I am treating all links as DPCM in the series [0] I sent.
Ahh, we want to consider codec-to-codec.. And I noticed multi-CPU / multi-Codec case too (= for normal connection). Do we have sample code ? > Mark
If we encode use case information in DT, it would become regid and is not flexible when the HW is giving us the all flexibility (at least in my case). Thus may lead to complications. If there is a way to configure PCM parameters from the user space it would help to simplify things. Then DT can just describe the HW links.
What kind of PCM parameters you want to get from user-space ?
Mark, this is question from me. In my understanding, CPU - Codec are merging each parameter automatically at runtime, for example rates / channels / formats (= snd_soc_runtime_calc_hw()). And system-clock / TDM-slot / DAI-format are specified via Sound Card.
I can understand that system-clock is specified via Sound Card, but TDM-slot / DAI-format can be adjusted from CPU / Codec driver automatically same as rates / channels / formats ? I'm not sure it is possible, or not, but I've been wondering for a long time.
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Morimoto-san,
...
If we encode use case information in DT, it would become regid and is not flexible when the HW is giving us the all flexibility (at least in my case). Thus may lead to complications. If there is a way to configure PCM parameters from the user space it would help to simplify things. Then DT can just describe the HW links.
What kind of PCM parameters you want to get from user-space ?
I was referring to channels, rate, sample size for PCM parameters which are part of hw_params(). Having these strictly defined in DT would limit from using the same audio path for different configurations. So far used mixer controls for overriding this info in hw_param(), but this idea is dropped as per Mark's suggestion earlier. The requirement here is, some components have the ability to alter these parameters. Subsequent components in the audio path should me made aware of this.
For example, SRC (sample rate converter) can change 'rate' info coming from hw_param(). Similarly 'mux/demux' can change channel info. Fixing one set of configuration in DT will limit the usage. If this is configurable from user space, then it would be lot more easier.
. . .
On Fri, Aug 21, 2020 at 01:58:15PM +0530, Sameer Pujar wrote:
For example, SRC (sample rate converter) can change 'rate' info coming from hw_param(). Similarly 'mux/demux' can change channel info. Fixing one set of configuration in DT will limit the usage. If this is configurable from user space, then it would be lot more easier.
We shouldn't be fixing the configuration in DT unless it's fixed in the hardware, in which case we should be able to figure this out through constraints at runtime anyway and not need to expose anything to userspace. This covers an awful lot of SRC and other conversion use cases, they're often there because there's some fixed DAI that means there's only one possible setting on one side of the convertor. A lot of the other cases where there's more flexibility can probably be handled in a similar fashion with something other than the SRC itself being configured - normally the goal is to fix a link in a particular configuration through software policy and we can work back to setting up the SRC in the same way as if it was a hardware constraint.
Hi
For more generic Auido Graph Sound Card, I'm thinking about Multi-CPU/Codec now. (Need to think about Codec-to-Codec later, but it is next task) I want to know your idea, opinion, etc, etc...
Please note that asoc_simple_init_dai_link_params() makes the link codec-to-codec if any component involved has 'non_legacy_dai_naming' flag enabled. It is used by both audio-graph and simple-card. So it is a combination of three (DPCM, normal and codec-to-codec). To avoid all this complication, I am treating all links as DPCM in the series [0] I sent.
Ahh, we want to consider codec-to-codec.. And I noticed multi-CPU / multi-Codec case too (= for normal connection). Do we have sample code ? > Mark
First idea is using "ports". In such case, I think multi-CPU side is not big issue, because "dais" selects "port" (and "ports") instead of "endpoint". (Note: here have no compatible to "-scu-" of audio-graph)
CPU0-1 <------> Codec0 CPU0-2 <---/ CPU0-3 <---/
sound { compatible = "audio-graph-card2";
=> dais = <&cpu0>; };
cpu-device { /* * use "ports" for Multi-CPU */ => cpu0: ports { port@0 { cpu0_1: endpoint { remote-endpoint = <&codec0_1>; }; }; port@1 { cpu0_2: endpoint { remote-endpoint = <&codec0_2>; }; }; port@2 { cpu0_3: endpoint { remote-endpoint = <&codec0_3>; }; }; };
/* * According to graph.txt, * we can use other "port" and/or "ports" for non multi-CPU. */ port@x { }; port@x { }; ports { port { }; ... }; };
codec-device { port { codec0_1: endpoint@0 { remote-endpoint = <&cpu0_1>; }; codec0_2: endpoint@0 { remote-endpoint = <&cpu0_2>; }; codec0_3: endpoint@0 { remote-endpoint = <&cpu0_3>; }; }; };
But multi-Codec side is difficult. Because it is selected via "endpoint" via CPU. No way to select it via "port" and/or "ports".
CPU1 <------> Codec1-1 ---> Codec1-2 ---> Codec1-3
sound { compatible = "audio-graph-card2";
dais = <&cpu1>; };
cpu-device { /* * CPU selects Codec by "endpoints". */ cpu1: port { cpu1_1: endpoint { remote-endpoint = <&codec1_1>; }; cpu1_2: endpoint { remote-endpoint = <&codec1_2>; }; cpu1_3: endpoint { remote-endpoint = <&codec1_3>; }; }; };
codec-device { /* * Codec is selected via CPU. * It is best if we can select "codec1" (= port / ports) from CPU. * But is selected via "endpoint" in reality, * using "ports" is not enough for Multi-Codec. * because see below */ codec1: ports { port@0 { codec1_1: endpoint { remote-endpoint = <&dai0_0>; }; }; port@1 { codec1_2: endpoint { remote-endpoint = <&dai0_1>; }; }; port@2 { codec1_3: endpoint { remote-endpoint = <&dai0_2>; }; }; };
/* * Because "ports" is used to for non Multi-xxx too. * How to know the "ports" was for Multi-xxx or not ? */ ports { port@1 { for device1 }; port@2 { for device2 }; port@3 { for device3 }; }; };
We might want to select Multi-CPU/Codec by using multi deivces ? in such case, using "ports" idea is not enough.
Using extra device like DSP can be more generic ?
<--- multi-CPU ---> ******* CPU0-1 <--> * * <--> Codec0 CPU0-2 <--> * * CPU0-3 <--> * * *******
sound { compatible = "audio-graph-card2";
dais = <&cpu0>; };
cpu-device { ports { /* for multi-CPU */ port@0 { cpu0_1: endpoint { remote-endpoint = <&multi_cpu_1>; }; }; port@1 { cpu0_2: endpoint { remote-endpoint = <&multi_cpu_2>; }; }; port@2 { cpu0_3: endpoint { remote-endpoint = <&multi_cpu_3>; }; };
/* for others */ port@3 { ... }; }; };
multi-cpu { compatible = "audio-graph-card2-multi";
/* CPU side */ cpu0: ports@0 { port@0 { multi_cpu_1: endpoint { remote-endpoint = <&cpu0_1>; }; }; port@1 { multi_cpu_2: endpoint { remote-endpoint = <&cpu0_2>; }; }; port@2 { multi_cpu_3: endpoint { remote-endpoint = <&cpu0_3>; }; }; };
/* Codec side */ ports@1 { port@0 { multi_cpu0: endpoint { remote-endpoint = <&codec0>; }; }; }; };
codec-device { ports { port@0 { codec0: endpoint { remote-endpoint = <&multi_cpu0>; }; }; port@1 { ... }; }; };
<--- multi-Codec ---> ******* CPU1 <--> * * <--> Codec1-1 * * <--> Codec1-2 * * <--> Codec1-3 *******
sound { compatible = "audio-graph-card2";
dais = <&cpu1>; };
cpu-device { ports { cpu1: port@0 { cpu1_0: endpoint { remote-endpoint = <&multi_codec1>; }; }; port@1 { ... }; }; };
multi-cpu { compatible = "audio-graph-card2-multi";
/* Front End */ ports@0 { port { multi_codec1: endpoint { remote-endpoint = <&cpu1_0>; }; }; };
/* Back End */ ports@1 { port@0 { multi_codec1_1: endpoint { remote-endpoint = <&codec1_1>; }; }; port@1 { multi_codec1_2: endpoint { remote-endpoint = <&codec1_2>; }; }; port@2 { multi_codec1_3: endpoint { remote-endpoint = <&codec1_3>; }; }; }; };
codec-device { ports { port@0 { codec1_1: endpoint { remote-endpoint = <&multi_codec1_1>; }; }; port@0 { codec1_2: endpoint { remote-endpoint = <&multi_codec1_2>; }; }; port@0 { codec1_3: endpoint { remote-endpoint = <&multi_codec1_3>; }; }; port@1 { ... }; }; };
Thank you for your help !!
Best regards --- Kuninori Morimoto
On Fri, Sep 25, 2020 at 10:43:59AM +0900, Kuninori Morimoto wrote:
But multi-Codec side is difficult. Because it is selected via "endpoint" via CPU. No way to select it via "port" and/or "ports".
Indeed.
We might want to select Multi-CPU/Codec by using multi deivces ? in such case, using "ports" idea is not enough.
Using extra device like DSP can be more generic ?
<--- multi-CPU ---> ******* CPU0-1 <--> * * <--> Codec0 CPU0-2 <--> * * CPU0-3 <--> * * *******
I think this is what we want for SoCs, represent the DSPs explicitly and then have the FEs and BEs all be ports on the DSP. I think a similar thing would also work for legacy (I2S & so on) DAIs where we've got more endpoints on the DAI - if we define slots on the DAI then from the point of view of the DT bindings it's just a very, very inflexible DSP:
CPU1 <--> DAI slot A <--> Codec1-1 -> DAI slot B <--> Codec1-2 CPU2 <--> DAI slot C <--> Codec1-3
or whatever. This doesn't allow for really complex setups that change the slot mapping at runtime (TBH those probably need custom cards anyway) but I think it should support most cases where TDM causes difficulties today. I'm not sure if we need this for more modern buses like SoundWire, I'd hope we can dynamically assign slots at runtime more easily, but ICBW.
Thanks for all your work on this!
On 9/25/20 2:22 PM, Mark Brown wrote:
On Fri, Sep 25, 2020 at 10:43:59AM +0900, Kuninori Morimoto wrote:
But multi-Codec side is difficult. Because it is selected via "endpoint" via CPU. No way to select it via "port" and/or "ports".
Indeed.
We might want to select Multi-CPU/Codec by using multi deivces ? in such case, using "ports" idea is not enough.
Using extra device like DSP can be more generic ?
<--- multi-CPU ---> ******* CPU0-1 <--> * * <--> Codec0 CPU0-2 <--> * * CPU0-3 <--> * * *******
I think this is what we want for SoCs, represent the DSPs explicitly and then have the FEs and BEs all be ports on the DSP. I think a similar thing would also work for legacy (I2S & so on) DAIs where we've got more endpoints on the DAI - if we define slots on the DAI then from the point of view of the DT bindings it's just a very, very inflexible DSP:
CPU1 <--> DAI slot A <--> Codec1-1 \-> DAI slot B <--> Codec1-2 CPU2 <--> DAI slot C <--> Codec1-3or whatever. This doesn't allow for really complex setups that change the slot mapping at runtime (TBH those probably need custom cards anyway) but I think it should support most cases where TDM causes difficulties today. I'm not sure if we need this for more modern buses like SoundWire, I'd hope we can dynamically assign slots at runtime more easily, but ICBW.
SoundWire doesn't have a notion of 'slot'. Instead you program the data ports for the type of audio data to be transmitted/received.
See some pointers at https://mipi.org/sites/default/files/MIPI-SoundWire-webinar-20150121-final.p... Pages 42-47 describe the main concepts.
The actual bit allocation can be done in different ways. On the Intel side, we use a dynamic allocation. It's my understanding that Qualcomm have a static allocation for their amplifier links.
In most cases, a sink port receives exactly what it needs, but for playback we have cases where all amplifiers receive the same data (we call this 'mirror mode', and each amplifier will be configured to render a specific channel from the data received. This is useful to deal with orientation/posture changes where the data transmitted on the wires doesn't need to be changed. This avoid dynamic re-configurations on the DSP + bus sides, only the amplifier settings need to be modified - typically via controls.
That said, the mapping of data ports between CPU and codec sides is rather static, mostly because devices typically dedicate specific data ports to specific functionality. SDCA will not change this, quite the opposite, the mapping between ports and audio functionality behind the port will be defined in platform firmware.
It's a bit of a stretch but conceptually there is some level of overlap between SoundWire data ports and TDM slots, e.g. if in a TDM link you used slots 4,5 for headset playback, you might use data port 2 on a SoundWire link. It's however a 'logical' mapping, the actual position of the bits in the frame is handled by the bit allocation.
Hope this helps! -Pierre
On Fri, Sep 25, 2020 at 03:04:37PM -0500, Pierre-Louis Bossart wrote:
On 9/25/20 2:22 PM, Mark Brown wrote:
difficulties today. I'm not sure if we need this for more modern buses like SoundWire, I'd hope we can dynamically assign slots at runtime more easily, but ICBW.
SoundWire doesn't have a notion of 'slot'. Instead you program the data ports for the type of audio data to be transmitted/received.
...
In most cases, a sink port receives exactly what it needs, but for playback we have cases where all amplifiers receive the same data (we call this 'mirror mode', and each amplifier will be configured to render a specific channel from the data received. This is useful to deal with
Channels are essentially the same as timeslots on a TDM bus TBH.
That said, the mapping of data ports between CPU and codec sides is rather static, mostly because devices typically dedicate specific data ports to specific functionality. SDCA will not change this, quite the opposite, the mapping between ports and audio functionality behind the port will be defined in platform firmware.
It's a bit of a stretch but conceptually there is some level of overlap between SoundWire data ports and TDM slots, e.g. if in a TDM link you used slots 4,5 for headset playback, you might use data port 2 on a SoundWire link. It's however a 'logical' mapping, the actual position of the bits in the frame is handled by the bit allocation.
If a device is hard limited to particular slots we can presumably discover that (either through the spec or by keying off the ID registers) and do the right thing? In any case if we need a firmware mapping for DT systems it sounds like something that works for TDM should be mappable onto SoundWire channels easily enough.
If a device is hard limited to particular slots we can presumably discover that (either through the spec or by keying off the ID registers) and do the right thing? In any case if we need a firmware mapping for DT systems it sounds like something that works for TDM should be mappable onto SoundWire channels easily enough.
yes, if we can define a notion of logical TDM 'slot group' (e.g. headphone, mic, streamA, <add tag here>, etc), then the mapping with SoundWire data ports is straightforward. The mapping would typically come from platform firmware, not by checking device registers.
The actual value of a TDM slot which defines the transport position is however irrelevant for SoundWire.
On Fri, Aug 21, 2020 at 7:16 AM Kuninori Morimoto kuninori.morimoto.gx@renesas.com wrote:
Hi Mark Cc Pierre-Louis, Sameer
Current audio-graph-card driver has DPCM support, but it is limited (= Mix/Mux/TDM-split/rate-convert/channel-convert). It was expanded forcibly expanded.
Because of it, the connection judgement for normal vs DPCM is tricky. I know Pierre-Louis want to use it for SOF, but something is missing, thus, can't use (?).
Hi Morimoto-san,
Thanks for having a look at this. I also tried to use generic machine driver (simple-card) with SOF but I had some trouble creating desired DPCM links.
Main limitation of simple-card is that there is no direct way of specifying DPCM links.
I had an attempt to introduce a more flexible approach for that here: https://lkml.org/lkml/2019/10/13/123
but obviously we can do better than that.
For now, I'm using only normal links with simple-audio card and it works fine for me with SOF, but in the future I think we might need to expand that.
Hi Daniel
Thank you for your feedback
Because of it, the connection judgement for normal vs DPCM is tricky. I know Pierre-Louis want to use it for SOF, but something is missing, thus, can't use (?).
(snip)
Thanks for having a look at this. I also tried to use generic machine driver (simple-card) with SOF but I had some trouble creating desired DPCM links.
Main limitation of simple-card is that there is no direct way of specifying DPCM links.
I had an attempt to introduce a more flexible approach for that here: https://lkml.org/lkml/2019/10/13/123
but obviously we can do better than that.
For now, I'm using only normal links with simple-audio card and it works fine for me with SOF, but in the future I think we might need to expand that.
OK, the issue is also normal vs DPCM judgement. Maybe having "dsp" can solve it, same as audio-graph-card ? I'm not sure we need to update simple-audio, too > Mark ?
Thank you for your help !!
Best regards --- Kuninori Morimoto
On Fri, Aug 21, 2020 at 10:25 AM Kuninori Morimoto kuninori.morimoto.gx@renesas.com wrote:
Hi Daniel
Thank you for your feedback
Because of it, the connection judgement for normal vs DPCM is tricky. I know Pierre-Louis want to use it for SOF, but something is missing, thus, can't use (?).
(snip)
Thanks for having a look at this. I also tried to use generic machine driver (simple-card) with SOF but I had some trouble creating desired DPCM links.
Main limitation of simple-card is that there is no direct way of specifying DPCM links.
I had an attempt to introduce a more flexible approach for that here: https://lkml.org/lkml/2019/10/13/123
but obviously we can do better than that.
For now, I'm using only normal links with simple-audio card and it works fine for me with SOF, but in the future I think we might need to expand that.
OK, the issue is also normal vs DPCM judgement. Maybe having "dsp" can solve it, same as audio-graph-card ? I'm not sure we need to update simple-audio, too > Mark ?
Yes, the condition choosing normal vs dpcm is fairly similar for simple-card and audio-graph-card:
if (dpcm_selectable && ((of_get_child_count(codec_port) > 1) || adata.convert_rate || adata.convert_channels)
Can you expand a little bit more about what having 'DSP' means?
On Fri, Aug 21, 2020 at 01:15:41PM +0900, Kuninori Morimoto wrote:
I'm thinking below style as new audio-graph-card2. It is not tricky, thus connection judgement for normal vs DSP is easy. Then, code can be more readable (= no guaranty :P) ?
Having an audio-graph-card2 isn't ideal but may be required at least during development :/ Ideally we'd be able to have the new driver parse both binding formats (or rather, have the new binding format be new use cases for the same binding format) and only use -card2 while it's in development.
| Front End PCMs | SoC DSP | Back End DAIs | Audio devices |
*************PCM0 <------------> * * <----DAI0-----> Codec Headset * * PCM1 <------------> * * <----DAI1-----> Codec Speakers * DSP * PCM2 <------------> * * <----DAI2-----> MODEM * * PCM3 <------------> * * <----DAI3-----> BT * * * * <----DAI4-----> DMIC * * * * <----DAI5-----> FM *************
I think your plan with this explicit representation of the DSP is where we want to get to, the issue is mapping this onto DPCM as it currently stands. The binding you proposed underneath looked sensible on first readthrough.
Hi Mark Cc Sameer
Having an audio-graph-card2 isn't ideal but may be required at least during development :/ Ideally we'd be able to have the new driver parse both binding formats (or rather, have the new binding format be new use cases for the same binding format) and only use -card2 while it's in development.
If you want to update current audio-graph-card without creating new audio-graph-card2, I'm OK about it. But need adjusting / agreement.
Current audio-graph-card "DSP" user is only me, and I'm using it only locally. Thus upstream doesn't get damage if I removed "audio-graph-scu-card" (= DSP use case) support.
OTOH, Sameer is posting patch for "-cc-" support. If it was accepted and he used it on upstream (= on tegra), keeping compatibility will be very difficult and/or code will be very confusable.
If Sameer can OK and wait new audio-graph-card, maybe we have no compatibility issue. But in such case, 1st version of new audio-graph-card might be not enough for him. Sameer need to waiting / testing / adjusting / etc, etc...
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Morimoto-san,
On 8/24/2020 5:55 AM, Kuninori Morimoto wrote:
External email: Use caution opening links or attachments
Hi Mark Cc Sameer
Having an audio-graph-card2 isn't ideal but may be required at least during development :/ Ideally we'd be able to have the new driver parse both binding formats (or rather, have the new binding format be new use cases for the same binding format) and only use -card2 while it's in development.
If you want to update current audio-graph-card without creating new audio-graph-card2, I'm OK about it. But need adjusting / agreement.
Current audio-graph-card "DSP" user is only me, and I'm using it only locally. Thus upstream doesn't get damage if I removed "audio-graph-scu-card" (= DSP use case) support.
OTOH, Sameer is posting patch for "-cc-" support. If it was accepted and he used it on upstream (= on tegra), keeping compatibility will be very difficult and/or code will be very confusable.
If Sameer can OK and wait new audio-graph-card, maybe we have no compatibility issue. But in such case, 1st version of new audio-graph-card might be not enough for him. Sameer need to waiting / testing / adjusting / etc, etc...
The series [0] introduces small deltas to resolve issues I am facing. As you see, most of the implementation is unchanged for the graph-card driver. Hence I am not sure if we need a new driver now. If at all it gets complicated in future, the "-cc-" compatible can be moved to new driver? Please note that the new "-cc-" compatibility is added to address following and some of these are discussed in [1]. - DPCM usage with component model (where there can be N number of components available and M (<= N) of them can be connected together to form an audio path). For example the path would be like, FE -> BE_1 -> BE_2 -> ... -> BE_M. - I am extending dpcm_path_get() for this reason and DAI ops get called for all connected components.
[0] https://lkml.org/lkml/2020/8/5/42 [1] https://lkml.org/lkml/2020/4/30/519
Thank you for your help !!
Best regards
Kuninori Morimoto
Hi Sameer
The series [0] introduces small deltas to resolve issues I am facing. As you see, most of the implementation is unchanged for the graph-card driver. Hence I am not sure if we need a new driver now.
Yes, maybe it is not needed *for now*, but will be issue in the future, because I can't have normal-link and DPCM-link in the same time, right ?
at all it gets complicated in future, the "-cc-" compatible can be moved to new driver? Please note that the new "-cc-" compatibility is added to address following and some of these are discussed in [1].
- DPCM usage with component model (where there can be N number of
components available and M (<= N) of them can be connected together to form an audio path). For example the path would be like, FE -> BE_1 -> BE_2 -> ... -> BE_M.
- I am extending dpcm_path_get() for this reason and DAI ops get
called for all connected components.
[0] https://lkml.org/lkml/2020/8/5/42 [1] https://lkml.org/lkml/2020/4/30/519
The difference between "-cc" and "card2" is DPCM link detection. "-cc-" will assume all are DPCM link, "card2" will detect both normal-link and DPCM-link via DT.
But, I guess new driver 1st version is focus to detecting normal-link and DPCM-link only.
This means it is not enough for your case, because I can't full reproduce your board/situation. Maybe you need some extra patch on "card2" which "-cc-" added to soc-xxx.c
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Morimoto-san,
CC Jon, Sharad
On 8/25/2020 6:29 AM, Kuninori Morimoto wrote:
External email: Use caution opening links or attachments
Hi Sameer
The series [0] introduces small deltas to resolve issues I am facing. As you see, most of the implementation is unchanged for the graph-card driver. Hence I am not sure if we need a new driver now.
Yes, maybe it is not needed *for now*, but will be issue in the future, because I can't have normal-link and DPCM-link in the same time, right ?
Yes I am forcing usage of DPCM for all links. May be the compatible "-cc-" should reflect this. The reason for doing so is, wanted to use DPCM interface with the component model as previously discussed with Mark in [1]. The existing detection mechanism did not work because, in my case, the HW links are one-to-one and the DT is described that way in [0].
at all it gets complicated in future, the "-cc-" compatible can be moved to new driver? Please note that the new "-cc-" compatibility is added to address following and some of these are discussed in [1].
- DPCM usage with component model (where there can be N number of
components available and M (<= N) of them can be connected together to form an audio path). For example the path would be like, FE -> BE_1 -> BE_2 -> ... -> BE_M.
- I am extending dpcm_path_get() for this reason and DAI ops get
called for all connected components.
[0] https://lkml.org/lkml/2020/8/5/42 [1] https://lkml.org/lkml/2020/4/30/519
The difference between "-cc" and "card2" is DPCM link detection. "-cc-" will assume all are DPCM link, "card2" will detect both normal-link and DPCM-link via DT.
But, I guess new driver 1st version is focus to detecting normal-link and DPCM-link only.
Do you plan to propose something like enum for card2 and has scope for extension, where link type can be normal/DPCM/codec-to-codec/etc., ? Since there are so many board variants and may have some specific requirements, I am wondering being able to detect link type from DT should be the only motivation for a new driver.
This means it is not enough for your case, because I can't full reproduce your board/situation. Maybe you need some extra patch on "card2" which "-cc-" added to soc-xxx.c
I am afraid that even the new driver won't work as it is for my case unless a similar compatible and flag exposed for it. So I am not sure how different it would be from [0].
Thanks, Sameer.
Hi Sameer
But, I guess new driver 1st version is focus to detecting normal-link and DPCM-link only.
Do you plan to propose something like enum for card2 and has scope for extension, where link type can be normal/DPCM/codec-to-codec/etc., ? Since there are so many board variants and may have some specific requirements, I am wondering being able to detect link type from DT should be the only motivation for a new driver.
I'm now thinking that new one can detect normal-link, DPCM-link, multi-CPU / multi-Codec, Codec-to-Codec as normal audio-graph feature. And has .hook callback for Eeach board specific feature. see my previous mail which was To:Pierre-Louis But, it is still under idea, not yet fixed.
I am afraid that even the new driver won't work as it is for my case unless a similar compatible and flag exposed for it. So I am not sure how different it would be from [0].
Yes, that is the reason why I'm discussing how to implement new driver. If we have "-cc-" and you use it on upsteam, we will keep current card driver as-is, and I will create new "card2" driver. Because keeping compatible will be difficult. But I'm not sure what should I do.
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Morimoto-san,
But, I guess new driver 1st version is focus to detecting normal-link and DPCM-link only.
Do you plan to propose something like enum for card2 and has scope for extension, where link type can be normal/DPCM/codec-to-codec/etc., ? Since there are so many board variants and may have some specific requirements, I am wondering being able to detect link type from DT should be the only motivation for a new driver.
I'm now thinking that new one can detect normal-link, DPCM-link, multi-CPU / multi-Codec, Codec-to-Codec as normal audio-graph feature. And has .hook callback for Eeach board specific feature. see my previous mail which was To:Pierre-Louis But, it is still under idea, not yet fixed.
If we plan to go this way, I think we need to consider board specific configuration at init time and one at runtime. In that case there could be multiple compatibles that would get added to the driver and various other requirements can be managed with behavioral flags instead from DT?
Thanks, Sameer.
Hi Sameer
If we plan to go this way, I think we need to consider board specific configuration at init time and one at runtime. In that case there could be multiple compatibles that would get added to the driver and various other requirements can be managed with behavioral flags instead from DT?
This is still just idea though... But for example, if you want to 1) basically, DT is almost audio-graph 2) but want to have customized operation for some part
And if "audio-graph-card2" driver has graph_init() exported function, you can create your own drviver, and use customized audio-graph by using .hooks.
This is just idea/sample, but I'm not sure this is enough or not, and/or if I can do what I want. But do you think it can solve your issue ?
-- own driver --- static const struct of_device_id graph_of_match[] = { => { .compatible = "sameer-audio-graph" }, {}, };
static audio_graph_hooks hooks = { .parse_of_hook_pre = xxx, => .parse_of_hook_post = sameer_parse_of_post, .dai_link_of_pre = xxx, .dai_link_of_post = xxx, => .init = sameer_init, ... };
=> int sameer_init(struct snd_soc_pcm_runtime *rtd) { /* * This will be called runtime init timing. * Call original asoc_simple_dai_init() first * and do own init, for example. */ asoc_simple_dai_init(rtd);
do_something_own_settings(rtd->xxx); }
=> static int sameer_parse_of_post(struct asoc_simple_priv *priv) { struct sameer_priv *my_priv = graph_priv_to_my_priv(priv);
/* * This will be called after audio_graph's graph_parse_of() */
/* * Customize own settings here. * * Special connection ? * Special Setings ? * Calculate something ? * Overwrite something ? */ }
static int sameer_probe(...) { struct sameer_priv *my_priv; struct asoc_simple_priv *graph_priv;
my_priv = zalloc(); graph_priv = my_priv_to_graph_priv(my_priv); => graph_priv->hooks = hooks
/* * Basically, it will do same as audio_graph, * but .hooks will be called if you specified */ => return graph_init(graph_priv); }
--- Kconfig ----
config SND_SAMEER_AUDIO_GRAPH_CARD tristate "Sameer's Audio Graph sound card support" => depends on SND_AUDIO_GRAPH_CARD ...
---- my-dt ----------
/* * DT setting is almost same as audio_graph * which is supporting normal and DPCM. * You can add own property which will be handled under .hook */ ************* PCM0 <------------> * * <----DAI0-----> * DSP * * * <----DAI1-----> ************* PCM1 <------------------------------------------> DAI2
sound { compatible = "sameer-audio-graph";
dais = <&PCM0, /* for DPCM */ &PCM1> /* for normal*/ };
front-end { ports { PCM0: port@0 { pcm0: endpoint { remote-endpoint = <&dsp_f0>; }; }; PCM1: port@1 { pcm1: endpoint { remote-endpoint = <&dai2>; }; }; }; };
dsp { compatible = "audio-graph-card2-dsp";
ports { /* Front End side */ port@0 { dsp_f0: endpoint { remote-endpoint = <&pcm0>; }; };
/* Back End side */ port@4 { dsp_b0: endpoint { remote-endpoint = <&dai0>; }; }; port@5 { dsp_b1: endpoint { remote-endpoint = <&dai1>; }; }; }; };
back-end { ports { port@0 { dai0: endpoint { remote-endpoint = <&dsp_b0>; }; }; port@1 { dai1: endpoint { remote-endpoint = <&dsp_b1>; }; }; }; };
codec { port { dai2: endpoint { remote-endpoint = <&pcm1>; }; }; };
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Morimoto-san,
If we plan to go this way, I think we need to consider board specific configuration at init time and one at runtime. In that case there could be multiple compatibles that would get added to the driver and various other requirements can be managed with behavioral flags instead from DT?
This is still just idea though... But for example, if you want to 1) basically, DT is almost audio-graph 2) but want to have customized operation for some part
And if "audio-graph-card2" driver has graph_init() exported function, you can create your own drviver, and use customized audio-graph by using .hooks.
What you are suggesting is 'audio-graph-card2' is a new restructured version of 'audio-graph-card' with some additional customization available for specific users. Do you think updating existing 'audio-graph-card' itself, with necessary hooks, would be too complicated to handle?
This is just idea/sample, but I'm not sure this is enough or not, and/or if I can do what I want. But do you think it can solve your issue ?
From a brief overview, it may solve my issue in customizing few stuff. But I am not too sure if we want to go that way, because eventually we end up in writing a separate machine driver for Tegra (though there can be common stuff used from the generic graph card). The original idea was to use 'audio-graph-card' and people facing similar issues could use "-cc-" compatible.
...
static audio_graph_hooks hooks = { .parse_of_hook_pre = xxx,=> .parse_of_hook_post = sameer_parse_of_post,
We may end up re-parsing the whole stuff under 'hook_post' (which appears redundant) because I am forcing DPCM and would like func_dpcm() to execute for all links. If I could set a flag under 'hooks' and if 'audio-graph-card' can use it to force DPCM, it would appear to be a better choice.
...
---- my-dt ----------
/* * DT setting is almost same as audio_graph * which is supporting normal and DPCM. * You can add own property which will be handled under .hook */ ************* PCM0 <------------> * * <----DAI0-----> * DSP * * * <----DAI1-----> ************* PCM1 <------------------------------------------> DAI2
I think, the main issue is you want to keep both normal/DPCM link detections, where as I am trying to force DPCM usage. That is mainly because I do have N components and would like to connect M(<=N) components with DPCM usage. My DT would be like this.
************* PCM0 <-------------> * * <------ DAI0 (component-0) ------> ... * * <------ DAI1 (component-1) ------> * Crossbar * ... PCMX <-------------> * * <------ DAIN (component-N) ------> * * *************
So at runtime my audio path could be, at t0: PCM0 -> DAI0 -> DAI1 at t1: PCM0 -> DAI1 -> DAIN
Audio path can have ideally any combination of DAIs (and components) in the path. DT may look like it is similar to having normal links, but conceptually any PCMx can be routed to any set of DAIs. DAIs can be resampler, mixer, multiplexer etc., in SoC. I hope this makes clear as to what I am looking for.
...
dsp { compatible = "audio-graph-card2-dsp";
Sorry I did not understand this. Do you intend to parse 'dsp' separately with some version of audio graph card? In my case 'dsp' is just a 'crossbar' and is registered as a component exposing all routes. However I have described links in the DT in a similar way where my 'crossbar' is exposing FEs and BEs like below.
ports { /* Front End side */ port@0 { dsp_f0: endpoint { remote-endpoint = <&pcm0>; }; }; /* Back End side */ port@4 { dsp_b0: endpoint { remote-endpoint = <&dai0>; }; }; port@5 { dsp_b1: endpoint { remote-endpoint = <&dai1>; }; }; }; }; back-end { ports { port@0 { dai0: endpoint { remote-endpoint = <&dsp_b0>; }; }; port@1 { dai1: endpoint { remote-endpoint = <&dsp_b1>; }; }; }; }; codec { port { dai2: endpoint { remote-endpoint = <&pcm1>; }; }; };Thank you for your help !!
Best regards
Kuninori Morimoto
Hi Sameer
What you are suggesting is 'audio-graph-card2' is a new restructured version of 'audio-graph-card' with some additional customization available for specific users. Do you think updating existing 'audio-graph-card' itself, with necessary hooks, would be too complicated to handle?
It depends on how new driver was implemented. If we need to keep current card, I will keep it as-is and do nothing anymore, and has customize option at card2. If we update current card, customize option will be added to it.
From a brief overview, it may solve my issue in customizing few stuff. But I am not too sure if we want to go that way, because eventually we end up in writing a separate machine driver for Tegra (though there can be common stuff used from the generic graph card). The original idea was to use 'audio-graph-card' and people facing similar issues could use "-cc-" compatible.
The biggest issue on current audio-graph-card is that DPCM feature is limited, and because of it, the link detection is very tricky. Your "-cc-" assumes all links are DPCM, but it is more limitation. We want to have more flexible/generic detection.
dsp { compatible = "audio-graph-card2-dsp";Sorry I did not understand this. Do you intend to parse 'dsp' separately with some version of audio graph card? In my case 'dsp' is just a 'crossbar' and is registered as a component exposing all routes. However I have described links in the DT in a similar way where my 'crossbar' is exposing FEs and BEs like below.
This compatible is used just for indicating audio-graph DSP. "audio-graph-card" will parse it if it was connected. If you confuse it, just ignore for now.
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Sameer
static audio_graph_hooks hooks = { .parse_of_hook_pre = xxx,=> .parse_of_hook_post = sameer_parse_of_post,
We may end up re-parsing the whole stuff under 'hook_post' (which appears redundant) because I am forcing DPCM and would like func_dpcm() to execute for all links. If I could set a flag under 'hooks' and if 'audio-graph-card' can use it to force DPCM, it would appear to be a better choice.
Asking how to judge link to driver is maybe nice idea.
I didn't mentioned in order not to not diverge, but I'm also thinking that some overwrite function itself as you mentioned.
if (priv->hooks->func_pre) priv->hooks->func_pre(...);
if (priv->hooks->func) priv->hooks->func(...); /* driver's function */ else graph_func(...); /* audio-graph function */ if (priv->hooks->func_post) priv->hooks->func_post(...);
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Mark Cc Sameer
Having an audio-graph-card2 isn't ideal but may be required at least during development :/ Ideally we'd be able to have the new driver parse both binding formats (or rather, have the new binding format be new use cases for the same binding format) and only use -card2 while it's in development.
If you want to update current audio-graph-card without creating new audio-graph-card2, I'm OK about it. But need adjusting / agreement.
Current audio-graph-card "DSP" user is only me, and I'm using it only locally. Thus upstream doesn't get damage if I removed "audio-graph-scu-card" (= DSP use case) support.
OTOH, Sameer is posting patch for "-cc-" support. If it was accepted and he used it on upstream (= on tegra), keeping compatibility will be very difficult and/or code will be very confusable.
If Sameer can OK and wait new audio-graph-card, maybe we have no compatibility issue. But in such case, 1st version of new audio-graph-card might be not enough for him. Sameer need to waiting / testing / adjusting / etc, etc...
The problem I'm feeling is that new card (= let's call it as card2 here) and Sameer's posting "-cc-" are conflicted / no-compatibility. It expands card1 DPCM detection.
The good things having "-cc-" is that he can use it immediately. the bad things is that it is dificult to keep compatibility between "card1 + -cc-" and "card2".
Creating card2 for development purpose is very welcome for me, and card1 user :) But after that, if we want to merge card1 and card2, we need to drop "-cc-" support unfortunately. If we want to keep "-cc-", we need to have both card1 / card2.
So, how to handle it ?
Thank you for your help !!
Best regards --- Kuninori Morimoto
Hi Mark, Morimoto-san,
Having an audio-graph-card2 isn't ideal but may be required at least during development :/ Ideally we'd be able to have the new driver parse both binding formats (or rather, have the new binding format be new use cases for the same binding format) and only use -card2 while it's in development.
If you want to update current audio-graph-card without creating new audio-graph-card2, I'm OK about it. But need adjusting / agreement.
Current audio-graph-card "DSP" user is only me, and I'm using it only locally. Thus upstream doesn't get damage if I removed "audio-graph-scu-card" (= DSP use case) support.
OTOH, Sameer is posting patch for "-cc-" support. If it was accepted and he used it on upstream (= on tegra), keeping compatibility will be very difficult and/or code will be very confusable.
If Sameer can OK and wait new audio-graph-card, maybe we have no compatibility issue. But in such case, 1st version of new audio-graph-card might be not enough for him. Sameer need to waiting / testing / adjusting / etc, etc...
The problem I'm feeling is that new card (= let's call it as card2 here) and Sameer's posting "-cc-" are conflicted / no-compatibility. It expands card1 DPCM detection.
The good things having "-cc-" is that he can use it immediately. the bad things is that it is dificult to keep compatibility between "card1 + -cc-" and "card2".
Creating card2 for development purpose is very welcome for me, and card1 user :) But after that, if we want to merge card1 and card2, we need to drop "-cc-" support unfortunately. If we want to keep "-cc-", we need to have both card1 / card2.
So, how to handle it ?
Need your help for clarity on above. Currently I cannot publish v3 for [0] unless above is resolved.
[0] https://lkml.org/lkml/2020/8/5/42
Thanks, Sameer.
Hi Morimoto-san,
I know Pierre-Louis want to use it for SOF, but something is missing, thus, can't use (?).
Here's a bit more background.
For SoundWire, we could have the following examples for amplifier connections, where in some cases we have two amplifiers located on separate links (top diagram) and in the second they are on the same link.
| FE PCMs | SoC DSP | BE DAIs | Audio devices |
************* PCM0 <---> * * DAI0 <---> Codec Headset * * PCM1 <---> * * DAI1 <---> Amp Left * DSP * PCM2 <---> * * DAI2 <---> Amp Right * * PCM3 <---> * * DAI3 <---> BT * * * * DAI4 <---> DMIC * * * * DAI5 <---> FM *************
************* PCM0 <---> * * DAI0 <---> Codec Headset * * PCM1 <---> * * DAI1 <---> Amp Left * DSP * | PCM2 <---> * * DAI2/ --> Amp Right * * PCM3 <---> * * DAI3 <---> BT * * * * DAI4 <---> DMIC * * * * DAI5 <---> FM *************
We'd need a means to express that the two amplifiers are really supposed to operate concurrently and be synchronized, and let the drivers know which configuration to use (multi-cpu/multi-codec or single CPU/multi-codec).
Currently we do this manually by hard-coding an 'endpoint group', see e.g. https://elixir.bootlin.com/linux/latest/source/sound/soc/intel/common/soc-ac..., and use this information in the machine driver to know which of the configurations to use.
At some point we'd like to read this information from platform firmware (will be ACPI for Intel but that's a detail) and create the dais in the right way by grouping endpoints on the same dailink when they are connected, and create separate dailink otherwise.
Note that we only want the connection between BEs and audio devices to be described in platform firmware, the topology inside the SOC DSP is handled with a topology file that can be changed at will to e.g. add/remove processing.
We really need to make sure the audio graph can be described in two parts, the bottom part related to hardware routing and layout, and the DSP part. Of course the topology part would have to be constrained to use the DAIs used in the lower level.
Hope this helps explain what I am looking for.
Thank you for starting this thread!
Regards
-Pierre
Hi ALSA ML
I'm thinking below style as new audio-graph-card2. It is not tricky, thus connection judgement for normal vs DSP is easy. Then, code can be more readable (= no guaranty :P) ?
I have proposed DPCM connection idea for audio-card2 before, and proposed Multi-CPU/Codec connection idea later. I'm not yet implemented these, but I want to merge / reuse these idea to keep simple code.
By this mail, I want to propose Codec2Codec idea. It is almost same idea with DPCM, Multi-CPU/Codec, and is last parts for missing feature.
I think this idea can be generic and easy to update in the future (if needed).
But, I want feedback, opinion. I wonder does it works for your use-case ?
-- Image ---
Codec2Codec
Codec1 <--> Codec2
-- DT Image --
+--+ => | |<--> Codec1 | |<--> Codec2 +--+
-- DT Sample --
sound { compatible = "audio-graph-card2";
dais = <&codec2codec>; };
CODEC2CODEC { compatible = "audio-graph-card2-codec2codec";
ports { codec2codec: port@0 { fe_ep: endpoint { remote-endpoint = <&codec1_ep>; }; }; port@1 { be_ep: endpoint { remote-endpoint = <&codec2_ep>; }; }; }; };
Codec1 { port@0 { codec1_ep: endpoint { remote-endpoint = <&fe_ep>; }; }; };
Codec2 { port@0 { codec2_ep: endpoint { remote-endpoint = <&be_ep>; }; }; };
-- Image ---
******* PCM0 <--> * * <--> DAI0: Codec Headset PCM1 <--> * * <--> DAI1: Codec Speakers PCM2 <--> * DSP * <--> DAI2: MODEM PCM3 <--> * * <--> DAI3: BT * * <--> DAI4: DMIC * * <--> DAI5: FM *******
-- DT Image --
+--+ **** +--+ => | |<--> PCM0 * * <-- (dummy) --> | | <= | |<-- (dummy) --> * * DAI0 <--> | | +--+ * * +--+ * * +--+ * * +--+ => | |<--> PCM1 * * <-- (dummy) --> | | <= | |<-- (dummy) --> * * DAI1 <--> | | +--+ * * +--+ ... ... ...
-- DT Sample --
sound { compatible = "audio-graph-card2";
/* indicate all Front-End, Back-End in DPCM case */ dais = <&dsp_fe0, &dsp_fe1, &dsp_fe2, &dsp_fe3, &dsp_be0, &dsp_be1, &dsp_be2, &dsp_be3, &dsp_be4, &dsp_be5>; };
DSP { compatible = "audio-graph-card2-dsp";
ports@0 { /* Front-End is pcm0_ep, Back-End is dummy */ dsp_fe0: port@0 { dsp_fe0_ep: endpoint { remote-endpoint = <&pcm0_ep>; }; }; port@1 { }; };
...
ports@4 { /* Front-End is dummy, Back-End is dai_ep */ dsp_be0: port@0 { }; port@1 { dsp_be0_ep: endpoint { remote-endpoint = <&dai0_ep>; }; }; };
...
};
CPU { ports { port@0 { pcm0_ep: endpoint { remote-endpoint = <&dsp_fe0_ep>; }; }; port@1 { pcm1_ep: endpoint { remote-endpoint = <&dsp_fe1_ep>; }; }; port@2 { pcm2_ep: endpoint { remote-endpoint = <&dsp_fe2_ep>; }; }; port@3 { pcm3_ep: endpoint { remote-endpoint = <&dsp_fe3_ep>; }; }; ... }; };
Codec { ports { port@0 { dai0_ep: endpoint { remote-endpoint = <&dsp_be0_ep>; }; }; port@1 { dai1_ep: endpoint { remote-endpoint = <&dsp_be1_ep>; }; }; port@2 { dai2_ep: endpoint { remote-endpoint = <&dsp_be2_ep>; }; }; port@3 { dai3_ep: endpoint { remote-endpoint = <&dsp_be3_ep>; }; }; port@4 { dai4_ep: endpoint { remote-endpoint = <&dsp_be4_ep>; }; }; port@5 { dai5_ep: endpoint { remote-endpoint = <&dsp_be5_ep>; }; }; ... }; };
-- Image ---
As Sample, Normal Audio-Graph
CPU <---> Codec (Same as before)
-- DT Sample --
sound { compatible = "audio-graph-card2";
dais = <&cpu>; };
CPU { cpu: port { cpu_ep: endpoint { remote-endpoint = <&codec_ep>; }; }; };
Codec { codec: port { codec_ep: endpoint { remote-endpoint = <&cpu_ep>; }; }; };
-- Image ---
Multi-CPU/Codec (complex connection as sample)
<- multi_CPU_0 -> ****** CPU01 <--> * * <--> Codec0 -> * * CPU02 <--> * * ******
<-- multi_codec_1 --> ****** CPU1 <--> * * <--> Codec11 * * <-/ * * <--> Codec12 ******
-- DT Image --
+--+ +--+ => | | <--> CPU01_1 => | | <--> CPU1 | | <--> CPU01_2 | | | | <--> CPU02 | | <--> Codec11_1 | | | | <--> Codec11_2 | | <--> Codec0 | | <--> Codec12 +--+ +--+
-- DT Sample --
sound { compatible = "audio-graph-card2";
dais = <&cpu_0, &cpu_1>; };
MULTI0 { compatible = "audio-graph-card2-multi";
cpu_0: ports@0 { port@0 { fe0_1_ep: endpoint { remote-endpoint = <&cpu01_1_ep>; }; }; port@1 { fe0_2_ep: endpoint { remote-endpoint = <&cpu01_2_ep>; }; }; port@2 { fe0_3_ep: endpoint { remote-endpoint = <&cpu02_ep>; }; }; }; ports@1 { port@0 { be0_ep: endpoint { remote-endpoint = <&codec0_ep>; }; }; }; };
MULTI1 { compatible = "audio-graph-card2-multi";
cpu_1: ports@0 { port@0 { fe1_ep: endpoint { remote-endpoint = <&cpu1_ep>; }; }; }; ports@1 { port@0 { be1_1_ep: endpoint { remote-endpoint = <&codec11_1_ep>; }; }; port@1 { be1_2_ep: endpoint { remote-endpoint = <&codec11_2_ep>; }; }; port@2 { be1_3_ep: endpoint { remote-endpoint = <&codec12_ep>; }; }; }; };
CPU01 { ports { port@0 { cpu01_1_ep: endpoint { remote-endpoint = <&fe0_1_ep>; }; }; port@1 { cpu01_2_ep: endpoint { remote-endpoint = <&fe0_2_ep>; }; }; ... }; };
CPU02 { ports { port@0 { cpu02_ep: endpoint { remote-endpoint = <&fe0_3_ep>; }; }; ... }; };
CPU1 { ports { port@0 { cpu1_ep: endpoint { remote-endpoint = <&fe1_ep>; }; }; ... }; };
Codec0 { ports { port@0 { codec0_ep: endpoint { remote-endpoint = <&be0_ep>; }; }; port@1 { ... }; }; };
Codec11 { ports { port@0 { codec11_1_ep: endpoint { remote-endpoint = <&be1_1_ep>; }; }; port@1 { codec11_2_ep: endpoint { remote-endpoint = <&be1_2_ep>; }; }; port@2 { ... }; }; };
Codec12 { ports { port@0 { codec12_ep: endpoint { remote-endpoint = <&be1_3_ep>; }; }; port@1 { ... }; }; };
On Tue, Oct 13, 2020 at 01:50:25PM +0900, Kuninori Morimoto wrote:
I'm thinking below style as new audio-graph-card2. It is not tricky, thus connection judgement for normal vs DSP is easy. Then, code can be more readable (= no guaranty :P) ?
I have proposed DPCM connection idea for audio-card2 before, and proposed Multi-CPU/Codec connection idea later. I'm not yet implemented these, but I want to merge / reuse these idea to keep simple code.
By this mail, I want to propose Codec2Codec idea. It is almost same idea with DPCM, Multi-CPU/Codec, and is last parts for missing feature.
I think this idea can be generic and easy to update in the future (if needed).
But, I want feedback, opinion. I wonder does it works for your use-case ?
-- Image ---
Codec2Codec
Codec1 <--> Codec2
-- DT Image --
+--+ => | |<--> Codec1 | |<--> Codec2 +--+
-- DT Sample --
sound { compatible = "audio-graph-card2";
dais = <&codec2codec>;};
CODEC2CODEC { compatible = "audio-graph-card2-codec2codec";
ports { codec2codec: port@0 { fe_ep: endpoint { remote-endpoint = <&codec1_ep>; }; }; port@1 { be_ep: endpoint { remote-endpoint = <&codec2_ep>; }; }; };};
I'm a bit confused about the use of FE and BE here since the two CODECs are peers rather than this being DPCM - I think that's just a bit cut'n'paste rather than anything else though? The other examples look sensible I think but this one feels odd.
-- Image ---
*******PCM0 <--> * * <--> DAI0: Codec Headset PCM1 <--> * * <--> DAI1: Codec Speakers PCM2 <--> * DSP * <--> DAI2: MODEM PCM3 <--> * * <--> DAI3: BT * * <--> DAI4: DMIC * * <--> DAI5: FM *******
DSP { compatible = "audio-graph-card2-dsp";
ports@0 { /* Front-End is pcm0_ep, Back-End is dummy */ dsp_fe0: port@0 { dsp_fe0_ep: endpoint { remote-endpoint = <&pcm0_ep>; }; }; port@1 { }; };
I'm not clear why we need to represent the dummy port here?
Hi Mark
Thank you for feedback
-- DT Image --
+--+ => | |<--> Codec1 | |<--> Codec2 +--+
-- DT Sample --
sound { compatible = "audio-graph-card2";
dais = <&codec2codec>;};
CODEC2CODEC { compatible = "audio-graph-card2-codec2codec";
ports { codec2codec: port@0 { fe_ep: endpoint { remote-endpoint = <&codec1_ep>; }; }; port@1 { be_ep: endpoint { remote-endpoint = <&codec2_ep>; }; }; };};
I'm a bit confused about the use of FE and BE here since the two CODECs are peers rather than this being DPCM - I think that's just a bit cut'n'paste rather than anything else though? The other examples look sensible I think but this one feels odd.
FE here is indicating CPU part's Codec, BE is for Codec part's Codec.
-- Image ---
*******PCM0 <--> * * <--> DAI0: Codec Headset PCM1 <--> * * <--> DAI1: Codec Speakers PCM2 <--> * DSP * <--> DAI2: MODEM PCM3 <--> * * <--> DAI3: BT * * <--> DAI4: DMIC * * <--> DAI5: FM *******
DSP { compatible = "audio-graph-card2-dsp";
ports@0 { /* Front-End is pcm0_ep, Back-End is dummy */ dsp_fe0: port@0 { dsp_fe0_ep: endpoint { remote-endpoint = <&pcm0_ep>; }; }; port@1 { }; };I'm not clear why we need to represent the dummy port here?
I'm thining that port@0 is IN (= CPU), port@1 is OUT (= Codec). Empty endpoint means not yet connected (= dummy).
It is difficut to know it was CPU-dummy or dummy-Codec without it.
Thank you for your help !!
Best regards --- Kuninori Morimoto
participants (5)
-
Daniel Baluta -
Kuninori Morimoto -
Mark Brown -
Pierre-Louis Bossart -
Sameer Pujar