Re: [alsa-devel] [RFC/PATCH v6 03/12] media: Entities, pads and links
Laurent Pinchart wrote:
A link is a point-to-point oriented connection between two pads, either on the same entity or on different entities. Data flows from a source pad to a sink pad.
Links are stored in the source entity.
In the descriptors of USB Audio and HDAudio devices, the links are stored only in the sink. But AFAICS this doesn't matter in the media driver API.
+Links have flags that describe the link capabilities and state.
- MEDIA_LINK_FLAG_ACTIVE indicates that the link is active and can be
- used to transfer media data. When two or more links target a sink pad,
- only one of them can be active at a time.
- MEDIA_LINK_FLAG_IMMUTABLE indicates that the link active state can't
- be modified at runtime. If MEDIA_LINK_FLAG_IMMUTABLE is set, then
- MEDIA_LINK_FLAG_ACTIVE must also be set since an immutable link is
- always active.
So the intended userspace API for controlling routing is to change the ACTIVE flag of links?
In USB and HD audio devices, all links are immutable, and the routing is controlled by 'selector' entities that activate exactly one of their input pads. In userspace, this entity shows up as a mixer control. I guess it would be possible to map the ACTIVE flag onto these controls.
Alternatively, entities can have 'mute' mixer controls associated with their pads. In this case, multiple unmuted inputs would be mixed together.
+#define MEDIA_ENTITY_TYPE_MASK 0x00ff0000 +#define MEDIA_ENTITY_SUBTYPE_MASK 0x0000ffff
+#define MEDIA_ENTITY_TYPE_NODE (1 << MEDIA_ENTITY_TYPE_SHIFT) ... +#define MEDIA_ENTITY_TYPE_NODE_ALSA (MEDIA_ENTITY_TYPE_NODE + 3)
ALSA has PCM and MIDI devices, and several types of mixer controls. (It also has hardware dependent and timer devices, but I don't think these would need topology information.) So we need at least these: MEDIA_ENTITY_TYPE_NODE_ALSA_PCM MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
Furthermore, topology information is also needed for entities not associated with a mixer control, such as microphones, speakers, jacks/ connectors, and effect units. These entities are defined in the USB and HD audio specifications, but are not yet handled by ALSA.
+struct media_entity { ...
- union {
/* Node specifications */struct {u32 major;u32 minor;} v4l;struct {u32 major;u32 minor;} fb;int alsa;int dvb;/* Sub-device specifications *//* Nothing needed yet */- };
ALSA devices are not addressed by their device node but with card/device/ subdevice numbers; mixer controls have numeric IDs, unique per card:
struct { int card; int device; int subdevice; } alsa_device; struct { int card; int numid; } alsa_control;
Regards, Clemens
On Thu, Nov 25, 2010 at 10:38:05AM +0100, Clemens Ladisch wrote:
In USB and HD audio devices, all links are immutable, and the routing is controlled by 'selector' entities that activate exactly one of their input pads. In userspace, this entity shows up as a mixer control. I guess it would be possible to map the ACTIVE flag onto these controls.
Ditto for ASoC, mostly.
Alternatively, entities can have 'mute' mixer controls associated with their pads. In this case, multiple unmuted inputs would be mixed together.
ALSA has PCM and MIDI devices, and several types of mixer controls. (It also has hardware dependent and timer devices, but I don't think these would need topology information.) So we need at least these: MEDIA_ENTITY_TYPE_NODE_ALSA_PCM MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
Furthermore, topology information is also needed for entities not associated with a mixer control, such as microphones, speakers, jacks/ connectors, and effect units. These entities are defined in the USB and HD audio specifications, but are not yet handled by ALSA.
All this and more in the embedded case - digital audio link nodes and DSP I/O nodes (could possibly do those as digital audio ones) spring to mind. Also bear in mind that embedded devices can get *very* large - a mobile phone audio system can have of the order of 100 nodes in the graph.
ALSA devices are not addressed by their device node but with card/device/ subdevice numbers; mixer controls have numeric IDs, unique per card:
struct { int card; int device; int subdevice; } alsa_device; struct { int card; int numid; } alsa_control;
For the embedded stuff we also have a bunch of stuff in the graph which may not be visible to userspace at all at present and would just have a string based identifier.
Hi Mark,
On Thursday 25 November 2010 14:41:35 Mark Brown wrote:
On Thu, Nov 25, 2010 at 10:38:05AM +0100, Clemens Ladisch wrote:
In USB and HD audio devices, all links are immutable, and the routing is controlled by 'selector' entities that activate exactly one of their input pads. In userspace, this entity shows up as a mixer control. I guess it would be possible to map the ACTIVE flag onto these controls.
Ditto for ASoC, mostly.
Alternatively, entities can have 'mute' mixer controls associated with their pads. In this case, multiple unmuted inputs would be mixed together.
ALSA has PCM and MIDI devices, and several types of mixer controls. (It also has hardware dependent and timer devices, but I don't think
these would need topology information.) So we need at least these: MEDIA_ENTITY_TYPE_NODE_ALSA_PCM MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
Furthermore, topology information is also needed for entities not associated with a mixer control, such as microphones, speakers, jacks/ connectors, and effect units. These entities are defined in the USB and HD audio specifications, but are not yet handled by ALSA.
All this and more in the embedded case - digital audio link nodes and DSP I/O nodes (could possibly do those as digital audio ones) spring to mind. Also bear in mind that embedded devices can get *very* large - a mobile phone audio system can have of the order of 100 nodes in the graph.
It depends on how you define nodes. I can certainly imagine a graph with 100 controls, but maybe several controls can be part of the same node ? On the video side we've decided to split entities depending on the possible data paths configurations. As I'm not a fluent ascii-art speaker, please have a look at pages 4 and 5 of http://www.ideasonboard.org/media/20101103-lpc- media.pdf
Page 4 shows the internal topology of the OMAP3 ISP. The major blocks in that diagram are reported as entities. Page 5 shows the internal topology of one of the blocks, the OMAP3 ISP preview engine. As you can see the pipeline is made of sub-blocks that implement a single image processing function. As the pipeline is linear (don't worry about the non-linear part in the beginning, it's just there to take into account link configurability at the higher level) we don't export all the sub-blocks as entities, but we expose the controls on the preview engine entity instead.
ALSA devices are not addressed by their device node but with card/device/
subdevice numbers; mixer controls have numeric IDs, unique per card: struct { int card; int device; int subdevice; } alsa_device; struct { int card; int numid; } alsa_control;
For the embedded stuff we also have a bunch of stuff in the graph which may not be visible to userspace at all at present and would just have a string based identifier.
That could be easily added (provided the string is not too long).
On Thu, Nov 25, 2010 at 04:29:53PM +0100, Laurent Pinchart wrote:
It depends on how you define nodes. I can certainly imagine a graph with 100 controls, but maybe several controls can be part of the same node ? On the video side we've decided to split entities depending on the possible data paths configurations. As I'm not a fluent ascii-art speaker, please have a look at pages 4 and 5 of http://www.ideasonboard.org/media/20101103-lpc- media.pdf
Please take a look at:
http://www.wolfsonmicro.com/products/WM8994
for an example of a modern smartphone CODEC. This has in the ballpark of 75-100 nodes (I've not counted exactly) represented in the routing graph in the driver today. This does not include any audio routing available in other components such as the CPU.
Hi Clemens,
Thanks a lot for the review.
On Thursday 25 November 2010 10:38:05 Clemens Ladisch wrote:
Laurent Pinchart wrote:
A link is a point-to-point oriented connection between two pads, either on the same entity or on different entities. Data flows from a source pad to a sink pad.
Links are stored in the source entity.
In the descriptors of USB Audio and HDAudio devices, the links are stored only in the sink. But AFAICS this doesn't matter in the media driver API.
Same for USB Video (UVC). The reason for that is that UAC and UVC have only 1- to-many connections (same for the media controller by the way), and it's easier to store that information in fixed-size structures if you store it in the sink entities.
On the kernel side, the media controller stores links in both source and sink entities for internal reasons. Links stored in the sink entity are called backlinks.
On the userspace side applications will only see "forward" links, stored in source entities. Applications will of course be free to store the links whereever they want in their internal data structures.
As you mentioned, I think it doesn't matter much anyway.
+Links have flags that describe the link capabilities and state.
- MEDIA_LINK_FLAG_ACTIVE indicates that the link is active and can be
- used to transfer media data. When two or more links target a sink pad,
- only one of them can be active at a time.
- MEDIA_LINK_FLAG_IMMUTABLE indicates that the link active state can't
- be modified at runtime. If MEDIA_LINK_FLAG_IMMUTABLE is set, then
- MEDIA_LINK_FLAG_ACTIVE must also be set since an immutable link is
- always active.
So the intended userspace API for controlling routing is to change the ACTIVE flag of links?
That's correct.
In USB and HD audio devices, all links are immutable, and the routing is controlled by 'selector' entities that activate exactly one of their input pads. In userspace, this entity shows up as a mixer control. I guess it would be possible to map the ACTIVE flag onto these controls.
Same for UVC once again. It would be quite easy to map selector units to link activation, but it doesn't even have to be done. We could keep the selector units in the graph with all links immutable, and use the existing APIs to control the selector units.
The alternative would have been to add selector units to the media controller. I've thought about it, but we have embedded hardware (at least on the video side) that have configurable links without selector units. The media controller would have had to create fake selector units.
Between creating fake selector and translating link activation to selector unit commands, I've decided to go for the second solution, as this will make the graph simpler. Once again we can decide to keep explicit selector units in the graph for UVC, UAC and HD audio devices. That's one I've done in a test implementation of the media controller API in the uvcvideo driver.
Alternatively, entities can have 'mute' mixer controls associated with their pads. In this case, multiple unmuted inputs would be mixed together.
I don't see any problem keeping those.
+#define MEDIA_ENTITY_TYPE_MASK 0x00ff0000 +#define MEDIA_ENTITY_SUBTYPE_MASK 0x0000ffff
+#define MEDIA_ENTITY_TYPE_NODE (1 << MEDIA_ENTITY_TYPE_SHIFT) ... +#define MEDIA_ENTITY_TYPE_NODE_ALSA (MEDIA_ENTITY_TYPE_NODE + 3)
ALSA has PCM and MIDI devices, and several types of mixer controls. (It also has hardware dependent and timer devices, but I don't think these would need topology information.) So we need at least these: MEDIA_ENTITY_TYPE_NODE_ALSA_PCM MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
I agree about PCM and MIDI, but I'm not sure about controls. If controls are part of an entity, the entity will be reported through the media controller API. If information about that entity can be queried through existing APIs (ALSA, V4L, ...) those APIs should be used. For instance, on the V4L side, V4L2 sub-devices are mapped to entities and also have a device node which can be used to enumerate the controls supported by the subdev. The media controller only reports the entity -> major:minor mapping to let applications find the device and query it directly.
I think we will need a new ioctl in the media controller API to report advanced information about an entity. This could be used to report controls implemented by an ALSA element if ALSA doesn't provide a way to do that directly. Another use case, which make me think that such an ioctl would be needed, is to report UVC extension units type (16-byte GUID) to userspace.
Furthermore, topology information is also needed for entities not associated with a mixer control, such as microphones, speakers, jacks/ connectors, and effect units. These entities are defined in the USB and HD audio specifications, but are not yet handled by ALSA.
Agreed, we will need to add new entity types and subtypes for those. The reason they're not part of this submission is that I wanted real use cases instead of coming up with a made-up list of entities I think would be useful.
+struct media_entity { ...
- union {
/* Node specifications */struct {u32 major;u32 minor;} v4l;struct {u32 major;u32 minor;} fb;int alsa;int dvb;/* Sub-device specifications *//* Nothing needed yet */- };
ALSA devices are not addressed by their device node but with card/device/ subdevice numbers; mixer controls have numeric IDs, unique per card:
struct { int card; int device; int subdevice; } alsa_device;
I will use this instead of 'int alsa'. Thanks a lot for the useful feedback.
struct { int card; int numid; } alsa_control;
Regarding controls, let's first see how they should be reported by userspace, as explained above.
On Thu, Nov 25, 2010 at 04:21:38PM +0100, Laurent Pinchart wrote:
On Thursday 25 November 2010 10:38:05 Clemens Ladisch wrote:
ALSA has PCM and MIDI devices, and several types of mixer controls. (It also has hardware dependent and timer devices, but I don't think these would need topology information.) So we need at least these: MEDIA_ENTITY_TYPE_NODE_ALSA_PCM MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
I agree about PCM and MIDI, but I'm not sure about controls. If controls are part of an entity, the entity will be reported through the media controller API. If information about that entity can be queried through existing APIs (ALSA, V4L, ...) those APIs should be used. For instance, on the V4L side, V4L2 sub-devices are mapped to entities and also have a device node which can be used to enumerate the controls supported by the subdev. The media controller only reports the entity -> major:minor mapping to let applications find the device and query it directly.
For audio we don't currently have a sensible API for associating controls with any sort of map of how the device is laid out, userspace has to play guessing games.
I think we will need a new ioctl in the media controller API to report advanced information about an entity. This could be used to report controls implemented by an ALSA element if ALSA doesn't provide a way to do that directly. Another use case, which make me think that such an ioctl would be needed, is to report UVC extension units type (16-byte GUID) to userspace.
That seems reasonable.
Laurent Pinchart wrote:
On Thursday 25 November 2010 10:38:05 Clemens Ladisch wrote:
MEDIA_ENTITY_TYPE_NODE_ALSA_PCM MEDIA_ENTITY_TYPE_NODE_ALSA_MIDI MEDIA_ENTITY_TYPE_SUBDEV_ALSA_CONTROL
I agree about PCM and MIDI, but I'm not sure about controls. If controls are part of an entity, the entity will be reported through the media controller API. If information about that entity can be queried through existing APIs (ALSA, V4L, ...) those APIs should be used.
At the moment, ALSA has no API for topology information.
I can certainly imagine a graph with 100 controls, but maybe several controls can be part of the same node ?
There is indeed no strict 1:1 relation; e.g., volume and mute are often part of the same node. So it looks we need some kind of separate ALSA node, which can be associated with mixer controls and/or other information.
ALSA already has is a method to query arbitrary (TLV) metadata for mixer controls; the entity/control relationship can be stored in the control.
I think we will need a new ioctl in the media controller API to report advanced information about an entity. This could be used to report controls implemented by an ALSA element if ALSA doesn't provide a way to do that directly.
This advanced information would always be specific to the entity type, so maybe this should be part of that subsystem's API. Otherwise, the media_entity would need a callback, or store a pointer to some memory block (which assumes that the information is always constant).
Furthermore, topology information is also needed for entities not associated with a mixer control, such as microphones, speakers, jacks/ connectors, and effect units. These entities are defined in the USB and HD audio specifications, but are not yet handled by ALSA.
Agreed, we will need to add new entity types and subtypes for those. The reason they're not part of this submission is that I wanted real use cases instead of coming up with a made-up list of entities I think would be useful.
The reason that I'm always mentioning the USB and HD audio specs is that those already define entities that should cover practically all of the audio needs. (And, of course, we want to be able to report those entities without having to do too many conversions.)
I'll see if I can draw up the ALSA-specific media stuff over the weekend.
Regards, Clemens
I wrote:
I'll see if I can draw up the ALSA-specific media stuff over the weekend.
Sorry, wrong weekend.
Anyway, below are some remarks and a patch.
* Entity types
TYPE_NODE was renamed to TYPE_DEVICE because "node" sounds like a node in a graph, which does not distinguish it from other entity types because all entities are part of the topology graph. I chose "device" as this type describes entities that are visible as some device node to other software.
TYPE_EXT describes entities that represent some interface to the external world, TYPE_INT those that are internal to the entire device. (I'm not sure if that distinction is very useful, but TYPE_SUBDEV seems to be an even more meaningless name.)
ALSA mixer controls are not directly represented; a better fit for the architecture of actual devices is that one or more mixer controls can be associated with an entity. (This can be done with a field of the mixer control.)
* Entity properties
There needs to be a mechanism to associate meta-information (properties) with entities. This information should be optional and extensible, but, when being handled inside the kernel, doesn't need to be more than a read-only blob. I think that something like ALSA's TLV format (used for mixer controls) can be used here. (I'm not mentioning the X-word here, except to note that the "M" stands for "markup".)
* Entity subtypes
EXT_JACK_ANALOG represents any analog audio and/or video connector. Properties for audio jacks would be jack type (TRS/RCA), color code, line level, position, etc.
EXT_JACK_DIGITAL represents a digital connector like S/PDIF (coax/ TOSLINK), ADAT, TDIF, or MADI.
EXT_JACK_BUS represents a bus like FireWire and comes from the USB audio spec. (I doubt that any devices with this entitiy will ever exist.)
EXT_INSTRUMENT represents something like an e-guitar, keyboard, or MIDI controller. (Instrument entities are typically audio sources and MIDI sources and sinks, but can also be audio sinks.)
EXT_SPEAKER also includes headphones; there might be made a case for having those as a separate subtype.
EXT_PLAYER represents a device like a CD/DVD/tape player. Recorders can also write to that device, so "player" might not be an ideal name.
EXT_BROADCAST represents devices like TV tuners, satellite receivers, cable tuners, or radios.
INT_SYNTHESIZER converts MIDI to audio.
INT_NOISE_SOURCE comes from the USB audio spec; this is not an attempt to describe the characteristics of consumer-grade devices :-) , but represents an internal noise source for level calibration or measurements.
INT_CONTROLS may have multiple independent controls (this is USB's Feature Unit); INT_EFFECT may have multiple controls that affect one single algorithm.
INT_CHANNEL_SPLIT/MERGE are needed for HDAudio devices, whose topology information has only stereo links.
* Entity specifications
While TYPE_DEVICE entities can be identified by their device node, other entities typcially have just a numeric ID. For that, it would be useful to make do without separate identification and let the driver choose the entity ID.
Signed-off-by: Clemens Ladisch clemens@ladisch.de
--- linux/include/linux/media.h +++ linux/include/linux/media.h @@ -46,16 +46,36 @@ struct media_device_info { #define MEDIA_ENTITY_TYPE_MASK 0x00ff0000 #define MEDIA_ENTITY_SUBTYPE_MASK 0x0000ffff
-#define MEDIA_ENTITY_TYPE_NODE (1 << MEDIA_ENTITY_TYPE_SHIFT) -#define MEDIA_ENTITY_TYPE_NODE_V4L (MEDIA_ENTITY_TYPE_NODE + 1) -#define MEDIA_ENTITY_TYPE_NODE_FB (MEDIA_ENTITY_TYPE_NODE + 2) -#define MEDIA_ENTITY_TYPE_NODE_ALSA (MEDIA_ENTITY_TYPE_NODE + 3) -#define MEDIA_ENTITY_TYPE_NODE_DVB (MEDIA_ENTITY_TYPE_NODE + 4) +#define MEDIA_ENTITY_TYPE_DEVICE (1 << MEDIA_ENTITY_TYPE_SHIFT) +#define MEDIA_ENTITY_TYPE_DEVICE_V4L (MEDIA_ENTITY_TYPE_DEVICE + 1) +#define MEDIA_ENTITY_TYPE_DEVICE_FB (MEDIA_ENTITY_TYPE_DEVICE + 2) +#define MEDIA_ENTITY_TYPE_DEVICE_DVB (MEDIA_ENTITY_TYPE_DEVICE + 3) +#define MEDIA_ENTITY_TYPE_DEVICE_ALSA_PCM (MEDIA_ENTITY_TYPE_DEVICE + 4) +#define MEDIA_ENTITY_TYPE_DEVICE_ALSA_MIDI (MEDIA_ENTITY_TYPE_DEVICE + 5)
-#define MEDIA_ENTITY_TYPE_SUBDEV (2 << MEDIA_ENTITY_TYPE_SHIFT) -#define MEDIA_ENTITY_TYPE_SUBDEV_SENSOR (MEDIA_ENTITY_TYPE_SUBDEV + 1) -#define MEDIA_ENTITY_TYPE_SUBDEV_FLASH (MEDIA_ENTITY_TYPE_SUBDEV + 2) -#define MEDIA_ENTITY_TYPE_SUBDEV_LENS (MEDIA_ENTITY_TYPE_SUBDEV + 3) +#define MEDIA_ENTITY_TYPE_EXT (2 << MEDIA_ENTITY_TYPE_SHIFT) +#define MEDIA_ENTITY_TYPE_EXT_SENSOR (MEDIA_ENTITY_TYPE_EXT + 1) +#define MEDIA_ENTITY_TYPE_EXT_FLASH (MEDIA_ENTITY_TYPE_EXT + 2) +#define MEDIA_ENTITY_TYPE_EXT_LENS (MEDIA_ENTITY_TYPE_EXT + 3) +#define MEDIA_ENTITY_TYPE_EXT_JACK_MIDI (MEDIA_ENTITY_TYPE_EXT + 4) +#define MEDIA_ENTITY_TYPE_EXT_JACK_ANALOG (MEDIA_ENTITY_TYPE_EXT + 5) +#define MEDIA_ENTITY_TYPE_EXT_JACK_DIGITAL (MEDIA_ENTITY_TYPE_EXT + 6) +#define MEDIA_ENTITY_TYPE_EXT_JACK_BUS (MEDIA_ENTITY_TYPE_EXT + 7) +#define MEDIA_ENTITY_TYPE_EXT_INSTRUMENT (MEDIA_ENTITY_TYPE_EXT + 8) +#define MEDIA_ENTITY_TYPE_EXT_SPEAKER (MEDIA_ENTITY_TYPE_EXT + 9) +#define MEDIA_ENTITY_TYPE_EXT_MICROPHONE (MEDIA_ENTITY_TYPE_EXT + 10) +#define MEDIA_ENTITY_TYPE_EXT_PLAYER (MEDIA_ENTITY_TYPE_EXT + 11) +#define MEDIA_ENTITY_TYPE_EXT_BROADCAST (MEDIA_ENTITY_TYPE_EXT + 12) + +#define MEDIA_ENTITY_TYPE_INT (3 << MEDIA_ENTITY_TYPE_SHIFT) +#define MEDIA_ENTITY_TYPE_INT_SYNTHESIZER (MEDIA_ENTITY_TYPE_INT + 1) +#define MEDIA_ENTITY_TYPE_INT_NOISE_SOURCE (MEDIA_ENTITY_TYPE_INT + 2) +#define MEDIA_ENTITY_TYPE_INT_MIXER (MEDIA_ENTITY_TYPE_INT + 3) +#define MEDIA_ENTITY_TYPE_INT_SELECTOR (MEDIA_ENTITY_TYPE_INT + 4) +#define MEDIA_ENTITY_TYPE_INT_CONTROLS (MEDIA_ENTITY_TYPE_INT + 5) +#define MEDIA_ENTITY_TYPE_INT_EFFECT (MEDIA_ENTITY_TYPE_INT + 6) +#define MEDIA_ENTITY_TYPE_INT_CHANNEL_SPLIT (MEDIA_ENTITY_TYPE_INT + 7) +#define MEDIA_ENTITY_TYPE_INT_CHANNEL_MERGE (MEDIA_ENTITY_TYPE_INT + 8)
#define MEDIA_ENTITY_FLAG_DEFAULT (1 << 0)
@@ -72,7 +92,7 @@ struct media_entity_desc { __u32 reserved[4];
union { - /* Node specifications */ + /* Device specifications */ struct { __u32 major; __u32 minor; @@ -81,11 +101,15 @@ struct media_entity_desc { __u32 major; __u32 minor; } fb; - int alsa; + struct { + __u32 card; + __u32 device; + __s32 subdevice; + } alsa; int dvb;
/* Sub-device specifications */ /* Nothing needed yet */ __u8 raw[184]; }; };
participants (3)
-
Clemens Ladisch -
Laurent Pinchart -
Mark Brown