Hi Mauro, Hans

On Sat, Feb 17, 2024 at 5:42 PM Mauro Carvalho Chehab mchehab@kernel.org wrote:

Em Thu, 18 Jan 2024 20:32:00 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

Audio signal processing has the requirement for memory to memory similar as Video.

This patch is to add this support in v4l2 framework, defined new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format for audio case usage.

The created audio device is named "/dev/v4l-audioX".

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../userspace-api/media/v4l/buffer.rst | 6 ++ .../media/v4l/dev-audio-mem2mem.rst | 71 +++++++++++++++++++ .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 ++ .../media/videodev2.h.rst.exceptions | 2 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 ++ drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +++ drivers/media/v4l2-core/v4l2-dev.c | 17 +++++ drivers/media/v4l2-core/v4l2-ioctl.c | 53 ++++++++++++++ include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 +++++++++ include/uapi/linux/videodev2.h | 17 +++++ 13 files changed, 222 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst

diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 52bbee81c080..a3754ca6f0d6 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -438,6 +438,12 @@ enum v4l2_buf_type * - ``V4L2_BUF_TYPE_META_OUTPUT`` - 14

...

   - Buffer for metadata output, see :ref:`metadata`.

• • • ``V4L2_BUF_TYPE_AUDIO_CAPTURE``

•  - 15
  

•  - Buffer for audio capture, see :ref:`audio`.
  

• • • ``V4L2_BUF_TYPE_AUDIO_OUTPUT``

•  - 16
  

Hmm... alsa APi define input/output as: enum { SNDRV_PCM_STREAM_PLAYBACK = 0, SNDRV_PCM_STREAM_CAPTURE, SNDRV_PCM_STREAM_LAST = SNDRV_PCM_STREAM_CAPTURE, };

I would use a namespace as close as possible to the ALSA API. Also, we're not talking about V4L2, but, instead audio. so, not sure if I like the prefix to start with V4L2_. Maybe ALSA_?

So, a better namespace would be:

    ${prefix}_BUF_TYPE_PCM_STREAM_PLAYBACK

and ${prefix}_BUF_TYPE_PCM_STREAM_CAPTURE

Ok, So can I use the name below? V4L2_BUF_TYPE_PCM_STREAM_PLAYBACK V4L2_BUF_TYPE_PCM_STREAM_CAPTURE

...

•  - Buffer for audio output, see :ref:`audio`.
  
  
  

.. _buffer-flags: diff --git a/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst new file mode 100644 index 000000000000..68faecfe3a02 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst @@ -0,0 +1,71 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later

+.. _audiomem2mem:

+******************************** +Audio Memory-To-Memory Interface +********************************

+An audio memory-to-memory device can compress, decompress, transform, or +otherwise convert audio data from one format into another format, in memory. +Such memory-to-memory devices set the ``V4L2_CAP_AUDIO_M2M`` capability. +Examples of memory-to-memory devices are audio codecs, audio preprocessing, +audio postprocessing.

+A memory-to-memory audio node supports both output (sending audio frames from +memory to the hardware) and capture (receiving the processed audio frames +from the hardware into memory) stream I/O. An application will have to +setup the stream I/O for both sides and finally call +:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` for both capture and output to +start the hardware.

+Memory-to-memory devices function as a shared resource: you can +open the audio node multiple times, each application setting up their +own properties that are local to the file handle, and each can use +it independently from the others. The driver will arbitrate access to +the hardware and reprogram it whenever another file handler gets access.

+Audio memory-to-memory devices are accessed through character device +special files named ``/dev/v4l-audio``

+Querying Capabilities +=====================

+Device nodes supporting the audio memory-to-memory interface set the +``V4L2_CAP_AUDIO_M2M`` flag in the ``device_caps`` field of the +:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP` +ioctl.

+Data Format Negotiation +=======================

+The audio device uses the :ref:`format` ioctls to select the capture format. +The audio buffer content format is bound to that selected format. In addition +to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be +supported as well.

+To use the :ref:`format` ioctls applications set the ``type`` field of the +:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` or to +``V4L2_BUF_TYPE_AUDIO_OUTPUT``. Both drivers and applications must set the +remainder of the :c:type:`v4l2_format` structure to 0.

+.. c:type:: v4l2_audio_format

+.. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|

+.. flat-table:: struct v4l2_audio_format

• :header-rows: 0
• :stub-columns: 0
• :widths: 1 1 2
• • • __u32

•  - ``pixelformat``
  

•  - The sample format, set by the application. see :ref:`pixfmt-audio`
  

pixelformat doesn't make any sense for audio: there are no pixels on a PCM stream. Please use call it, instead: `snd_pcm_format`, making it match the values for snd_pcm_format_t.

Yet, I would keep defining it as u32 (or u64?) instead of using a typedef int field there (snd_pcm_format_t), as the size of integer is different on 32 and 64 bit kernels.

...

• • • __u32

•  - ``channels``
  

•  - The channel number, set by the application. channel number range is
  

•    [1, 32].
  

Why not start on zero?

channels in audio are at least 1.

...

• • • __u32

•  - ``buffersize``
  

•  - Maximum buffer size in bytes required for data. The value is set by the
  

•    driver.
  

diff --git a/Documentation/userspace-api/media/v4l/devices.rst b/Documentation/userspace-api/media/v4l/devices.rst index 8bfbad65a9d4..758bd90f1c26 100644 --- a/Documentation/userspace-api/media/v4l/devices.rst +++ b/Documentation/userspace-api/media/v4l/devices.rst @@ -24,3 +24,4 @@ Interfaces dev-event dev-subdev dev-meta

• dev-audio-mem2mem

diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst index 000c154b0f98..42deb07f4ff4 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst @@ -96,6 +96,8 @@ the ``mbus_code`` field is handled differently: ``V4L2_BUF_TYPE_VIDEO_OVERLAY``, ``V4L2_BUF_TYPE_SDR_CAPTURE``, ``V4L2_BUF_TYPE_SDR_OUTPUT``,

• ``V4L2_BUF_TYPE_AUDIO_CAPTURE``,
  

• ``V4L2_BUF_TYPE_AUDIO_OUTPUT``,
  ``V4L2_BUF_TYPE_META_CAPTURE`` and
  ``V4L2_BUF_TYPE_META_OUTPUT``.
  See :c:type:`v4l2_buf_type`.
  

diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst index 675c385e5aca..528fd9df41aa 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst @@ -130,6 +130,10 @@ The format as returned by :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` must be identical - ``meta`` - Definition of a metadata format, see :ref:`meta-formats`, used by metadata capture devices.

• • • struct :c:type:`v4l2_audio_format`

•  - ``audio``
  

•  - Definition of a audio data format, see :ref:`audiomem2mem`, used by
  

•    audio memory-to-memory devices
  

  • • __u8
    • ``raw_data``\ [200]
    • Place holder for future extensions.

diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index da6d0b8e4c2c..e61152bb80d1 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -29,6 +29,8 @@ replace symbol V4L2_FIELD_SEQ_TB :c:type:`v4l2_field` replace symbol V4L2_FIELD_TOP :c:type:`v4l2_field`

# Documented enum v4l2_buf_type +replace symbol V4L2_BUF_TYPE_AUDIO_CAPTURE :c:type:`v4l2_buf_type` +replace symbol V4L2_BUF_TYPE_AUDIO_OUTPUT :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_META_CAPTURE :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_META_OUTPUT :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_SDR_CAPTURE :c:type:`v4l2_buf_type` diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index 54d572c3b515..16ab204d15b0 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -791,6 +791,10 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create) case V4L2_BUF_TYPE_META_OUTPUT: requested_sizes[0] = f->fmt.meta.buffersize; break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         requested_sizes[0] = f->fmt.audio.buffersize;
  

•         break;
  default:
          return -EINVAL;
  }
  

diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c index 8c07400bd280..5e94db8dfdae 100644 --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c @@ -101,6 +101,7 @@ struct v4l2_format32 { struct v4l2_sliced_vbi_format sliced; struct v4l2_sdr_format sdr; struct v4l2_meta_format meta;

•         struct v4l2_audio_format        audio;
          __u8    raw_data[200];        /* user-defined */
  } fmt;
  

}; @@ -166,6 +167,10 @@ static int get_v4l2_format32(struct v4l2_format *p64, case V4L2_BUF_TYPE_META_OUTPUT: return copy_from_user(&p64->fmt.meta, &p32->fmt.meta, sizeof(p64->fmt.meta)) ? -EFAULT : 0;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         return copy_from_user(&p64->fmt.audio, &p32->fmt.audio,
  

•                               sizeof(p64->fmt.audio)) ? -EFAULT : 0;
  default:
          return -EINVAL;
  }
  

@@ -216,6 +221,10 @@ static int put_v4l2_format32(struct v4l2_format *p64, case V4L2_BUF_TYPE_META_OUTPUT: return copy_to_user(&p32->fmt.meta, &p64->fmt.meta, sizeof(p64->fmt.meta)) ? -EFAULT : 0;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         return copy_to_user(&p32->fmt.audio, &p64->fmt.audio,
  

•                             sizeof(p64->fmt.audio)) ? -EFAULT : 0;
  default:
          return -EINVAL;
  }
  

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c index d13954bd31fd..bac008fcedc6 100644 --- a/drivers/media/v4l2-core/v4l2-dev.c +++ b/drivers/media/v4l2-core/v4l2-dev.c @@ -553,6 +553,7 @@ static void determine_valid_ioctls(struct video_device *vdev) bool is_tch = vdev->vfl_type == VFL_TYPE_TOUCH; bool is_meta = vdev->vfl_type == VFL_TYPE_VIDEO && (vdev->device_caps & meta_caps);

• bool is_audio = vdev->vfl_type == VFL_TYPE_AUDIO;
  bool is_rx = vdev->vfl_dir != VFL_DIR_TX;
  bool is_tx = vdev->vfl_dir != VFL_DIR_RX;
  bool is_io_mc = vdev->device_caps & V4L2_CAP_IO_MC;
  

@@ -666,6 +667,19 @@ static void determine_valid_ioctls(struct video_device *vdev) SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_meta_out); SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_meta_out); }

• if (is_audio && is_rx) {
  

•         /* audio capture specific ioctls */
  

•         SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_audio_cap);
  

•         SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_audio_cap);
  

•         SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_audio_cap);
  

•         SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_audio_cap);
  

• } else if (is_audio && is_tx) {
  

•         /* audio output specific ioctls */
  

•         SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_audio_out);
  

•         SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_audio_out);
  

•         SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_audio_out);
  

•         SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_audio_out);
  

• }
  if (is_vbi) {
          /* vbi specific ioctls */
          if ((is_rx && (ops->vidioc_g_fmt_vbi_cap ||
  

@@ -929,6 +943,9 @@ int __video_register_device(struct video_device *vdev, case VFL_TYPE_TOUCH: name_base = "v4l-touch"; break;

• case VFL_TYPE_AUDIO:
  

•         name_base = "v4l-audio";
  

•         break;
  default:
          pr_err("%s called with unknown type: %d\n",
                 __func__, type);
  

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 33076af4dfdb..e7be7c2f302d 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -188,6 +188,8 @@ const char *v4l2_type_names[] = { [V4L2_BUF_TYPE_SDR_OUTPUT] = "sdr-out", [V4L2_BUF_TYPE_META_CAPTURE] = "meta-cap", [V4L2_BUF_TYPE_META_OUTPUT] = "meta-out",

• [V4L2_BUF_TYPE_AUDIO_CAPTURE]      = "audio-cap",
  

• [V4L2_BUF_TYPE_AUDIO_OUTPUT]       = "audio-out",
  

}; EXPORT_SYMBOL(v4l2_type_names);

@@ -276,6 +278,7 @@ static void v4l_print_format(const void *arg, bool write_only) const struct v4l2_sliced_vbi_format *sliced; const struct v4l2_window *win; const struct v4l2_meta_format *meta;

• const struct v4l2_audio_format *audio;
  u32 pixelformat;
  u32 planes;
  unsigned i;
  

@@ -346,6 +349,13 @@ static void v4l_print_format(const void *arg, bool write_only) pr_cont(", dataformat=%p4cc, buffersize=%u\n", &pixelformat, meta->buffersize); break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         audio = &p->fmt.audio;
  

•         pixelformat = audio->audioformat;
  

•         pr_cont(", format=%p4cc, channels=%u, buffersize=%u\n",
  

•                 &pixelformat, audio->channels, audio->buffersize);
  

•         break;
  }
  

}

@@ -927,6 +937,7 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type) bool is_tch = vfd->vfl_type == VFL_TYPE_TOUCH; bool is_meta = vfd->vfl_type == VFL_TYPE_VIDEO && (vfd->device_caps & meta_caps);

• bool is_audio = vfd->vfl_type == VFL_TYPE_AUDIO;
  bool is_rx = vfd->vfl_dir != VFL_DIR_TX;
  bool is_tx = vfd->vfl_dir != VFL_DIR_RX;
  
  

@@ -992,6 +1003,14 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type) if (is_meta && is_tx && ops->vidioc_g_fmt_meta_out) return 0; break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

•         if (is_audio && is_rx && ops->vidioc_g_fmt_audio_cap)
  

•                 return 0;
  

•         break;
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         if (is_audio && is_tx && ops->vidioc_g_fmt_audio_out)
  

•                 return 0;
  

•         break;
  default:
          break;
  }
  

@@ -1597,6 +1616,16 @@ static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, break; ret = ops->vidioc_enum_fmt_meta_out(file, fh, arg); break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

•         if (unlikely(!ops->vidioc_enum_fmt_audio_cap))
  

•                 break;
  

•         ret = ops->vidioc_enum_fmt_audio_cap(file, fh, arg);
  

•         break;
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         if (unlikely(!ops->vidioc_enum_fmt_audio_out))
  

•                 break;
  

•         ret = ops->vidioc_enum_fmt_audio_out(file, fh, arg);
  

•         break;
  }
  if (ret == 0)
          v4l_fill_fmtdesc(p);
  

@@ -1673,6 +1702,10 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops, return ops->vidioc_g_fmt_meta_cap(file, fh, arg); case V4L2_BUF_TYPE_META_OUTPUT: return ops->vidioc_g_fmt_meta_out(file, fh, arg);

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

•         return ops->vidioc_g_fmt_audio_cap(file, fh, arg);
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         return ops->vidioc_g_fmt_audio_out(file, fh, arg);
  }
  return -EINVAL;
  

} @@ -1784,6 +1817,16 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops, break; memset_after(p, 0, fmt.meta); return ops->vidioc_s_fmt_meta_out(file, fh, arg);

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

•         if (unlikely(!ops->vidioc_s_fmt_audio_cap))
  

•                 break;
  

•         memset_after(p, 0, fmt.audio);
  

•         return ops->vidioc_s_fmt_audio_cap(file, fh, arg);
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         if (unlikely(!ops->vidioc_s_fmt_audio_out))
  

•                 break;
  

•         memset_after(p, 0, fmt.audio);
  

•         return ops->vidioc_s_fmt_audio_out(file, fh, arg);
  }
  return -EINVAL;
  

} @@ -1892,6 +1935,16 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops, break; memset_after(p, 0, fmt.meta); return ops->vidioc_try_fmt_meta_out(file, fh, arg);

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
  

•         if (unlikely(!ops->vidioc_try_fmt_audio_cap))
  

•                 break;
  

•         memset_after(p, 0, fmt.audio);
  

•         return ops->vidioc_try_fmt_audio_cap(file, fh, arg);
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:
  

•         if (unlikely(!ops->vidioc_try_fmt_audio_out))
  

•                 break;
  

•         memset_after(p, 0, fmt.audio);
  

•         return ops->vidioc_try_fmt_audio_out(file, fh, arg);
  }
  return -EINVAL;
  

} diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h index d82dfdbf6e58..82b63f82d43f 100644 --- a/include/media/v4l2-dev.h +++ b/include/media/v4l2-dev.h @@ -30,6 +30,7 @@

• @VFL_TYPE_SUBDEV: for V4L2 subdevices
• @VFL_TYPE_SDR: for Software Defined Radio tuners
• @VFL_TYPE_TOUCH: for touch sensors

• • @VFL_TYPE_AUDIO: for audio memory-to-memory devices
  • @VFL_TYPE_MAX: number of VFL types, must always be last in the enum
  */

enum vfl_devnode_type { @@ -39,6 +40,7 @@ enum vfl_devnode_type { VFL_TYPE_SUBDEV, VFL_TYPE_SDR, VFL_TYPE_TOUCH,

• VFL_TYPE_AUDIO,
  VFL_TYPE_MAX /* Shall be the last one */
  

};

diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h index edb733f21604..f840cf740ce1 100644 --- a/include/media/v4l2-ioctl.h +++ b/include/media/v4l2-ioctl.h @@ -45,6 +45,12 @@ struct v4l2_fh;

• @vidioc_enum_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic
• for metadata output

• • @vidioc_enum_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic
• • for audio capture
• • @vidioc_enum_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic
• • for audio output
  • @vidioc_g_fmt_vid_cap: pointer to the function that implements
  • :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
  • in single plane mode

@@ -79,6 +85,10 @@ struct v4l2_fh;

• :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
• @vidioc_g_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for metadata output

• • @vidioc_g_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for audio capture
• • @vidioc_g_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for audio output
  • @vidioc_s_fmt_vid_cap: pointer to the function that implements
  • :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
  • in single plane mode

@@ -113,6 +123,10 @@ struct v4l2_fh;

• :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
• @vidioc_s_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for metadata output

• • @vidioc_s_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for audio capture
• • @vidioc_s_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for audio output
  • @vidioc_try_fmt_vid_cap: pointer to the function that implements
  • :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
  • in single plane mode

@@ -149,6 +163,10 @@ struct v4l2_fh;

• :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
• @vidioc_try_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata output

• • @vidioc_try_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for audio capture
• • @vidioc_try_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for audio output
  • @vidioc_reqbufs: pointer to the function that implements
  • :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  • @vidioc_querybuf: pointer to the function that implements

@@ -315,6 +333,10 @@ struct v4l2_ioctl_ops { struct v4l2_fmtdesc *f); int (*vidioc_enum_fmt_meta_out)(struct file *file, void *fh, struct v4l2_fmtdesc *f);

• int (*vidioc_enum_fmt_audio_cap)(struct file *file, void *fh,
  

•                                  struct v4l2_fmtdesc *f);
  

• int (*vidioc_enum_fmt_audio_out)(struct file *file, void *fh,
  

•                                  struct v4l2_fmtdesc *f);
  
  /* VIDIOC_G_FMT handlers */
  int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
  

@@ -345,6 +367,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_g_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f);

• int (*vidioc_g_fmt_audio_cap)(struct file *file, void *fh,
  

•                               struct v4l2_format *f);
  

• int (*vidioc_g_fmt_audio_out)(struct file *file, void *fh,
  

•                               struct v4l2_format *f);
  
  /* VIDIOC_S_FMT handlers */
  int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
  

@@ -375,6 +401,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_s_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f);

• int (*vidioc_s_fmt_audio_cap)(struct file *file, void *fh,
  

•                               struct v4l2_format *f);
  

• int (*vidioc_s_fmt_audio_out)(struct file *file, void *fh,
  

•                               struct v4l2_format *f);
  
  /* VIDIOC_TRY_FMT handlers */
  int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
  

@@ -405,6 +435,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_try_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f);

• int (*vidioc_try_fmt_audio_cap)(struct file *file, void *fh,
  

•                                 struct v4l2_format *f);
  

• int (*vidioc_try_fmt_audio_out)(struct file *file, void *fh,
  

•                                 struct v4l2_format *f);
  
  /* Buffer handlers */
  int (*vidioc_reqbufs)(struct file *file, void *fh,
  

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 6cd65969c2b5..57c82eb158f1 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -153,6 +153,8 @@ enum v4l2_buf_type { V4L2_BUF_TYPE_SDR_OUTPUT = 12, V4L2_BUF_TYPE_META_CAPTURE = 13, V4L2_BUF_TYPE_META_OUTPUT = 14,

• V4L2_BUF_TYPE_AUDIO_CAPTURE        = 15,
  

• V4L2_BUF_TYPE_AUDIO_OUTPUT         = 16,
  /* Deprecated, do not use */
  V4L2_BUF_TYPE_PRIVATE              = 0x80,
  

}; @@ -169,6 +171,7 @@ enum v4l2_buf_type { || (type) == V4L2_BUF_TYPE_VBI_OUTPUT \ || (type) == V4L2_BUF_TYPE_SLICED_VBI_OUTPUT \ || (type) == V4L2_BUF_TYPE_SDR_OUTPUT \

•  || (type) == V4L2_BUF_TYPE_AUDIO_OUTPUT                \
   || (type) == V4L2_BUF_TYPE_META_OUTPUT)
  
  

#define V4L2_TYPE_IS_CAPTURE(type) (!V4L2_TYPE_IS_OUTPUT(type)) @@ -2423,6 +2426,18 @@ struct v4l2_meta_format { __u32 buffersize; } __attribute__ ((packed));

+/**

• • struct v4l2_audio_format - audio data format definition
• • @audioformat: little endian four character code (fourcc)

Don't use fourcc here. Instead, use the types defined at snd_pcm_format_t.

In the earlier version of this patch series, fourcc is not used in this struct v4l2_audio_format.

I use "__u32 audioformat", the value is from snd_pcm_format_t. Using "__u32" is in order to avoid including the include/uapi/sound/asound.h header.

And defined V4L2_AUDIO_FMT_LPCM for the v4l2_fmtdesc.pixelformat

#define V4L2_AUDIO_FMT_LPCM v4l2_fourcc('L', 'P', 'C', 'M') /* audio lpcm */

After discussing with Hans, I use the current way for v4l audio.

Hans: Not sure if you can accept going back to the previous solution? You have already done similar changes in v4l-utils.

Best regards Shengjiu Wang

...

• • @channels: channel numbers
• • @buffersize: maximum size in bytes required for data
• */

+struct v4l2_audio_format {

• __u32                           audioformat;
  

• __u32                           channels;
  

• __u32                           buffersize;
  

+} __attribute__ ((packed));

/**

• struct v4l2_format - stream data format
• @type: enum v4l2_buf_type; type of the data stream

@@ -2431,6 +2446,7 @@ struct v4l2_meta_format {

• @win: definition of an overlaid image
• @vbi: raw VBI capture or output parameters
• @sliced: sliced VBI capture or output parameters

• • @audio: definition of an audio format
  • @raw_data: placeholder for future extensions and custom formats
  • @fmt: union of @pix, @pix_mp, @win, @vbi, @sliced, @sdr, @meta

  •       and @raw_data
    

@@ -2445,6 +2461,7 @@ struct v4l2_format { struct v4l2_sliced_vbi_format sliced; /* V4L2_BUF_TYPE_SLICED_VBI_CAPTURE */ struct v4l2_sdr_format sdr; /* V4L2_BUF_TYPE_SDR_CAPTURE */ struct v4l2_meta_format meta; /* V4L2_BUF_TYPE_META_CAPTURE */

•         struct v4l2_audio_format        audio;   /* V4L2_BUF_TYPE_AUDIO_CAPTURE */
          __u8    raw_data[200];                   /* user-defined */
  } fmt;
  

};

Thanks, Mauro

On Wed, Feb 21, 2024 at 12:30 PM Tomasz Figa tfiga@chromium.org wrote:

On Sat, Feb 17, 2024 at 6:42 PM Mauro Carvalho Chehab mchehab@kernel.org wrote:

...

Em Thu, 18 Jan 2024 20:32:00 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

Audio signal processing has the requirement for memory to memory similar as Video.

This patch is to add this support in v4l2 framework, defined new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format for audio case usage.

The created audio device is named "/dev/v4l-audioX".

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../userspace-api/media/v4l/buffer.rst | 6 ++ .../media/v4l/dev-audio-mem2mem.rst | 71 +++++++++++++++++++ .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 ++ .../media/videodev2.h.rst.exceptions | 2 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 ++ drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +++ drivers/media/v4l2-core/v4l2-dev.c | 17 +++++ drivers/media/v4l2-core/v4l2-ioctl.c | 53 ++++++++++++++ include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 +++++++++ include/uapi/linux/videodev2.h | 17 +++++ 13 files changed, 222 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst

diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 52bbee81c080..a3754ca6f0d6 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -438,6 +438,12 @@ enum v4l2_buf_type * - ``V4L2_BUF_TYPE_META_OUTPUT`` - 14

...

   - Buffer for metadata output, see :ref:`metadata`.

• • • ``V4L2_BUF_TYPE_AUDIO_CAPTURE``

•  - 15
  

•  - Buffer for audio capture, see :ref:`audio`.
  

• • • ``V4L2_BUF_TYPE_AUDIO_OUTPUT``

•  - 16
  

Hmm... alsa APi define input/output as: enum { SNDRV_PCM_STREAM_PLAYBACK = 0, SNDRV_PCM_STREAM_CAPTURE, SNDRV_PCM_STREAM_LAST = SNDRV_PCM_STREAM_CAPTURE, };

I would use a namespace as close as possible to the ALSA API. Also, we're not talking about V4L2, but, instead audio. so, not sure if I like the prefix to start with V4L2_. Maybe ALSA_?

So, a better namespace would be:

    ${prefix}_BUF_TYPE_PCM_STREAM_PLAYBACK

and ${prefix}_BUF_TYPE_PCM_STREAM_CAPTURE

The API is still V4L2, and all the other non-video buf types also use the V4L2_ prefix, so perhaps that's good here as well?

Whether AUDIO or PCM_STREAM makes more sense goes outside of my expertise. Subjectively, a PCM stream sounds more specific than an audio stream. Do those buf types also support non-PCM audio streams?

Currently I use it for PCM, but I think it can also be used for non-PCM. So use the below name? V4L2_BUF_TYPE_AUDIO_CAPTURE V4L2_BUF_TYPE_AUDIO_PLAYBACK

...

...

•  - Buffer for audio output, see :ref:`audio`.
  
  
  

.. _buffer-flags: diff --git a/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst new file mode 100644 index 000000000000..68faecfe3a02 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst @@ -0,0 +1,71 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later

+.. _audiomem2mem:

+******************************** +Audio Memory-To-Memory Interface +********************************

+An audio memory-to-memory device can compress, decompress, transform, or +otherwise convert audio data from one format into another format, in memory. +Such memory-to-memory devices set the ``V4L2_CAP_AUDIO_M2M`` capability. +Examples of memory-to-memory devices are audio codecs, audio preprocessing, +audio postprocessing.

+A memory-to-memory audio node supports both output (sending audio frames from +memory to the hardware) and capture (receiving the processed audio frames +from the hardware into memory) stream I/O. An application will have to +setup the stream I/O for both sides and finally call +:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` for both capture and output to +start the hardware.

+Memory-to-memory devices function as a shared resource: you can +open the audio node multiple times, each application setting up their +own properties that are local to the file handle, and each can use +it independently from the others. The driver will arbitrate access to +the hardware and reprogram it whenever another file handler gets access.

+Audio memory-to-memory devices are accessed through character device +special files named ``/dev/v4l-audio``

+Querying Capabilities +=====================

+Device nodes supporting the audio memory-to-memory interface set the +``V4L2_CAP_AUDIO_M2M`` flag in the ``device_caps`` field of the +:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP` +ioctl.

+Data Format Negotiation +=======================

+The audio device uses the :ref:`format` ioctls to select the capture format. +The audio buffer content format is bound to that selected format. In addition +to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be +supported as well.

+To use the :ref:`format` ioctls applications set the ``type`` field of the +:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` or to +``V4L2_BUF_TYPE_AUDIO_OUTPUT``. Both drivers and applications must set the +remainder of the :c:type:`v4l2_format` structure to 0.

+.. c:type:: v4l2_audio_format

+.. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|

+.. flat-table:: struct v4l2_audio_format

• :header-rows: 0
• :stub-columns: 0
• :widths: 1 1 2
• • • __u32

•  - ``pixelformat``
  

•  - The sample format, set by the application. see :ref:`pixfmt-audio`
  

pixelformat doesn't make any sense for audio: there are no pixels on a PCM stream. Please use call it, instead: `snd_pcm_format`, making it match the values for snd_pcm_format_t.

+1

FWIW v4l2_meta_format uses the name "dataformat".

Actually, I just realized that the C code actually uses the name "audioformat". Tbh., after reading the kerneldoc comment, my subjective preference would be on "sample_format", since that's exactly what it is.

Ok, I will change it to sampleformat.

Best Regards Shengjiu Wang

...

Yet, I would keep defining it as u32 (or u64?) instead of using a typedef int field there (snd_pcm_format_t), as the size of integer is different on 32 and 64 bit kernels.

+1

Best regards, Tomasz

On 17/02/2024 10:42, Mauro Carvalho Chehab wrote:

Em Thu, 18 Jan 2024 20:32:00 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

Audio signal processing has the requirement for memory to memory similar as Video.

This patch is to add this support in v4l2 framework, defined new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format for audio case usage.

The created audio device is named "/dev/v4l-audioX".

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../userspace-api/media/v4l/buffer.rst | 6 ++ .../media/v4l/dev-audio-mem2mem.rst | 71 +++++++++++++++++++ .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 ++ .../media/videodev2.h.rst.exceptions | 2 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 ++ drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +++ drivers/media/v4l2-core/v4l2-dev.c | 17 +++++ drivers/media/v4l2-core/v4l2-ioctl.c | 53 ++++++++++++++ include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 +++++++++ include/uapi/linux/videodev2.h | 17 +++++ 13 files changed, 222 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst

diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 52bbee81c080..a3754ca6f0d6 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -438,6 +438,12 @@ enum v4l2_buf_type * - ``V4L2_BUF_TYPE_META_OUTPUT`` - 14

...

   - Buffer for metadata output, see :ref:`metadata`.

• • • ``V4L2_BUF_TYPE_AUDIO_CAPTURE``

•  - 15
  

•  - Buffer for audio capture, see :ref:`audio`.
  

• • • ``V4L2_BUF_TYPE_AUDIO_OUTPUT``

•  - 16
  

Hmm... alsa APi define input/output as: enum { SNDRV_PCM_STREAM_PLAYBACK = 0, SNDRV_PCM_STREAM_CAPTURE, SNDRV_PCM_STREAM_LAST = SNDRV_PCM_STREAM_CAPTURE, };

I would use a namespace as close as possible to the ALSA API. Also, we're not talking about V4L2, but, instead audio. so, not sure if I like the prefix to start with V4L2_. Maybe ALSA_?

So, a better namespace would be:

${prefix}_BUF_TYPE_PCM_STREAM_PLAYBACK and ${prefix}_BUF_TYPE_PCM_STREAM_CAPTURE

...

•  - Buffer for audio output, see :ref:`audio`.
  
  
  

.. _buffer-flags: diff --git a/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst new file mode 100644 index 000000000000..68faecfe3a02 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst @@ -0,0 +1,71 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later

+.. _audiomem2mem:

+******************************** +Audio Memory-To-Memory Interface +********************************

+An audio memory-to-memory device can compress, decompress, transform, or +otherwise convert audio data from one format into another format, in memory. +Such memory-to-memory devices set the ``V4L2_CAP_AUDIO_M2M`` capability. +Examples of memory-to-memory devices are audio codecs, audio preprocessing, +audio postprocessing.

+A memory-to-memory audio node supports both output (sending audio frames from +memory to the hardware) and capture (receiving the processed audio frames +from the hardware into memory) stream I/O. An application will have to +setup the stream I/O for both sides and finally call +:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` for both capture and output to +start the hardware.

+Memory-to-memory devices function as a shared resource: you can +open the audio node multiple times, each application setting up their +own properties that are local to the file handle, and each can use +it independently from the others. The driver will arbitrate access to +the hardware and reprogram it whenever another file handler gets access.

+Audio memory-to-memory devices are accessed through character device +special files named ``/dev/v4l-audio``

+Querying Capabilities +=====================

+Device nodes supporting the audio memory-to-memory interface set the +``V4L2_CAP_AUDIO_M2M`` flag in the ``device_caps`` field of the +:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP` +ioctl.

+Data Format Negotiation +=======================

+The audio device uses the :ref:`format` ioctls to select the capture format. +The audio buffer content format is bound to that selected format. In addition +to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be +supported as well.

+To use the :ref:`format` ioctls applications set the ``type`` field of the +:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` or to +``V4L2_BUF_TYPE_AUDIO_OUTPUT``. Both drivers and applications must set the +remainder of the :c:type:`v4l2_format` structure to 0.

+.. c:type:: v4l2_audio_format

+.. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|

+.. flat-table:: struct v4l2_audio_format

• :header-rows: 0
• :stub-columns: 0
• :widths: 1 1 2
• • • __u32

•  - ``pixelformat``
  

•  - The sample format, set by the application. see :ref:`pixfmt-audio`
  

pixelformat doesn't make any sense for audio: there are no pixels on a

Ah, that's a doc bug: this field is called audioformat in the struct, not pixelformat. I missed this one in my review.

Regards,

Hans

PCM stream. Please use call it, instead: `snd_pcm_format`, making it match the values for snd_pcm_format_t.

Yet, I would keep defining it as u32 (or u64?) instead of using a typedef int field there (snd_pcm_format_t), as the size of integer is different on 32 and 64 bit kernels.

...

• • • __u32

•  - ``channels``
  

•  - The channel number, set by the application. channel number range is
  

•    [1, 32].
  

Why not start on zero?

...

• • • __u32

•  - ``buffersize``
  

•  - Maximum buffer size in bytes required for data. The value is set by the
  

•    driver.
  

diff --git a/Documentation/userspace-api/media/v4l/devices.rst b/Documentation/userspace-api/media/v4l/devices.rst index 8bfbad65a9d4..758bd90f1c26 100644 --- a/Documentation/userspace-api/media/v4l/devices.rst +++ b/Documentation/userspace-api/media/v4l/devices.rst @@ -24,3 +24,4 @@ Interfaces dev-event dev-subdev dev-meta

• dev-audio-mem2mem

diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst index 000c154b0f98..42deb07f4ff4 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst @@ -96,6 +96,8 @@ the ``mbus_code`` field is handled differently: ``V4L2_BUF_TYPE_VIDEO_OVERLAY``, ``V4L2_BUF_TYPE_SDR_CAPTURE``, ``V4L2_BUF_TYPE_SDR_OUTPUT``,

• ``V4L2_BUF_TYPE_AUDIO_CAPTURE``,
• ``V4L2_BUF_TYPE_AUDIO_OUTPUT``, ``V4L2_BUF_TYPE_META_CAPTURE`` and ``V4L2_BUF_TYPE_META_OUTPUT``. See :c:type:`v4l2_buf_type`.

diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst index 675c385e5aca..528fd9df41aa 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst @@ -130,6 +130,10 @@ The format as returned by :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` must be identical - ``meta`` - Definition of a metadata format, see :ref:`meta-formats`, used by metadata capture devices.

• • • struct :c:type:`v4l2_audio_format`

•  - ``audio``
  

•  - Definition of a audio data format, see :ref:`audiomem2mem`, used by
  

•    audio memory-to-memory devices
  

  • • __u8
    • ``raw_data``\ [200]
    • Place holder for future extensions.

diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index da6d0b8e4c2c..e61152bb80d1 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -29,6 +29,8 @@ replace symbol V4L2_FIELD_SEQ_TB :c:type:`v4l2_field` replace symbol V4L2_FIELD_TOP :c:type:`v4l2_field`

# Documented enum v4l2_buf_type +replace symbol V4L2_BUF_TYPE_AUDIO_CAPTURE :c:type:`v4l2_buf_type` +replace symbol V4L2_BUF_TYPE_AUDIO_OUTPUT :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_META_CAPTURE :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_META_OUTPUT :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_SDR_CAPTURE :c:type:`v4l2_buf_type` diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index 54d572c3b515..16ab204d15b0 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -791,6 +791,10 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create) case V4L2_BUF_TYPE_META_OUTPUT: requested_sizes[0] = f->fmt.meta.buffersize; break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• requested_sizes[0] = f->fmt.audio.buffersize;
  

• break;
  

  default: return -EINVAL; }

diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c index 8c07400bd280..5e94db8dfdae 100644 --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c @@ -101,6 +101,7 @@ struct v4l2_format32 { struct v4l2_sliced_vbi_format sliced; struct v4l2_sdr_format sdr; struct v4l2_meta_format meta;

• struct v4l2_audio_format	audio;
  

  __u8 raw_data[200]; /* user-defined */ } fmt;

}; @@ -166,6 +167,10 @@ static int get_v4l2_format32(struct v4l2_format *p64, case V4L2_BUF_TYPE_META_OUTPUT: return copy_from_user(&p64->fmt.meta, &p32->fmt.meta, sizeof(p64->fmt.meta)) ? -EFAULT : 0;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• return copy_from_user(&p64->fmt.audio, &p32->fmt.audio,
  

• 		      sizeof(p64->fmt.audio)) ? -EFAULT : 0;
  

  default: return -EINVAL; }

@@ -216,6 +221,10 @@ static int put_v4l2_format32(struct v4l2_format *p64, case V4L2_BUF_TYPE_META_OUTPUT: return copy_to_user(&p32->fmt.meta, &p64->fmt.meta, sizeof(p64->fmt.meta)) ? -EFAULT : 0;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• return copy_to_user(&p32->fmt.audio, &p64->fmt.audio,
  

• 		    sizeof(p64->fmt.audio)) ? -EFAULT : 0;
  

  default: return -EINVAL; }

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c index d13954bd31fd..bac008fcedc6 100644 --- a/drivers/media/v4l2-core/v4l2-dev.c +++ b/drivers/media/v4l2-core/v4l2-dev.c @@ -553,6 +553,7 @@ static void determine_valid_ioctls(struct video_device *vdev) bool is_tch = vdev->vfl_type == VFL_TYPE_TOUCH; bool is_meta = vdev->vfl_type == VFL_TYPE_VIDEO && (vdev->device_caps & meta_caps);

• bool is_audio = vdev->vfl_type == VFL_TYPE_AUDIO; bool is_rx = vdev->vfl_dir != VFL_DIR_TX; bool is_tx = vdev->vfl_dir != VFL_DIR_RX; bool is_io_mc = vdev->device_caps & V4L2_CAP_IO_MC;

@@ -666,6 +667,19 @@ static void determine_valid_ioctls(struct video_device *vdev) SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_meta_out); SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_meta_out); }

• if (is_audio && is_rx) {

• /* audio capture specific ioctls */
  

• SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_audio_cap);
  

• SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_audio_cap);
  

• SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_audio_cap);
  

• SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_audio_cap);
  

• } else if (is_audio && is_tx) {

• /* audio output specific ioctls */
  

• SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_audio_out);
  

• SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_audio_out);
  

• SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_audio_out);
  

• SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_audio_out);
  

• } if (is_vbi) { /* vbi specific ioctls */ if ((is_rx && (ops->vidioc_g_fmt_vbi_cap ||

@@ -929,6 +943,9 @@ int __video_register_device(struct video_device *vdev, case VFL_TYPE_TOUCH: name_base = "v4l-touch"; break;

• case VFL_TYPE_AUDIO:

• name_base = "v4l-audio";
  

• break;
  

  default: pr_err("%s called with unknown type: %d\n", __func__, type);

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 33076af4dfdb..e7be7c2f302d 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -188,6 +188,8 @@ const char *v4l2_type_names[] = { [V4L2_BUF_TYPE_SDR_OUTPUT] = "sdr-out", [V4L2_BUF_TYPE_META_CAPTURE] = "meta-cap", [V4L2_BUF_TYPE_META_OUTPUT] = "meta-out",

• [V4L2_BUF_TYPE_AUDIO_CAPTURE] = "audio-cap",
• [V4L2_BUF_TYPE_AUDIO_OUTPUT] = "audio-out",

}; EXPORT_SYMBOL(v4l2_type_names);

@@ -276,6 +278,7 @@ static void v4l_print_format(const void *arg, bool write_only) const struct v4l2_sliced_vbi_format *sliced; const struct v4l2_window *win; const struct v4l2_meta_format *meta;

• const struct v4l2_audio_format *audio; u32 pixelformat; u32 planes; unsigned i;

@@ -346,6 +349,13 @@ static void v4l_print_format(const void *arg, bool write_only) pr_cont(", dataformat=%p4cc, buffersize=%u\n", &pixelformat, meta->buffersize); break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:
• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• audio = &p->fmt.audio;
  

• pixelformat = audio->audioformat;
  

• pr_cont(", format=%p4cc, channels=%u, buffersize=%u\n",
  

• 	&pixelformat, audio->channels, audio->buffersize);
  

• break;
  

  }

}

@@ -927,6 +937,7 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type) bool is_tch = vfd->vfl_type == VFL_TYPE_TOUCH; bool is_meta = vfd->vfl_type == VFL_TYPE_VIDEO && (vfd->device_caps & meta_caps);

• bool is_audio = vfd->vfl_type == VFL_TYPE_AUDIO; bool is_rx = vfd->vfl_dir != VFL_DIR_TX; bool is_tx = vfd->vfl_dir != VFL_DIR_RX;

@@ -992,6 +1003,14 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type) if (is_meta && is_tx && ops->vidioc_g_fmt_meta_out) return 0; break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:

• if (is_audio && is_rx && ops->vidioc_g_fmt_audio_cap)
  

• 	return 0;
  

• break;
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• if (is_audio && is_tx && ops->vidioc_g_fmt_audio_out)
  

• 	return 0;
  

• break;
  

  default: break; }

@@ -1597,6 +1616,16 @@ static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, break; ret = ops->vidioc_enum_fmt_meta_out(file, fh, arg); break;

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:

• if (unlikely(!ops->vidioc_enum_fmt_audio_cap))
  

• 	break;
  

• ret = ops->vidioc_enum_fmt_audio_cap(file, fh, arg);
  

• break;
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• if (unlikely(!ops->vidioc_enum_fmt_audio_out))
  

• 	break;
  

• ret = ops->vidioc_enum_fmt_audio_out(file, fh, arg);
  

• break;
  

  } if (ret == 0) v4l_fill_fmtdesc(p);

@@ -1673,6 +1702,10 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops, return ops->vidioc_g_fmt_meta_cap(file, fh, arg); case V4L2_BUF_TYPE_META_OUTPUT: return ops->vidioc_g_fmt_meta_out(file, fh, arg);

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:

• return ops->vidioc_g_fmt_audio_cap(file, fh, arg);
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• return ops->vidioc_g_fmt_audio_out(file, fh, arg);
  

  } return -EINVAL;

} @@ -1784,6 +1817,16 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops, break; memset_after(p, 0, fmt.meta); return ops->vidioc_s_fmt_meta_out(file, fh, arg);

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:

• if (unlikely(!ops->vidioc_s_fmt_audio_cap))
  

• 	break;
  

• memset_after(p, 0, fmt.audio);
  

• return ops->vidioc_s_fmt_audio_cap(file, fh, arg);
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• if (unlikely(!ops->vidioc_s_fmt_audio_out))
  

• 	break;
  

• memset_after(p, 0, fmt.audio);
  

• return ops->vidioc_s_fmt_audio_out(file, fh, arg);
  

  } return -EINVAL;

} @@ -1892,6 +1935,16 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops, break; memset_after(p, 0, fmt.meta); return ops->vidioc_try_fmt_meta_out(file, fh, arg);

• case V4L2_BUF_TYPE_AUDIO_CAPTURE:

• if (unlikely(!ops->vidioc_try_fmt_audio_cap))
  

• 	break;
  

• memset_after(p, 0, fmt.audio);
  

• return ops->vidioc_try_fmt_audio_cap(file, fh, arg);
  

• case V4L2_BUF_TYPE_AUDIO_OUTPUT:

• if (unlikely(!ops->vidioc_try_fmt_audio_out))
  

• 	break;
  

• memset_after(p, 0, fmt.audio);
  

• return ops->vidioc_try_fmt_audio_out(file, fh, arg);
  

  } return -EINVAL;

} diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h index d82dfdbf6e58..82b63f82d43f 100644 --- a/include/media/v4l2-dev.h +++ b/include/media/v4l2-dev.h @@ -30,6 +30,7 @@

• @VFL_TYPE_SUBDEV: for V4L2 subdevices
• @VFL_TYPE_SDR: for Software Defined Radio tuners
• @VFL_TYPE_TOUCH: for touch sensors

• • @VFL_TYPE_AUDIO: for audio memory-to-memory devices
  • @VFL_TYPE_MAX: number of VFL types, must always be last in the enum
  */

enum vfl_devnode_type { @@ -39,6 +40,7 @@ enum vfl_devnode_type { VFL_TYPE_SUBDEV, VFL_TYPE_SDR, VFL_TYPE_TOUCH,

• VFL_TYPE_AUDIO, VFL_TYPE_MAX /* Shall be the last one */

};

diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h index edb733f21604..f840cf740ce1 100644 --- a/include/media/v4l2-ioctl.h +++ b/include/media/v4l2-ioctl.h @@ -45,6 +45,12 @@ struct v4l2_fh;

• @vidioc_enum_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic
• for metadata output

• • @vidioc_enum_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic
• • for audio capture
• • @vidioc_enum_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic
• • for audio output
  • @vidioc_g_fmt_vid_cap: pointer to the function that implements
  • :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
  • in single plane mode

@@ -79,6 +85,10 @@ struct v4l2_fh;

• :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
• @vidioc_g_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for metadata output

• • @vidioc_g_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for audio capture
• • @vidioc_g_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for audio output
  • @vidioc_s_fmt_vid_cap: pointer to the function that implements
  • :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
  • in single plane mode

@@ -113,6 +123,10 @@ struct v4l2_fh;

• :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
• @vidioc_s_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for metadata output

• • @vidioc_s_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for audio capture
• • @vidioc_s_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for audio output
  • @vidioc_try_fmt_vid_cap: pointer to the function that implements
  • :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
  • in single plane mode

@@ -149,6 +163,10 @@ struct v4l2_fh;

• :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata capture
• @vidioc_try_fmt_meta_out: pointer to the function that implements
• :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata output

• • @vidioc_try_fmt_audio_cap: pointer to the function that implements
• • :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for audio capture
• • @vidioc_try_fmt_audio_out: pointer to the function that implements
• • :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for audio output
  • @vidioc_reqbufs: pointer to the function that implements
  • :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  • @vidioc_querybuf: pointer to the function that implements

@@ -315,6 +333,10 @@ struct v4l2_ioctl_ops { struct v4l2_fmtdesc *f); int (*vidioc_enum_fmt_meta_out)(struct file *file, void *fh, struct v4l2_fmtdesc *f);

• int (*vidioc_enum_fmt_audio_cap)(struct file *file, void *fh,

• 			 struct v4l2_fmtdesc *f);
  

• int (*vidioc_enum_fmt_audio_out)(struct file *file, void *fh,

• 			 struct v4l2_fmtdesc *f);
  
  

  /* VIDIOC_G_FMT handlers */ int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,

@@ -345,6 +367,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_g_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f);

• int (*vidioc_g_fmt_audio_cap)(struct file *file, void *fh,

• 		      struct v4l2_format *f);
  

• int (*vidioc_g_fmt_audio_out)(struct file *file, void *fh,

• 		      struct v4l2_format *f);
  
  

  /* VIDIOC_S_FMT handlers */ int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,

@@ -375,6 +401,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_s_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f);

• int (*vidioc_s_fmt_audio_cap)(struct file *file, void *fh,

• 		      struct v4l2_format *f);
  

• int (*vidioc_s_fmt_audio_out)(struct file *file, void *fh,

• 		      struct v4l2_format *f);
  
  

  /* VIDIOC_TRY_FMT handlers */ int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,

@@ -405,6 +435,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_try_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f);

• int (*vidioc_try_fmt_audio_cap)(struct file *file, void *fh,

• 			struct v4l2_format *f);
  

• int (*vidioc_try_fmt_audio_out)(struct file *file, void *fh,

• 			struct v4l2_format *f);
  
  

  /* Buffer handlers */ int (*vidioc_reqbufs)(struct file *file, void *fh,

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 6cd65969c2b5..57c82eb158f1 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -153,6 +153,8 @@ enum v4l2_buf_type { V4L2_BUF_TYPE_SDR_OUTPUT = 12, V4L2_BUF_TYPE_META_CAPTURE = 13, V4L2_BUF_TYPE_META_OUTPUT = 14,

• V4L2_BUF_TYPE_AUDIO_CAPTURE = 15,
• V4L2_BUF_TYPE_AUDIO_OUTPUT = 16, /* Deprecated, do not use */ V4L2_BUF_TYPE_PRIVATE = 0x80,

}; @@ -169,6 +171,7 @@ enum v4l2_buf_type { || (type) == V4L2_BUF_TYPE_VBI_OUTPUT \ || (type) == V4L2_BUF_TYPE_SLICED_VBI_OUTPUT \ || (type) == V4L2_BUF_TYPE_SDR_OUTPUT \

• || (type) == V4L2_BUF_TYPE_AUDIO_OUTPUT		\
  

  || (type) == V4L2_BUF_TYPE_META_OUTPUT)

#define V4L2_TYPE_IS_CAPTURE(type) (!V4L2_TYPE_IS_OUTPUT(type)) @@ -2423,6 +2426,18 @@ struct v4l2_meta_format { __u32 buffersize; } __attribute__ ((packed));

+/**

• • struct v4l2_audio_format - audio data format definition
• • @audioformat: little endian four character code (fourcc)

Don't use fourcc here. Instead, use the types defined at snd_pcm_format_t.

...

• • @channels: channel numbers
• • @buffersize: maximum size in bytes required for data
• */

+struct v4l2_audio_format {

• __u32 audioformat;
• __u32 channels;
• __u32 buffersize;

+} __attribute__ ((packed));

/**

• struct v4l2_format - stream data format
• @type: enum v4l2_buf_type; type of the data stream

@@ -2431,6 +2446,7 @@ struct v4l2_meta_format {

• @win: definition of an overlaid image
• @vbi: raw VBI capture or output parameters
• @sliced: sliced VBI capture or output parameters

• • @audio: definition of an audio format
  • @raw_data: placeholder for future extensions and custom formats
  • @fmt: union of @pix, @pix_mp, @win, @vbi, @sliced, @sdr, @meta

  • and @raw_data
    

@@ -2445,6 +2461,7 @@ struct v4l2_format { struct v4l2_sliced_vbi_format sliced; /* V4L2_BUF_TYPE_SLICED_VBI_CAPTURE */ struct v4l2_sdr_format sdr; /* V4L2_BUF_TYPE_SDR_CAPTURE */ struct v4l2_meta_format meta; /* V4L2_BUF_TYPE_META_CAPTURE */

• struct v4l2_audio_format	audio;   /* V4L2_BUF_TYPE_AUDIO_CAPTURE */
  

  __u8 raw_data[200]; /* user-defined */ } fmt;

};

Thanks, Mauro

Hi Mauro

On Sat, Feb 17, 2024 at 5:19 PM Mauro Carvalho Chehab mchehab@kernel.org wrote:

Em Thu, 18 Jan 2024 20:32:01 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

The audio sample format definition is from alsa, the header file is include/uapi/sound/asound.h, but don't include this header file directly, because in user space, there is another copy in alsa-lib. There will be conflict in userspace for include videodev2.h & asound.h and asoundlib.h

Here still use the fourcc format.

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../userspace-api/media/v4l/pixfmt-audio.rst | 87 +++++++++++++++++++ .../userspace-api/media/v4l/pixfmt.rst | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 13 +++ include/uapi/linux/videodev2.h | 23 +++++ 4 files changed, 124 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst

diff --git a/Documentation/userspace-api/media/v4l/pixfmt-audio.rst b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst new file mode 100644 index 000000000000..04b4a7fbd8f4 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst @@ -0,0 +1,87 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later

+.. _pixfmt-audio:

+************* +Audio Formats +*************

+These formats are used for :ref:`audiomem2mem` interface only.

+.. tabularcolumns:: |p{5.8cm}|p{1.2cm}|p{10.3cm}|

+.. cssclass:: longtable

+.. flat-table:: Audio Format

• :header-rows: 1
• :stub-columns: 0
• :widths: 3 1 4
• • • Identifier

•  - Code
  

•  - Details
  

• • .. _V4L2-AUDIO-FMT-S8:

•  - ``V4L2_AUDIO_FMT_S8``
  

•  - 'S8'
  

•  - Corresponds to SNDRV_PCM_FORMAT_S8 in ALSA
  

• • .. _V4L2-AUDIO-FMT-S16-LE:

Hmm... why can't we just use SNDRV_*_FORMAT_*? Those are already part of an uAPI header. No need to add any abstraction here and/or redefine what is there already at include/uapi/sound/asound.h.

Actually I try to avoid including the include/uapi/sound/asound.h. Because in user space, there is another copy in alsa-lib (asoundlib.h). There will be conflict in userspace when including videodev2.h and asoundlib.h.

And in the V4l framework, the fourcc type is commonly used in other cases (video, radio, touch, meta....), to avoid changing common code a lot, so I think using fourcc definition for audio may be simpler.

Best regards Shengjiu Wang

On 19/02/2024 13:56, Mauro Carvalho Chehab wrote:

Em Mon, 19 Feb 2024 12:05:02 +0800 Shengjiu Wang shengjiu.wang@gmail.com escreveu:

...

Hi Mauro

On Sat, Feb 17, 2024 at 5:19 PM Mauro Carvalho Chehab mchehab@kernel.org wrote:

...

Em Thu, 18 Jan 2024 20:32:01 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

The audio sample format definition is from alsa, the header file is include/uapi/sound/asound.h, but don't include this header file directly, because in user space, there is another copy in alsa-lib. There will be conflict in userspace for include videodev2.h & asound.h and asoundlib.h

Here still use the fourcc format.

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../userspace-api/media/v4l/pixfmt-audio.rst | 87 +++++++++++++++++++ .../userspace-api/media/v4l/pixfmt.rst | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 13 +++ include/uapi/linux/videodev2.h | 23 +++++ 4 files changed, 124 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst

diff --git a/Documentation/userspace-api/media/v4l/pixfmt-audio.rst b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst new file mode 100644 index 000000000000..04b4a7fbd8f4 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst @@ -0,0 +1,87 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later

+.. _pixfmt-audio:

+************* +Audio Formats +*************

+These formats are used for :ref:`audiomem2mem` interface only.

+.. tabularcolumns:: |p{5.8cm}|p{1.2cm}|p{10.3cm}|

+.. cssclass:: longtable

+.. flat-table:: Audio Format

• :header-rows: 1
• :stub-columns: 0
• :widths: 3 1 4
• • • Identifier

•  - Code
  

•  - Details
  

• • .. _V4L2-AUDIO-FMT-S8:

•  - ``V4L2_AUDIO_FMT_S8``
  

•  - 'S8'
  

•  - Corresponds to SNDRV_PCM_FORMAT_S8 in ALSA
  

• • .. _V4L2-AUDIO-FMT-S16-LE:

Hmm... why can't we just use SNDRV_*_FORMAT_*? Those are already part of an uAPI header. No need to add any abstraction here and/or redefine what is there already at include/uapi/sound/asound.h.

Actually I try to avoid including the include/uapi/sound/asound.h. Because in user space, there is another copy in alsa-lib (asoundlib.h). There will be conflict in userspace when including videodev2.h and asoundlib.h.

Well, alsasoundlib.h seems to be using the same definitions: https://github.com/michaelwu/alsa-lib/blob/master/include/pcm.h

So, I can't see what would be the actual issue, as both userspace library and ALSA internal headers use the same magic numbers.

You can still do things like:

#ifdef __KERNEL__ # include <sound/asound.h> #else # include <asoundlib.h> #endif

To avoid such kind of conflicts, if you need to have it included on some header file. Yet, I can't see why you would need that.

IMO, at uAPI headers, you just need to declare the uAPI audiofmt field to be either __u32 or __u64, pointing it to where this value comes from (on both userspace and Kernelspace. E. g.:

/**

• struct v4l2_audio_format - audio data format definition
• @audioformat:
• an integer number matching the fields inside
• enum snd_pcm_format_t (e. g. `SNDRV_PCM_FORMAT_*`), as defined
• in include/uapi/sound/asound.h and

•  https://www.alsa-project.org/alsa-doc/alsa-lib/group___p_c_m.html#gaa14b7f26877a812acbb39811364177f8.
  

• @channels: channel numbers
• @buffersize: maximum size in bytes required for data

*/ struct v4l2_audio_format { __u32 audioformat; __u32 channels; __u32 buffersize; } __attribute__ ((packed));

Then, at documentation you just need to point to where the possible values for SNDRV_PCM_FORMAT_ are defined. No need to document them one by one.

With such definition, you'll only need to include sound/asound.h within the kAPI scope.

...

And in the V4l framework, the fourcc type is commonly used in other cases (video, radio, touch, meta....), to avoid changing common code a lot, so I think using fourcc definition for audio may be simpler.

Those are real video streams (or a video-related streams, in the case of metadata) where fourcc is widely used. There, it makes sense. However, ALSA format definitions are already being used for a long time. There's no sense on trying to reinvent it - or having an abstract layer to convert from/to fourcc <==> enum snd_pcm_format_t. Just use what is there already.

The problem is that within V4L2 we use fourcc consistently to describe a format, including in VIDIOC_ENUM_FMT. And the expectation is that the fourcc can be printed to a human readable string (there is even a printk format for that these days).

But the pcm values are all small integers (and can even be 0!), and printing the fourcc will give garbage. It doesn't work well at all with the V4L2 API. But by having a straightforward conversion between the pcm identifier and a fourcc it was really easy to deal with this.

There might even be applications today that call VIDIOC_ENUM_FMT to see what is supported and fail if it is not a proper fourcc is returned.

It will certainly report nonsense in v4l_print_fmtdesc() (v4l2-ioctl.c).

One of the early versions of this patch series did precisely what you request, but it just doesn't work well within the V4L2 uAPI.

Regards,

Hans

Thanks, Mauro

On 22/02/2024 04:50, Shengjiu Wang wrote:

On Wed, Feb 21, 2024 at 7:10 PM Hans Verkuil hverkuil@xs4all.nl wrote:

...

On 19/02/2024 13:56, Mauro Carvalho Chehab wrote:

...

Em Mon, 19 Feb 2024 12:05:02 +0800 Shengjiu Wang shengjiu.wang@gmail.com escreveu:

...

Hi Mauro

On Sat, Feb 17, 2024 at 5:19 PM Mauro Carvalho Chehab mchehab@kernel.org wrote:

...

Em Thu, 18 Jan 2024 20:32:01 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

The audio sample format definition is from alsa, the header file is include/uapi/sound/asound.h, but don't include this header file directly, because in user space, there is another copy in alsa-lib. There will be conflict in userspace for include videodev2.h & asound.h and asoundlib.h

Here still use the fourcc format.

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../userspace-api/media/v4l/pixfmt-audio.rst | 87 +++++++++++++++++++ .../userspace-api/media/v4l/pixfmt.rst | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 13 +++ include/uapi/linux/videodev2.h | 23 +++++ 4 files changed, 124 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst

diff --git a/Documentation/userspace-api/media/v4l/pixfmt-audio.rst b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst new file mode 100644 index 000000000000..04b4a7fbd8f4 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst @@ -0,0 +1,87 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later

+.. _pixfmt-audio:

+************* +Audio Formats +*************

+These formats are used for :ref:`audiomem2mem` interface only.

+.. tabularcolumns:: |p{5.8cm}|p{1.2cm}|p{10.3cm}|

+.. cssclass:: longtable

+.. flat-table:: Audio Format

• :header-rows: 1
• :stub-columns: 0
• :widths: 3 1 4
• • • Identifier

•  - Code
  

•  - Details
  

• • .. _V4L2-AUDIO-FMT-S8:

•  - ``V4L2_AUDIO_FMT_S8``
  

•  - 'S8'
  

•  - Corresponds to SNDRV_PCM_FORMAT_S8 in ALSA
  

• • .. _V4L2-AUDIO-FMT-S16-LE:

Hmm... why can't we just use SNDRV_*_FORMAT_*? Those are already part of an uAPI header. No need to add any abstraction here and/or redefine what is there already at include/uapi/sound/asound.h.

Actually I try to avoid including the include/uapi/sound/asound.h. Because in user space, there is another copy in alsa-lib (asoundlib.h). There will be conflict in userspace when including videodev2.h and asoundlib.h.

Well, alsasoundlib.h seems to be using the same definitions: https://github.com/michaelwu/alsa-lib/blob/master/include/pcm.h

So, I can't see what would be the actual issue, as both userspace library and ALSA internal headers use the same magic numbers.

You can still do things like:

  #ifdef __KERNEL__
  #  include <sound/asound.h>
  #else
  #  include <asoundlib.h>
  #endif

To avoid such kind of conflicts, if you need to have it included on some header file. Yet, I can't see why you would need that.

IMO, at uAPI headers, you just need to declare the uAPI audiofmt field to be either __u32 or __u64, pointing it to where this value comes from (on both userspace and Kernelspace. E. g.:

/**

• struct v4l2_audio_format - audio data format definition
• @audioformat:
• an integer number matching the fields inside
• enum snd_pcm_format_t (e. g. `SNDRV_PCM_FORMAT_*`), as defined
• in include/uapi/sound/asound.h and

•  https://www.alsa-project.org/alsa-doc/alsa-lib/group___p_c_m.html#gaa14b7f26877a812acbb39811364177f8.
  

• @channels: channel numbers
• @buffersize: maximum size in bytes required for data

*/ struct v4l2_audio_format { __u32 audioformat; __u32 channels; __u32 buffersize; } __attribute__ ((packed));

Then, at documentation you just need to point to where the possible values for SNDRV_PCM_FORMAT_ are defined. No need to document them one by one.

With such definition, you'll only need to include sound/asound.h within the kAPI scope.

...

And in the V4l framework, the fourcc type is commonly used in other cases (video, radio, touch, meta....), to avoid changing common code a lot, so I think using fourcc definition for audio may be simpler.

Those are real video streams (or a video-related streams, in the case of metadata) where fourcc is widely used. There, it makes sense. However, ALSA format definitions are already being used for a long time. There's no sense on trying to reinvent it - or having an abstract layer to convert from/to fourcc <==> enum snd_pcm_format_t. Just use what is there already.

The problem is that within V4L2 we use fourcc consistently to describe a format, including in VIDIOC_ENUM_FMT. And the expectation is that the fourcc can be printed to a human readable string (there is even a printk format for that these days).

But the pcm values are all small integers (and can even be 0!), and printing the fourcc will give garbage. It doesn't work well at all with the V4L2 API. But by having a straightforward conversion between the pcm identifier and a fourcc it was really easy to deal with this.

There might even be applications today that call VIDIOC_ENUM_FMT to see what is supported and fail if it is not a proper fourcc is returned.

It will certainly report nonsense in v4l_print_fmtdesc() (v4l2-ioctl.c).

One of the early versions of this patch series did precisely what you request, but it just doesn't work well within the V4L2 uAPI.

Thanks all.

So can I reach the conclusion below?

1. Keep using the fourcc definition for v4l2_audio_format.audioformat.
2. Keep the name V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT.
3. Fix Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst to change 'pixelformat' to 'audioformat'.
4. Update Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst to add more description for clock drift.

Yes, let's go with that.

Regards,

Hans

Best Regards Shengjiu Wang

...

Regards,

    Hans

...

Thanks, Mauro

On Sat, Feb 17, 2024 at 5:57 PM Mauro Carvalho Chehab mchehab@kernel.org wrote:

Em Thu, 18 Jan 2024 20:32:03 +0800 Shengjiu Wang shengjiu.wang@nxp.com escreveu:

...

Add V4L2_CID_M2M_AUDIO_SOURCE_RATE and V4L2_CID_M2M_AUDIO_DEST_RATE new IDs for rate control.

Add V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET and V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET for clock drift.

Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com

.../media/v4l/ext-ctrls-audio-m2m.rst | 20 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 6 ++++++ include/uapi/linux/v4l2-controls.h | 5 +++++ 3 files changed, 31 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst index 82d2ecedbfee..de579ab8fb94 100644 --- a/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst @@ -19,3 +19,23 @@ Audio M2M Control IDs The Audio M2M class descriptor. Calling :ref:`VIDIOC_QUERYCTRL` for this control will return a description of this control class.

+.. _v4l2-audio-asrc:

+``V4L2_CID_M2M_AUDIO_SOURCE_RATE (integer menu)``

• Sets the audio source sample rate, unit is Hz

+``V4L2_CID_M2M_AUDIO_DEST_RATE (integer menu)``

• Sets the audio destination sample rate, unit is Hz

+``V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET (fixed point)``

• Sets the offset from the audio source sample rate, unit is Hz.
• The offset compensates for any clock drift. The actual source audio
• sample rate is the ideal source audio sample rate from
• ``V4L2_CID_M2M_AUDIO_SOURCE_RATE`` plus this fixed point offset.

+``V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET (fixed point)``

• Sets the offset from the audio destination sample rate, unit is Hz.
• The offset compensates for any clock drift. The actual destination audio
• sample rate is the ideal source audio sample rate from
• ``V4L2_CID_M2M_AUDIO_DEST_RATE`` plus this fixed point offset.

Hmm... first of all, controls on V4L2 API can either be get or set. So, starting the sentence with "Set" sounds an assumption that may be wrong.

Ok, will update the description.

Also, I would explain a little bit more about the frequency offset values, as clock drift adjustment on PCM streams is something that can be done using different approaches.

I'm assuming that what you wanted here is to use it to check if the video and audio clocks have some drift, and reducing or increasing the audio sample rate dynamically to ensure that such drift will stay constraint to a maximum allowed drift measured in mili or nano seconds. So, userspace would be expected to be monitoring such drift and increasing/decreasing the sample frequency as needed to maintain such constraint.

Is that the way such uAPI is expected to work?

Yes. Userspace should monitor the drift, get the offset based on the common sample rate (8k, 44.1k, 48k...) then send it to the driver. The offset is a fixed point. it is base on the patch: https://patchwork.kernel.org/project/linux-media/patch/cec82507-ced9-4e7d-80...

Best regards Shengjiu Wang

...

diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 2a85ea3dc92f..91e1f5348c23 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1245,6 +1245,8 @@ const char *v4l2_ctrl_get_name(u32 id)

  /* Audio M2M controls */
  case V4L2_CID_M2M_AUDIO_CLASS:  return "Audio M2M Controls";

• case V4L2_CID_M2M_AUDIO_SOURCE_RATE:    return "Audio Source Sample Rate";
  

• case V4L2_CID_M2M_AUDIO_DEST_RATE:      return "Audio Destination Sample Rate";
  default:
          return NULL;
  }
  

@@ -1606,6 +1608,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;

• case V4L2_CID_M2M_AUDIO_SOURCE_RATE:
  

• case V4L2_CID_M2M_AUDIO_DEST_RATE:
  

•         *type = V4L2_CTRL_TYPE_INTEGER_MENU;
  

•         break;
  default:
          *type = V4L2_CTRL_TYPE_INTEGER;
          break;
  

diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index a8b4b830c757..30129ccdc282 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -3495,6 +3495,11 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_M2M_AUDIO_CLASS_BASE (V4L2_CTRL_CLASS_M2M_AUDIO | 0x900) #define V4L2_CID_M2M_AUDIO_CLASS (V4L2_CTRL_CLASS_M2M_AUDIO | 1)

+#define V4L2_CID_M2M_AUDIO_SOURCE_RATE (V4L2_CID_M2M_AUDIO_CLASS_BASE + 0) +#define V4L2_CID_M2M_AUDIO_DEST_RATE (V4L2_CID_M2M_AUDIO_CLASS_BASE + 1) +#define V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET (V4L2_CID_M2M_AUDIO_CLASS_BASE + 2) +#define V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET (V4L2_CID_M2M_AUDIO_CLASS_BASE + 3)

/* MPEG-compression definitions kept for backwards compatibility */ #ifndef __KERNEL__ #define V4L2_CTRL_CLASS_MPEG V4L2_CTRL_CLASS_CODEC

Thanks, Mauro