[RFC PATCH v6 00/11] Add audio support in v4l2 framework
Audio signal processing also has the requirement for memory to memory similar as Video.
This asrc memory to memory (memory ->asrc->memory) case is a non real time use case.
User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case.
It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case.
Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver.
The main change is in the v4l2 side, A /dev/vl4-audioX will be created, user applications only use the ioctl of v4l2 framework.
Other change is to add memory to memory support for two kinds of i.MX ASRC module.
changes in v6: - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT - fix warning by kernel test rebot - remove some unused format V4L2_AUDIO_FMT_XX - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. - rename audm2m to viaudm2m.
changes in v5: - remove V4L2_AUDIO_FMT_LPCM - define audio pixel format like V4L2_AUDIO_FMT_S8... - remove rate and format in struct v4l2_audio_format. - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls - updata document accordingly.
changes in v4: - update document style - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
changes in v3: - Modify documents for adding audio m2m support - Add audio virtual m2m driver - Defined V4L2_AUDIO_FMT_LPCM format type for audio. - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. - with modification in v4l-utils, pass v4l2-compliance test.
changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder
Shengjiu Wang (11): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag media: v4l2: Add audio capture and output support media: uapi: define audio sample format fourcc type media: uapi: Add audio rate controls support media: imx-asrc: Add memory to memory driver media: viaudm2m: add virtual driver for audio memory to memory
.../userspace-api/media/v4l/buffer.rst | 6 + .../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/dev-audio-mem2mem.rst | 71 + .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 + .../userspace-api/media/v4l/pixfmt-audio.rst | 202 +++ .../userspace-api/media/v4l/pixfmt.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + .../media/v4l/vidioc-querycap.rst | 3 + .../media/v4l/vidioc-queryctrl.rst | 7 + .../media/videodev2.h.rst.exceptions | 4 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1248 +++++++++++++++++ drivers/media/test-drivers/Kconfig | 9 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/viaudm2m.c | 707 ++++++++++ drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 + drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 + drivers/media/v4l2-core/v4l2-dev.c | 17 + drivers/media/v4l2-core/v4l2-ioctl.c | 89 ++ include/media/v4l2-ctrls.h | 2 + include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 60 + include/uapi/linux/v4l2-controls.h | 13 + include/uapi/linux/videodev2.h | 69 + sound/soc/fsl/fsl_asrc.c | 144 ++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 212 +++ sound/soc/fsl/fsl_easrc.h | 6 +- 35 files changed, 2984 insertions(+), 3 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst create mode 100644 drivers/media/platform/nxp/imx-asrc.c create mode 100644 drivers/media/test-drivers/viaudm2m.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
ASRC can be used on memory to memory case, define several functions for m2m usage.
m2m_prepare: prepare for the start step m2m_start: the start step m2m_unprepare: unprepare for stop step, optional m2m_stop: stop step m2m_check_format: check format is supported or not m2m_calc_out_len: calculate output length according to input length m2m_get_maxburst: burst size for dma m2m_pair_suspend: suspend function of pair, optional. m2m_pair_resume: resume function of pair get_output_fifo_size: get remaining data size in FIFO
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- sound/soc/fsl/fsl_asrc.c | 126 ++++++++++++++++++++++++++++++++ sound/soc/fsl/fsl_asrc.h | 2 + sound/soc/fsl/fsl_asrc_common.h | 37 ++++++++++ 3 files changed, 165 insertions(+)
diff --git a/sound/soc/fsl/fsl_asrc.c b/sound/soc/fsl/fsl_asrc.c index b793263291dc..7d8643ee0ba0 100644 --- a/sound/soc/fsl/fsl_asrc.c +++ b/sound/soc/fsl/fsl_asrc.c @@ -1063,6 +1063,124 @@ static int fsl_asrc_get_fifo_addr(u8 dir, enum asrc_pair_index index) return REG_ASRDx(dir, index); }
+/* Get sample numbers in FIFO */ +static unsigned int fsl_asrc_get_output_fifo_size(struct fsl_asrc_pair *pair) +{ + struct fsl_asrc *asrc = pair->asrc; + enum asrc_pair_index index = pair->index; + u32 val; + + regmap_read(asrc->regmap, REG_ASRFST(index), &val); + + val &= ASRFSTi_OUTPUT_FIFO_MASK; + + return val >> ASRFSTi_OUTPUT_FIFO_SHIFT; +} + +static int fsl_asrc_m2m_prepare(struct fsl_asrc_pair *pair) +{ + struct fsl_asrc_pair_priv *pair_priv = pair->private; + struct fsl_asrc *asrc = pair->asrc; + struct device *dev = &asrc->pdev->dev; + struct asrc_config config; + int ret; + + /* fill config */ + config.pair = pair->index; + config.channel_num = pair->channels; + config.input_sample_rate = pair->rate[IN]; + config.output_sample_rate = pair->rate[OUT]; + config.input_format = pair->sample_format[IN]; + config.output_format = pair->sample_format[OUT]; + config.inclk = INCLK_NONE; + config.outclk = OUTCLK_ASRCK1_CLK; + + pair_priv->config = &config; + ret = fsl_asrc_config_pair(pair, true); + if (ret) { + dev_err(dev, "failed to config pair: %d\n", ret); + return ret; + } + + pair->first_convert = 1; + + return 0; +} + +static int fsl_asrc_m2m_start(struct fsl_asrc_pair *pair) +{ + if (pair->first_convert) { + fsl_asrc_start_pair(pair); + pair->first_convert = 0; + } + /* + * Clear DMA request during the stall state of ASRC: + * During STALL state, the remaining in input fifo would never be + * smaller than the input threshold while the output fifo would not + * be bigger than output one. Thus the DMA request would be cleared. + */ + fsl_asrc_set_watermarks(pair, ASRC_FIFO_THRESHOLD_MIN, + ASRC_FIFO_THRESHOLD_MAX); + + /* Update the real input threshold to raise DMA request */ + fsl_asrc_set_watermarks(pair, ASRC_M2M_INPUTFIFO_WML, + ASRC_M2M_OUTPUTFIFO_WML); + + return 0; +} + +static int fsl_asrc_m2m_stop(struct fsl_asrc_pair *pair) +{ + if (!pair->first_convert) { + fsl_asrc_stop_pair(pair); + pair->first_convert = 1; + } + + return 0; +} + +/* calculate capture data length according to output data length and sample rate */ +static int fsl_asrc_m2m_calc_out_len(struct fsl_asrc_pair *pair, int input_buffer_length) +{ + unsigned int in_width, out_width; + unsigned int channels = pair->channels; + unsigned int in_samples, out_samples; + unsigned int out_length; + + in_width = snd_pcm_format_physical_width(pair->sample_format[IN]) / 8; + out_width = snd_pcm_format_physical_width(pair->sample_format[OUT]) / 8; + + in_samples = input_buffer_length / in_width / channels; + out_samples = pair->rate[OUT] * in_samples / pair->rate[IN]; + out_length = (out_samples - ASRC_OUTPUT_LAST_SAMPLE) * out_width * channels; + + return out_length; +} + +static int fsl_asrc_m2m_get_maxburst(u8 dir, struct fsl_asrc_pair *pair) +{ + struct fsl_asrc *asrc = pair->asrc; + struct fsl_asrc_priv *asrc_priv = asrc->private; + int wml = (dir == IN) ? ASRC_M2M_INPUTFIFO_WML : ASRC_M2M_OUTPUTFIFO_WML; + + if (!asrc_priv->soc->use_edma) + return wml * pair->channels; + else + return 1; +} + +static int fsl_asrc_m2m_pair_resume(struct fsl_asrc_pair *pair) +{ + struct fsl_asrc *asrc = pair->asrc; + int i; + + for (i = 0; i < pair->channels * 4; i++) + regmap_write(asrc->regmap, REG_ASRDI(pair->index), 0); + + pair->first_convert = 1; + return 0; +} + static int fsl_asrc_runtime_resume(struct device *dev); static int fsl_asrc_runtime_suspend(struct device *dev);
@@ -1147,6 +1265,14 @@ static int fsl_asrc_probe(struct platform_device *pdev) asrc->get_fifo_addr = fsl_asrc_get_fifo_addr; asrc->pair_priv_size = sizeof(struct fsl_asrc_pair_priv);
+ asrc->m2m_prepare = fsl_asrc_m2m_prepare; + asrc->m2m_start = fsl_asrc_m2m_start; + asrc->m2m_stop = fsl_asrc_m2m_stop; + asrc->get_output_fifo_size = fsl_asrc_get_output_fifo_size; + asrc->m2m_calc_out_len = fsl_asrc_m2m_calc_out_len; + asrc->m2m_get_maxburst = fsl_asrc_m2m_get_maxburst; + asrc->m2m_pair_resume = fsl_asrc_m2m_pair_resume; + if (of_device_is_compatible(np, "fsl,imx35-asrc")) { asrc_priv->clk_map[IN] = input_clk_map_imx35; asrc_priv->clk_map[OUT] = output_clk_map_imx35; diff --git a/sound/soc/fsl/fsl_asrc.h b/sound/soc/fsl/fsl_asrc.h index 86d2422ad606..1c492eb237f5 100644 --- a/sound/soc/fsl/fsl_asrc.h +++ b/sound/soc/fsl/fsl_asrc.h @@ -12,6 +12,8 @@
#include "fsl_asrc_common.h"
+#define ASRC_M2M_INPUTFIFO_WML 0x4 +#define ASRC_M2M_OUTPUTFIFO_WML 0x2 #define ASRC_DMA_BUFFER_NUM 2 #define ASRC_INPUTFIFO_THRESHOLD 32 #define ASRC_OUTPUTFIFO_THRESHOLD 32 diff --git a/sound/soc/fsl/fsl_asrc_common.h b/sound/soc/fsl/fsl_asrc_common.h index 7e1c13ca37f1..3b53d366182f 100644 --- a/sound/soc/fsl/fsl_asrc_common.h +++ b/sound/soc/fsl/fsl_asrc_common.h @@ -34,6 +34,12 @@ enum asrc_pair_index { * @pos: hardware pointer position * @req_dma_chan: flag to release dev_to_dev chan * @private: pair private area + * @complete: dma task complete + * @sample_format: format of m2m + * @rate: rate of m2m + * @buf_len: buffer length of m2m + * @first_convert: start of conversion + * @req_pair: flag for request pair */ struct fsl_asrc_pair { struct fsl_asrc *asrc; @@ -49,6 +55,14 @@ struct fsl_asrc_pair { bool req_dma_chan;
void *private; + + /* used for m2m */ + struct completion complete[2]; + snd_pcm_format_t sample_format[2]; + unsigned int rate[2]; + unsigned int buf_len[2]; + unsigned int first_convert; + bool req_pair; };
/** @@ -72,6 +86,16 @@ struct fsl_asrc_pair { * @request_pair: function pointer * @release_pair: function pointer * @get_fifo_addr: function pointer + * @m2m_prepare: function pointer + * @m2m_start: function pointer + * @m2m_unprepare: function pointer + * @m2m_stop: function pointer + * @m2m_calc_out_len: function pointer + * @m2m_get_maxburst: function pointer + * @m2m_pair_suspend: function pointer + * @m2m_pair_resume: function pointer + * @m2m_set_ratio_mod: function pointer + * @get_output_fifo_size: function pointer * @pair_priv_size: size of pair private struct. * @private: private data structure */ @@ -97,6 +121,19 @@ struct fsl_asrc { int (*request_pair)(int channels, struct fsl_asrc_pair *pair); void (*release_pair)(struct fsl_asrc_pair *pair); int (*get_fifo_addr)(u8 dir, enum asrc_pair_index index); + + int (*m2m_prepare)(struct fsl_asrc_pair *pair); + int (*m2m_start)(struct fsl_asrc_pair *pair); + int (*m2m_unprepare)(struct fsl_asrc_pair *pair); + int (*m2m_stop)(struct fsl_asrc_pair *pair); + + int (*m2m_calc_out_len)(struct fsl_asrc_pair *pair, int input_buffer_length); + int (*m2m_get_maxburst)(u8 dir, struct fsl_asrc_pair *pair); + int (*m2m_pair_suspend)(struct fsl_asrc_pair *pair); + int (*m2m_pair_resume)(struct fsl_asrc_pair *pair); + int (*m2m_set_ratio_mod)(struct fsl_asrc_pair *pair, int val); + + unsigned int (*get_output_fifo_size)(struct fsl_asrc_pair *pair); size_t pair_priv_size;
void *private;
On Fri, Oct 13, 2023 at 04:30:55PM +0800, Shengjiu Wang wrote:
ASRC can be used on memory to memory case, define several functions for m2m usage.
Acked-by: Mark Brown broonie@kernel.org
ASRC can be used on memory to memory case, define several functions for m2m usage and export them as function pointer.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- sound/soc/fsl/fsl_easrc.c | 193 ++++++++++++++++++++++++++++++++++++++ sound/soc/fsl/fsl_easrc.h | 4 + 2 files changed, 197 insertions(+)
diff --git a/sound/soc/fsl/fsl_easrc.c b/sound/soc/fsl/fsl_easrc.c index ba62995c909a..dc603bb383e2 100644 --- a/sound/soc/fsl/fsl_easrc.c +++ b/sound/soc/fsl/fsl_easrc.c @@ -1861,6 +1861,190 @@ static int fsl_easrc_get_fifo_addr(u8 dir, enum asrc_pair_index index) return REG_EASRC_FIFO(dir, index); }
+/* Get sample numbers in FIFO */ +static unsigned int fsl_easrc_get_output_fifo_size(struct fsl_asrc_pair *pair) +{ + struct fsl_asrc *asrc = pair->asrc; + enum asrc_pair_index index = pair->index; + u32 val; + + regmap_read(asrc->regmap, REG_EASRC_SFS(index), &val); + val &= EASRC_SFS_NSGO_MASK; + + return val >> EASRC_SFS_NSGO_SHIFT; +} + +static int fsl_easrc_m2m_prepare(struct fsl_asrc_pair *pair) +{ + struct fsl_easrc_ctx_priv *ctx_priv = pair->private; + struct fsl_asrc *asrc = pair->asrc; + struct device *dev = &asrc->pdev->dev; + int ret; + + ctx_priv->in_params.sample_rate = pair->rate[IN]; + ctx_priv->in_params.sample_format = pair->sample_format[IN]; + ctx_priv->out_params.sample_rate = pair->rate[OUT]; + ctx_priv->out_params.sample_format = pair->sample_format[OUT]; + + ctx_priv->in_params.fifo_wtmk = FSL_EASRC_INPUTFIFO_WML; + ctx_priv->out_params.fifo_wtmk = FSL_EASRC_OUTPUTFIFO_WML; + /* Fill the right half of the re-sampler with zeros */ + ctx_priv->rs_init_mode = 0x2; + /* Zero fill the right half of the prefilter */ + ctx_priv->pf_init_mode = 0x2; + + ret = fsl_easrc_set_ctx_format(pair, + &ctx_priv->in_params.sample_format, + &ctx_priv->out_params.sample_format); + if (ret) { + dev_err(dev, "failed to set context format: %d\n", ret); + return ret; + } + + ret = fsl_easrc_config_context(asrc, pair->index); + if (ret) { + dev_err(dev, "failed to config context %d\n", ret); + return ret; + } + + ctx_priv->in_params.iterations = 1; + ctx_priv->in_params.group_len = pair->channels; + ctx_priv->in_params.access_len = pair->channels; + ctx_priv->out_params.iterations = 1; + ctx_priv->out_params.group_len = pair->channels; + ctx_priv->out_params.access_len = pair->channels; + + ret = fsl_easrc_set_ctx_organziation(pair); + if (ret) { + dev_err(dev, "failed to set fifo organization\n"); + return ret; + } + + /* The context start flag */ + pair->first_convert = 1; + return 0; +} + +static int fsl_easrc_m2m_start(struct fsl_asrc_pair *pair) +{ + /* start context once */ + if (pair->first_convert) { + fsl_easrc_start_context(pair); + pair->first_convert = 0; + } + + return 0; +} + +static int fsl_easrc_m2m_stop(struct fsl_asrc_pair *pair) +{ + /* Stop pair/context */ + if (!pair->first_convert) { + fsl_easrc_stop_context(pair); + pair->first_convert = 1; + } + + return 0; +} + +/* calculate capture data length according to output data length and sample rate */ +static int fsl_easrc_m2m_calc_out_len(struct fsl_asrc_pair *pair, int input_buffer_length) +{ + struct fsl_asrc *easrc = pair->asrc; + struct fsl_easrc_priv *easrc_priv = easrc->private; + struct fsl_easrc_ctx_priv *ctx_priv = pair->private; + unsigned int in_rate = ctx_priv->in_params.norm_rate; + unsigned int out_rate = ctx_priv->out_params.norm_rate; + unsigned int channels = pair->channels; + unsigned int in_samples, out_samples; + unsigned int in_width, out_width; + unsigned int out_length; + unsigned int frac_bits; + u64 val1, val2; + + switch (easrc_priv->rs_num_taps) { + case EASRC_RS_32_TAPS: + /* integer bits = 5; */ + frac_bits = 39; + break; + case EASRC_RS_64_TAPS: + /* integer bits = 6; */ + frac_bits = 38; + break; + case EASRC_RS_128_TAPS: + /* integer bits = 7; */ + frac_bits = 37; + break; + default: + return -EINVAL; + } + + val1 = (u64)in_rate << frac_bits; + do_div(val1, out_rate); + val1 = val1 + (ctx_priv->ratio_mod << (frac_bits - 31)); + + in_width = snd_pcm_format_physical_width(ctx_priv->in_params.sample_format) / 8; + out_width = snd_pcm_format_physical_width(ctx_priv->out_params.sample_format) / 8; + + ctx_priv->in_filled_len += input_buffer_length; + if (ctx_priv->in_filled_len <= ctx_priv->in_filled_sample * in_width * channels) { + out_length = 0; + } else { + in_samples = ctx_priv->in_filled_len / (in_width * channels) - + ctx_priv->in_filled_sample; + + /* right shift 12 bit to make ratio in 32bit space */ + val2 = (u64)in_samples << (frac_bits - 12); + val1 = val1 >> 12; + do_div(val2, val1); + out_samples = val2; + + out_length = out_samples * out_width * channels; + ctx_priv->in_filled_len = ctx_priv->in_filled_sample * in_width * channels; + } + + return out_length; +} + +static int fsl_easrc_m2m_get_maxburst(u8 dir, struct fsl_asrc_pair *pair) +{ + struct fsl_easrc_ctx_priv *ctx_priv = pair->private; + + if (dir == IN) + return ctx_priv->in_params.fifo_wtmk * pair->channels; + else + return ctx_priv->out_params.fifo_wtmk * pair->channels; +} + +static int fsl_easrc_m2m_pair_suspend(struct fsl_asrc_pair *pair) +{ + fsl_easrc_stop_context(pair); + + return 0; +} + +static int fsl_easrc_m2m_pair_resume(struct fsl_asrc_pair *pair) +{ + struct fsl_easrc_ctx_priv *ctx_priv = pair->private; + + pair->first_convert = 1; + ctx_priv->in_filled_len = 0; + + return 0; +} + +/* val is Q31 */ +static int fsl_easrc_m2m_set_ratio_mod(struct fsl_asrc_pair *pair, int val) +{ + struct fsl_easrc_ctx_priv *ctx_priv = pair->private; + struct fsl_asrc *easrc = pair->asrc; + + ctx_priv->ratio_mod += val; + regmap_write(easrc->regmap, REG_EASRC_RUC(pair->index), EASRC_RSUC_RS_RM(val)); + + return 0; +} + static const struct of_device_id fsl_easrc_dt_ids[] = { { .compatible = "fsl,imx8mn-easrc",}, {} @@ -1926,6 +2110,15 @@ static int fsl_easrc_probe(struct platform_device *pdev) easrc->release_pair = fsl_easrc_release_context; easrc->get_fifo_addr = fsl_easrc_get_fifo_addr; easrc->pair_priv_size = sizeof(struct fsl_easrc_ctx_priv); + easrc->m2m_prepare = fsl_easrc_m2m_prepare; + easrc->m2m_start = fsl_easrc_m2m_start; + easrc->m2m_stop = fsl_easrc_m2m_stop; + easrc->get_output_fifo_size = fsl_easrc_get_output_fifo_size; + easrc->m2m_calc_out_len = fsl_easrc_m2m_calc_out_len; + easrc->m2m_get_maxburst = fsl_easrc_m2m_get_maxburst; + easrc->m2m_pair_suspend = fsl_easrc_m2m_pair_suspend; + easrc->m2m_pair_resume = fsl_easrc_m2m_pair_resume; + easrc->m2m_set_ratio_mod = fsl_easrc_m2m_set_ratio_mod;
easrc_priv->rs_num_taps = EASRC_RS_32_TAPS; easrc_priv->const_coeff = 0x3FF0000000000000; diff --git a/sound/soc/fsl/fsl_easrc.h b/sound/soc/fsl/fsl_easrc.h index 7c70dac52713..c9f770862662 100644 --- a/sound/soc/fsl/fsl_easrc.h +++ b/sound/soc/fsl/fsl_easrc.h @@ -601,6 +601,8 @@ struct fsl_easrc_slot { * @out_missed_sample: sample missed in output * @st1_addexp: exponent added for stage1 * @st2_addexp: exponent added for stage2 + * @ratio_mod: update ratio + * @in_filled_len: input filled length */ struct fsl_easrc_ctx_priv { struct fsl_easrc_io_params in_params; @@ -618,6 +620,8 @@ struct fsl_easrc_ctx_priv { int out_missed_sample; int st1_addexp; int st2_addexp; + int ratio_mod; + unsigned int in_filled_len; };
/**
On Fri, Oct 13, 2023 at 04:30:56PM +0800, Shengjiu Wang wrote:
ASRC can be used on memory to memory case, define several functions for m2m usage and export them as function pointer.
Acked-by: Mark Brown broonie@kernel.org
Move fsl_asrc_common.h to include/sound that it can be included from other drivers.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- {sound/soc/fsl => include/sound}/fsl_asrc_common.h | 0 sound/soc/fsl/fsl_asrc.h | 2 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.h | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-) rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (100%)
diff --git a/sound/soc/fsl/fsl_asrc_common.h b/include/sound/fsl_asrc_common.h similarity index 100% rename from sound/soc/fsl/fsl_asrc_common.h rename to include/sound/fsl_asrc_common.h diff --git a/sound/soc/fsl/fsl_asrc.h b/sound/soc/fsl/fsl_asrc.h index 1c492eb237f5..66544624de7b 100644 --- a/sound/soc/fsl/fsl_asrc.h +++ b/sound/soc/fsl/fsl_asrc.h @@ -10,7 +10,7 @@ #ifndef _FSL_ASRC_H #define _FSL_ASRC_H
-#include "fsl_asrc_common.h" +#include <sound/fsl_asrc_common.h>
#define ASRC_M2M_INPUTFIFO_WML 0x4 #define ASRC_M2M_OUTPUTFIFO_WML 0x2 diff --git a/sound/soc/fsl/fsl_asrc_dma.c b/sound/soc/fsl/fsl_asrc_dma.c index 05a7d1588d20..b034fee3f1f4 100644 --- a/sound/soc/fsl/fsl_asrc_dma.c +++ b/sound/soc/fsl/fsl_asrc_dma.c @@ -12,7 +12,7 @@ #include <sound/dmaengine_pcm.h> #include <sound/pcm_params.h>
-#include "fsl_asrc_common.h" +#include <sound/fsl_asrc_common.h>
#define FSL_ASRC_DMABUF_SIZE (256 * 1024)
diff --git a/sound/soc/fsl/fsl_easrc.h b/sound/soc/fsl/fsl_easrc.h index c9f770862662..a24e540876a4 100644 --- a/sound/soc/fsl/fsl_easrc.h +++ b/sound/soc/fsl/fsl_easrc.h @@ -9,7 +9,7 @@ #include <sound/asound.h> #include <linux/dma/imx-dma.h>
-#include "fsl_asrc_common.h" +#include <sound/fsl_asrc_common.h>
/* EASRC Register Map */
On Fri, Oct 13, 2023 at 04:30:57PM +0800, Shengjiu Wang wrote:
Move fsl_asrc_common.h to include/sound that it can be included from other drivers.
Acked-by: Mark Brown broonie@kernel.org
Register m2m platform device, that user can use M2M feature.
Defined platform data structure and platform driver name.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- include/sound/fsl_asrc_common.h | 23 +++++++++++++++++++++++ sound/soc/fsl/fsl_asrc.c | 18 ++++++++++++++++++ 2 files changed, 41 insertions(+)
diff --git a/include/sound/fsl_asrc_common.h b/include/sound/fsl_asrc_common.h index 3b53d366182f..c709b8906929 100644 --- a/include/sound/fsl_asrc_common.h +++ b/include/sound/fsl_asrc_common.h @@ -71,6 +71,7 @@ struct fsl_asrc_pair { * @dma_params_rx: DMA parameters for receive channel * @dma_params_tx: DMA parameters for transmit channel * @pdev: platform device pointer + * @m2m_pdev: m2m platform device pointer * @regmap: regmap handler * @paddr: physical address to the base address of registers * @mem_clk: clock source to access register @@ -103,6 +104,7 @@ struct fsl_asrc { struct snd_dmaengine_dai_dma_data dma_params_rx; struct snd_dmaengine_dai_dma_data dma_params_tx; struct platform_device *pdev; + struct platform_device *m2m_pdev; struct regmap *regmap; unsigned long paddr; struct clk *mem_clk; @@ -139,6 +141,27 @@ struct fsl_asrc { void *private; };
+/** + * struct fsl_asrc_m2m_pdata - platform data + * @asrc: pointer to struct fsl_asrc + * @fmt_in: input sample format + * @fmt_out: output sample format + * @chan_min: minimum channel number + * @chan_max: maximum channel number + * @rate_min: minimum rate + * @rate_max: maximum rete + */ +struct fsl_asrc_m2m_pdata { + struct fsl_asrc *asrc; + u64 fmt_in; + u64 fmt_out; + int chan_min; + int chan_max; + int rate_min; + int rate_max; +}; + +#define M2M_DRV_NAME "fsl_asrc_m2m" #define DRV_NAME "fsl-asrc-dai" extern struct snd_soc_component_driver fsl_asrc_component;
diff --git a/sound/soc/fsl/fsl_asrc.c b/sound/soc/fsl/fsl_asrc.c index 7d8643ee0ba0..5ecb5d869607 100644 --- a/sound/soc/fsl/fsl_asrc.c +++ b/sound/soc/fsl/fsl_asrc.c @@ -1187,6 +1187,7 @@ static int fsl_asrc_runtime_suspend(struct device *dev); static int fsl_asrc_probe(struct platform_device *pdev) { struct device_node *np = pdev->dev.of_node; + struct fsl_asrc_m2m_pdata m2m_pdata; struct fsl_asrc_priv *asrc_priv; struct fsl_asrc *asrc; struct resource *res; @@ -1368,6 +1369,18 @@ static int fsl_asrc_probe(struct platform_device *pdev) goto err_pm_get_sync; }
+ m2m_pdata.asrc = asrc; + m2m_pdata.fmt_in = FSL_ASRC_FORMATS; + m2m_pdata.fmt_out = FSL_ASRC_FORMATS | SNDRV_PCM_FMTBIT_S8; + m2m_pdata.rate_min = 5512; + m2m_pdata.rate_max = 192000; + m2m_pdata.chan_min = 1; + m2m_pdata.chan_max = 10; + asrc->m2m_pdev = platform_device_register_data(&pdev->dev, + M2M_DRV_NAME, + PLATFORM_DEVID_AUTO, + &m2m_pdata, + sizeof(m2m_pdata)); return 0;
err_pm_get_sync: @@ -1380,6 +1393,11 @@ static int fsl_asrc_probe(struct platform_device *pdev)
static void fsl_asrc_remove(struct platform_device *pdev) { + struct fsl_asrc *asrc = dev_get_drvdata(&pdev->dev); + + if (asrc->m2m_pdev && !IS_ERR(asrc->m2m_pdev)) + platform_device_unregister(asrc->m2m_pdev); + pm_runtime_disable(&pdev->dev); if (!pm_runtime_status_suspended(&pdev->dev)) fsl_asrc_runtime_suspend(&pdev->dev);
On Fri, Oct 13, 2023 at 04:30:58PM +0800, Shengjiu Wang wrote:
Register m2m platform device, that user can use M2M feature.
Defined platform data structure and platform driver name.
Acked-by: Mark Brown broonie@kernel.org
Register m2m platform device,that user can use M2M feature.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- sound/soc/fsl/fsl_easrc.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
diff --git a/sound/soc/fsl/fsl_easrc.c b/sound/soc/fsl/fsl_easrc.c index dc603bb383e2..8ba6f045f425 100644 --- a/sound/soc/fsl/fsl_easrc.c +++ b/sound/soc/fsl/fsl_easrc.c @@ -2054,6 +2054,7 @@ MODULE_DEVICE_TABLE(of, fsl_easrc_dt_ids); static int fsl_easrc_probe(struct platform_device *pdev) { struct fsl_easrc_priv *easrc_priv; + struct fsl_asrc_m2m_pdata m2m_pdata; struct device *dev = &pdev->dev; struct fsl_asrc *easrc; struct resource *res; @@ -2169,11 +2170,29 @@ static int fsl_easrc_probe(struct platform_device *pdev) return ret; }
+ m2m_pdata.asrc = easrc; + m2m_pdata.fmt_in = FSL_EASRC_FORMATS; + m2m_pdata.fmt_out = FSL_EASRC_FORMATS | SNDRV_PCM_FMTBIT_IEC958_SUBFRAME_LE; + m2m_pdata.rate_min = 8000; + m2m_pdata.rate_max = 768000; + m2m_pdata.chan_min = 1; + m2m_pdata.chan_max = 32; + easrc->m2m_pdev = platform_device_register_data(&pdev->dev, + M2M_DRV_NAME, + PLATFORM_DEVID_AUTO, + &m2m_pdata, + sizeof(m2m_pdata)); + return 0; }
static void fsl_easrc_remove(struct platform_device *pdev) { + struct fsl_asrc *easrc = dev_get_drvdata(&pdev->dev); + + if (easrc->m2m_pdev && !IS_ERR(easrc->m2m_pdev)) + platform_device_unregister(easrc->m2m_pdev); + pm_runtime_disable(&pdev->dev); }
On Fri, Oct 13, 2023 at 04:30:59PM +0800, Shengjiu Wang wrote:
Register m2m platform device,that user can use M2M feature.
Acked-by: Mark Brown broonie@kernel.org
V4L2_CAP_AUDIO_M2M is similar to V4L2_CAP_VIDEO_M2M flag.
It is used for audio memory to memory case.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- Documentation/userspace-api/media/v4l/vidioc-querycap.rst | 3 +++ Documentation/userspace-api/media/videodev2.h.rst.exceptions | 1 + include/uapi/linux/videodev2.h | 1 + 3 files changed, 5 insertions(+)
diff --git a/Documentation/userspace-api/media/v4l/vidioc-querycap.rst b/Documentation/userspace-api/media/v4l/vidioc-querycap.rst index 6c57b8428356..0b3cefefc86b 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-querycap.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-querycap.rst @@ -259,6 +259,9 @@ specification the ioctl returns an ``EINVAL`` error code. video topology configuration, including which I/O entity is routed to the input/output, is configured by userspace via the Media Controller. See :ref:`media_controller`. + * - ``V4L2_CAP_AUDIO_M2M`` + - 0x40000000 + - The device supports the audio Memory-To-Memory interface. * - ``V4L2_CAP_DEVICE_CAPS`` - 0x80000000 - The driver fills the ``device_caps`` field. This capability can diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index 3e58aac4ef0b..da6d0b8e4c2c 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -197,6 +197,7 @@ replace define V4L2_CAP_META_OUTPUT device-capabilities replace define V4L2_CAP_DEVICE_CAPS device-capabilities replace define V4L2_CAP_TOUCH device-capabilities replace define V4L2_CAP_IO_MC device-capabilities +replace define V4L2_CAP_AUDIO_M2M device-capabilities
# V4L2 pix flags replace define V4L2_PIX_FMT_PRIV_MAGIC :c:type:`v4l2_pix_format` diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 78260e5d9985..3decf7d73870 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -508,6 +508,7 @@ struct v4l2_capability { #define V4L2_CAP_TOUCH 0x10000000 /* Is a touch device */
#define V4L2_CAP_IO_MC 0x20000000 /* Is input/output controlled by the media controller */ +#define V4L2_CAP_AUDIO_M2M 0x40000000 /* audio memory to memory */
#define V4L2_CAP_DEVICE_CAPS 0x80000000 /* sets device capabilities field */
Audio signal processing has the requirement for memory to memory similar as Video.
This patch is to add this support in v4l2 framework, defined new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format for audio case usage.
The created audio device is named "/dev/v4l-audioX".
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- .../userspace-api/media/v4l/buffer.rst | 6 ++ .../media/v4l/dev-audio-mem2mem.rst | 71 +++++++++++++++++++ .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 ++ .../media/videodev2.h.rst.exceptions | 2 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 ++ drivers/media/v4l2-core/v4l2-dev.c | 17 +++++ drivers/media/v4l2-core/v4l2-ioctl.c | 53 ++++++++++++++ include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 +++++++++ include/uapi/linux/videodev2.h | 17 +++++ 12 files changed, 213 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 04dec3e570ed..80cf2cb20dfe 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -438,6 +438,12 @@ enum v4l2_buf_type * - ``V4L2_BUF_TYPE_META_OUTPUT`` - 14 - Buffer for metadata output, see :ref:`metadata`. + * - ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` + - 15 + - Buffer for audio capture, see :ref:`audio`. + * - ``V4L2_BUF_TYPE_AUDIO_OUTPUT`` + - 16 + - Buffer for audio output, see :ref:`audio`.
.. _buffer-flags: diff --git a/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst new file mode 100644 index 000000000000..2ea493d0a73b --- /dev/null +++ b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst @@ -0,0 +1,71 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later + +.. _audiomem2mem: + +******************************** +Audio Memory-To-Memory Interface +******************************** + +A audio memory-to-memory device can compress, decompress, transform, or +otherwise convert audio data from one format into another format, in memory. +Such memory-to-memory devices set the ``V4L2_CAP_AUDIO_M2M`` capability. +Examples of memory-to-memory devices are codecs, audio preprocessing, +audio postprocessing. + +A memory-to-memory audio node supports both output (sending frames from +memory to the hardware) and capture (receiving the processed frames +from the hardware into memory) stream I/O. An application will have to +setup the stream I/O for both sides and finally call +:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` for both capture and output to +start the hardware. + +Memory-to-memory devices function as a shared resource: you can +open the audio node multiple times, each application setting up their +own properties that are local to the file handle, and each can use +it independently from the others. The driver will arbitrate access to +the hardware and reprogram it whenever another file handler gets access. + +Audio memory-to-memory devices are accessed through character device +special files named ``/dev/v4l-audio`` + +Querying Capabilities +===================== + +Device nodes supporting the audio memory-to-memory interface set the +``V4L2_CAP_AUDIO_M2M`` flag in the ``device_caps`` field of the +:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP` +ioctl. + +Data Format Negotiation +======================= + +The audio device uses the :ref:`format` ioctls to select the capture format. +The audio buffer content format is bound to that selected format. In addition +to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be +supported as well. + +To use the :ref:`format` ioctls applications set the ``type`` field of the +:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` or to +``V4L2_BUF_TYPE_AUDIO_OUTPUT``. Both drivers and applications must set the +remainder of the :c:type:`v4l2_format` structure to 0. + +.. c:type:: v4l2_audio_format + +.. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}| + +.. flat-table:: struct v4l2_audio_format + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - __u32 + - ``pixelformat`` + - The sample format, set by the application. see :ref:`pixfmt-audio` + * - __u32 + - ``channels`` + - The channel number, set by the application. channel number range is + [1, 32]. + * - __u32 + - ``buffersize`` + - Maximum buffer size in bytes required for data. The value is set by the + driver. diff --git a/Documentation/userspace-api/media/v4l/devices.rst b/Documentation/userspace-api/media/v4l/devices.rst index 8bfbad65a9d4..758bd90f1c26 100644 --- a/Documentation/userspace-api/media/v4l/devices.rst +++ b/Documentation/userspace-api/media/v4l/devices.rst @@ -24,3 +24,4 @@ Interfaces dev-event dev-subdev dev-meta + dev-audio-mem2mem diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst index 000c154b0f98..42deb07f4ff4 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst @@ -96,6 +96,8 @@ the ``mbus_code`` field is handled differently: ``V4L2_BUF_TYPE_VIDEO_OVERLAY``, ``V4L2_BUF_TYPE_SDR_CAPTURE``, ``V4L2_BUF_TYPE_SDR_OUTPUT``, + ``V4L2_BUF_TYPE_AUDIO_CAPTURE``, + ``V4L2_BUF_TYPE_AUDIO_OUTPUT``, ``V4L2_BUF_TYPE_META_CAPTURE`` and ``V4L2_BUF_TYPE_META_OUTPUT``. See :c:type:`v4l2_buf_type`. diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst index 675c385e5aca..528fd9df41aa 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-fmt.rst @@ -130,6 +130,10 @@ The format as returned by :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` must be identical - ``meta`` - Definition of a metadata format, see :ref:`meta-formats`, used by metadata capture devices. + * - struct :c:type:`v4l2_audio_format` + - ``audio`` + - Definition of a audio data format, see :ref:`audiomem2mem`, used by + audio memory-to-memory devices * - __u8 - ``raw_data``\ [200] - Place holder for future extensions. diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index da6d0b8e4c2c..e61152bb80d1 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -29,6 +29,8 @@ replace symbol V4L2_FIELD_SEQ_TB :c:type:`v4l2_field` replace symbol V4L2_FIELD_TOP :c:type:`v4l2_field`
# Documented enum v4l2_buf_type +replace symbol V4L2_BUF_TYPE_AUDIO_CAPTURE :c:type:`v4l2_buf_type` +replace symbol V4L2_BUF_TYPE_AUDIO_OUTPUT :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_META_CAPTURE :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_META_OUTPUT :c:type:`v4l2_buf_type` replace symbol V4L2_BUF_TYPE_SDR_CAPTURE :c:type:`v4l2_buf_type` diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index c7a54d82a55e..12f2be2773a2 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -785,6 +785,10 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create) case V4L2_BUF_TYPE_META_OUTPUT: requested_sizes[0] = f->fmt.meta.buffersize; break; + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + requested_sizes[0] = f->fmt.audio.buffersize; + break; default: return -EINVAL; } diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c index f81279492682..b92c760b611a 100644 --- a/drivers/media/v4l2-core/v4l2-dev.c +++ b/drivers/media/v4l2-core/v4l2-dev.c @@ -553,6 +553,7 @@ static void determine_valid_ioctls(struct video_device *vdev) bool is_tch = vdev->vfl_type == VFL_TYPE_TOUCH; bool is_meta = vdev->vfl_type == VFL_TYPE_VIDEO && (vdev->device_caps & meta_caps); + bool is_audio = vdev->vfl_type == VFL_TYPE_AUDIO; bool is_rx = vdev->vfl_dir != VFL_DIR_TX; bool is_tx = vdev->vfl_dir != VFL_DIR_RX; bool is_io_mc = vdev->device_caps & V4L2_CAP_IO_MC; @@ -664,6 +665,19 @@ static void determine_valid_ioctls(struct video_device *vdev) SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_meta_out); SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_meta_out); } + if (is_audio && is_rx) { + /* audio capture specific ioctls */ + SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_audio_cap); + SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_audio_cap); + SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_audio_cap); + SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_audio_cap); + } else if (is_audio && is_tx) { + /* audio output specific ioctls */ + SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_audio_out); + SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_audio_out); + SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_audio_out); + SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_audio_out); + } if (is_vbi) { /* vbi specific ioctls */ if ((is_rx && (ops->vidioc_g_fmt_vbi_cap || @@ -927,6 +941,9 @@ int __video_register_device(struct video_device *vdev, case VFL_TYPE_TOUCH: name_base = "v4l-touch"; break; + case VFL_TYPE_AUDIO: + name_base = "v4l-audio"; + break; default: pr_err("%s called with unknown type: %d\n", __func__, type); diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index f4d9d6279094..5d088e6c43e4 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -188,6 +188,8 @@ const char *v4l2_type_names[] = { [V4L2_BUF_TYPE_SDR_OUTPUT] = "sdr-out", [V4L2_BUF_TYPE_META_CAPTURE] = "meta-cap", [V4L2_BUF_TYPE_META_OUTPUT] = "meta-out", + [V4L2_BUF_TYPE_AUDIO_CAPTURE] = "audio-cap", + [V4L2_BUF_TYPE_AUDIO_OUTPUT] = "audio-out", }; EXPORT_SYMBOL(v4l2_type_names);
@@ -276,6 +278,7 @@ static void v4l_print_format(const void *arg, bool write_only) const struct v4l2_sliced_vbi_format *sliced; const struct v4l2_window *win; const struct v4l2_meta_format *meta; + const struct v4l2_audio_format *audio; u32 pixelformat; u32 planes; unsigned i; @@ -346,6 +349,13 @@ static void v4l_print_format(const void *arg, bool write_only) pr_cont(", dataformat=%p4cc, buffersize=%u\n", &pixelformat, meta->buffersize); break; + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + audio = &p->fmt.audio; + pixelformat = audio->audioformat; + pr_cont(", format=%p4cc, channels=%u, buffersize=%u\n", + &pixelformat, audio->channels, audio->buffersize); + break; } }
@@ -927,6 +937,7 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type) bool is_tch = vfd->vfl_type == VFL_TYPE_TOUCH; bool is_meta = vfd->vfl_type == VFL_TYPE_VIDEO && (vfd->device_caps & meta_caps); + bool is_audio = vfd->vfl_type == VFL_TYPE_AUDIO; bool is_rx = vfd->vfl_dir != VFL_DIR_TX; bool is_tx = vfd->vfl_dir != VFL_DIR_RX;
@@ -992,6 +1003,14 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type) if (is_meta && is_tx && ops->vidioc_g_fmt_meta_out) return 0; break; + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + if (is_audio && is_rx && ops->vidioc_g_fmt_audio_cap) + return 0; + break; + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + if (is_audio && is_tx && ops->vidioc_g_fmt_audio_out) + return 0; + break; default: break; } @@ -1596,6 +1615,16 @@ static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, break; ret = ops->vidioc_enum_fmt_meta_out(file, fh, arg); break; + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + if (unlikely(!ops->vidioc_enum_fmt_audio_cap)) + break; + ret = ops->vidioc_enum_fmt_audio_cap(file, fh, arg); + break; + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + if (unlikely(!ops->vidioc_enum_fmt_audio_out)) + break; + ret = ops->vidioc_enum_fmt_audio_out(file, fh, arg); + break; } if (ret == 0) v4l_fill_fmtdesc(p); @@ -1672,6 +1701,10 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops, return ops->vidioc_g_fmt_meta_cap(file, fh, arg); case V4L2_BUF_TYPE_META_OUTPUT: return ops->vidioc_g_fmt_meta_out(file, fh, arg); + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + return ops->vidioc_g_fmt_audio_cap(file, fh, arg); + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + return ops->vidioc_g_fmt_audio_out(file, fh, arg); } return -EINVAL; } @@ -1783,6 +1816,16 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops, break; memset_after(p, 0, fmt.meta); return ops->vidioc_s_fmt_meta_out(file, fh, arg); + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + if (unlikely(!ops->vidioc_s_fmt_audio_cap)) + break; + memset_after(p, 0, fmt.audio); + return ops->vidioc_s_fmt_audio_cap(file, fh, arg); + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + if (unlikely(!ops->vidioc_s_fmt_audio_out)) + break; + memset_after(p, 0, fmt.audio); + return ops->vidioc_s_fmt_audio_out(file, fh, arg); } return -EINVAL; } @@ -1891,6 +1934,16 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops, break; memset_after(p, 0, fmt.meta); return ops->vidioc_try_fmt_meta_out(file, fh, arg); + case V4L2_BUF_TYPE_AUDIO_CAPTURE: + if (unlikely(!ops->vidioc_try_fmt_audio_cap)) + break; + memset_after(p, 0, fmt.audio); + return ops->vidioc_try_fmt_audio_cap(file, fh, arg); + case V4L2_BUF_TYPE_AUDIO_OUTPUT: + if (unlikely(!ops->vidioc_try_fmt_audio_out)) + break; + memset_after(p, 0, fmt.audio); + return ops->vidioc_try_fmt_audio_out(file, fh, arg); } return -EINVAL; } diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h index e0a13505f88d..1adef2a90bd5 100644 --- a/include/media/v4l2-dev.h +++ b/include/media/v4l2-dev.h @@ -30,6 +30,7 @@ * @VFL_TYPE_SUBDEV: for V4L2 subdevices * @VFL_TYPE_SDR: for Software Defined Radio tuners * @VFL_TYPE_TOUCH: for touch sensors + * @VFL_TYPE_AUDIO: for audio memory-to-memory devices * @VFL_TYPE_MAX: number of VFL types, must always be last in the enum */ enum vfl_devnode_type { @@ -39,6 +40,7 @@ enum vfl_devnode_type { VFL_TYPE_SUBDEV, VFL_TYPE_SDR, VFL_TYPE_TOUCH, + VFL_TYPE_AUDIO, VFL_TYPE_MAX /* Shall be the last one */ };
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h index edb733f21604..f840cf740ce1 100644 --- a/include/media/v4l2-ioctl.h +++ b/include/media/v4l2-ioctl.h @@ -45,6 +45,12 @@ struct v4l2_fh; * @vidioc_enum_fmt_meta_out: pointer to the function that implements * :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic * for metadata output + * @vidioc_enum_fmt_audio_cap: pointer to the function that implements + * :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic + * for audio capture + * @vidioc_enum_fmt_audio_out: pointer to the function that implements + * :ref:`VIDIOC_ENUM_FMT <vidioc_enum_fmt>` ioctl logic + * for audio output * @vidioc_g_fmt_vid_cap: pointer to the function that implements * :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture * in single plane mode @@ -79,6 +85,10 @@ struct v4l2_fh; * :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for metadata capture * @vidioc_g_fmt_meta_out: pointer to the function that implements * :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for metadata output + * @vidioc_g_fmt_audio_cap: pointer to the function that implements + * :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for audio capture + * @vidioc_g_fmt_audio_out: pointer to the function that implements + * :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for audio output * @vidioc_s_fmt_vid_cap: pointer to the function that implements * :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture * in single plane mode @@ -113,6 +123,10 @@ struct v4l2_fh; * :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for metadata capture * @vidioc_s_fmt_meta_out: pointer to the function that implements * :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for metadata output + * @vidioc_s_fmt_audio_cap: pointer to the function that implements + * :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for audio capture + * @vidioc_s_fmt_audio_out: pointer to the function that implements + * :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for audio output * @vidioc_try_fmt_vid_cap: pointer to the function that implements * :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture * in single plane mode @@ -149,6 +163,10 @@ struct v4l2_fh; * :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata capture * @vidioc_try_fmt_meta_out: pointer to the function that implements * :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for metadata output + * @vidioc_try_fmt_audio_cap: pointer to the function that implements + * :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for audio capture + * @vidioc_try_fmt_audio_out: pointer to the function that implements + * :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for audio output * @vidioc_reqbufs: pointer to the function that implements * :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl * @vidioc_querybuf: pointer to the function that implements @@ -315,6 +333,10 @@ struct v4l2_ioctl_ops { struct v4l2_fmtdesc *f); int (*vidioc_enum_fmt_meta_out)(struct file *file, void *fh, struct v4l2_fmtdesc *f); + int (*vidioc_enum_fmt_audio_cap)(struct file *file, void *fh, + struct v4l2_fmtdesc *f); + int (*vidioc_enum_fmt_audio_out)(struct file *file, void *fh, + struct v4l2_fmtdesc *f);
/* VIDIOC_G_FMT handlers */ int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh, @@ -345,6 +367,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_g_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f); + int (*vidioc_g_fmt_audio_cap)(struct file *file, void *fh, + struct v4l2_format *f); + int (*vidioc_g_fmt_audio_out)(struct file *file, void *fh, + struct v4l2_format *f);
/* VIDIOC_S_FMT handlers */ int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh, @@ -375,6 +401,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_s_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f); + int (*vidioc_s_fmt_audio_cap)(struct file *file, void *fh, + struct v4l2_format *f); + int (*vidioc_s_fmt_audio_out)(struct file *file, void *fh, + struct v4l2_format *f);
/* VIDIOC_TRY_FMT handlers */ int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh, @@ -405,6 +435,10 @@ struct v4l2_ioctl_ops { struct v4l2_format *f); int (*vidioc_try_fmt_meta_out)(struct file *file, void *fh, struct v4l2_format *f); + int (*vidioc_try_fmt_audio_cap)(struct file *file, void *fh, + struct v4l2_format *f); + int (*vidioc_try_fmt_audio_out)(struct file *file, void *fh, + struct v4l2_format *f);
/* Buffer handlers */ int (*vidioc_reqbufs)(struct file *file, void *fh, diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 3decf7d73870..b0ddb7319d36 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -153,6 +153,8 @@ enum v4l2_buf_type { V4L2_BUF_TYPE_SDR_OUTPUT = 12, V4L2_BUF_TYPE_META_CAPTURE = 13, V4L2_BUF_TYPE_META_OUTPUT = 14, + V4L2_BUF_TYPE_AUDIO_CAPTURE = 15, + V4L2_BUF_TYPE_AUDIO_OUTPUT = 16, /* Deprecated, do not use */ V4L2_BUF_TYPE_PRIVATE = 0x80, }; @@ -169,6 +171,7 @@ enum v4l2_buf_type { || (type) == V4L2_BUF_TYPE_VBI_OUTPUT \ || (type) == V4L2_BUF_TYPE_SLICED_VBI_OUTPUT \ || (type) == V4L2_BUF_TYPE_SDR_OUTPUT \ + || (type) == V4L2_BUF_TYPE_AUDIO_OUTPUT \ || (type) == V4L2_BUF_TYPE_META_OUTPUT)
#define V4L2_TYPE_IS_CAPTURE(type) (!V4L2_TYPE_IS_OUTPUT(type)) @@ -2418,6 +2421,18 @@ struct v4l2_meta_format { __u32 buffersize; } __attribute__ ((packed));
+/** + * struct v4l2_audio_format - audio data format definition + * @audioformat: little endian four character code (fourcc) + * @channels: channel numbers + * @buffersize: maximum size in bytes required for data + */ +struct v4l2_audio_format { + __u32 audioformat; + __u32 channels; + __u32 buffersize; +} __attribute__ ((packed)); + /** * struct v4l2_format - stream data format * @type: enum v4l2_buf_type; type of the data stream @@ -2426,6 +2441,7 @@ struct v4l2_meta_format { * @win: definition of an overlaid image * @vbi: raw VBI capture or output parameters * @sliced: sliced VBI capture or output parameters + * @audio: definition of an audio format * @raw_data: placeholder for future extensions and custom formats * @fmt: union of @pix, @pix_mp, @win, @vbi, @sliced, @sdr, @meta * and @raw_data @@ -2440,6 +2456,7 @@ struct v4l2_format { struct v4l2_sliced_vbi_format sliced; /* V4L2_BUF_TYPE_SLICED_VBI_CAPTURE */ struct v4l2_sdr_format sdr; /* V4L2_BUF_TYPE_SDR_CAPTURE */ struct v4l2_meta_format meta; /* V4L2_BUF_TYPE_META_CAPTURE */ + struct v4l2_audio_format audio; /* V4L2_BUF_TYPE_AUDIO_CAPTURE */ __u8 raw_data[200]; /* user-defined */ } fmt; };
On 13/10/2023 10:31, Shengjiu Wang wrote:
Audio signal processing has the requirement for memory to memory similar as Video.
This patch is to add this support in v4l2 framework, defined new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format for audio case usage.
The created audio device is named "/dev/v4l-audioX".
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/buffer.rst | 6 ++ .../media/v4l/dev-audio-mem2mem.rst | 71 +++++++++++++++++++ .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 ++ .../media/videodev2.h.rst.exceptions | 2 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 ++ drivers/media/v4l2-core/v4l2-dev.c | 17 +++++ drivers/media/v4l2-core/v4l2-ioctl.c | 53 ++++++++++++++ include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 +++++++++ include/uapi/linux/videodev2.h | 17 +++++ 12 files changed, 213 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 04dec3e570ed..80cf2cb20dfe 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -438,6 +438,12 @@ enum v4l2_buf_type * - ``V4L2_BUF_TYPE_META_OUTPUT`` - 14 - Buffer for metadata output, see :ref:`metadata`.
- ``V4L2_BUF_TYPE_AUDIO_CAPTURE``
- 15- Buffer for audio capture, see :ref:`audio`.
- ``V4L2_BUF_TYPE_AUDIO_OUTPUT``
- 16- Buffer for audio output, see :ref:`audio`... _buffer-flags: diff --git a/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst new file mode 100644 index 000000000000..2ea493d0a73b --- /dev/null +++ b/Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst @@ -0,0 +1,71 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _audiomem2mem:
+******************************** +Audio Memory-To-Memory Interface +********************************
+A audio memory-to-memory device can compress, decompress, transform, or
A -> An
+otherwise convert audio data from one format into another format, in memory. +Such memory-to-memory devices set the ``V4L2_CAP_AUDIO_M2M`` capability. +Examples of memory-to-memory devices are codecs, audio preprocessing,
codecs -> audio codecs
Reason: within V4L2 'codec' refers to a video codec by default, so for audio codecs it is better to be explicit and mention 'audio'.
+audio postprocessing.
+A memory-to-memory audio node supports both output (sending frames from
I think it is better to write 'audio frames' instead of just 'frames', for the same reason as why I prefer 'audio codec'. It makes it explicit that we are dealing with audio.
+memory to the hardware) and capture (receiving the processed frames
audio frames
+from the hardware into memory) stream I/O. An application will have to +setup the stream I/O for both sides and finally call +:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` for both capture and output to +start the hardware.
+Memory-to-memory devices function as a shared resource: you can +open the audio node multiple times, each application setting up their +own properties that are local to the file handle, and each can use +it independently from the others. The driver will arbitrate access to +the hardware and reprogram it whenever another file handler gets access.
+Audio memory-to-memory devices are accessed through character device +special files named ``/dev/v4l-audio``
+Querying Capabilities +=====================
+Device nodes supporting the audio memory-to-memory interface set the +``V4L2_CAP_AUDIO_M2M`` flag in the ``device_caps`` field of the +:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP` +ioctl.
+Data Format Negotiation +=======================
+The audio device uses the :ref:`format` ioctls to select the capture format. +The audio buffer content format is bound to that selected format. In addition +to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be +supported as well.
+To use the :ref:`format` ioctls applications set the ``type`` field of the +:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` or to +``V4L2_BUF_TYPE_AUDIO_OUTPUT``. Both drivers and applications must set the +remainder of the :c:type:`v4l2_format` structure to 0.
+.. c:type:: v4l2_audio_format
+.. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|
+.. flat-table:: struct v4l2_audio_format
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
- ``pixelformat``- The sample format, set by the application. see :ref:`pixfmt-audio`
- __u32
- ``channels``- The channel number, set by the application. channel number range is[1, 32].
- __u32
- ``buffersize``- Maximum buffer size in bytes required for data. The value is set by thedriver.
<snip>
Regards,
Hans
The audio sample format definition is from alsa, the header file is include/uapi/sound/asound.h, but don't include this header file directly, because in user space, there is another copy in alsa-lib. There will be conflict in userspace for include videodev2.h & asound.h and asoundlib.h
Here still use the fourcc format.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- .../userspace-api/media/v4l/pixfmt-audio.rst | 202 ++++++++++++++++++ .../userspace-api/media/v4l/pixfmt.rst | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 36 ++++ include/uapi/linux/videodev2.h | 48 +++++ 4 files changed, 287 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-audio.rst b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst new file mode 100644 index 000000000000..ac89b2c4b594 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst @@ -0,0 +1,202 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later + +.. _pixfmt-audio: + +************* +Audio Formats +************* + +These formats are used for :ref:`audiomem2mem` interface only. + +.. tabularcolumns:: |p{5.8cm}|p{1.2cm}|p{10.3cm}| + +.. cssclass:: longtable + +.. flat-table:: Audio Format + :header-rows: 1 + :stub-columns: 0 + :widths: 3 1 4 + + * - Identifier + - Code + - Details + * .. _V4L2-AUDIO-FMT-S8: + + - ``V4L2_AUDIO_FMT_S8`` + - 'S8' + - Corresponds to SNDRV_PCM_FORMAT_S8 in ALSA + * .. _V4L2-AUDIO-FMT-U8: + + - ``V4L2_AUDIO_FMT_U8`` + - 'U8' + - Corresponds to SNDRV_PCM_FORMAT_U8 in ALSA + * .. _V4L2-AUDIO-FMT-S16-LE: + + - ``V4L2_AUDIO_FMT_S16_LE`` + - 'S16_LE' + - Corresponds to SNDRV_PCM_FORMAT_S16_LE in ALSA + * .. _V4L2-AUDIO-FMT-S16-BE: + + - ``V4L2_AUDIO_FMT_S16_BE`` + - 'S16_BE' + - Corresponds to SNDRV_PCM_FORMAT_S16_BE in ALSA + * .. _V4L2-AUDIO-FMT-U16-LE: + + - ``V4L2_AUDIO_FMT_U16_LE`` + - 'U16_LE' + - Corresponds to SNDRV_PCM_FORMAT_U16_LE in ALSA + * .. _V4L2-AUDIO-FMT-U16-BE: + + - ``V4L2_AUDIO_FMT_U16_BE`` + - 'U16_BE' + - Corresponds to SNDRV_PCM_FORMAT_U16_BE in ALSA + * .. _V4L2-AUDIO-FMT-S24-LE: + + - ``V4L2_AUDIO_FMT_S24_LE`` + - 'S24_LE' + - Corresponds to SNDRV_PCM_FORMAT_S24_LE in ALSA + * .. _V4L2-AUDIO-FMT-S24-BE: + + - ``V4L2_AUDIO_FMT_S24_BE`` + - 'S24_BE' + - Corresponds to SNDRV_PCM_FORMAT_S24_BE in ALSA + * .. _V4L2-AUDIO-FMT-U24-LE: + + - ``V4L2_AUDIO_FMT_U24_LE`` + - 'U24_LE' + - Corresponds to SNDRV_PCM_FORMAT_U24_LE in ALSA + * .. _V4L2-AUDIO-FMT-U24-BE: + + - ``V4L2_AUDIO_FMT_U24_BE`` + - 'U24_BE' + - Corresponds to SNDRV_PCM_FORMAT_U24_BE in ALSA + * .. _V4L2-AUDIO-FMT-S32-LE: + + - ``V4L2_AUDIO_FMT_S32_LE`` + - 'S32_LE' + - Corresponds to SNDRV_PCM_FORMAT_S32_LE in ALSA + * .. _V4L2-AUDIO-FMT-S32-BE: + + - ``V4L2_AUDIO_FMT_S32_BE`` + - 'S32_BE' + - Corresponds to SNDRV_PCM_FORMAT_S32_BE in ALSA + * .. _V4L2-AUDIO-FMT-U32-LE: + + - ``V4L2_AUDIO_FMT_U32_LE`` + - 'U32_LE' + - Corresponds to SNDRV_PCM_FORMAT_U32_LE in ALSA + * .. _V4L2-AUDIO-FMT-U32-BE: + + - ``V4L2_AUDIO_FMT_U32_BE`` + - 'U32_BE' + - Corresponds to SNDRV_PCM_FORMAT_U32_BE in ALSA + * .. _V4L2-AUDIO-FMT-FLOAT-LE: + + - ``V4L2_AUDIO_FMT_FLOAT_LE`` + - 'FLOAT_LE' + - Corresponds to SNDRV_PCM_FORMAT_FLOAT_LE in ALSA + * .. _V4L2-AUDIO-FMT-FLOAT-BE: + + - ``V4L2_AUDIO_FMT_FLOAT_BE`` + - 'FLOAT_BE' + - Corresponds to SNDRV_PCM_FORMAT_FLOAT_BE in ALSA + * .. _V4L2-AUDIO-FMT-FLOAT64-LE: + + - ``V4L2_AUDIO_FMT_FLOAT64_LE`` + - 'FLOAT64_LE' + - Corresponds to SNDRV_PCM_FORMAT_FLOAT64_LE in ALSA + * .. _V4L2-AUDIO-FMT-FLOAT64-BE: + + - ``V4L2_AUDIO_FMT_FLOAT64_BE`` + - 'FLOAT64_BE' + - Corresponds to SNDRV_PCM_FORMAT_FLOAT64_BE in ALSA + * .. _V4L2-AUDIO-FMT-IEC958-SUBFRAME-LE: + + - ``V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE`` + - 'IEC958_SUBFRAME_LE' + - Corresponds to SNDRV_PCM_FORMAT_IEC958_SUBFRAME_LE in ALSA + * .. _V4L2-AUDIO-FMT-IEC958-SUBFRAME-BE: + + - ``V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE`` + - 'IEC958_SUBFRAME_BE' + - Corresponds to SNDRV_PCM_FORMAT_IEC958_SUBFRAME_BE in ALSA + * .. _V4L2-AUDIO-FMT-S20-LE: + + - ``V4L2_AUDIO_FMT_S20_LE`` + - 'S20_LE' + - Corresponds to SNDRV_PCM_FORMAT_S20_LE in ALSA + * .. _V4L2-AUDIO-FMT-S20-BE: + + - ``V4L2_AUDIO_FMT_S20_BE`` + - 'S20_BE' + - Corresponds to SNDRV_PCM_FORMAT_S20_BE in ALSA + * .. _V4L2-AUDIO-FMT-U20-LE: + + - ``V4L2_AUDIO_FMT_U20_LE`` + - 'U20_LE' + - Corresponds to SNDRV_PCM_FORMAT_U20_LE in ALSA + * .. _V4L2-AUDIO-FMT-U20-BE: + + - ``V4L2_AUDIO_FMT_U20_BE`` + - 'U20_BE' + - Corresponds to SNDRV_PCM_FORMAT_U20_BE in ALSA + * .. _V4L2-AUDIO-FMT-S24-3LE: + + - ``V4L2_AUDIO_FMT_S24_3LE`` + - 'S24_3LE' + - Corresponds to SNDRV_PCM_FORMAT_S24_3LE in ALSA + * .. _V4L2-AUDIO-FMT-S24-3BE: + + - ``V4L2_AUDIO_FMT_S24_3BE`` + - 'S24_3BE' + - Corresponds to SNDRV_PCM_FORMAT_S24_3BE in ALSA + * .. _V4L2-AUDIO-FMT-U24-3LE: + + - ``V4L2_AUDIO_FMT_U24_3LE`` + - 'U24_3LE' + - Corresponds to SNDRV_PCM_FORMAT_U24_3LE in ALSA + * .. _V4L2-AUDIO-FMT-U24-3BE: + + - ``V4L2_AUDIO_FMT_U24_3BE`` + - 'U24_3BE' + - Corresponds to SNDRV_PCM_FORMAT_U24_3BE in ALSA + * .. _V4L2-AUDIO-FMT-S20-3LE: + + - ``V4L2_AUDIO_FMT_S20_3LE`` + - 'S20_3LE' + - Corresponds to SNDRV_PCM_FORMAT_S24_3LE in ALSA + * .. _V4L2-AUDIO-FMT-S20-3BE: + + - ``V4L2_AUDIO_FMT_S20_3BE`` + - 'S20_3BE' + - Corresponds to SNDRV_PCM_FORMAT_S20_3BE in ALSA + * .. _V4L2-AUDIO-FMT-U20-3LE: + + - ``V4L2_AUDIO_FMT_U20_3LE`` + - 'U20_3LE' + - Corresponds to SNDRV_PCM_FORMAT_U20_3LE in ALSA + * .. _V4L2-AUDIO-FMT-U20-3BE: + + - ``V4L2_AUDIO_FMT_U20_3BE`` + - 'U20_3BE' + - Corresponds to SNDRV_PCM_FORMAT_U20_3BE in ALSA + * .. _V4L2-AUDIO-FMT-S18-3LE: + + - ``V4L2_AUDIO_FMT_S18_3LE`` + - 'S18_3LE' + - Corresponds to SNDRV_PCM_FORMAT_S18_3LE in ALSA + * .. _V4L2-AUDIO-FMT-S18-3BE: + + - ``V4L2_AUDIO_FMT_S18_3BE`` + - 'S18_3BE' + - Corresponds to SNDRV_PCM_FORMAT_S18_3BE in ALSA + * .. _V4L2-AUDIO-FMT-U18-3LE: + + - ``V4L2_AUDIO_FMT_U18_3LE`` + - 'U18_3LE' + - Corresponds to SNDRV_PCM_FORMAT_U18_3LE in ALSA + * .. _V4L2-AUDIO-FMT-U18-3BE: + + - ``V4L2_AUDIO_FMT_U18_3BE`` + - 'U18_3BE' + - Corresponds to SNDRV_PCM_FORMAT_U18_3BE in ALSA diff --git a/Documentation/userspace-api/media/v4l/pixfmt.rst b/Documentation/userspace-api/media/v4l/pixfmt.rst index 11dab4a90630..2eb6fdd3b43d 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt.rst @@ -36,3 +36,4 @@ see also :ref:`VIDIOC_G_FBUF <VIDIOC_G_FBUF>`.) colorspaces colorspaces-defs colorspaces-details + pixfmt-audio diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 5d088e6c43e4..31e443c644db 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1471,6 +1471,42 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_Y210: descr = "10-bit YUYV Packed"; break; case V4L2_PIX_FMT_Y212: descr = "12-bit YUYV Packed"; break; case V4L2_PIX_FMT_Y216: descr = "16-bit YUYV Packed"; break; + case V4L2_AUDIO_FMT_S8: descr = "8-bit Signed"; break; + case V4L2_AUDIO_FMT_U8: descr = "8-bit Unsigned"; break; + case V4L2_AUDIO_FMT_S16_LE: descr = "16-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S16_BE: descr = "16-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U16_LE: descr = "16-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U16_BE: descr = "16-bit Unsigned BE"; break; + case V4L2_AUDIO_FMT_S24_LE: descr = "24(32)-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S24_BE: descr = "24(32)-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U24_LE: descr = "24(32)-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U24_BE: descr = "24(32)-bit Unsigned BE"; break; + case V4L2_AUDIO_FMT_S32_LE: descr = "32-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S32_BE: descr = "32-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U32_LE: descr = "32-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U32_BE: descr = "32-bit Unsigned BE"; break; + case V4L2_AUDIO_FMT_FLOAT_LE: descr = "32-bit Float LE"; break; + case V4L2_AUDIO_FMT_FLOAT_BE: descr = "32-bit Float BE"; break; + case V4L2_AUDIO_FMT_FLOAT64_LE: descr = "64-bit Float LE"; break; + case V4L2_AUDIO_FMT_FLOAT64_BE: descr = "64-bit Float BE"; break; + case V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE: descr = "32-bit IEC958 LE"; break; + case V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE: descr = "32-bit IEC958 BE"; break; + case V4L2_AUDIO_FMT_S20_LE: descr = "20-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S20_BE: descr = "20-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U20_LE: descr = "20-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U20_BE: descr = "20-bit Unsigned BE"; break; + case V4L2_AUDIO_FMT_S24_3LE: descr = "24(24)-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S24_3BE: descr = "24(24)-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U24_3LE: descr = "24(24)-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U24_3BE: descr = "24(24)-bit Unsigned BE"; break; + case V4L2_AUDIO_FMT_S20_3LE: descr = "20(24)-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S20_3BE: descr = "20(24)-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U20_3LE: descr = "20(24)-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U20_3BE: descr = "20(24)-bit Unsigned BE"; break; + case V4L2_AUDIO_FMT_S18_3LE: descr = "18(24)-bit Signed LE"; break; + case V4L2_AUDIO_FMT_S18_3BE: descr = "18(24)-bit Signed BE"; break; + case V4L2_AUDIO_FMT_U18_3LE: descr = "18(24)-bit Unsigned LE"; break; + case V4L2_AUDIO_FMT_U18_3BE: descr = "18(24)-bit Unsigned BE"; break;
default: /* Compressed formats */ diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index b0ddb7319d36..2ac7b989394c 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -842,6 +842,54 @@ struct v4l2_pix_format { #define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */ #define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
+/* + * Audio-data formats + * All these audio formats use a fourcc starting with 'AU' + * followed by the SNDRV_PCM_FORMAT_ value from asound.h. + */ +#define V4L2_AUDIO_FMT_S8 v4l2_fourcc('A', 'U', '0', '0') +#define V4L2_AUDIO_FMT_U8 v4l2_fourcc('A', 'U', '0', '1') +#define V4L2_AUDIO_FMT_S16_LE v4l2_fourcc('A', 'U', '0', '2') +#define V4L2_AUDIO_FMT_S16_BE v4l2_fourcc('A', 'U', '0', '3') +#define V4L2_AUDIO_FMT_U16_LE v4l2_fourcc('A', 'U', '0', '4') +#define V4L2_AUDIO_FMT_U16_BE v4l2_fourcc('A', 'U', '0', '5') +#define V4L2_AUDIO_FMT_S24_LE v4l2_fourcc('A', 'U', '0', '6') +#define V4L2_AUDIO_FMT_S24_BE v4l2_fourcc('A', 'U', '0', '7') +#define V4L2_AUDIO_FMT_U24_LE v4l2_fourcc('A', 'U', '0', '8') +#define V4L2_AUDIO_FMT_U24_BE v4l2_fourcc('A', 'U', '0', '9') + +#define V4L2_AUDIO_FMT_S32_LE v4l2_fourcc('A', 'U', '1', '0') +#define V4L2_AUDIO_FMT_S32_BE v4l2_fourcc('A', 'U', '1', '1') +#define V4L2_AUDIO_FMT_U32_LE v4l2_fourcc('A', 'U', '1', '2') +#define V4L2_AUDIO_FMT_U32_BE v4l2_fourcc('A', 'U', '1', '3') +#define V4L2_AUDIO_FMT_FLOAT_LE v4l2_fourcc('A', 'U', '1', '4') +#define V4L2_AUDIO_FMT_FLOAT_BE v4l2_fourcc('A', 'U', '1', '5') +#define V4L2_AUDIO_FMT_FLOAT64_LE v4l2_fourcc('A', 'U', '1', '6') +#define V4L2_AUDIO_FMT_FLOAT64_BE v4l2_fourcc('A', 'U', '1', '7') +#define V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE v4l2_fourcc('A', 'U', '1', '8') +#define V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE v4l2_fourcc('A', 'U', '1', '9') + +#define V4L2_AUDIO_FMT_S20_LE v4l2_fourcc('A', 'U', '2', '5') +#define V4L2_AUDIO_FMT_S20_BE v4l2_fourcc('A', 'U', '2', '6') +#define V4L2_AUDIO_FMT_U20_LE v4l2_fourcc('A', 'U', '2', '7') +#define V4L2_AUDIO_FMT_U20_BE v4l2_fourcc('A', 'U', '2', '8') + +#define V4L2_AUDIO_FMT_S24_3LE v4l2_fourcc('A', 'U', '3', '2') +#define V4L2_AUDIO_FMT_S24_3BE v4l2_fourcc('A', 'U', '3', '3') +#define V4L2_AUDIO_FMT_U24_3LE v4l2_fourcc('A', 'U', '3', '4') +#define V4L2_AUDIO_FMT_U24_3BE v4l2_fourcc('A', 'U', '3', '5') +#define V4L2_AUDIO_FMT_S20_3LE v4l2_fourcc('A', 'U', '3', '6') +#define V4L2_AUDIO_FMT_S20_3BE v4l2_fourcc('A', 'U', '3', '7') +#define V4L2_AUDIO_FMT_U20_3LE v4l2_fourcc('A', 'U', '3', '8') +#define V4L2_AUDIO_FMT_U20_3BE v4l2_fourcc('A', 'U', '3', '9') +#define V4L2_AUDIO_FMT_S18_3LE v4l2_fourcc('A', 'U', '4', '0') +#define V4L2_AUDIO_FMT_S18_3BE v4l2_fourcc('A', 'U', '4', '1') +#define V4L2_AUDIO_FMT_U18_3LE v4l2_fourcc('A', 'U', '4', '2') +#define V4L2_AUDIO_FMT_U18_3BE v4l2_fourcc('A', 'U', '4', '3') + +#define v4l2_fourcc_to_audfmt(X) \ + (((((X) >> 16) & 0xFF) - '0') * 10 + ((((X) >> 24) & 0xFF) - '0')) + /* priv field value to indicates that subsequent fields are valid. */ #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
On 13/10/2023 10:31, Shengjiu Wang wrote:
The audio sample format definition is from alsa, the header file is include/uapi/sound/asound.h, but don't include this header file directly, because in user space, there is another copy in alsa-lib. There will be conflict in userspace for include videodev2.h & asound.h and asoundlib.h
Here still use the fourcc format.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/pixfmt-audio.rst | 202 ++++++++++++++++++ .../userspace-api/media/v4l/pixfmt.rst | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 36 ++++ include/uapi/linux/videodev2.h | 48 +++++ 4 files changed, 287 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-audio.rst b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst new file mode 100644 index 000000000000..ac89b2c4b594 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst @@ -0,0 +1,202 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _pixfmt-audio:
+************* +Audio Formats +*************
+These formats are used for :ref:`audiomem2mem` interface only.
+.. tabularcolumns:: |p{5.8cm}|p{1.2cm}|p{10.3cm}|
+.. cssclass:: longtable
+.. flat-table:: Audio Format
- :header-rows: 1
- :stub-columns: 0
- :widths: 3 1 4
- Identifier
- Code- Details
- .. _V4L2-AUDIO-FMT-S8:
- ``V4L2_AUDIO_FMT_S8``- 'S8'- Corresponds to SNDRV_PCM_FORMAT_S8 in ALSA
- .. _V4L2-AUDIO-FMT-U8:
- ``V4L2_AUDIO_FMT_U8``- 'U8'- Corresponds to SNDRV_PCM_FORMAT_U8 in ALSA
- .. _V4L2-AUDIO-FMT-S16-LE:
- ``V4L2_AUDIO_FMT_S16_LE``- 'S16_LE'- Corresponds to SNDRV_PCM_FORMAT_S16_LE in ALSA
- .. _V4L2-AUDIO-FMT-S16-BE:
- ``V4L2_AUDIO_FMT_S16_BE``- 'S16_BE'- Corresponds to SNDRV_PCM_FORMAT_S16_BE in ALSA
- .. _V4L2-AUDIO-FMT-U16-LE:
- ``V4L2_AUDIO_FMT_U16_LE``- 'U16_LE'- Corresponds to SNDRV_PCM_FORMAT_U16_LE in ALSA
- .. _V4L2-AUDIO-FMT-U16-BE:
- ``V4L2_AUDIO_FMT_U16_BE``- 'U16_BE'- Corresponds to SNDRV_PCM_FORMAT_U16_BE in ALSA
- .. _V4L2-AUDIO-FMT-S24-LE:
- ``V4L2_AUDIO_FMT_S24_LE``- 'S24_LE'- Corresponds to SNDRV_PCM_FORMAT_S24_LE in ALSA
- .. _V4L2-AUDIO-FMT-S24-BE:
- ``V4L2_AUDIO_FMT_S24_BE``- 'S24_BE'- Corresponds to SNDRV_PCM_FORMAT_S24_BE in ALSA
- .. _V4L2-AUDIO-FMT-U24-LE:
- ``V4L2_AUDIO_FMT_U24_LE``- 'U24_LE'- Corresponds to SNDRV_PCM_FORMAT_U24_LE in ALSA
- .. _V4L2-AUDIO-FMT-U24-BE:
- ``V4L2_AUDIO_FMT_U24_BE``- 'U24_BE'- Corresponds to SNDRV_PCM_FORMAT_U24_BE in ALSA
- .. _V4L2-AUDIO-FMT-S32-LE:
- ``V4L2_AUDIO_FMT_S32_LE``- 'S32_LE'- Corresponds to SNDRV_PCM_FORMAT_S32_LE in ALSA
- .. _V4L2-AUDIO-FMT-S32-BE:
- ``V4L2_AUDIO_FMT_S32_BE``- 'S32_BE'- Corresponds to SNDRV_PCM_FORMAT_S32_BE in ALSA
- .. _V4L2-AUDIO-FMT-U32-LE:
- ``V4L2_AUDIO_FMT_U32_LE``- 'U32_LE'- Corresponds to SNDRV_PCM_FORMAT_U32_LE in ALSA
- .. _V4L2-AUDIO-FMT-U32-BE:
- ``V4L2_AUDIO_FMT_U32_BE``- 'U32_BE'- Corresponds to SNDRV_PCM_FORMAT_U32_BE in ALSA
- .. _V4L2-AUDIO-FMT-FLOAT-LE:
- ``V4L2_AUDIO_FMT_FLOAT_LE``- 'FLOAT_LE'- Corresponds to SNDRV_PCM_FORMAT_FLOAT_LE in ALSA
- .. _V4L2-AUDIO-FMT-FLOAT-BE:
- ``V4L2_AUDIO_FMT_FLOAT_BE``- 'FLOAT_BE'- Corresponds to SNDRV_PCM_FORMAT_FLOAT_BE in ALSA
- .. _V4L2-AUDIO-FMT-FLOAT64-LE:
- ``V4L2_AUDIO_FMT_FLOAT64_LE``- 'FLOAT64_LE'- Corresponds to SNDRV_PCM_FORMAT_FLOAT64_LE in ALSA
- .. _V4L2-AUDIO-FMT-FLOAT64-BE:
- ``V4L2_AUDIO_FMT_FLOAT64_BE``- 'FLOAT64_BE'- Corresponds to SNDRV_PCM_FORMAT_FLOAT64_BE in ALSA
- .. _V4L2-AUDIO-FMT-IEC958-SUBFRAME-LE:
- ``V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE``- 'IEC958_SUBFRAME_LE'- Corresponds to SNDRV_PCM_FORMAT_IEC958_SUBFRAME_LE in ALSA
- .. _V4L2-AUDIO-FMT-IEC958-SUBFRAME-BE:
- ``V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE``- 'IEC958_SUBFRAME_BE'- Corresponds to SNDRV_PCM_FORMAT_IEC958_SUBFRAME_BE in ALSA
- .. _V4L2-AUDIO-FMT-S20-LE:
- ``V4L2_AUDIO_FMT_S20_LE``- 'S20_LE'- Corresponds to SNDRV_PCM_FORMAT_S20_LE in ALSA
- .. _V4L2-AUDIO-FMT-S20-BE:
- ``V4L2_AUDIO_FMT_S20_BE``- 'S20_BE'- Corresponds to SNDRV_PCM_FORMAT_S20_BE in ALSA
- .. _V4L2-AUDIO-FMT-U20-LE:
- ``V4L2_AUDIO_FMT_U20_LE``- 'U20_LE'- Corresponds to SNDRV_PCM_FORMAT_U20_LE in ALSA
- .. _V4L2-AUDIO-FMT-U20-BE:
- ``V4L2_AUDIO_FMT_U20_BE``- 'U20_BE'- Corresponds to SNDRV_PCM_FORMAT_U20_BE in ALSA
- .. _V4L2-AUDIO-FMT-S24-3LE:
- ``V4L2_AUDIO_FMT_S24_3LE``- 'S24_3LE'- Corresponds to SNDRV_PCM_FORMAT_S24_3LE in ALSA
- .. _V4L2-AUDIO-FMT-S24-3BE:
- ``V4L2_AUDIO_FMT_S24_3BE``- 'S24_3BE'- Corresponds to SNDRV_PCM_FORMAT_S24_3BE in ALSA
- .. _V4L2-AUDIO-FMT-U24-3LE:
- ``V4L2_AUDIO_FMT_U24_3LE``- 'U24_3LE'- Corresponds to SNDRV_PCM_FORMAT_U24_3LE in ALSA
- .. _V4L2-AUDIO-FMT-U24-3BE:
- ``V4L2_AUDIO_FMT_U24_3BE``- 'U24_3BE'- Corresponds to SNDRV_PCM_FORMAT_U24_3BE in ALSA
- .. _V4L2-AUDIO-FMT-S20-3LE:
- ``V4L2_AUDIO_FMT_S20_3LE``- 'S20_3LE'- Corresponds to SNDRV_PCM_FORMAT_S24_3LE in ALSA
- .. _V4L2-AUDIO-FMT-S20-3BE:
- ``V4L2_AUDIO_FMT_S20_3BE``- 'S20_3BE'- Corresponds to SNDRV_PCM_FORMAT_S20_3BE in ALSA
- .. _V4L2-AUDIO-FMT-U20-3LE:
- ``V4L2_AUDIO_FMT_U20_3LE``- 'U20_3LE'- Corresponds to SNDRV_PCM_FORMAT_U20_3LE in ALSA
- .. _V4L2-AUDIO-FMT-U20-3BE:
- ``V4L2_AUDIO_FMT_U20_3BE``- 'U20_3BE'- Corresponds to SNDRV_PCM_FORMAT_U20_3BE in ALSA
- .. _V4L2-AUDIO-FMT-S18-3LE:
- ``V4L2_AUDIO_FMT_S18_3LE``- 'S18_3LE'- Corresponds to SNDRV_PCM_FORMAT_S18_3LE in ALSA
- .. _V4L2-AUDIO-FMT-S18-3BE:
- ``V4L2_AUDIO_FMT_S18_3BE``- 'S18_3BE'- Corresponds to SNDRV_PCM_FORMAT_S18_3BE in ALSA
- .. _V4L2-AUDIO-FMT-U18-3LE:
- ``V4L2_AUDIO_FMT_U18_3LE``- 'U18_3LE'- Corresponds to SNDRV_PCM_FORMAT_U18_3LE in ALSA
- .. _V4L2-AUDIO-FMT-U18-3BE:
- ``V4L2_AUDIO_FMT_U18_3BE``- 'U18_3BE'- Corresponds to SNDRV_PCM_FORMAT_U18_3BE in ALSAdiff --git a/Documentation/userspace-api/media/v4l/pixfmt.rst b/Documentation/userspace-api/media/v4l/pixfmt.rst index 11dab4a90630..2eb6fdd3b43d 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt.rst @@ -36,3 +36,4 @@ see also :ref:`VIDIOC_G_FBUF <VIDIOC_G_FBUF>`.) colorspaces colorspaces-defs colorspaces-details
- pixfmt-audio
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 5d088e6c43e4..31e443c644db 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1471,6 +1471,42 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_Y210: descr = "10-bit YUYV Packed"; break; case V4L2_PIX_FMT_Y212: descr = "12-bit YUYV Packed"; break; case V4L2_PIX_FMT_Y216: descr = "16-bit YUYV Packed"; break;
- case V4L2_AUDIO_FMT_S8: descr = "8-bit Signed"; break;
- case V4L2_AUDIO_FMT_U8: descr = "8-bit Unsigned"; break;
- case V4L2_AUDIO_FMT_S16_LE: descr = "16-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S16_BE: descr = "16-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U16_LE: descr = "16-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U16_BE: descr = "16-bit Unsigned BE"; break;
- case V4L2_AUDIO_FMT_S24_LE: descr = "24(32)-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S24_BE: descr = "24(32)-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U24_LE: descr = "24(32)-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U24_BE: descr = "24(32)-bit Unsigned BE"; break;
- case V4L2_AUDIO_FMT_S32_LE: descr = "32-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S32_BE: descr = "32-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U32_LE: descr = "32-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U32_BE: descr = "32-bit Unsigned BE"; break;
- case V4L2_AUDIO_FMT_FLOAT_LE: descr = "32-bit Float LE"; break;
- case V4L2_AUDIO_FMT_FLOAT_BE: descr = "32-bit Float BE"; break;
- case V4L2_AUDIO_FMT_FLOAT64_LE: descr = "64-bit Float LE"; break;
- case V4L2_AUDIO_FMT_FLOAT64_BE: descr = "64-bit Float BE"; break;
- case V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE: descr = "32-bit IEC958 LE"; break;
- case V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE: descr = "32-bit IEC958 BE"; break;
- case V4L2_AUDIO_FMT_S20_LE: descr = "20-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S20_BE: descr = "20-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U20_LE: descr = "20-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U20_BE: descr = "20-bit Unsigned BE"; break;
- case V4L2_AUDIO_FMT_S24_3LE: descr = "24(24)-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S24_3BE: descr = "24(24)-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U24_3LE: descr = "24(24)-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U24_3BE: descr = "24(24)-bit Unsigned BE"; break;
- case V4L2_AUDIO_FMT_S20_3LE: descr = "20(24)-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S20_3BE: descr = "20(24)-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U20_3LE: descr = "20(24)-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U20_3BE: descr = "20(24)-bit Unsigned BE"; break;
- case V4L2_AUDIO_FMT_S18_3LE: descr = "18(24)-bit Signed LE"; break;
- case V4L2_AUDIO_FMT_S18_3BE: descr = "18(24)-bit Signed BE"; break;
- case V4L2_AUDIO_FMT_U18_3LE: descr = "18(24)-bit Unsigned LE"; break;
- case V4L2_AUDIO_FMT_U18_3BE: descr = "18(24)-bit Unsigned BE"; break;
This still has several formats that are not used by the initial drivers imx-asrc and viaudm2m. Please just add the ones that are actually used.
We had bad experiences in the past with defined-but-never-used formats, so we are fairly strict about that these days.
default: /* Compressed formats */ diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index b0ddb7319d36..2ac7b989394c 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -842,6 +842,54 @@ struct v4l2_pix_format { #define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */ #define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
+/*
- Audio-data formats
- All these audio formats use a fourcc starting with 'AU'
- followed by the SNDRV_PCM_FORMAT_ value from asound.h.
- */
+#define V4L2_AUDIO_FMT_S8 v4l2_fourcc('A', 'U', '0', '0') +#define V4L2_AUDIO_FMT_U8 v4l2_fourcc('A', 'U', '0', '1') +#define V4L2_AUDIO_FMT_S16_LE v4l2_fourcc('A', 'U', '0', '2') +#define V4L2_AUDIO_FMT_S16_BE v4l2_fourcc('A', 'U', '0', '3') +#define V4L2_AUDIO_FMT_U16_LE v4l2_fourcc('A', 'U', '0', '4') +#define V4L2_AUDIO_FMT_U16_BE v4l2_fourcc('A', 'U', '0', '5') +#define V4L2_AUDIO_FMT_S24_LE v4l2_fourcc('A', 'U', '0', '6') +#define V4L2_AUDIO_FMT_S24_BE v4l2_fourcc('A', 'U', '0', '7') +#define V4L2_AUDIO_FMT_U24_LE v4l2_fourcc('A', 'U', '0', '8') +#define V4L2_AUDIO_FMT_U24_BE v4l2_fourcc('A', 'U', '0', '9')
+#define V4L2_AUDIO_FMT_S32_LE v4l2_fourcc('A', 'U', '1', '0') +#define V4L2_AUDIO_FMT_S32_BE v4l2_fourcc('A', 'U', '1', '1') +#define V4L2_AUDIO_FMT_U32_LE v4l2_fourcc('A', 'U', '1', '2') +#define V4L2_AUDIO_FMT_U32_BE v4l2_fourcc('A', 'U', '1', '3') +#define V4L2_AUDIO_FMT_FLOAT_LE v4l2_fourcc('A', 'U', '1', '4') +#define V4L2_AUDIO_FMT_FLOAT_BE v4l2_fourcc('A', 'U', '1', '5') +#define V4L2_AUDIO_FMT_FLOAT64_LE v4l2_fourcc('A', 'U', '1', '6') +#define V4L2_AUDIO_FMT_FLOAT64_BE v4l2_fourcc('A', 'U', '1', '7') +#define V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE v4l2_fourcc('A', 'U', '1', '8') +#define V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE v4l2_fourcc('A', 'U', '1', '9')
+#define V4L2_AUDIO_FMT_S20_LE v4l2_fourcc('A', 'U', '2', '5') +#define V4L2_AUDIO_FMT_S20_BE v4l2_fourcc('A', 'U', '2', '6') +#define V4L2_AUDIO_FMT_U20_LE v4l2_fourcc('A', 'U', '2', '7') +#define V4L2_AUDIO_FMT_U20_BE v4l2_fourcc('A', 'U', '2', '8')
+#define V4L2_AUDIO_FMT_S24_3LE v4l2_fourcc('A', 'U', '3', '2') +#define V4L2_AUDIO_FMT_S24_3BE v4l2_fourcc('A', 'U', '3', '3') +#define V4L2_AUDIO_FMT_U24_3LE v4l2_fourcc('A', 'U', '3', '4') +#define V4L2_AUDIO_FMT_U24_3BE v4l2_fourcc('A', 'U', '3', '5') +#define V4L2_AUDIO_FMT_S20_3LE v4l2_fourcc('A', 'U', '3', '6') +#define V4L2_AUDIO_FMT_S20_3BE v4l2_fourcc('A', 'U', '3', '7') +#define V4L2_AUDIO_FMT_U20_3LE v4l2_fourcc('A', 'U', '3', '8') +#define V4L2_AUDIO_FMT_U20_3BE v4l2_fourcc('A', 'U', '3', '9') +#define V4L2_AUDIO_FMT_S18_3LE v4l2_fourcc('A', 'U', '4', '0') +#define V4L2_AUDIO_FMT_S18_3BE v4l2_fourcc('A', 'U', '4', '1') +#define V4L2_AUDIO_FMT_U18_3LE v4l2_fourcc('A', 'U', '4', '2') +#define V4L2_AUDIO_FMT_U18_3BE v4l2_fourcc('A', 'U', '4', '3')
+#define v4l2_fourcc_to_audfmt(X) \
- (((((X) >> 16) & 0xFF) - '0') * 10 + ((((X) >> 24) & 0xFF) - '0'))
X -> fourcc 0xFF -> 0xff
/* priv field value to indicates that subsequent fields are valid. */ #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
Regards,
Hans
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- .../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry + ext-ctrls-fixed-point fourcc format planar-apis diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later + +.. _fixed-point-controls: + +*************************** +Fixed Point Control Reference +*************************** + +These controls are intended to support an asynchronous sample +rate converter. + +.. _v4l2-audio-asrc: + +``V4L2_CID_ASRC_SOURCE_RATE`` + sets the resampler source rate. + +``V4L2_CID_ASRC_DEST_RATE`` + sets the resampler destination rate. + +.. c:type:: v4l2_ctrl_fixed_point + +.. cssclass:: longtable + +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| + +.. flat-table:: struct v4l2_ctrl_fixed_point + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - __u32 + - ``integer`` + - integer part of fixed point value. + * - __s32 + - ``fractional`` + - fractional part of fixed point value, which is Q31. diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. + * - struct :c:type:`v4l2_ctrl_fixed_point` * + - ``p_fixed_point`` + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. * - void * - ``ptr`` - A pointer to a compound type which can be an N-dimensional array diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders. + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` + - n/a + - n/a + - n/a + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has + integer part and fractional part, i.e. audio sample rate. +
.. raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break; + case V4L2_CTRL_TYPE_FIXED_POINT: + break;
default: return -EINVAL; @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break; + case V4L2_CTRL_TYPE_FIXED_POINT: + elem_size = sizeof(struct v4l2_ctrl_fixed_point); + break; default: if (type < V4L2_CTRL_COMPOUND_TYPES) elem_size = sizeof(s32); diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break; + case V4L2_CID_ASRC_SOURCE_RATE: + case V4L2_CID_ASRC_DEST_RATE: + *type = V4L2_CTRL_TYPE_FIXED_POINT; + break; default: *type = V4L2_CTRL_TYPE_INTEGER; break; diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device; * @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure. * @p_av1_frame: Pointer to an AV1 frame structure. * @p_av1_film_grain: Pointer to an AV1 film grain structure. + * @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point. * @p: Pointer to a compound value. * @p_const: Pointer to a constant compound value. */ @@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain; + struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const; }; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/** + * struct v4l2_ctrl_fixed_point - fixed point parameter. + * + * @rate_integer: integer part of fixed point value. + * @rate_fractional: fractional part of fixed point value + */ +struct v4l2_ctrl_fixed_point { + __u32 integer; + __u32 fractional; +}; + #endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain; + struct v4l2_ctrl_fixed_point __user *p_fixed_point; void __user *ptr; }; } __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283, + + V4L2_CTRL_TYPE_FIXED_POINT = 0x290, };
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote:
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry
- ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
fourcc format planar-apisdiff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _fixed-point-controls:
+*************************** +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
+***************************
+These controls are intended to support an asynchronous sample +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
+.. _v4l2-audio-asrc:
+``V4L2_CID_ASRC_SOURCE_RATE``
- sets the resampler source rate.
+``V4L2_CID_ASRC_DEST_RATE``
- sets the resampler destination rate.
Document the unit (Hz) for these two controls.
+.. c:type:: v4l2_ctrl_fixed_point
+.. cssclass:: longtable
+.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}|
+.. flat-table:: struct v4l2_ctrl_fixed_point
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
Hmm, shouldn't this be __s32?
- ``integer``- integer part of fixed point value.
- __s32
and this __u32?
You want to be able to use this generic type as a signed value.
- ``fractional``- fractional part of fixed point value, which is Q31.diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``.
- struct :c:type:`v4l2_ctrl_fixed_point` *
- ``p_fixed_point``- A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control isof type ``V4L2_CTRL_TYPE_FIXED_POINT``.
- void *
- ``ptr``
- A pointer to a compound type which can be an N-dimensional array
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders.
- ``V4L2_CTRL_TYPE_FIXED_POINT``
- n/a- n/a- n/a- A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which hasinteger part and fractional part, i.e. audio sample rate... raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break;
- case V4L2_CTRL_TYPE_FIXED_POINT:
break;
Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,
(10, because it is no longer a compound type).
default: return -EINVAL; @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break;
- case V4L2_CTRL_TYPE_FIXED_POINT:
elem_size = sizeof(struct v4l2_ctrl_fixed_point);break;diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;
- case V4L2_CID_ASRC_SOURCE_RATE:
- case V4L2_CID_ASRC_DEST_RATE:
*type = V4L2_CTRL_TYPE_FIXED_POINT;break;diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device;
- @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure.
- @p_av1_frame: Pointer to an AV1 frame structure.
- @p_av1_film_grain: Pointer to an AV1 film grain structure.
*/
- @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point.
- @p: Pointer to a compound value.
- @p_const: Pointer to a constant compound value.
@@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain;
- struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const;
}; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
1) Add the new M2M_AUDIO control class, 2) Add the new V4L2_CTRL_TYPE_FIXED_POINT type, 3) Add the new controls.
These are all independent changes, so separating them makes it easier to review.
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/**
- struct v4l2_ctrl_fixed_point - fixed point parameter.
- @rate_integer: integer part of fixed point value.
- @rate_fractional: fractional part of fixed point value
- */
+struct v4l2_ctrl_fixed_point {
- __u32 integer;
__s32?
- __u32 fractional;
+};
#endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain;
struct v4l2_ctrl_fixed_point __user *p_fixed_point;} __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283,
- V4L2_CTRL_TYPE_FIXED_POINT = 0x290,
};
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote:
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote:
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry
- ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
fourcc format planar-apisdiff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _fixed-point-controls:
+*************************** +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
+***************************
+These controls are intended to support an asynchronous sample +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
+.. _v4l2-audio-asrc:
+``V4L2_CID_ASRC_SOURCE_RATE``
- sets the resampler source rate.
+``V4L2_CID_ASRC_DEST_RATE``
- sets the resampler destination rate.
Document the unit (Hz) for these two controls.
+.. c:type:: v4l2_ctrl_fixed_point
+.. cssclass:: longtable
+.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}|
+.. flat-table:: struct v4l2_ctrl_fixed_point
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
Hmm, shouldn't this be __s32?
- ``integer``- integer part of fixed point value.
- __s32
and this __u32?
You want to be able to use this generic type as a signed value.
- ``fractional``- fractional part of fixed point value, which is Q31.diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``.
- struct :c:type:`v4l2_ctrl_fixed_point` *
- ``p_fixed_point``- A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control isof type ``V4L2_CTRL_TYPE_FIXED_POINT``.
- void *
- ``ptr``
- A pointer to a compound type which can be an N-dimensional array
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders.
- ``V4L2_CTRL_TYPE_FIXED_POINT``
- n/a- n/a- n/a- A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which hasinteger part and fractional part, i.e. audio sample rate... raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break;
case V4L2_CTRL_TYPE_FIXED_POINT:break;Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,(10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
best regards wang shengjiu
default: return -EINVAL;@@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break;
case V4L2_CTRL_TYPE_FIXED_POINT:elem_size = sizeof(struct v4l2_ctrl_fixed_point);break; default: if (type < V4L2_CTRL_COMPOUND_TYPES) elem_size = sizeof(s32);diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;
case V4L2_CID_ASRC_SOURCE_RATE:case V4L2_CID_ASRC_DEST_RATE:*type = V4L2_CTRL_TYPE_FIXED_POINT;break; default: *type = V4L2_CTRL_TYPE_INTEGER; break;diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device;
- @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure.
- @p_av1_frame: Pointer to an AV1 frame structure.
- @p_av1_film_grain: Pointer to an AV1 film grain structure.
*/
- @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point.
- @p: Pointer to a compound value.
- @p_const: Pointer to a constant compound value.
@@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain;
struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const;}; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
- Add the new M2M_AUDIO control class,
- Add the new V4L2_CTRL_TYPE_FIXED_POINT type,
- Add the new controls.
These are all independent changes, so separating them makes it easier to review.
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/**
- struct v4l2_ctrl_fixed_point - fixed point parameter.
- @rate_integer: integer part of fixed point value.
- @rate_fractional: fractional part of fixed point value
- */
+struct v4l2_ctrl_fixed_point {
__u32 integer;__s32?
__u32 fractional;+};
#endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain;
struct v4l2_ctrl_fixed_point __user *p_fixed_point; void __user *ptr; };} __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283,
V4L2_CTRL_TYPE_FIXED_POINT = 0x290,};
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On 17/10/2023 15:11, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote:
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote:
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry
- ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
fourcc format planar-apisdiff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _fixed-point-controls:
+*************************** +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
+***************************
+These controls are intended to support an asynchronous sample +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
+.. _v4l2-audio-asrc:
+``V4L2_CID_ASRC_SOURCE_RATE``
- sets the resampler source rate.
+``V4L2_CID_ASRC_DEST_RATE``
- sets the resampler destination rate.
Document the unit (Hz) for these two controls.
+.. c:type:: v4l2_ctrl_fixed_point
+.. cssclass:: longtable
+.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}|
+.. flat-table:: struct v4l2_ctrl_fixed_point
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
Hmm, shouldn't this be __s32?
- ``integer``- integer part of fixed point value.
- __s32
and this __u32?
You want to be able to use this generic type as a signed value.
- ``fractional``- fractional part of fixed point value, which is Q31.diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``.
- struct :c:type:`v4l2_ctrl_fixed_point` *
- ``p_fixed_point``- A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control isof type ``V4L2_CTRL_TYPE_FIXED_POINT``.
- void *
- ``ptr``
- A pointer to a compound type which can be an N-dimensional array
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders.
- ``V4L2_CTRL_TYPE_FIXED_POINT``
- n/a- n/a- n/a- A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which hasinteger part and fractional part, i.e. audio sample rate... raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break;
case V4L2_CTRL_TYPE_FIXED_POINT:break;Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,(10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
Regards,
Hans
best regards wang shengjiu
default: return -EINVAL;@@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break;
case V4L2_CTRL_TYPE_FIXED_POINT:elem_size = sizeof(struct v4l2_ctrl_fixed_point);break; default: if (type < V4L2_CTRL_COMPOUND_TYPES) elem_size = sizeof(s32);diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;
case V4L2_CID_ASRC_SOURCE_RATE:case V4L2_CID_ASRC_DEST_RATE:*type = V4L2_CTRL_TYPE_FIXED_POINT;break; default: *type = V4L2_CTRL_TYPE_INTEGER; break;diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device;
- @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure.
- @p_av1_frame: Pointer to an AV1 frame structure.
- @p_av1_film_grain: Pointer to an AV1 film grain structure.
*/
- @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point.
- @p: Pointer to a compound value.
- @p_const: Pointer to a constant compound value.
@@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain;
struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const;}; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
- Add the new M2M_AUDIO control class,
- Add the new V4L2_CTRL_TYPE_FIXED_POINT type,
- Add the new controls.
These are all independent changes, so separating them makes it easier to review.
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/**
- struct v4l2_ctrl_fixed_point - fixed point parameter.
- @rate_integer: integer part of fixed point value.
- @rate_fractional: fractional part of fixed point value
- */
+struct v4l2_ctrl_fixed_point {
__u32 integer;__s32?
__u32 fractional;+};
#endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain;
struct v4l2_ctrl_fixed_point __user *p_fixed_point; void __user *ptr; };} __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283,
V4L2_CTRL_TYPE_FIXED_POINT = 0x290,};
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote:
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote:
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry
- ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
fourcc format planar-apisdiff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _fixed-point-controls:
+*************************** +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
+***************************
+These controls are intended to support an asynchronous sample +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
+.. _v4l2-audio-asrc:
+``V4L2_CID_ASRC_SOURCE_RATE``
- sets the resampler source rate.
+``V4L2_CID_ASRC_DEST_RATE``
- sets the resampler destination rate.
Document the unit (Hz) for these two controls.
+.. c:type:: v4l2_ctrl_fixed_point
+.. cssclass:: longtable
+.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}|
+.. flat-table:: struct v4l2_ctrl_fixed_point
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
Hmm, shouldn't this be __s32?
- ``integer``- integer part of fixed point value.
- __s32
and this __u32?
You want to be able to use this generic type as a signed value.
- ``fractional``- fractional part of fixed point value, which is Q31.diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``.
- struct :c:type:`v4l2_ctrl_fixed_point` *
- ``p_fixed_point``- A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control isof type ``V4L2_CTRL_TYPE_FIXED_POINT``.
- void *
- ``ptr``
- A pointer to a compound type which can be an N-dimensional array
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders.
- ``V4L2_CTRL_TYPE_FIXED_POINT``
- n/a- n/a- n/a- A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which hasinteger part and fractional part, i.e. audio sample rate... raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break;
case V4L2_CTRL_TYPE_FIXED_POINT:break;Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,(10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Best regards Wang Shengjiu
Regards,
Hansbest regards wang shengjiu
default: return -EINVAL;@@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break;
case V4L2_CTRL_TYPE_FIXED_POINT:elem_size = sizeof(struct v4l2_ctrl_fixed_point);break; default: if (type < V4L2_CTRL_COMPOUND_TYPES) elem_size = sizeof(s32);diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;
case V4L2_CID_ASRC_SOURCE_RATE:case V4L2_CID_ASRC_DEST_RATE:*type = V4L2_CTRL_TYPE_FIXED_POINT;break; default: *type = V4L2_CTRL_TYPE_INTEGER; break;diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device;
- @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure.
- @p_av1_frame: Pointer to an AV1 frame structure.
- @p_av1_film_grain: Pointer to an AV1 film grain structure.
*/
- @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point.
- @p: Pointer to a compound value.
- @p_const: Pointer to a constant compound value.
@@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain;
struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const;}; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
- Add the new M2M_AUDIO control class,
- Add the new V4L2_CTRL_TYPE_FIXED_POINT type,
- Add the new controls.
These are all independent changes, so separating them makes it easier to review.
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/**
- struct v4l2_ctrl_fixed_point - fixed point parameter.
- @rate_integer: integer part of fixed point value.
- @rate_fractional: fractional part of fixed point value
- */
+struct v4l2_ctrl_fixed_point {
__u32 integer;__s32?
__u32 fractional;+};
#endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain;
struct v4l2_ctrl_fixed_point __user *p_fixed_point; void __user *ptr; };} __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283,
V4L2_CTRL_TYPE_FIXED_POINT = 0x290,};
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote:
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote:
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry
- ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
fourcc format planar-apisdiff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _fixed-point-controls:
+*************************** +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
+***************************
+These controls are intended to support an asynchronous sample +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
+.. _v4l2-audio-asrc:
+``V4L2_CID_ASRC_SOURCE_RATE``
- sets the resampler source rate.
+``V4L2_CID_ASRC_DEST_RATE``
- sets the resampler destination rate.
Document the unit (Hz) for these two controls.
+.. c:type:: v4l2_ctrl_fixed_point
+.. cssclass:: longtable
+.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}|
+.. flat-table:: struct v4l2_ctrl_fixed_point
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
Hmm, shouldn't this be __s32?
- ``integer``- integer part of fixed point value.
- __s32
and this __u32?
You want to be able to use this generic type as a signed value.
- ``fractional``- fractional part of fixed point value, which is Q31.diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``.
- struct :c:type:`v4l2_ctrl_fixed_point` *
- ``p_fixed_point``- A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control isof type ``V4L2_CTRL_TYPE_FIXED_POINT``.
- void *
- ``ptr``
- A pointer to a compound type which can be an N-dimensional array
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders.
- ``V4L2_CTRL_TYPE_FIXED_POINT``
- n/a- n/a- n/a- A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which hasinteger part and fractional part, i.e. audio sample rate... raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break;
case V4L2_CTRL_TYPE_FIXED_POINT:break;Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,(10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
All of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
best regards wang shengjiu
Best regards Wang Shengjiu
Regards,
Hansbest regards wang shengjiu
default: return -EINVAL;@@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break;
case V4L2_CTRL_TYPE_FIXED_POINT:elem_size = sizeof(struct v4l2_ctrl_fixed_point);break; default: if (type < V4L2_CTRL_COMPOUND_TYPES) elem_size = sizeof(s32);diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;
case V4L2_CID_ASRC_SOURCE_RATE:case V4L2_CID_ASRC_DEST_RATE:*type = V4L2_CTRL_TYPE_FIXED_POINT;break; default: *type = V4L2_CTRL_TYPE_INTEGER; break;diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device;
- @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure.
- @p_av1_frame: Pointer to an AV1 frame structure.
- @p_av1_film_grain: Pointer to an AV1 film grain structure.
*/
- @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point.
- @p: Pointer to a compound value.
- @p_const: Pointer to a constant compound value.
@@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain;
struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const;}; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
- Add the new M2M_AUDIO control class,
- Add the new V4L2_CTRL_TYPE_FIXED_POINT type,
- Add the new controls.
These are all independent changes, so separating them makes it easier to review.
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/**
- struct v4l2_ctrl_fixed_point - fixed point parameter.
- @rate_integer: integer part of fixed point value.
- @rate_fractional: fractional part of fixed point value
- */
+struct v4l2_ctrl_fixed_point {
__u32 integer;__s32?
__u32 fractional;+};
#endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain;
struct v4l2_ctrl_fixed_point __user *p_fixed_point; void __user *ptr; };} __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283,
V4L2_CTRL_TYPE_FIXED_POINT = 0x290,};
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote:
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote:
Fixed point controls are used by the user to configure the audio sample rate to driver.
Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE new IDs for ASRC rate control.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
.../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ .../media/v4l/vidioc-queryctrl.rst | 7 ++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ include/media/v4l2-ctrls.h | 2 ++ include/uapi/linux/v4l2-controls.h | 13 +++++++ include/uapi/linux/videodev2.h | 3 ++ 10 files changed, 76 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst index ea0435182e44..35707edffb13 100644 --- a/Documentation/userspace-api/media/v4l/common.rst +++ b/Documentation/userspace-api/media/v4l/common.rst @@ -52,6 +52,7 @@ applicable to all devices. ext-ctrls-fm-rx ext-ctrls-detect ext-ctrls-colorimetry
- ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
fourcc format planar-apisdiff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst new file mode 100644 index 000000000000..2ef6e250580c --- /dev/null +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. _fixed-point-controls:
+*************************** +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
+***************************
+These controls are intended to support an asynchronous sample +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
+.. _v4l2-audio-asrc:
+``V4L2_CID_ASRC_SOURCE_RATE``
- sets the resampler source rate.
+``V4L2_CID_ASRC_DEST_RATE``
- sets the resampler destination rate.
Document the unit (Hz) for these two controls.
+.. c:type:: v4l2_ctrl_fixed_point
+.. cssclass:: longtable
+.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}|
+.. flat-table:: struct v4l2_ctrl_fixed_point
- :header-rows: 0
- :stub-columns: 0
- :widths: 1 1 2
- __u32
Hmm, shouldn't this be __s32?
- ``integer``- integer part of fixed point value.
- __s32
and this __u32?
You want to be able to use this generic type as a signed value.
- ``fractional``- fractional part of fixed point value, which is Q31.diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index f9f73530a6be..1811dabf5c74 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -295,6 +295,10 @@ still cause this situation. - ``p_av1_film_grain`` - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``.
- struct :c:type:`v4l2_ctrl_fixed_point` *
- ``p_fixed_point``- A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control isof type ``V4L2_CTRL_TYPE_FIXED_POINT``.
- void *
- ``ptr``
- A pointer to a compound type which can be an N-dimensional array
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index 4d38acafe8e1..9285f4f39eed 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain parameters for stateless video decoders.
- ``V4L2_CTRL_TYPE_FIXED_POINT``
- n/a- n/a- n/a- A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which hasinteger part and fractional part, i.e. audio sample rate... raw:: latex
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index e61152bb80d1..2faa5a2015eb 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type`
# V4L2 capability defines replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index a662fb60f73f..7a616ac91059 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, if (!area->width || !area->height) return -EINVAL; break;
case V4L2_CTRL_TYPE_FIXED_POINT:break;Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,(10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
Regards,
Hans
All of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
best regards wang shengjiu
Best regards Wang Shengjiu
Regards,
Hansbest regards wang shengjiu
default: return -EINVAL;@@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_AREA: elem_size = sizeof(struct v4l2_area); break;
case V4L2_CTRL_TYPE_FIXED_POINT:elem_size = sizeof(struct v4l2_ctrl_fixed_point);break; default: if (type < V4L2_CTRL_COMPOUND_TYPES) elem_size = sizeof(s32);diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 8696eb1cdd61..d8f232df6b6a 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; break;
case V4L2_CID_ASRC_SOURCE_RATE:case V4L2_CID_ASRC_DEST_RATE:*type = V4L2_CTRL_TYPE_FIXED_POINT;break; default: *type = V4L2_CTRL_TYPE_INTEGER; break;diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h index 59679a42b3e7..645e4cccafc7 100644 --- a/include/media/v4l2-ctrls.h +++ b/include/media/v4l2-ctrls.h @@ -56,6 +56,7 @@ struct video_device;
- @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure.
- @p_av1_frame: Pointer to an AV1 frame structure.
- @p_av1_film_grain: Pointer to an AV1 film grain structure.
*/
- @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point.
- @p: Pointer to a compound value.
- @p_const: Pointer to a constant compound value.
@@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame *p_av1_frame; struct v4l2_ctrl_av1_film_grain *p_av1_film_grain;
struct v4l2_ctrl_fixed_point *p_fixed_point; void *p; const void *p_const;}; diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index c3604a0a3e30..91096259e3ea 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -112,6 +112,8 @@ enum v4l2_colorfx {
/* last CID + 1 */ #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
- Add the new M2M_AUDIO control class,
- Add the new V4L2_CTRL_TYPE_FIXED_POINT type,
- Add the new controls.
These are all independent changes, so separating them makes it easier to review.
/* USER-class private control IDs */
@@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE #endif
+/**
- struct v4l2_ctrl_fixed_point - fixed point parameter.
- @rate_integer: integer part of fixed point value.
- @rate_fractional: fractional part of fixed point value
- */
+struct v4l2_ctrl_fixed_point {
__u32 integer;__s32?
__u32 fractional;+};
#endif diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 2ac7b989394c..3ef32c09c2fa 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; struct v4l2_ctrl_av1_frame __user *p_av1_frame; struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain;
struct v4l2_ctrl_fixed_point __user *p_fixed_point; void __user *ptr; };} __attribute__ ((packed)); @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, V4L2_CTRL_TYPE_AV1_FRAME = 0x282, V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283,
V4L2_CTRL_TYPE_FIXED_POINT = 0x290,};
/* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote:
Hi Shengjiu,
On 13/10/2023 10:31, Shengjiu Wang wrote: > Fixed point controls are used by the user to configure > the audio sample rate to driver. > > Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE > new IDs for ASRC rate control. > > Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com > --- > .../userspace-api/media/v4l/common.rst | 1 + > .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ > .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ > .../media/v4l/vidioc-queryctrl.rst | 7 ++++ > .../media/videodev2.h.rst.exceptions | 1 + > drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ > include/media/v4l2-ctrls.h | 2 ++ > include/uapi/linux/v4l2-controls.h | 13 +++++++ > include/uapi/linux/videodev2.h | 3 ++ > 10 files changed, 76 insertions(+) > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst > > diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst > index ea0435182e44..35707edffb13 100644 > --- a/Documentation/userspace-api/media/v4l/common.rst > +++ b/Documentation/userspace-api/media/v4l/common.rst > @@ -52,6 +52,7 @@ applicable to all devices. > ext-ctrls-fm-rx > ext-ctrls-detect > ext-ctrls-colorimetry > + ext-ctrls-fixed-point
Rename this to ext-ctrls-audio-m2m.
> fourcc > format > planar-apis > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst > new file mode 100644 > index 000000000000..2ef6e250580c > --- /dev/null > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst > @@ -0,0 +1,36 @@ > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later > + > +.. _fixed-point-controls: > + > +*************************** > +Fixed Point Control Reference
This is for audio controls. "Fixed Point" is just the type, and it doesn't make sense to group fixed point controls. But it does make sense to group the audio controls.
V4L2 controls can be grouped into classes. Basically it is a way to put controls into categories, and for each category there is also a control that gives a description of the class (see 2.15.15 in https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...)
If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that they are grouped based on what class of control they are.
So I think it would be a good idea to create a new control class for M2M audio controls, instead of just adding them to the catch-all 'User Controls' class.
Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how it is done.
M2M_AUDIO would probably be a good name for the class.
> +*************************** > + > +These controls are intended to support an asynchronous sample > +rate converter.
Add ' (ASRC).' at the end to indicate the common abbreviation for that.
> + > +.. _v4l2-audio-asrc: > + > +``V4L2_CID_ASRC_SOURCE_RATE`` > + sets the resampler source rate. > + > +``V4L2_CID_ASRC_DEST_RATE`` > + sets the resampler destination rate.
Document the unit (Hz) for these two controls.
> + > +.. c:type:: v4l2_ctrl_fixed_point > + > +.. cssclass:: longtable > + > +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| > + > +.. flat-table:: struct v4l2_ctrl_fixed_point > + :header-rows: 0 > + :stub-columns: 0 > + :widths: 1 1 2 > + > + * - __u32
Hmm, shouldn't this be __s32?
> + - ``integer`` > + - integer part of fixed point value. > + * - __s32
and this __u32?
You want to be able to use this generic type as a signed value.
> + - ``fractional`` > + - fractional part of fixed point value, which is Q31. > diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst > index f9f73530a6be..1811dabf5c74 100644 > --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst > +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst > @@ -295,6 +295,10 @@ still cause this situation. > - ``p_av1_film_grain`` > - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is > of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. > + * - struct :c:type:`v4l2_ctrl_fixed_point` * > + - ``p_fixed_point`` > + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is > + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. > * - void * > - ``ptr`` > - A pointer to a compound type which can be an N-dimensional array > diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst > index 4d38acafe8e1..9285f4f39eed 100644 > --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst > +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst > @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. > - n/a > - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain > parameters for stateless video decoders. > + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` > + - n/a > + - n/a > + - n/a > + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has > + integer part and fractional part, i.e. audio sample rate. > + > > .. raw:: latex > > diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions > index e61152bb80d1..2faa5a2015eb 100644 > --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions > +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions > @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` > replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` > replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` > replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` > +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` > > # V4L2 capability defines > replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities > diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c > index a662fb60f73f..7a616ac91059 100644 > --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c > +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c > @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, > if (!area->width || !area->height) > return -EINVAL; > break; > + case V4L2_CTRL_TYPE_FIXED_POINT: > + break;
Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL':
https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040...
since min and max values are perfectly fine for a fixed point value.
Even a step value (currently not supported in that patch) would make sense.
But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, why not represent the fixed point value as a Q31.32. Then the standard minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64.
Except that both userspace and drivers need to multiply it with 2^-32 to get the actual value.
So in enum v4l2_ctrl_type add:
V4L2_CTRL_TYPE_FIXED_POINT = 10,(10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
best regards wang shengjiu
Best regards Wang Shengjiu
Regards,
Hansbest regards wang shengjiu
> > default: > return -EINVAL; > @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, > case V4L2_CTRL_TYPE_AREA: > elem_size = sizeof(struct v4l2_area); > break; > + case V4L2_CTRL_TYPE_FIXED_POINT: > + elem_size = sizeof(struct v4l2_ctrl_fixed_point); > + break; > default: > if (type < V4L2_CTRL_COMPOUND_TYPES) > elem_size = sizeof(s32); > diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c > index 8696eb1cdd61..d8f232df6b6a 100644 > --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c > +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c > @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, > case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: > *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; > break; > + case V4L2_CID_ASRC_SOURCE_RATE: > + case V4L2_CID_ASRC_DEST_RATE: > + *type = V4L2_CTRL_TYPE_FIXED_POINT; > + break; > default: > *type = V4L2_CTRL_TYPE_INTEGER; > break; > diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h > index 59679a42b3e7..645e4cccafc7 100644 > --- a/include/media/v4l2-ctrls.h > +++ b/include/media/v4l2-ctrls.h > @@ -56,6 +56,7 @@ struct video_device; > * @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure. > * @p_av1_frame: Pointer to an AV1 frame structure. > * @p_av1_film_grain: Pointer to an AV1 film grain structure. > + * @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point. > * @p: Pointer to a compound value. > * @p_const: Pointer to a constant compound value. > */ > @@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { > struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; > struct v4l2_ctrl_av1_frame *p_av1_frame; > struct v4l2_ctrl_av1_film_grain *p_av1_film_grain; > + struct v4l2_ctrl_fixed_point *p_fixed_point; > void *p; > const void *p_const; > }; > diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h > index c3604a0a3e30..91096259e3ea 100644 > --- a/include/uapi/linux/v4l2-controls.h > +++ b/include/uapi/linux/v4l2-controls.h > @@ -112,6 +112,8 @@ enum v4l2_colorfx { > > /* last CID + 1 */ > #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) > +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) > +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46)
This patch needs to be split in three parts:
- Add the new M2M_AUDIO control class,
- Add the new V4L2_CTRL_TYPE_FIXED_POINT type,
- Add the new controls.
These are all independent changes, so separating them makes it easier to review.
> > /* USER-class private control IDs */ > > @@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { > #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE > #endif > > +/** > + * struct v4l2_ctrl_fixed_point - fixed point parameter. > + * > + * @rate_integer: integer part of fixed point value. > + * @rate_fractional: fractional part of fixed point value > + */ > +struct v4l2_ctrl_fixed_point { > + __u32 integer;
__s32?
> + __u32 fractional; > +}; > + > #endif > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h > index 2ac7b989394c..3ef32c09c2fa 100644 > --- a/include/uapi/linux/videodev2.h > +++ b/include/uapi/linux/videodev2.h > @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { > struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; > struct v4l2_ctrl_av1_frame __user *p_av1_frame; > struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain; > + struct v4l2_ctrl_fixed_point __user *p_fixed_point; > void __user *ptr; > }; > } __attribute__ ((packed)); > @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { > V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, > V4L2_CTRL_TYPE_AV1_FRAME = 0x282, > V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283, > + > + V4L2_CTRL_TYPE_FIXED_POINT = 0x290, > }; > > /* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */
Regards,
Hans
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: > > Hi Shengjiu, > > On 13/10/2023 10:31, Shengjiu Wang wrote: >> Fixed point controls are used by the user to configure >> the audio sample rate to driver. >> >> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >> new IDs for ASRC rate control. >> >> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >> --- >> .../userspace-api/media/v4l/common.rst | 1 + >> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >> .../media/videodev2.h.rst.exceptions | 1 + >> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >> include/media/v4l2-ctrls.h | 2 ++ >> include/uapi/linux/v4l2-controls.h | 13 +++++++ >> include/uapi/linux/videodev2.h | 3 ++ >> 10 files changed, 76 insertions(+) >> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >> >> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >> index ea0435182e44..35707edffb13 100644 >> --- a/Documentation/userspace-api/media/v4l/common.rst >> +++ b/Documentation/userspace-api/media/v4l/common.rst >> @@ -52,6 +52,7 @@ applicable to all devices. >> ext-ctrls-fm-rx >> ext-ctrls-detect >> ext-ctrls-colorimetry >> + ext-ctrls-fixed-point > > Rename this to ext-ctrls-audio-m2m. > >> fourcc >> format >> planar-apis >> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >> new file mode 100644 >> index 000000000000..2ef6e250580c >> --- /dev/null >> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >> @@ -0,0 +1,36 @@ >> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >> + >> +.. _fixed-point-controls: >> + >> +*************************** >> +Fixed Point Control Reference > > This is for audio controls. "Fixed Point" is just the type, and it doesn't make > sense to group fixed point controls. But it does make sense to group the audio > controls. > > V4L2 controls can be grouped into classes. Basically it is a way to put controls > into categories, and for each category there is also a control that gives a > description of the class (see 2.15.15 in > https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) > > If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that > they are grouped based on what class of control they are. > > So I think it would be a good idea to create a new control class for M2M audio controls, > instead of just adding them to the catch-all 'User Controls' class. > > Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how > it is done. > > M2M_AUDIO would probably be a good name for the class. > >> +*************************** >> + >> +These controls are intended to support an asynchronous sample >> +rate converter. > > Add ' (ASRC).' at the end to indicate the common abbreviation for > that. > >> + >> +.. _v4l2-audio-asrc: >> + >> +``V4L2_CID_ASRC_SOURCE_RATE`` >> + sets the resampler source rate. >> + >> +``V4L2_CID_ASRC_DEST_RATE`` >> + sets the resampler destination rate. > > Document the unit (Hz) for these two controls. > >> + >> +.. c:type:: v4l2_ctrl_fixed_point >> + >> +.. cssclass:: longtable >> + >> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >> + >> +.. flat-table:: struct v4l2_ctrl_fixed_point >> + :header-rows: 0 >> + :stub-columns: 0 >> + :widths: 1 1 2 >> + >> + * - __u32 > > Hmm, shouldn't this be __s32? > >> + - ``integer`` >> + - integer part of fixed point value. >> + * - __s32 > > and this __u32? > > You want to be able to use this generic type as a signed value. > >> + - ``fractional`` >> + - fractional part of fixed point value, which is Q31. >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >> index f9f73530a6be..1811dabf5c74 100644 >> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >> @@ -295,6 +295,10 @@ still cause this situation. >> - ``p_av1_film_grain`` >> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >> + - ``p_fixed_point`` >> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >> * - void * >> - ``ptr`` >> - A pointer to a compound type which can be an N-dimensional array >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >> index 4d38acafe8e1..9285f4f39eed 100644 >> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >> - n/a >> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >> parameters for stateless video decoders. >> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >> + - n/a >> + - n/a >> + - n/a >> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >> + integer part and fractional part, i.e. audio sample rate. >> + >> >> .. raw:: latex >> >> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >> index e61152bb80d1..2faa5a2015eb 100644 >> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >> >> # V4L2 capability defines >> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >> index a662fb60f73f..7a616ac91059 100644 >> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >> if (!area->width || !area->height) >> return -EINVAL; >> break; >> + case V4L2_CTRL_TYPE_FIXED_POINT: >> + break; > > Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': > > https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... > > since min and max values are perfectly fine for a fixed point value. > > Even a step value (currently not supported in that patch) would make sense. > > But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, > why not represent the fixed point value as a Q31.32. Then the standard > minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. > > Except that both userspace and drivers need to multiply it with 2^-32 to get the actual > value. > > So in enum v4l2_ctrl_type add: > > V4L2_CTRL_TYPE_FIXED_POINT = 10, > > (10, because it is no longer a compound type).
Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64?
The reason I use the 'integer' and 'fractional' is that I want 'integer' to be the normal sample rate, for example 48kHz. The 'fractional' is the difference with normal sample rate.
For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655.
So if we use s64 for rate, then in driver need to convert the rate to the closed normal sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
Regards,
Hans
best regards wang shengjiu
Best regards Wang Shengjiu
Regards,
Hansbest regards wang shengjiu
> >> >> default: >> return -EINVAL; >> @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, >> case V4L2_CTRL_TYPE_AREA: >> elem_size = sizeof(struct v4l2_area); >> break; >> + case V4L2_CTRL_TYPE_FIXED_POINT: >> + elem_size = sizeof(struct v4l2_ctrl_fixed_point); >> + break; >> default: >> if (type < V4L2_CTRL_COMPOUND_TYPES) >> elem_size = sizeof(s32); >> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >> index 8696eb1cdd61..d8f232df6b6a 100644 >> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c >> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >> @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, >> case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: >> *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; >> break; >> + case V4L2_CID_ASRC_SOURCE_RATE: >> + case V4L2_CID_ASRC_DEST_RATE: >> + *type = V4L2_CTRL_TYPE_FIXED_POINT; >> + break; >> default: >> *type = V4L2_CTRL_TYPE_INTEGER; >> break; >> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h >> index 59679a42b3e7..645e4cccafc7 100644 >> --- a/include/media/v4l2-ctrls.h >> +++ b/include/media/v4l2-ctrls.h >> @@ -56,6 +56,7 @@ struct video_device; >> * @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure. >> * @p_av1_frame: Pointer to an AV1 frame structure. >> * @p_av1_film_grain: Pointer to an AV1 film grain structure. >> + * @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point. >> * @p: Pointer to a compound value. >> * @p_const: Pointer to a constant compound value. >> */ >> @@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { >> struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; >> struct v4l2_ctrl_av1_frame *p_av1_frame; >> struct v4l2_ctrl_av1_film_grain *p_av1_film_grain; >> + struct v4l2_ctrl_fixed_point *p_fixed_point; >> void *p; >> const void *p_const; >> }; >> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h >> index c3604a0a3e30..91096259e3ea 100644 >> --- a/include/uapi/linux/v4l2-controls.h >> +++ b/include/uapi/linux/v4l2-controls.h >> @@ -112,6 +112,8 @@ enum v4l2_colorfx { >> >> /* last CID + 1 */ >> #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) >> +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) >> +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46) > > This patch needs to be split in three parts: > > 1) Add the new M2M_AUDIO control class, > 2) Add the new V4L2_CTRL_TYPE_FIXED_POINT type, > 3) Add the new controls. > > These are all independent changes, so separating them makes it easier to > review. > >> >> /* USER-class private control IDs */ >> >> @@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { >> #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE >> #endif >> >> +/** >> + * struct v4l2_ctrl_fixed_point - fixed point parameter. >> + * >> + * @rate_integer: integer part of fixed point value. >> + * @rate_fractional: fractional part of fixed point value >> + */ >> +struct v4l2_ctrl_fixed_point { >> + __u32 integer; > > __s32? > >> + __u32 fractional; >> +}; >> + >> #endif >> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h >> index 2ac7b989394c..3ef32c09c2fa 100644 >> --- a/include/uapi/linux/videodev2.h >> +++ b/include/uapi/linux/videodev2.h >> @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { >> struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; >> struct v4l2_ctrl_av1_frame __user *p_av1_frame; >> struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain; >> + struct v4l2_ctrl_fixed_point __user *p_fixed_point; >> void __user *ptr; >> }; >> } __attribute__ ((packed)); >> @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { >> V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, >> V4L2_CTRL_TYPE_AV1_FRAME = 0x282, >> V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283, >> + >> + V4L2_CTRL_TYPE_FIXED_POINT = 0x290, >> }; >> >> /* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */ > > Regards, > > Hans
On Wed, Oct 18, 2023 at 3:58 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote: > On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: >> >> Hi Shengjiu, >> >> On 13/10/2023 10:31, Shengjiu Wang wrote: >>> Fixed point controls are used by the user to configure >>> the audio sample rate to driver. >>> >>> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >>> new IDs for ASRC rate control. >>> >>> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >>> --- >>> .../userspace-api/media/v4l/common.rst | 1 + >>> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >>> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >>> .../media/videodev2.h.rst.exceptions | 1 + >>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >>> include/media/v4l2-ctrls.h | 2 ++ >>> include/uapi/linux/v4l2-controls.h | 13 +++++++ >>> include/uapi/linux/videodev2.h | 3 ++ >>> 10 files changed, 76 insertions(+) >>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>> >>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >>> index ea0435182e44..35707edffb13 100644 >>> --- a/Documentation/userspace-api/media/v4l/common.rst >>> +++ b/Documentation/userspace-api/media/v4l/common.rst >>> @@ -52,6 +52,7 @@ applicable to all devices. >>> ext-ctrls-fm-rx >>> ext-ctrls-detect >>> ext-ctrls-colorimetry >>> + ext-ctrls-fixed-point >> >> Rename this to ext-ctrls-audio-m2m. >> >>> fourcc >>> format >>> planar-apis >>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>> new file mode 100644 >>> index 000000000000..2ef6e250580c >>> --- /dev/null >>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>> @@ -0,0 +1,36 @@ >>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>> + >>> +.. _fixed-point-controls: >>> + >>> +*************************** >>> +Fixed Point Control Reference >> >> This is for audio controls. "Fixed Point" is just the type, and it doesn't make >> sense to group fixed point controls. But it does make sense to group the audio >> controls. >> >> V4L2 controls can be grouped into classes. Basically it is a way to put controls >> into categories, and for each category there is also a control that gives a >> description of the class (see 2.15.15 in >> https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) >> >> If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that >> they are grouped based on what class of control they are. >> >> So I think it would be a good idea to create a new control class for M2M audio controls, >> instead of just adding them to the catch-all 'User Controls' class. >> >> Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how >> it is done. >> >> M2M_AUDIO would probably be a good name for the class. >> >>> +*************************** >>> + >>> +These controls are intended to support an asynchronous sample >>> +rate converter. >> >> Add ' (ASRC).' at the end to indicate the common abbreviation for >> that. >> >>> + >>> +.. _v4l2-audio-asrc: >>> + >>> +``V4L2_CID_ASRC_SOURCE_RATE`` >>> + sets the resampler source rate. >>> + >>> +``V4L2_CID_ASRC_DEST_RATE`` >>> + sets the resampler destination rate. >> >> Document the unit (Hz) for these two controls. >> >>> + >>> +.. c:type:: v4l2_ctrl_fixed_point >>> + >>> +.. cssclass:: longtable >>> + >>> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >>> + >>> +.. flat-table:: struct v4l2_ctrl_fixed_point >>> + :header-rows: 0 >>> + :stub-columns: 0 >>> + :widths: 1 1 2 >>> + >>> + * - __u32 >> >> Hmm, shouldn't this be __s32? >> >>> + - ``integer`` >>> + - integer part of fixed point value. >>> + * - __s32 >> >> and this __u32? >> >> You want to be able to use this generic type as a signed value. >> >>> + - ``fractional`` >>> + - fractional part of fixed point value, which is Q31. >>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>> index f9f73530a6be..1811dabf5c74 100644 >>> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>> @@ -295,6 +295,10 @@ still cause this situation. >>> - ``p_av1_film_grain`` >>> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >>> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >>> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >>> + - ``p_fixed_point`` >>> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >>> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >>> * - void * >>> - ``ptr`` >>> - A pointer to a compound type which can be an N-dimensional array >>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>> index 4d38acafe8e1..9285f4f39eed 100644 >>> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >>> - n/a >>> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >>> parameters for stateless video decoders. >>> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >>> + - n/a >>> + - n/a >>> + - n/a >>> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >>> + integer part and fractional part, i.e. audio sample rate. >>> + >>> >>> .. raw:: latex >>> >>> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>> index e61152bb80d1..2faa5a2015eb 100644 >>> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >>> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >>> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >>> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >>> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >>> >>> # V4L2 capability defines >>> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>> index a662fb60f73f..7a616ac91059 100644 >>> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >>> if (!area->width || !area->height) >>> return -EINVAL; >>> break; >>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>> + break; >> >> Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': >> >> https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... >> >> since min and max values are perfectly fine for a fixed point value. >> >> Even a step value (currently not supported in that patch) would make sense. >> >> But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, >> why not represent the fixed point value as a Q31.32. Then the standard >> minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. >> >> Except that both userspace and drivers need to multiply it with 2^-32 to get the actual >> value. >> >> So in enum v4l2_ctrl_type add: >> >> V4L2_CTRL_TYPE_FIXED_POINT = 10, >> >> (10, because it is no longer a compound type). > > Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64? > > The reason I use the 'integer' and 'fractional' is that I want > 'integer' to be the normal sample > rate, for example 48kHz. The 'fractional' is the difference with > normal sample rate. > > For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655. > > So if we use s64 for rate, then in driver need to convert the rate to > the closed normal > sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
Right, the source and dest rates needn't change in same time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
I will use:
V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET
'OFFSET' is V4L2_CTRL_TYPE_FIXED_POINT, which is Q31.32.
best regards wang shengjiu
Regards,
Hansbest regards wang shengjiu
Best regards Wang Shengjiu
Regards,
Hans> > best regards > wang shengjiu > >> >>> >>> default: >>> return -EINVAL; >>> @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, >>> case V4L2_CTRL_TYPE_AREA: >>> elem_size = sizeof(struct v4l2_area); >>> break; >>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>> + elem_size = sizeof(struct v4l2_ctrl_fixed_point); >>> + break; >>> default: >>> if (type < V4L2_CTRL_COMPOUND_TYPES) >>> elem_size = sizeof(s32); >>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>> index 8696eb1cdd61..d8f232df6b6a 100644 >>> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>> @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, >>> case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: >>> *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; >>> break; >>> + case V4L2_CID_ASRC_SOURCE_RATE: >>> + case V4L2_CID_ASRC_DEST_RATE: >>> + *type = V4L2_CTRL_TYPE_FIXED_POINT; >>> + break; >>> default: >>> *type = V4L2_CTRL_TYPE_INTEGER; >>> break; >>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h >>> index 59679a42b3e7..645e4cccafc7 100644 >>> --- a/include/media/v4l2-ctrls.h >>> +++ b/include/media/v4l2-ctrls.h >>> @@ -56,6 +56,7 @@ struct video_device; >>> * @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure. >>> * @p_av1_frame: Pointer to an AV1 frame structure. >>> * @p_av1_film_grain: Pointer to an AV1 film grain structure. >>> + * @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point. >>> * @p: Pointer to a compound value. >>> * @p_const: Pointer to a constant compound value. >>> */ >>> @@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { >>> struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; >>> struct v4l2_ctrl_av1_frame *p_av1_frame; >>> struct v4l2_ctrl_av1_film_grain *p_av1_film_grain; >>> + struct v4l2_ctrl_fixed_point *p_fixed_point; >>> void *p; >>> const void *p_const; >>> }; >>> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h >>> index c3604a0a3e30..91096259e3ea 100644 >>> --- a/include/uapi/linux/v4l2-controls.h >>> +++ b/include/uapi/linux/v4l2-controls.h >>> @@ -112,6 +112,8 @@ enum v4l2_colorfx { >>> >>> /* last CID + 1 */ >>> #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) >>> +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) >>> +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46) >> >> This patch needs to be split in three parts: >> >> 1) Add the new M2M_AUDIO control class, >> 2) Add the new V4L2_CTRL_TYPE_FIXED_POINT type, >> 3) Add the new controls. >> >> These are all independent changes, so separating them makes it easier to >> review. >> >>> >>> /* USER-class private control IDs */ >>> >>> @@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { >>> #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE >>> #endif >>> >>> +/** >>> + * struct v4l2_ctrl_fixed_point - fixed point parameter. >>> + * >>> + * @rate_integer: integer part of fixed point value. >>> + * @rate_fractional: fractional part of fixed point value >>> + */ >>> +struct v4l2_ctrl_fixed_point { >>> + __u32 integer; >> >> __s32? >> >>> + __u32 fractional; >>> +}; >>> + >>> #endif >>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h >>> index 2ac7b989394c..3ef32c09c2fa 100644 >>> --- a/include/uapi/linux/videodev2.h >>> +++ b/include/uapi/linux/videodev2.h >>> @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { >>> struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; >>> struct v4l2_ctrl_av1_frame __user *p_av1_frame; >>> struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain; >>> + struct v4l2_ctrl_fixed_point __user *p_fixed_point; >>> void __user *ptr; >>> }; >>> } __attribute__ ((packed)); >>> @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { >>> V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, >>> V4L2_CTRL_TYPE_AV1_FRAME = 0x282, >>> V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283, >>> + >>> + V4L2_CTRL_TYPE_FIXED_POINT = 0x290, >>> }; >>> >>> /* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */ >> >> Regards, >> >> Hans
On 18/10/2023 14:52, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:58 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote: > > On 17/10/2023 15:11, Shengjiu Wang wrote: >> On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: >>> >>> Hi Shengjiu, >>> >>> On 13/10/2023 10:31, Shengjiu Wang wrote: >>>> Fixed point controls are used by the user to configure >>>> the audio sample rate to driver. >>>> >>>> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >>>> new IDs for ASRC rate control. >>>> >>>> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >>>> --- >>>> .../userspace-api/media/v4l/common.rst | 1 + >>>> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >>>> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >>>> .../media/videodev2.h.rst.exceptions | 1 + >>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >>>> include/media/v4l2-ctrls.h | 2 ++ >>>> include/uapi/linux/v4l2-controls.h | 13 +++++++ >>>> include/uapi/linux/videodev2.h | 3 ++ >>>> 10 files changed, 76 insertions(+) >>>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>> >>>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >>>> index ea0435182e44..35707edffb13 100644 >>>> --- a/Documentation/userspace-api/media/v4l/common.rst >>>> +++ b/Documentation/userspace-api/media/v4l/common.rst >>>> @@ -52,6 +52,7 @@ applicable to all devices. >>>> ext-ctrls-fm-rx >>>> ext-ctrls-detect >>>> ext-ctrls-colorimetry >>>> + ext-ctrls-fixed-point >>> >>> Rename this to ext-ctrls-audio-m2m. >>> >>>> fourcc >>>> format >>>> planar-apis >>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>> new file mode 100644 >>>> index 000000000000..2ef6e250580c >>>> --- /dev/null >>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>> @@ -0,0 +1,36 @@ >>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>> + >>>> +.. _fixed-point-controls: >>>> + >>>> +*************************** >>>> +Fixed Point Control Reference >>> >>> This is for audio controls. "Fixed Point" is just the type, and it doesn't make >>> sense to group fixed point controls. But it does make sense to group the audio >>> controls. >>> >>> V4L2 controls can be grouped into classes. Basically it is a way to put controls >>> into categories, and for each category there is also a control that gives a >>> description of the class (see 2.15.15 in >>> https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) >>> >>> If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that >>> they are grouped based on what class of control they are. >>> >>> So I think it would be a good idea to create a new control class for M2M audio controls, >>> instead of just adding them to the catch-all 'User Controls' class. >>> >>> Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how >>> it is done. >>> >>> M2M_AUDIO would probably be a good name for the class. >>> >>>> +*************************** >>>> + >>>> +These controls are intended to support an asynchronous sample >>>> +rate converter. >>> >>> Add ' (ASRC).' at the end to indicate the common abbreviation for >>> that. >>> >>>> + >>>> +.. _v4l2-audio-asrc: >>>> + >>>> +``V4L2_CID_ASRC_SOURCE_RATE`` >>>> + sets the resampler source rate. >>>> + >>>> +``V4L2_CID_ASRC_DEST_RATE`` >>>> + sets the resampler destination rate. >>> >>> Document the unit (Hz) for these two controls. >>> >>>> + >>>> +.. c:type:: v4l2_ctrl_fixed_point >>>> + >>>> +.. cssclass:: longtable >>>> + >>>> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >>>> + >>>> +.. flat-table:: struct v4l2_ctrl_fixed_point >>>> + :header-rows: 0 >>>> + :stub-columns: 0 >>>> + :widths: 1 1 2 >>>> + >>>> + * - __u32 >>> >>> Hmm, shouldn't this be __s32? >>> >>>> + - ``integer`` >>>> + - integer part of fixed point value. >>>> + * - __s32 >>> >>> and this __u32? >>> >>> You want to be able to use this generic type as a signed value. >>> >>>> + - ``fractional`` >>>> + - fractional part of fixed point value, which is Q31. >>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>> index f9f73530a6be..1811dabf5c74 100644 >>>> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>> @@ -295,6 +295,10 @@ still cause this situation. >>>> - ``p_av1_film_grain`` >>>> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >>>> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >>>> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >>>> + - ``p_fixed_point`` >>>> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >>>> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >>>> * - void * >>>> - ``ptr`` >>>> - A pointer to a compound type which can be an N-dimensional array >>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>> index 4d38acafe8e1..9285f4f39eed 100644 >>>> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >>>> - n/a >>>> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >>>> parameters for stateless video decoders. >>>> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >>>> + - n/a >>>> + - n/a >>>> + - n/a >>>> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >>>> + integer part and fractional part, i.e. audio sample rate. >>>> + >>>> >>>> .. raw:: latex >>>> >>>> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>> index e61152bb80d1..2faa5a2015eb 100644 >>>> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >>>> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >>>> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >>>> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >>>> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >>>> >>>> # V4L2 capability defines >>>> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>> index a662fb60f73f..7a616ac91059 100644 >>>> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >>>> if (!area->width || !area->height) >>>> return -EINVAL; >>>> break; >>>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>>> + break; >>> >>> Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': >>> >>> https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... >>> >>> since min and max values are perfectly fine for a fixed point value. >>> >>> Even a step value (currently not supported in that patch) would make sense. >>> >>> But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, >>> why not represent the fixed point value as a Q31.32. Then the standard >>> minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. >>> >>> Except that both userspace and drivers need to multiply it with 2^-32 to get the actual >>> value. >>> >>> So in enum v4l2_ctrl_type add: >>> >>> V4L2_CTRL_TYPE_FIXED_POINT = 10, >>> >>> (10, because it is no longer a compound type). >> >> Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64? >> >> The reason I use the 'integer' and 'fractional' is that I want >> 'integer' to be the normal sample >> rate, for example 48kHz. The 'fractional' is the difference with >> normal sample rate. >> >> For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655. >> >> So if we use s64 for rate, then in driver need to convert the rate to >> the closed normal >> sample rate + fractional. > > That wasn't what the documentation said :-) > > So this is really two controls: one for the 'normal sample rate' (whatever 'normal' > means in this context) and the offset to the actual sample rate. > > Presumably the 'normal' sample rate is set once, while the offset changes > regularly. > > But why do you need the 'normal' sample rate? With audio resampling I assume > you resample from one rate to another, so why do you need a third 'normal' > rate? >
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
Right, the source and dest rates needn't change in same time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
I will use:
V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET
'OFFSET' is V4L2_CTRL_TYPE_FIXED_POINT, which is Q31.32.
So now I come back to my original question: why do you need both the rate and the offset? Isn't it enough to set just the rates, as long as that is in fixed point format?
Why does the driver need both the 'ideal' rate + the offset?
I'm not opposed to this, I'm just trying to understand whether this makes sense.
Can't you take e.g. the source and dest rate as starting points when you start streaming? And every time userspace updates one or both of these rates you calculate the ratio_mod compared to the previous rates?
Or is there a reason why you need the ideal rates as well? E.g. 48000 or 44100, etc.
Regards,
Hans
On Wed, Oct 18, 2023 at 9:09 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 14:52, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:58 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote: > > On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote: >> >> On 17/10/2023 15:11, Shengjiu Wang wrote: >>> On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: >>>> >>>> Hi Shengjiu, >>>> >>>> On 13/10/2023 10:31, Shengjiu Wang wrote: >>>>> Fixed point controls are used by the user to configure >>>>> the audio sample rate to driver. >>>>> >>>>> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >>>>> new IDs for ASRC rate control. >>>>> >>>>> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >>>>> --- >>>>> .../userspace-api/media/v4l/common.rst | 1 + >>>>> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >>>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >>>>> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >>>>> .../media/videodev2.h.rst.exceptions | 1 + >>>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >>>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >>>>> include/media/v4l2-ctrls.h | 2 ++ >>>>> include/uapi/linux/v4l2-controls.h | 13 +++++++ >>>>> include/uapi/linux/videodev2.h | 3 ++ >>>>> 10 files changed, 76 insertions(+) >>>>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>>> >>>>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >>>>> index ea0435182e44..35707edffb13 100644 >>>>> --- a/Documentation/userspace-api/media/v4l/common.rst >>>>> +++ b/Documentation/userspace-api/media/v4l/common.rst >>>>> @@ -52,6 +52,7 @@ applicable to all devices. >>>>> ext-ctrls-fm-rx >>>>> ext-ctrls-detect >>>>> ext-ctrls-colorimetry >>>>> + ext-ctrls-fixed-point >>>> >>>> Rename this to ext-ctrls-audio-m2m. >>>> >>>>> fourcc >>>>> format >>>>> planar-apis >>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>>> new file mode 100644 >>>>> index 000000000000..2ef6e250580c >>>>> --- /dev/null >>>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>>> @@ -0,0 +1,36 @@ >>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>> + >>>>> +.. _fixed-point-controls: >>>>> + >>>>> +*************************** >>>>> +Fixed Point Control Reference >>>> >>>> This is for audio controls. "Fixed Point" is just the type, and it doesn't make >>>> sense to group fixed point controls. But it does make sense to group the audio >>>> controls. >>>> >>>> V4L2 controls can be grouped into classes. Basically it is a way to put controls >>>> into categories, and for each category there is also a control that gives a >>>> description of the class (see 2.15.15 in >>>> https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) >>>> >>>> If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that >>>> they are grouped based on what class of control they are. >>>> >>>> So I think it would be a good idea to create a new control class for M2M audio controls, >>>> instead of just adding them to the catch-all 'User Controls' class. >>>> >>>> Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how >>>> it is done. >>>> >>>> M2M_AUDIO would probably be a good name for the class. >>>> >>>>> +*************************** >>>>> + >>>>> +These controls are intended to support an asynchronous sample >>>>> +rate converter. >>>> >>>> Add ' (ASRC).' at the end to indicate the common abbreviation for >>>> that. >>>> >>>>> + >>>>> +.. _v4l2-audio-asrc: >>>>> + >>>>> +``V4L2_CID_ASRC_SOURCE_RATE`` >>>>> + sets the resampler source rate. >>>>> + >>>>> +``V4L2_CID_ASRC_DEST_RATE`` >>>>> + sets the resampler destination rate. >>>> >>>> Document the unit (Hz) for these two controls. >>>> >>>>> + >>>>> +.. c:type:: v4l2_ctrl_fixed_point >>>>> + >>>>> +.. cssclass:: longtable >>>>> + >>>>> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >>>>> + >>>>> +.. flat-table:: struct v4l2_ctrl_fixed_point >>>>> + :header-rows: 0 >>>>> + :stub-columns: 0 >>>>> + :widths: 1 1 2 >>>>> + >>>>> + * - __u32 >>>> >>>> Hmm, shouldn't this be __s32? >>>> >>>>> + - ``integer`` >>>>> + - integer part of fixed point value. >>>>> + * - __s32 >>>> >>>> and this __u32? >>>> >>>> You want to be able to use this generic type as a signed value. >>>> >>>>> + - ``fractional`` >>>>> + - fractional part of fixed point value, which is Q31. >>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>>> index f9f73530a6be..1811dabf5c74 100644 >>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>>> @@ -295,6 +295,10 @@ still cause this situation. >>>>> - ``p_av1_film_grain`` >>>>> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >>>>> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >>>>> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >>>>> + - ``p_fixed_point`` >>>>> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >>>>> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >>>>> * - void * >>>>> - ``ptr`` >>>>> - A pointer to a compound type which can be an N-dimensional array >>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>>> index 4d38acafe8e1..9285f4f39eed 100644 >>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>>> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >>>>> - n/a >>>>> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >>>>> parameters for stateless video decoders. >>>>> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >>>>> + - n/a >>>>> + - n/a >>>>> + - n/a >>>>> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >>>>> + integer part and fractional part, i.e. audio sample rate. >>>>> + >>>>> >>>>> .. raw:: latex >>>>> >>>>> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>>> index e61152bb80d1..2faa5a2015eb 100644 >>>>> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>>> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>>> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >>>>> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >>>>> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >>>>> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >>>>> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >>>>> >>>>> # V4L2 capability defines >>>>> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >>>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>>> index a662fb60f73f..7a616ac91059 100644 >>>>> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>>> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >>>>> if (!area->width || !area->height) >>>>> return -EINVAL; >>>>> break; >>>>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>>>> + break; >>>> >>>> Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': >>>> >>>> https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... >>>> >>>> since min and max values are perfectly fine for a fixed point value. >>>> >>>> Even a step value (currently not supported in that patch) would make sense. >>>> >>>> But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, >>>> why not represent the fixed point value as a Q31.32. Then the standard >>>> minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. >>>> >>>> Except that both userspace and drivers need to multiply it with 2^-32 to get the actual >>>> value. >>>> >>>> So in enum v4l2_ctrl_type add: >>>> >>>> V4L2_CTRL_TYPE_FIXED_POINT = 10, >>>> >>>> (10, because it is no longer a compound type). >>> >>> Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64? >>> >>> The reason I use the 'integer' and 'fractional' is that I want >>> 'integer' to be the normal sample >>> rate, for example 48kHz. The 'fractional' is the difference with >>> normal sample rate. >>> >>> For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655. >>> >>> So if we use s64 for rate, then in driver need to convert the rate to >>> the closed normal >>> sample rate + fractional. >> >> That wasn't what the documentation said :-) >> >> So this is really two controls: one for the 'normal sample rate' (whatever 'normal' >> means in this context) and the offset to the actual sample rate. >> >> Presumably the 'normal' sample rate is set once, while the offset changes >> regularly. >> >> But why do you need the 'normal' sample rate? With audio resampling I assume >> you resample from one rate to another, so why do you need a third 'normal' >> rate? >> > > 'Normal' rate is used to select the prefilter table. >
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
Right, the source and dest rates needn't change in same time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
I will use:
V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET
'OFFSET' is V4L2_CTRL_TYPE_FIXED_POINT, which is Q31.32.
So now I come back to my original question: why do you need both the rate and the offset? Isn't it enough to set just the rates, as long as that is in fixed point format?
Why does the driver need both the 'ideal' rate + the offset?
I'm not opposed to this, I'm just trying to understand whether this makes sense.
Can't you take e.g. the source and dest rate as starting points when you start streaming? And every time userspace updates one or both of these rates you calculate the ratio_mod compared to the previous rates?
Or is there a reason why you need the ideal rates as well? E.g. 48000 or 44100, etc.
ideal rates is used to select prefilter table. the prefilter table is indexed by ideal rates.
The asrc has two part: prefilter and resampler filter. The convert ratio is applied to rasampler filter part.
best regards wang shengjiu
On 18/10/2023 15:50, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 9:09 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 14:52, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:58 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote: > On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote: >> >> On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote: >>> >>> On 17/10/2023 15:11, Shengjiu Wang wrote: >>>> On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: >>>>> >>>>> Hi Shengjiu, >>>>> >>>>> On 13/10/2023 10:31, Shengjiu Wang wrote: >>>>>> Fixed point controls are used by the user to configure >>>>>> the audio sample rate to driver. >>>>>> >>>>>> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >>>>>> new IDs for ASRC rate control. >>>>>> >>>>>> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >>>>>> --- >>>>>> .../userspace-api/media/v4l/common.rst | 1 + >>>>>> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >>>>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >>>>>> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >>>>>> .../media/videodev2.h.rst.exceptions | 1 + >>>>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >>>>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >>>>>> include/media/v4l2-ctrls.h | 2 ++ >>>>>> include/uapi/linux/v4l2-controls.h | 13 +++++++ >>>>>> include/uapi/linux/videodev2.h | 3 ++ >>>>>> 10 files changed, 76 insertions(+) >>>>>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>>>> >>>>>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >>>>>> index ea0435182e44..35707edffb13 100644 >>>>>> --- a/Documentation/userspace-api/media/v4l/common.rst >>>>>> +++ b/Documentation/userspace-api/media/v4l/common.rst >>>>>> @@ -52,6 +52,7 @@ applicable to all devices. >>>>>> ext-ctrls-fm-rx >>>>>> ext-ctrls-detect >>>>>> ext-ctrls-colorimetry >>>>>> + ext-ctrls-fixed-point >>>>> >>>>> Rename this to ext-ctrls-audio-m2m. >>>>> >>>>>> fourcc >>>>>> format >>>>>> planar-apis >>>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>>>> new file mode 100644 >>>>>> index 000000000000..2ef6e250580c >>>>>> --- /dev/null >>>>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>>>> @@ -0,0 +1,36 @@ >>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>> + >>>>>> +.. _fixed-point-controls: >>>>>> + >>>>>> +*************************** >>>>>> +Fixed Point Control Reference >>>>> >>>>> This is for audio controls. "Fixed Point" is just the type, and it doesn't make >>>>> sense to group fixed point controls. But it does make sense to group the audio >>>>> controls. >>>>> >>>>> V4L2 controls can be grouped into classes. Basically it is a way to put controls >>>>> into categories, and for each category there is also a control that gives a >>>>> description of the class (see 2.15.15 in >>>>> https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) >>>>> >>>>> If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that >>>>> they are grouped based on what class of control they are. >>>>> >>>>> So I think it would be a good idea to create a new control class for M2M audio controls, >>>>> instead of just adding them to the catch-all 'User Controls' class. >>>>> >>>>> Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how >>>>> it is done. >>>>> >>>>> M2M_AUDIO would probably be a good name for the class. >>>>> >>>>>> +*************************** >>>>>> + >>>>>> +These controls are intended to support an asynchronous sample >>>>>> +rate converter. >>>>> >>>>> Add ' (ASRC).' at the end to indicate the common abbreviation for >>>>> that. >>>>> >>>>>> + >>>>>> +.. _v4l2-audio-asrc: >>>>>> + >>>>>> +``V4L2_CID_ASRC_SOURCE_RATE`` >>>>>> + sets the resampler source rate. >>>>>> + >>>>>> +``V4L2_CID_ASRC_DEST_RATE`` >>>>>> + sets the resampler destination rate. >>>>> >>>>> Document the unit (Hz) for these two controls. >>>>> >>>>>> + >>>>>> +.. c:type:: v4l2_ctrl_fixed_point >>>>>> + >>>>>> +.. cssclass:: longtable >>>>>> + >>>>>> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >>>>>> + >>>>>> +.. flat-table:: struct v4l2_ctrl_fixed_point >>>>>> + :header-rows: 0 >>>>>> + :stub-columns: 0 >>>>>> + :widths: 1 1 2 >>>>>> + >>>>>> + * - __u32 >>>>> >>>>> Hmm, shouldn't this be __s32? >>>>> >>>>>> + - ``integer`` >>>>>> + - integer part of fixed point value. >>>>>> + * - __s32 >>>>> >>>>> and this __u32? >>>>> >>>>> You want to be able to use this generic type as a signed value. >>>>> >>>>>> + - ``fractional`` >>>>>> + - fractional part of fixed point value, which is Q31. >>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>>>> index f9f73530a6be..1811dabf5c74 100644 >>>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>>>> @@ -295,6 +295,10 @@ still cause this situation. >>>>>> - ``p_av1_film_grain`` >>>>>> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >>>>>> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >>>>>> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >>>>>> + - ``p_fixed_point`` >>>>>> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >>>>>> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >>>>>> * - void * >>>>>> - ``ptr`` >>>>>> - A pointer to a compound type which can be an N-dimensional array >>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>>>> index 4d38acafe8e1..9285f4f39eed 100644 >>>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>>>> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >>>>>> - n/a >>>>>> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >>>>>> parameters for stateless video decoders. >>>>>> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >>>>>> + - n/a >>>>>> + - n/a >>>>>> + - n/a >>>>>> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >>>>>> + integer part and fractional part, i.e. audio sample rate. >>>>>> + >>>>>> >>>>>> .. raw:: latex >>>>>> >>>>>> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>>>> index e61152bb80d1..2faa5a2015eb 100644 >>>>>> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>>>> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>>>> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >>>>>> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >>>>>> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >>>>>> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >>>>>> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >>>>>> >>>>>> # V4L2 capability defines >>>>>> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>>>> index a662fb60f73f..7a616ac91059 100644 >>>>>> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>>>> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >>>>>> if (!area->width || !area->height) >>>>>> return -EINVAL; >>>>>> break; >>>>>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>>>>> + break; >>>>> >>>>> Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': >>>>> >>>>> https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... >>>>> >>>>> since min and max values are perfectly fine for a fixed point value. >>>>> >>>>> Even a step value (currently not supported in that patch) would make sense. >>>>> >>>>> But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, >>>>> why not represent the fixed point value as a Q31.32. Then the standard >>>>> minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. >>>>> >>>>> Except that both userspace and drivers need to multiply it with 2^-32 to get the actual >>>>> value. >>>>> >>>>> So in enum v4l2_ctrl_type add: >>>>> >>>>> V4L2_CTRL_TYPE_FIXED_POINT = 10, >>>>> >>>>> (10, because it is no longer a compound type). >>>> >>>> Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64? >>>> >>>> The reason I use the 'integer' and 'fractional' is that I want >>>> 'integer' to be the normal sample >>>> rate, for example 48kHz. The 'fractional' is the difference with >>>> normal sample rate. >>>> >>>> For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655. >>>> >>>> So if we use s64 for rate, then in driver need to convert the rate to >>>> the closed normal >>>> sample rate + fractional. >>> >>> That wasn't what the documentation said :-) >>> >>> So this is really two controls: one for the 'normal sample rate' (whatever 'normal' >>> means in this context) and the offset to the actual sample rate. >>> >>> Presumably the 'normal' sample rate is set once, while the offset changes >>> regularly. >>> >>> But why do you need the 'normal' sample rate? With audio resampling I assume >>> you resample from one rate to another, so why do you need a third 'normal' >>> rate? >>> >> >> 'Normal' rate is used to select the prefilter table. >> > > Currently I think we may define > V4L2_CID_M2M_AUDIO_SOURCE_RATE > V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
> V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
Hans> > All of them can be V4L2_CTRL_TYPE_INTEGER. > > RATIO_MOD was defined in the very beginning version. > I think it is better to let users calculate this value. > > The reason is: > if we define the offset for source rate and dest rate in > driver separately, when offset of source rate is set, > driver don't know if it needs to wait or not the dest rate > offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
Right, the source and dest rates needn't change in same time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
I will use:
V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET
'OFFSET' is V4L2_CTRL_TYPE_FIXED_POINT, which is Q31.32.
So now I come back to my original question: why do you need both the rate and the offset? Isn't it enough to set just the rates, as long as that is in fixed point format?
Why does the driver need both the 'ideal' rate + the offset?
I'm not opposed to this, I'm just trying to understand whether this makes sense.
Can't you take e.g. the source and dest rate as starting points when you start streaming? And every time userspace updates one or both of these rates you calculate the ratio_mod compared to the previous rates?
Or is there a reason why you need the ideal rates as well? E.g. 48000 or 44100, etc.
ideal rates is used to select prefilter table. the prefilter table is indexed by ideal rates.
The asrc has two part: prefilter and resampler filter. The convert ratio is applied to rasampler filter part.
Ah! But doesn't this also imply that there are only a limited number of supported ideal rates? I see the supported_asrc_rate array in sound/soc/fsl/fsl_asrc.c, which I suspect is in fact that list?
That means that the SOURCE/DEST_RATE controls aren't integers, but instead should be of type V4L2_CTRL_TYPE_INTEGER_MENU.
How flexible is the resampler? E.g. I see that 12 and 16 kHz are supported, so can I select 12 kHz as the ideal rate, and an offset of 2000 Hz to convert from 14 kHz to something else? Or is it much more limited?
I ask, because if it is really flexible, then it is easy to just select 14 kHz and let the driver find the closest ideal rate.
If, however, it only can support a small range around each ideal rate (e.g. 11900-12100 Hz around the 12000 Hz ideal rate), then userspace needs to know those ideal rates, and that needs to be exposed through an INTEGER_MENU control.
Regards,
Hans
On Wed, Oct 18, 2023 at 3:58 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 17/10/2023 15:11, Shengjiu Wang wrote: > On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: >> >> Hi Shengjiu, >> >> On 13/10/2023 10:31, Shengjiu Wang wrote: >>> Fixed point controls are used by the user to configure >>> the audio sample rate to driver. >>> >>> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >>> new IDs for ASRC rate control. >>> >>> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >>> --- >>> .../userspace-api/media/v4l/common.rst | 1 + >>> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >>> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >>> .../media/videodev2.h.rst.exceptions | 1 + >>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >>> include/media/v4l2-ctrls.h | 2 ++ >>> include/uapi/linux/v4l2-controls.h | 13 +++++++ >>> include/uapi/linux/videodev2.h | 3 ++ >>> 10 files changed, 76 insertions(+) >>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>> >>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >>> index ea0435182e44..35707edffb13 100644 >>> --- a/Documentation/userspace-api/media/v4l/common.rst >>> +++ b/Documentation/userspace-api/media/v4l/common.rst >>> @@ -52,6 +52,7 @@ applicable to all devices. >>> ext-ctrls-fm-rx >>> ext-ctrls-detect >>> ext-ctrls-colorimetry >>> + ext-ctrls-fixed-point >> >> Rename this to ext-ctrls-audio-m2m. >> >>> fourcc >>> format >>> planar-apis >>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>> new file mode 100644 >>> index 000000000000..2ef6e250580c >>> --- /dev/null >>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>> @@ -0,0 +1,36 @@ >>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>> + >>> +.. _fixed-point-controls: >>> + >>> +*************************** >>> +Fixed Point Control Reference >> >> This is for audio controls. "Fixed Point" is just the type, and it doesn't make >> sense to group fixed point controls. But it does make sense to group the audio >> controls. >> >> V4L2 controls can be grouped into classes. Basically it is a way to put controls >> into categories, and for each category there is also a control that gives a >> description of the class (see 2.15.15 in >> https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) >> >> If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that >> they are grouped based on what class of control they are. >> >> So I think it would be a good idea to create a new control class for M2M audio controls, >> instead of just adding them to the catch-all 'User Controls' class. >> >> Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how >> it is done. >> >> M2M_AUDIO would probably be a good name for the class. >> >>> +*************************** >>> + >>> +These controls are intended to support an asynchronous sample >>> +rate converter. >> >> Add ' (ASRC).' at the end to indicate the common abbreviation for >> that. >> >>> + >>> +.. _v4l2-audio-asrc: >>> + >>> +``V4L2_CID_ASRC_SOURCE_RATE`` >>> + sets the resampler source rate. >>> + >>> +``V4L2_CID_ASRC_DEST_RATE`` >>> + sets the resampler destination rate. >> >> Document the unit (Hz) for these two controls. >> >>> + >>> +.. c:type:: v4l2_ctrl_fixed_point >>> + >>> +.. cssclass:: longtable >>> + >>> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >>> + >>> +.. flat-table:: struct v4l2_ctrl_fixed_point >>> + :header-rows: 0 >>> + :stub-columns: 0 >>> + :widths: 1 1 2 >>> + >>> + * - __u32 >> >> Hmm, shouldn't this be __s32? >> >>> + - ``integer`` >>> + - integer part of fixed point value. >>> + * - __s32 >> >> and this __u32? >> >> You want to be able to use this generic type as a signed value. >> >>> + - ``fractional`` >>> + - fractional part of fixed point value, which is Q31. >>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>> index f9f73530a6be..1811dabf5c74 100644 >>> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>> @@ -295,6 +295,10 @@ still cause this situation. >>> - ``p_av1_film_grain`` >>> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >>> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >>> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >>> + - ``p_fixed_point`` >>> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >>> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >>> * - void * >>> - ``ptr`` >>> - A pointer to a compound type which can be an N-dimensional array >>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>> index 4d38acafe8e1..9285f4f39eed 100644 >>> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >>> - n/a >>> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >>> parameters for stateless video decoders. >>> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >>> + - n/a >>> + - n/a >>> + - n/a >>> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >>> + integer part and fractional part, i.e. audio sample rate. >>> + >>> >>> .. raw:: latex >>> >>> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>> index e61152bb80d1..2faa5a2015eb 100644 >>> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >>> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >>> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >>> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >>> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >>> >>> # V4L2 capability defines >>> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>> index a662fb60f73f..7a616ac91059 100644 >>> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >>> if (!area->width || !area->height) >>> return -EINVAL; >>> break; >>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>> + break; >> >> Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': >> >> https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... >> >> since min and max values are perfectly fine for a fixed point value. >> >> Even a step value (currently not supported in that patch) would make sense. >> >> But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, >> why not represent the fixed point value as a Q31.32. Then the standard >> minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. >> >> Except that both userspace and drivers need to multiply it with 2^-32 to get the actual >> value. >> >> So in enum v4l2_ctrl_type add: >> >> V4L2_CTRL_TYPE_FIXED_POINT = 10, >> >> (10, because it is no longer a compound type). > > Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64? > > The reason I use the 'integer' and 'fractional' is that I want > 'integer' to be the normal sample > rate, for example 48kHz. The 'fractional' is the difference with > normal sample rate. > > For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655. > > So if we use s64 for rate, then in driver need to convert the rate to > the closed normal > sample rate + fractional.
That wasn't what the documentation said :-)
So this is really two controls: one for the 'normal sample rate' (whatever 'normal' means in this context) and the offset to the actual sample rate.
Presumably the 'normal' sample rate is set once, while the offset changes regularly.
But why do you need the 'normal' sample rate? With audio resampling I assume you resample from one rate to another, so why do you need a third 'normal' rate?
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
Can we use V4L2_CTRL_TYPE_INTEGER64 for Q31.32? or still need to define V4L2_CTRL_TYPE_FIXED_POINT?
best regards wang shengjiu
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
Regards,
Hansbest regards wang shengjiu
Best regards Wang Shengjiu
Regards,
Hans> > best regards > wang shengjiu > >> >>> >>> default: >>> return -EINVAL; >>> @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, >>> case V4L2_CTRL_TYPE_AREA: >>> elem_size = sizeof(struct v4l2_area); >>> break; >>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>> + elem_size = sizeof(struct v4l2_ctrl_fixed_point); >>> + break; >>> default: >>> if (type < V4L2_CTRL_COMPOUND_TYPES) >>> elem_size = sizeof(s32); >>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>> index 8696eb1cdd61..d8f232df6b6a 100644 >>> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>> @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, >>> case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: >>> *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; >>> break; >>> + case V4L2_CID_ASRC_SOURCE_RATE: >>> + case V4L2_CID_ASRC_DEST_RATE: >>> + *type = V4L2_CTRL_TYPE_FIXED_POINT; >>> + break; >>> default: >>> *type = V4L2_CTRL_TYPE_INTEGER; >>> break; >>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h >>> index 59679a42b3e7..645e4cccafc7 100644 >>> --- a/include/media/v4l2-ctrls.h >>> +++ b/include/media/v4l2-ctrls.h >>> @@ -56,6 +56,7 @@ struct video_device; >>> * @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure. >>> * @p_av1_frame: Pointer to an AV1 frame structure. >>> * @p_av1_film_grain: Pointer to an AV1 film grain structure. >>> + * @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point. >>> * @p: Pointer to a compound value. >>> * @p_const: Pointer to a constant compound value. >>> */ >>> @@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { >>> struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; >>> struct v4l2_ctrl_av1_frame *p_av1_frame; >>> struct v4l2_ctrl_av1_film_grain *p_av1_film_grain; >>> + struct v4l2_ctrl_fixed_point *p_fixed_point; >>> void *p; >>> const void *p_const; >>> }; >>> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h >>> index c3604a0a3e30..91096259e3ea 100644 >>> --- a/include/uapi/linux/v4l2-controls.h >>> +++ b/include/uapi/linux/v4l2-controls.h >>> @@ -112,6 +112,8 @@ enum v4l2_colorfx { >>> >>> /* last CID + 1 */ >>> #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) >>> +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) >>> +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46) >> >> This patch needs to be split in three parts: >> >> 1) Add the new M2M_AUDIO control class, >> 2) Add the new V4L2_CTRL_TYPE_FIXED_POINT type, >> 3) Add the new controls. >> >> These are all independent changes, so separating them makes it easier to >> review. >> >>> >>> /* USER-class private control IDs */ >>> >>> @@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { >>> #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE >>> #endif >>> >>> +/** >>> + * struct v4l2_ctrl_fixed_point - fixed point parameter. >>> + * >>> + * @rate_integer: integer part of fixed point value. >>> + * @rate_fractional: fractional part of fixed point value >>> + */ >>> +struct v4l2_ctrl_fixed_point { >>> + __u32 integer; >> >> __s32? >> >>> + __u32 fractional; >>> +}; >>> + >>> #endif >>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h >>> index 2ac7b989394c..3ef32c09c2fa 100644 >>> --- a/include/uapi/linux/videodev2.h >>> +++ b/include/uapi/linux/videodev2.h >>> @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { >>> struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; >>> struct v4l2_ctrl_av1_frame __user *p_av1_frame; >>> struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain; >>> + struct v4l2_ctrl_fixed_point __user *p_fixed_point; >>> void __user *ptr; >>> }; >>> } __attribute__ ((packed)); >>> @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { >>> V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, >>> V4L2_CTRL_TYPE_AV1_FRAME = 0x282, >>> V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283, >>> + >>> + V4L2_CTRL_TYPE_FIXED_POINT = 0x290, >>> }; >>> >>> /* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */ >> >> Regards, >> >> Hans
On 19/10/2023 12:44, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:58 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:40, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 3:31 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 18/10/2023 09:23, Shengjiu Wang wrote:
On Wed, Oct 18, 2023 at 10:27 AM Shengjiu Wang shengjiu.wang@gmail.com wrote:
On Tue, Oct 17, 2023 at 9:37 PM Hans Verkuil hverkuil@xs4all.nl wrote: > > On 17/10/2023 15:11, Shengjiu Wang wrote: >> On Mon, Oct 16, 2023 at 9:16 PM Hans Verkuil hverkuil@xs4all.nl wrote: >>> >>> Hi Shengjiu, >>> >>> On 13/10/2023 10:31, Shengjiu Wang wrote: >>>> Fixed point controls are used by the user to configure >>>> the audio sample rate to driver. >>>> >>>> Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE >>>> new IDs for ASRC rate control. >>>> >>>> Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com >>>> --- >>>> .../userspace-api/media/v4l/common.rst | 1 + >>>> .../media/v4l/ext-ctrls-fixed-point.rst | 36 +++++++++++++++++++ >>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +++ >>>> .../media/v4l/vidioc-queryctrl.rst | 7 ++++ >>>> .../media/videodev2.h.rst.exceptions | 1 + >>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 +++ >>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 +++ >>>> include/media/v4l2-ctrls.h | 2 ++ >>>> include/uapi/linux/v4l2-controls.h | 13 +++++++ >>>> include/uapi/linux/videodev2.h | 3 ++ >>>> 10 files changed, 76 insertions(+) >>>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>> >>>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst >>>> index ea0435182e44..35707edffb13 100644 >>>> --- a/Documentation/userspace-api/media/v4l/common.rst >>>> +++ b/Documentation/userspace-api/media/v4l/common.rst >>>> @@ -52,6 +52,7 @@ applicable to all devices. >>>> ext-ctrls-fm-rx >>>> ext-ctrls-detect >>>> ext-ctrls-colorimetry >>>> + ext-ctrls-fixed-point >>> >>> Rename this to ext-ctrls-audio-m2m. >>> >>>> fourcc >>>> format >>>> planar-apis >>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>> new file mode 100644 >>>> index 000000000000..2ef6e250580c >>>> --- /dev/null >>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-fixed-point.rst >>>> @@ -0,0 +1,36 @@ >>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>> + >>>> +.. _fixed-point-controls: >>>> + >>>> +*************************** >>>> +Fixed Point Control Reference >>> >>> This is for audio controls. "Fixed Point" is just the type, and it doesn't make >>> sense to group fixed point controls. But it does make sense to group the audio >>> controls. >>> >>> V4L2 controls can be grouped into classes. Basically it is a way to put controls >>> into categories, and for each category there is also a control that gives a >>> description of the class (see 2.15.15 in >>> https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...) >>> >>> If you use e.g. 'v4l2-ctl -l' to list all the controls, then you will see that >>> they are grouped based on what class of control they are. >>> >>> So I think it would be a good idea to create a new control class for M2M audio controls, >>> instead of just adding them to the catch-all 'User Controls' class. >>> >>> Search e.g. for V4L2_CTRL_CLASS_COLORIMETRY and V4L2_CID_COLORIMETRY_CLASS to see how >>> it is done. >>> >>> M2M_AUDIO would probably be a good name for the class. >>> >>>> +*************************** >>>> + >>>> +These controls are intended to support an asynchronous sample >>>> +rate converter. >>> >>> Add ' (ASRC).' at the end to indicate the common abbreviation for >>> that. >>> >>>> + >>>> +.. _v4l2-audio-asrc: >>>> + >>>> +``V4L2_CID_ASRC_SOURCE_RATE`` >>>> + sets the resampler source rate. >>>> + >>>> +``V4L2_CID_ASRC_DEST_RATE`` >>>> + sets the resampler destination rate. >>> >>> Document the unit (Hz) for these two controls. >>> >>>> + >>>> +.. c:type:: v4l2_ctrl_fixed_point >>>> + >>>> +.. cssclass:: longtable >>>> + >>>> +.. tabularcolumns:: |p{1.5cm}|p{5.8cm}|p{10.0cm}| >>>> + >>>> +.. flat-table:: struct v4l2_ctrl_fixed_point >>>> + :header-rows: 0 >>>> + :stub-columns: 0 >>>> + :widths: 1 1 2 >>>> + >>>> + * - __u32 >>> >>> Hmm, shouldn't this be __s32? >>> >>>> + - ``integer`` >>>> + - integer part of fixed point value. >>>> + * - __s32 >>> >>> and this __u32? >>> >>> You want to be able to use this generic type as a signed value. >>> >>>> + - ``fractional`` >>>> + - fractional part of fixed point value, which is Q31. >>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>> index f9f73530a6be..1811dabf5c74 100644 >>>> --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst >>>> @@ -295,6 +295,10 @@ still cause this situation. >>>> - ``p_av1_film_grain`` >>>> - A pointer to a struct :c:type:`v4l2_ctrl_av1_film_grain`. Valid if this control is >>>> of type ``V4L2_CTRL_TYPE_AV1_FILM_GRAIN``. >>>> + * - struct :c:type:`v4l2_ctrl_fixed_point` * >>>> + - ``p_fixed_point`` >>>> + - A pointer to a struct :c:type:`v4l2_ctrl_fixed_point`. Valid if this control is >>>> + of type ``V4L2_CTRL_TYPE_FIXED_POINT``. >>>> * - void * >>>> - ``ptr`` >>>> - A pointer to a compound type which can be an N-dimensional array >>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>> index 4d38acafe8e1..9285f4f39eed 100644 >>>> --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst >>>> @@ -549,6 +549,13 @@ See also the examples in :ref:`control`. >>>> - n/a >>>> - A struct :c:type:`v4l2_ctrl_av1_film_grain`, containing AV1 Film Grain >>>> parameters for stateless video decoders. >>>> + * - ``V4L2_CTRL_TYPE_FIXED_POINT`` >>>> + - n/a >>>> + - n/a >>>> + - n/a >>>> + - A struct :c:type:`v4l2_ctrl_fixed_point`, containing parameter which has >>>> + integer part and fractional part, i.e. audio sample rate. >>>> + >>>> >>>> .. raw:: latex >>>> >>>> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>> index e61152bb80d1..2faa5a2015eb 100644 >>>> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions >>>> @@ -167,6 +167,7 @@ replace symbol V4L2_CTRL_TYPE_AV1_SEQUENCE :c:type:`v4l2_ctrl_type` >>>> replace symbol V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY :c:type:`v4l2_ctrl_type` >>>> replace symbol V4L2_CTRL_TYPE_AV1_FRAME :c:type:`v4l2_ctrl_type` >>>> replace symbol V4L2_CTRL_TYPE_AV1_FILM_GRAIN :c:type:`v4l2_ctrl_type` >>>> +replace symbol V4L2_CTRL_TYPE_FIXED_POINT :c:type:`v4l2_ctrl_type` >>>> >>>> # V4L2 capability defines >>>> replace define V4L2_CAP_VIDEO_CAPTURE device-capabilities >>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>> index a662fb60f73f..7a616ac91059 100644 >>>> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c >>>> @@ -1168,6 +1168,8 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, >>>> if (!area->width || !area->height) >>>> return -EINVAL; >>>> break; >>>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>>> + break; >>> >>> Hmm, this would need this patch 'v4l2-ctrls: add support for V4L2_CTRL_WHICH_MIN/MAX_VAL': >>> >>> https://patchwork.linuxtv.org/project/linux-media/patch/20231010022136.15040... >>> >>> since min and max values are perfectly fine for a fixed point value. >>> >>> Even a step value (currently not supported in that patch) would make sense. >>> >>> But I wonder if we couldn't simplify this: instead of creating a v4l2_ctrl_fixed_point, >>> why not represent the fixed point value as a Q31.32. Then the standard >>> minimum/maximum/step values can be used, and it acts like a regular V4L2_TYPE_INTEGER64. >>> >>> Except that both userspace and drivers need to multiply it with 2^-32 to get the actual >>> value. >>> >>> So in enum v4l2_ctrl_type add: >>> >>> V4L2_CTRL_TYPE_FIXED_POINT = 10, >>> >>> (10, because it is no longer a compound type). >> >> Seems we don't need V4L2_CTRL_TYPE_FIXED_POINT, just use V4L2_TYPE_INTEGER64? >> >> The reason I use the 'integer' and 'fractional' is that I want >> 'integer' to be the normal sample >> rate, for example 48kHz. The 'fractional' is the difference with >> normal sample rate. >> >> For example, the rate = 47998.12345. so integer = 48000, fractional= -1.87655. >> >> So if we use s64 for rate, then in driver need to convert the rate to >> the closed normal >> sample rate + fractional. > > That wasn't what the documentation said :-) > > So this is really two controls: one for the 'normal sample rate' (whatever 'normal' > means in this context) and the offset to the actual sample rate. > > Presumably the 'normal' sample rate is set once, while the offset changes > regularly. > > But why do you need the 'normal' sample rate? With audio resampling I assume > you resample from one rate to another, so why do you need a third 'normal' > rate? >
'Normal' rate is used to select the prefilter table.
Currently I think we may define V4L2_CID_M2M_AUDIO_SOURCE_RATE V4L2_CID_M2M_AUDIO_DEST_RATE
That makes sense.
V4L2_CID_M2M_AUDIO_ASRC_RATIO_MOD
OK, can you document this control? Just write it down in the reply, I just want to understand how the integer value you set here is used.
It is Q31 value. It is equal to: in_rate_new / out_rate_new - in_rate_old / out_rate_old
So that's not an integer. Also, Q31 is limited to -1...1, and I think that's too limiting.
For this having a Q31.32 fixed point type still makes a lot of sense.
Can we use V4L2_CTRL_TYPE_INTEGER64 for Q31.32?
No.
or still need to define V4L2_CTRL_TYPE_FIXED_POINT?
Yes, we need that. For the most part V4L2_CTRL_TYPE_FIXED_POINT will be handled the same as V4L2_CTRL_TYPE_INTEGER64 internally, so that makes life a lot easier.
Regards,
Hans
best regards wang shengjiu
I still feel this is a overly complicated API.
See more below...
Best regards Wang shengjiu
Regards,
HansAll of them can be V4L2_CTRL_TYPE_INTEGER.
RATIO_MOD was defined in the very beginning version. I think it is better to let users calculate this value.
The reason is: if we define the offset for source rate and dest rate in driver separately, when offset of source rate is set, driver don't know if it needs to wait or not the dest rate offset, then go to calculate the ratio_mod.
Ah, in order to update the ratio mod userspace needs to set both source and dest rate at the same time to avoid race conditions.
That is perfectly possible in the V4L2 control framework. See:
https://linuxtv.org/downloads/v4l-dvb-apis-new/driver-api/v4l2-controls.html...
In practice, isn't it likely that you would fix either the source or destination rate, and let the other rate fluctuate? It kind of feels weird to me that both source AND destination rates can fluctuate over time.
In any case, with a control cluster it doesn't really matter, you can set one rate or both rates, and it will be handled atomically.
I feel that the RATIO_MOD control is too hardware specific. This is something that should be hidden in the driver.
Regards,
Hansbest regards wang shengjiu
Best regards Wang Shengjiu
> Regards, > > Hans > >> >> best regards >> wang shengjiu >> >>> >>>> >>>> default: >>>> return -EINVAL; >>>> @@ -1868,6 +1870,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, >>>> case V4L2_CTRL_TYPE_AREA: >>>> elem_size = sizeof(struct v4l2_area); >>>> break; >>>> + case V4L2_CTRL_TYPE_FIXED_POINT: >>>> + elem_size = sizeof(struct v4l2_ctrl_fixed_point); >>>> + break; >>>> default: >>>> if (type < V4L2_CTRL_COMPOUND_TYPES) >>>> elem_size = sizeof(s32); >>>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>>> index 8696eb1cdd61..d8f232df6b6a 100644 >>>> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>>> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c >>>> @@ -1602,6 +1602,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, >>>> case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY: >>>> *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY; >>>> break; >>>> + case V4L2_CID_ASRC_SOURCE_RATE: >>>> + case V4L2_CID_ASRC_DEST_RATE: >>>> + *type = V4L2_CTRL_TYPE_FIXED_POINT; >>>> + break; >>>> default: >>>> *type = V4L2_CTRL_TYPE_INTEGER; >>>> break; >>>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h >>>> index 59679a42b3e7..645e4cccafc7 100644 >>>> --- a/include/media/v4l2-ctrls.h >>>> +++ b/include/media/v4l2-ctrls.h >>>> @@ -56,6 +56,7 @@ struct video_device; >>>> * @p_av1_tile_group_entry: Pointer to an AV1 tile group entry structure. >>>> * @p_av1_frame: Pointer to an AV1 frame structure. >>>> * @p_av1_film_grain: Pointer to an AV1 film grain structure. >>>> + * @p_fixed_point: Pointer to a struct v4l2_ctrl_fixed_point. >>>> * @p: Pointer to a compound value. >>>> * @p_const: Pointer to a constant compound value. >>>> */ >>>> @@ -89,6 +90,7 @@ union v4l2_ctrl_ptr { >>>> struct v4l2_ctrl_av1_tile_group_entry *p_av1_tile_group_entry; >>>> struct v4l2_ctrl_av1_frame *p_av1_frame; >>>> struct v4l2_ctrl_av1_film_grain *p_av1_film_grain; >>>> + struct v4l2_ctrl_fixed_point *p_fixed_point; >>>> void *p; >>>> const void *p_const; >>>> }; >>>> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h >>>> index c3604a0a3e30..91096259e3ea 100644 >>>> --- a/include/uapi/linux/v4l2-controls.h >>>> +++ b/include/uapi/linux/v4l2-controls.h >>>> @@ -112,6 +112,8 @@ enum v4l2_colorfx { >>>> >>>> /* last CID + 1 */ >>>> #define V4L2_CID_LASTP1 (V4L2_CID_BASE+44) >>>> +#define V4L2_CID_ASRC_SOURCE_RATE (V4L2_CID_BASE + 45) >>>> +#define V4L2_CID_ASRC_DEST_RATE (V4L2_CID_BASE + 46) >>> >>> This patch needs to be split in three parts: >>> >>> 1) Add the new M2M_AUDIO control class, >>> 2) Add the new V4L2_CTRL_TYPE_FIXED_POINT type, >>> 3) Add the new controls. >>> >>> These are all independent changes, so separating them makes it easier to >>> review. >>> >>>> >>>> /* USER-class private control IDs */ >>>> >>>> @@ -3488,4 +3490,15 @@ struct v4l2_ctrl_av1_film_grain { >>>> #define V4L2_CID_MPEG_MFC51_BASE V4L2_CID_CODEC_MFC51_BASE >>>> #endif >>>> >>>> +/** >>>> + * struct v4l2_ctrl_fixed_point - fixed point parameter. >>>> + * >>>> + * @rate_integer: integer part of fixed point value. >>>> + * @rate_fractional: fractional part of fixed point value >>>> + */ >>>> +struct v4l2_ctrl_fixed_point { >>>> + __u32 integer; >>> >>> __s32? >>> >>>> + __u32 fractional; >>>> +}; >>>> + >>>> #endif >>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h >>>> index 2ac7b989394c..3ef32c09c2fa 100644 >>>> --- a/include/uapi/linux/videodev2.h >>>> +++ b/include/uapi/linux/videodev2.h >>>> @@ -1888,6 +1888,7 @@ struct v4l2_ext_control { >>>> struct v4l2_ctrl_av1_tile_group_entry __user *p_av1_tile_group_entry; >>>> struct v4l2_ctrl_av1_frame __user *p_av1_frame; >>>> struct v4l2_ctrl_av1_film_grain __user *p_av1_film_grain; >>>> + struct v4l2_ctrl_fixed_point __user *p_fixed_point; >>>> void __user *ptr; >>>> }; >>>> } __attribute__ ((packed)); >>>> @@ -1966,6 +1967,8 @@ enum v4l2_ctrl_type { >>>> V4L2_CTRL_TYPE_AV1_TILE_GROUP_ENTRY = 0x281, >>>> V4L2_CTRL_TYPE_AV1_FRAME = 0x282, >>>> V4L2_CTRL_TYPE_AV1_FILM_GRAIN = 0x283, >>>> + >>>> + V4L2_CTRL_TYPE_FIXED_POINT = 0x290, >>>> }; >>>> >>>> /* Used in the VIDIOC_QUERYCTRL ioctl for querying controls */ >>> >>> Regards, >>> >>> Hans >
Implement the ASRC memory to memory function using the v4l2 framework, user can use this function with v4l2 ioctl interface.
User send the output and capture buffer to driver and driver store the converted data to the capture buffer.
This feature can be shared by ASRC and EASRC drivers
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1248 +++++++++++++++++++++++++ 3 files changed, 1261 insertions(+) create mode 100644 drivers/media/platform/nxp/imx-asrc.c
diff --git a/drivers/media/platform/nxp/Kconfig b/drivers/media/platform/nxp/Kconfig index 40e3436669e2..8234644ee341 100644 --- a/drivers/media/platform/nxp/Kconfig +++ b/drivers/media/platform/nxp/Kconfig @@ -67,3 +67,15 @@ config VIDEO_MX2_EMMAPRP
source "drivers/media/platform/nxp/dw100/Kconfig" source "drivers/media/platform/nxp/imx-jpeg/Kconfig" + +config VIDEO_IMX_ASRC + tristate "NXP i.MX ASRC M2M support" + depends on V4L_MEM2MEM_DRIVERS + depends on MEDIA_SUPPORT + select VIDEOBUF2_DMA_CONTIG + select V4L2_MEM2MEM_DEV + help + Say Y if you want to add ASRC M2M support for NXP CPUs. + It is a complement for ASRC M2P and ASRC P2M features. + This option is only useful for out-of-tree drivers since + in-tree drivers select it automatically. diff --git a/drivers/media/platform/nxp/Makefile b/drivers/media/platform/nxp/Makefile index 4d90eb713652..1325675e34f5 100644 --- a/drivers/media/platform/nxp/Makefile +++ b/drivers/media/platform/nxp/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX8MQ_MIPI_CSI2) += imx8mq-mipi-csi2.o obj-$(CONFIG_VIDEO_IMX_MIPI_CSIS) += imx-mipi-csis.o obj-$(CONFIG_VIDEO_IMX_PXP) += imx-pxp.o obj-$(CONFIG_VIDEO_MX2_EMMAPRP) += mx2_emmaprp.o +obj-$(CONFIG_VIDEO_IMX_ASRC) += imx-asrc.o diff --git a/drivers/media/platform/nxp/imx-asrc.c b/drivers/media/platform/nxp/imx-asrc.c new file mode 100644 index 000000000000..373ca2b5ec90 --- /dev/null +++ b/drivers/media/platform/nxp/imx-asrc.c @@ -0,0 +1,1248 @@ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2014-2016 Freescale Semiconductor, Inc. +// Copyright (C) 2019-2023 NXP +// +// Freescale ASRC Memory to Memory (M2M) driver + +#include <linux/dma/imx-dma.h> +#include <linux/pm_runtime.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-device.h> +#include <media/v4l2-event.h> +#include <media/v4l2-fh.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-mem2mem.h> +#include <media/videobuf2-dma-contig.h> +#include <sound/dmaengine_pcm.h> +#include <sound/fsl_asrc_common.h> + +#define V4L_CAP OUT +#define V4L_OUT IN + +#define ASRC_xPUT_DMA_CALLBACK(dir) \ + (((dir) == V4L_OUT) ? asrc_input_dma_callback \ + : asrc_output_dma_callback) + +#define DIR_STR(dir) (dir) == V4L_OUT ? "out" : "cap" + +#define ASRC_M2M_BUFFER_SIZE (512 * 1024) +#define ASRC_M2M_PERIOD_SIZE (48 * 1024) +#define ASRC_M2M_SG_NUM (20) + +struct asrc_fmt { + u32 fourcc; + snd_pcm_format_t format; +}; + +struct asrc_pair_m2m { + struct fsl_asrc_pair *pair; + struct asrc_m2m *m2m; + struct v4l2_fh fh; + struct v4l2_ctrl_handler ctrl_handler; + int channels[2]; + struct v4l2_ctrl_fixed_point src_rate; + struct v4l2_ctrl_fixed_point dst_rate; + +}; + +struct asrc_m2m { + struct fsl_asrc_m2m_pdata pdata; + struct v4l2_device v4l2_dev; + struct v4l2_m2m_dev *m2m_dev; + struct video_device *dec_vdev; + struct mutex mlock; /* v4l2 ioctls serialization */ + struct platform_device *pdev; +}; + +static struct asrc_fmt formats[] = { + { + .fourcc = V4L2_AUDIO_FMT_S8, + }, + { + .fourcc = V4L2_AUDIO_FMT_S16_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U16_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S24_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S24_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U24_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U24_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S32_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U32_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S20_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U20_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_FLOAT_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE, + }, +}; + +#define NUM_FORMATS ARRAY_SIZE(formats) + +static snd_pcm_format_t convert_fourcc(u32 fourcc) { + + return (__force snd_pcm_format_t)v4l2_fourcc_to_audfmt(fourcc); +} + +static u32 find_fourcc(snd_pcm_format_t format) +{ + struct asrc_fmt *fmt; + unsigned int k; + + for (k = 0; k < NUM_FORMATS; k++) { + fmt = &formats[k]; + fmt->format = convert_fourcc(fmt->fourcc); + if (fmt->format == format) + break; + } + + if (k == NUM_FORMATS) + return 0; + + return formats[k].fourcc; +} + +static snd_pcm_format_t find_format(u32 fourcc) +{ + struct asrc_fmt *fmt; + unsigned int k; + + for (k = 0; k < NUM_FORMATS; k++) { + fmt = &formats[k]; + if (fmt->fourcc == fourcc) + break; + } + + if (k == NUM_FORMATS) + return 0; + + formats[k].format = convert_fourcc(formats[k].fourcc); + + return formats[k].format; +} + +static int asrc_check_format(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 format) +{ + struct asrc_m2m *m2m = pair_m2m->m2m; + struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata; + struct fsl_asrc_pair *pair = pair_m2m->pair; + u64 format_bit = 0; + int i; + + for (i = 0; i < NUM_FORMATS; ++i) { + if (formats[i].fourcc == format) { + formats[i].format = convert_fourcc(formats[i].fourcc); + format_bit = pcm_format_to_bits(formats[i].format); + break; + } + } + + if (dir == IN && !(format_bit & pdata->fmt_in)) + return find_fourcc(pair->sample_format[V4L_OUT]); + else if (dir == OUT && !(format_bit & pdata->fmt_out)) + return find_fourcc(pair->sample_format[V4L_CAP]); + else + return format; +} + +static int asrc_check_channel(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 channels) +{ + struct asrc_m2m *m2m = pair_m2m->m2m; + struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata; + struct fsl_asrc_pair *pair = pair_m2m->pair; + + if (channels < pdata->chan_min || channels > pdata->chan_max) + return pair->channels; + else + return channels; +} + +static int asrc_check_rate(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 rate) +{ + struct asrc_m2m *m2m = pair_m2m->m2m; + struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata; + struct fsl_asrc_pair *pair = pair_m2m->pair; + + if (rate < pdata->rate_min || rate > pdata->rate_max) + return pair->rate[dir]; + else + return rate; +} + +static inline struct asrc_pair_m2m *asrc_m2m_fh_to_ctx(struct v4l2_fh *fh) +{ + return container_of(fh, struct asrc_pair_m2m, fh); +} + +/** + * asrc_read_last_fifo: read all the remaining data from FIFO + * @pair: Structure pointer of fsl_asrc_pair + * @dma_vaddr: virtual address of capture buffer + * @length: payload length of capture buffer + */ +static void asrc_read_last_fifo(struct fsl_asrc_pair *pair, void *dma_vaddr, u32 *length) +{ + struct fsl_asrc *asrc = pair->asrc; + enum asrc_pair_index index = pair->index; + u32 i, reg, size, t_size = 0, width; + u32 *reg32 = NULL; + u16 *reg16 = NULL; + u8 *reg24 = NULL; + + width = snd_pcm_format_physical_width(pair->sample_format[V4L_CAP]); + if (width == 32) + reg32 = dma_vaddr + *length; + else if (width == 16) + reg16 = dma_vaddr + *length; + else + reg24 = dma_vaddr + *length; +retry: + size = asrc->get_output_fifo_size(pair); + if (size + *length > ASRC_M2M_BUFFER_SIZE) + goto end; + + for (i = 0; i < size * pair->channels; i++) { + regmap_read(asrc->regmap, asrc->get_fifo_addr(OUT, index), ®); + if (reg32) { + *(reg32) = reg; + reg32++; + } else if (reg16) { + *(reg16) = (u16)reg; + reg16++; + } else { + *reg24++ = (u8)reg; + *reg24++ = (u8)(reg >> 8); + *reg24++ = (u8)(reg >> 16); + } + } + t_size += size; + + /* In case there is data left in FIFO */ + if (size) + goto retry; +end: + /* Update payload length */ + if (reg32) + *length += t_size * pair->channels * 4; + else if (reg16) + *length += t_size * pair->channels * 2; + else + *length += t_size * pair->channels * 3; +} + +static int asrc_m2m_start_streaming(struct vb2_queue *q, unsigned int count) +{ + struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q); + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct asrc_m2m *m2m = pair_m2m->m2m; + struct fsl_asrc *asrc = pair->asrc; + struct device *dev = &m2m->pdev->dev; + struct vb2_v4l2_buffer *buf; + bool request_flag = false; + int ret; + + dev_dbg(dev, "Start streaming pair=%p, %d\n", pair, q->type); + + ret = pm_runtime_get_sync(dev); + if (ret < 0) { + dev_err(dev, "Failed to power up asrc\n"); + goto err_pm_runtime; + } + + /* Request asrc pair/context */ + if (!pair->req_pair) { + /* flag for error handler of this function */ + request_flag = true; + + ret = asrc->request_pair(pair->channels, pair); + if (ret) { + dev_err(dev, "failed to request pair: %d\n", ret); + goto err_request_pair; + } + + ret = asrc->m2m_prepare(pair); + if (ret) { + dev_err(dev, "failed to start pair part one: %d\n", ret); + goto err_start_part_one; + } + + pair->req_pair = true; + } + + /* Request dma channels */ + if (V4L2_TYPE_IS_OUTPUT(q->type)) { + pair->dma_chan[V4L_OUT] = asrc->get_dma_channel(pair, IN); + if (!pair->dma_chan[V4L_OUT]) { + dev_err(dev, "[ctx%d] failed to get input DMA channel\n", pair->index); + ret = -EBUSY; + goto err_dma_channel; + } + } else { + pair->dma_chan[V4L_CAP] = asrc->get_dma_channel(pair, OUT); + if (!pair->dma_chan[V4L_CAP]) { + dev_err(dev, "[ctx%d] failed to get output DMA channel\n", pair->index); + ret = -EBUSY; + goto err_dma_channel; + } + } + + v4l2_m2m_update_start_streaming_state(pair_m2m->fh.m2m_ctx, q); + + return 0; + +err_dma_channel: + if (request_flag && asrc->m2m_unprepare) + asrc->m2m_unprepare(pair); +err_start_part_one: + if (request_flag) + asrc->release_pair(pair); +err_request_pair: + pm_runtime_put_sync(dev); +err_pm_runtime: + /* Release buffers */ + if (V4L2_TYPE_IS_OUTPUT(q->type)) { + while ((buf = v4l2_m2m_src_buf_remove(pair_m2m->fh.m2m_ctx))) + v4l2_m2m_buf_done(buf, VB2_BUF_STATE_QUEUED); + } else { + while ((buf = v4l2_m2m_dst_buf_remove(pair_m2m->fh.m2m_ctx))) + v4l2_m2m_buf_done(buf, VB2_BUF_STATE_QUEUED); + } + return ret; +} + +static void asrc_m2m_stop_streaming(struct vb2_queue *q) +{ + struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q); + struct asrc_m2m *m2m = pair_m2m->m2m; + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct fsl_asrc *asrc = pair->asrc; + struct device *dev = &m2m->pdev->dev; + + dev_dbg(dev, "Stop streaming pair=%p, %d\n", pair, q->type); + + v4l2_m2m_update_stop_streaming_state(pair_m2m->fh.m2m_ctx, q); + + /* Stop & release pair/context */ + if (asrc->m2m_stop) + asrc->m2m_stop(pair); + + if (pair->req_pair) { + if (asrc->m2m_unprepare) + asrc->m2m_unprepare(pair); + asrc->release_pair(pair); + pair->req_pair = false; + } + + /* Release dma channel */ + if (V4L2_TYPE_IS_OUTPUT(q->type)) { + if (pair->dma_chan[V4L_OUT]) + dma_release_channel(pair->dma_chan[V4L_OUT]); + } else { + if (pair->dma_chan[V4L_CAP]) + dma_release_channel(pair->dma_chan[V4L_CAP]); + } + + pm_runtime_put_sync(dev); +} + +static int asrc_m2m_queue_setup(struct vb2_queue *q, + unsigned int *num_buffers, unsigned int *num_planes, + unsigned int sizes[], struct device *alloc_devs[]) +{ + struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q); + struct fsl_asrc_pair *pair = pair_m2m->pair; + + /* single buffer */ + *num_planes = 1; + + /* + * The capture buffer size depends on output buffer size + * and the convert ratio. + * + * Here just use a fix length for capture and output buffer. + * User need to care about it. + */ + + if (V4L2_TYPE_IS_OUTPUT(q->type)) + sizes[0] = pair->buf_len[V4L_OUT]; + else + sizes[0] = pair->buf_len[V4L_CAP]; + + return 0; +} + +static void asrc_m2m_buf_queue(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); + struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(vb->vb2_queue); + + /* queue buffer */ + v4l2_m2m_buf_queue(pair_m2m->fh.m2m_ctx, vbuf); +} + +static const struct vb2_ops asrc_m2m_qops = { + .wait_prepare = vb2_ops_wait_prepare, + .wait_finish = vb2_ops_wait_finish, + .start_streaming = asrc_m2m_start_streaming, + .stop_streaming = asrc_m2m_stop_streaming, + .queue_setup = asrc_m2m_queue_setup, + .buf_queue = asrc_m2m_buf_queue, +}; + +/* Init video buffer queue for src and dst. */ +static int asrc_m2m_queue_init(void *priv, struct vb2_queue *src_vq, + struct vb2_queue *dst_vq) +{ + struct asrc_pair_m2m *pair_m2m = priv; + struct asrc_m2m *m2m = pair_m2m->m2m; + int ret; + + src_vq->type = V4L2_BUF_TYPE_AUDIO_OUTPUT; + src_vq->io_modes = VB2_MMAP | VB2_DMABUF; + src_vq->drv_priv = pair_m2m; + src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + src_vq->ops = &asrc_m2m_qops; + src_vq->mem_ops = &vb2_dma_contig_memops; + src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + src_vq->lock = &m2m->mlock; + src_vq->dev = &m2m->pdev->dev; + src_vq->min_buffers_needed = 1; + + ret = vb2_queue_init(src_vq); + if (ret) + return ret; + + dst_vq->type = V4L2_BUF_TYPE_AUDIO_CAPTURE; + dst_vq->io_modes = VB2_MMAP | VB2_DMABUF; + dst_vq->drv_priv = pair_m2m; + dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + dst_vq->ops = &asrc_m2m_qops; + dst_vq->mem_ops = &vb2_dma_contig_memops; + dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + dst_vq->lock = &m2m->mlock; + dst_vq->dev = &m2m->pdev->dev; + dst_vq->min_buffers_needed = 1; + + ret = vb2_queue_init(dst_vq); + return ret; +} + +static int asrc_m2m_op_s_ctrl(struct v4l2_ctrl *ctrl) +{ + struct asrc_pair_m2m *pair_m2m = + container_of(ctrl->handler, struct asrc_pair_m2m, ctrl_handler); + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct fsl_asrc *asrc = pair->asrc; + int src_rate_int, src_rate_frac; + int dst_rate_int, dst_rate_frac; + int new_rate, new_frac; + u64 src_rate, dst_rate; + u64 ratio_pre, ratio_cur; + s64 ratio_diff; + int ret = 0; + + switch (ctrl->id) { + case V4L2_CID_ASRC_SOURCE_RATE: + new_rate = ctrl->p_new.p_fixed_point->integer; + new_frac = ctrl->p_new.p_fixed_point->fractional; + src_rate_int = asrc_check_rate(pair_m2m, IN, new_rate); + if (src_rate_int != new_rate || + (pair_m2m->src_rate.integer > 0 && + src_rate_int != pair_m2m->src_rate.integer)) + return -EINVAL; + + pair->rate[V4L_OUT] = src_rate_int; + + if (new_frac != pair_m2m->src_rate.fractional && + new_rate == pair_m2m->src_rate.integer && + pair_m2m->dst_rate.integer > 0) { + /* + * use maximum rate 768kHz as limitation, then we can shift right 21 bit for + * division + */ + src_rate_frac = pair_m2m->src_rate.fractional; + src_rate = ((s64)src_rate_int << 31) + src_rate_frac; + dst_rate_int = pair_m2m->dst_rate.integer; + dst_rate_frac = pair_m2m->dst_rate.fractional; + dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20; + do_div(src_rate, dst_rate); + ratio_pre = src_rate; + + src_rate_frac = new_frac; + src_rate = ((s64)src_rate_int << 31) + src_rate_frac; + dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20; + do_div(src_rate, dst_rate); + ratio_cur = src_rate; + + ratio_diff = ratio_cur - ratio_pre; + asrc->m2m_set_ratio_mod(pair, ratio_diff << 11); + } + + pair_m2m->src_rate.integer = new_rate; + pair_m2m->src_rate.fractional = new_frac; + + break; + case V4L2_CID_ASRC_DEST_RATE: + new_rate = ctrl->p_new.p_fixed_point->integer; + new_frac = ctrl->p_new.p_fixed_point->fractional; + + dst_rate_int = asrc_check_rate(pair_m2m, OUT, new_rate); + if (dst_rate_int != new_rate || + (pair_m2m->dst_rate.integer > 0 && + dst_rate_int != pair_m2m->dst_rate.integer)) + return -EINVAL; + + pair->rate[V4L_CAP] = dst_rate_int; + + if (new_frac != pair_m2m->dst_rate.fractional && + new_rate == pair_m2m->dst_rate.integer && + pair_m2m->src_rate.integer > 0) { + /* + * use maximum rate 768kHz as limitation, then we can shift right 21 bit for + * division + */ + src_rate_int = pair_m2m->src_rate.integer; + src_rate_frac = pair_m2m->src_rate.fractional; + src_rate = ((s64)src_rate_int << 31) + src_rate_frac; + dst_rate_frac = pair_m2m->dst_rate.fractional; + dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20; + do_div(src_rate, dst_rate); + ratio_pre = src_rate; + + src_rate = ((s64)src_rate_int << 31) + src_rate_frac; + dst_rate_int = new_rate; + dst_rate_frac = new_frac; + dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20; + do_div(src_rate, dst_rate); + ratio_cur = src_rate; + + ratio_diff = ratio_cur - ratio_pre; + /* convert ratio_diff to Q31*/ + asrc->m2m_set_ratio_mod(pair, ratio_diff << 11); + } + + pair_m2m->dst_rate.integer = new_rate; + pair_m2m->dst_rate.fractional = new_frac; + + break; + default: + ret = -EINVAL; + break; + } + + return ret; +} + +static const struct v4l2_ctrl_ops asrc_m2m_ctrl_ops = { + .s_ctrl = asrc_m2m_op_s_ctrl, +}; + +static const struct v4l2_ctrl_config asrc_src_rate_control = { + .ops = &asrc_m2m_ctrl_ops, + .id = V4L2_CID_ASRC_SOURCE_RATE, + .name = "asrc source sample rate", + .type = V4L2_CTRL_TYPE_FIXED_POINT, + .min = 0, + .max = 0x7fffffff, + .def = 8000, + .flags = V4L2_CTRL_FLAG_UPDATE, +}; + +static const struct v4l2_ctrl_config asrc_dst_rate_control = { + .ops = &asrc_m2m_ctrl_ops, + .id = V4L2_CID_ASRC_DEST_RATE, + .name = "asrc dest sample rate", + .type = V4L2_CTRL_TYPE_FIXED_POINT, + .min = 0, + .max = 0x7fffffff, + .def = 8000, + .flags = V4L2_CTRL_FLAG_UPDATE, +}; + +/* system callback for open() */ +static int asrc_m2m_open(struct file *file) +{ + struct asrc_m2m *m2m = video_drvdata(file); + struct fsl_asrc *asrc = m2m->pdata.asrc; + struct video_device *vdev = video_devdata(file); + struct fsl_asrc_pair *pair; + struct asrc_pair_m2m *pair_m2m; + int ret = 0; + + if (mutex_lock_interruptible(&m2m->mlock)) + return -ERESTARTSYS; + + pair = kzalloc(sizeof(*pair) + asrc->pair_priv_size, GFP_KERNEL); + if (!pair) { + ret = -ENOMEM; + goto err_alloc_pair; + } + + pair_m2m = kzalloc(sizeof(*pair_m2m), GFP_KERNEL); + if (!pair_m2m) { + ret = -ENOMEM; + goto err_alloc_pair_m2m; + } + + pair->private = (void *)pair + sizeof(struct fsl_asrc_pair); + pair->asrc = asrc; + + pair->buf_len[V4L_OUT] = ASRC_M2M_BUFFER_SIZE; + pair->buf_len[V4L_CAP] = ASRC_M2M_BUFFER_SIZE; + + pair->channels = 2; + pair->rate[V4L_OUT] = 8000; + pair->rate[V4L_CAP] = 8000; + pair->sample_format[V4L_OUT] = SNDRV_PCM_FORMAT_S16_LE; + pair->sample_format[V4L_CAP] = SNDRV_PCM_FORMAT_S16_LE; + + init_completion(&pair->complete[V4L_OUT]); + init_completion(&pair->complete[V4L_CAP]); + + v4l2_fh_init(&pair_m2m->fh, vdev); + v4l2_fh_add(&pair_m2m->fh); + file->private_data = &pair_m2m->fh; + + pair_m2m->pair = pair; + pair_m2m->m2m = m2m; + /* m2m context init */ + pair_m2m->fh.m2m_ctx = v4l2_m2m_ctx_init(m2m->m2m_dev, pair_m2m, + asrc_m2m_queue_init); + if (IS_ERR(pair_m2m->fh.m2m_ctx)) { + ret = PTR_ERR(pair_m2m->fh.m2m_ctx); + goto err_ctx_init; + } + + v4l2_ctrl_handler_init(&pair_m2m->ctrl_handler, 2); + + /* use V4L2_CID_GAIN for ratio update control */ + v4l2_ctrl_new_custom(&pair_m2m->ctrl_handler, &asrc_src_rate_control, NULL); + v4l2_ctrl_new_custom(&pair_m2m->ctrl_handler, &asrc_dst_rate_control, NULL); + + if (pair_m2m->ctrl_handler.error) { + ret = pair_m2m->ctrl_handler.error; + v4l2_ctrl_handler_free(&pair_m2m->ctrl_handler); + goto err_ctrl_handler; + } + + pair_m2m->fh.ctrl_handler = &pair_m2m->ctrl_handler; + + mutex_unlock(&m2m->mlock); + + return 0; + +err_ctrl_handler: + v4l2_m2m_ctx_release(pair_m2m->fh.m2m_ctx); +err_ctx_init: + v4l2_fh_del(&pair_m2m->fh); + v4l2_fh_exit(&pair_m2m->fh); + kfree(pair_m2m); +err_alloc_pair_m2m: + kfree(pair); +err_alloc_pair: + mutex_unlock(&m2m->mlock); + return ret; +} + +static int asrc_m2m_release(struct file *file) +{ + struct asrc_m2m *m2m = video_drvdata(file); + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(file->private_data); + struct fsl_asrc_pair *pair = pair_m2m->pair; + + mutex_lock(&m2m->mlock); + v4l2_ctrl_handler_free(&pair_m2m->ctrl_handler); + v4l2_m2m_ctx_release(pair_m2m->fh.m2m_ctx); + v4l2_fh_del(&pair_m2m->fh); + v4l2_fh_exit(&pair_m2m->fh); + kfree(pair_m2m); + kfree(pair); + mutex_unlock(&m2m->mlock); + + return 0; +} + +static const struct v4l2_file_operations asrc_m2m_fops = { + .owner = THIS_MODULE, + .open = asrc_m2m_open, + .release = asrc_m2m_release, + .poll = v4l2_m2m_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = v4l2_m2m_fop_mmap, +}; + +static int asrc_m2m_querycap(struct file *file, void *priv, + struct v4l2_capability *cap) +{ + strscpy(cap->driver, "asrc m2m", sizeof(cap->driver)); + strscpy(cap->card, "asrc m2m", sizeof(cap->card)); + cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_AUDIO_M2M; + cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS; + + return 0; +} + +static int enum_fmt(struct v4l2_fmtdesc *f, u64 fmtbit) +{ + int i, num; + struct asrc_fmt *fmt; + + num = 0; + + for (i = 0; i < NUM_FORMATS; ++i) { + formats[i].format = convert_fourcc(formats[i].fourcc); + if (pcm_format_to_bits(formats[i].format) & fmtbit) { + if (num == f->index) + break; + /* + * Correct type but haven't reached our index yet, + * just increment per-type index + */ + ++num; + } + } + + if (i < NUM_FORMATS) { + /* Format found */ + fmt = &formats[i]; + f->pixelformat = fmt->fourcc; + return 0; + } + + return -EINVAL; +} + +static int asrc_m2m_enum_fmt_aud_cap(struct file *file, void *fh, + struct v4l2_fmtdesc *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + struct asrc_m2m *m2m = pair_m2m->m2m; + + return enum_fmt(f, m2m->pdata.fmt_out); +} + +static int asrc_m2m_enum_fmt_aud_out(struct file *file, void *fh, + struct v4l2_fmtdesc *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + struct asrc_m2m *m2m = pair_m2m->m2m; + + return enum_fmt(f, m2m->pdata.fmt_in); +} + +static int asrc_m2m_g_fmt_aud_cap(struct file *file, void *fh, + struct v4l2_format *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + struct fsl_asrc_pair *pair = pair_m2m->pair; + + f->fmt.audio.channels = pair->channels; + f->fmt.audio.buffersize = pair->buf_len[V4L_CAP]; + f->fmt.audio.audioformat = find_fourcc(pair->sample_format[V4L_CAP]); + + return 0; +} + +static int asrc_m2m_g_fmt_aud_out(struct file *file, void *fh, + struct v4l2_format *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + struct fsl_asrc_pair *pair = pair_m2m->pair; + + f->fmt.audio.channels = pair->channels; + f->fmt.audio.buffersize = pair->buf_len[V4L_OUT]; + f->fmt.audio.audioformat = find_fourcc(pair->sample_format[V4L_OUT]); + + return 0; +} + +/* output for asrc */ +static int asrc_m2m_s_fmt_aud_cap(struct file *file, void *fh, + struct v4l2_format *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct asrc_m2m *m2m = pair_m2m->m2m; + struct device *dev = &m2m->pdev->dev; + + f->fmt.audio.audioformat = asrc_check_format(pair_m2m, OUT, f->fmt.audio.audioformat); + f->fmt.audio.channels = asrc_check_channel(pair_m2m, OUT, f->fmt.audio.channels); + + if (pair_m2m->channels[V4L_CAP] > 0 && + pair_m2m->channels[V4L_CAP] != f->fmt.audio.channels) { + dev_err(dev, "channels don't match for cap and out\n"); + return -EINVAL; + } + + pair_m2m->channels[V4L_CAP] = f->fmt.audio.channels; + pair->channels = f->fmt.audio.channels; + pair->sample_format[V4L_CAP] = find_format(f->fmt.audio.audioformat); + + return 0; +} + +/* input for asrc */ +static int asrc_m2m_s_fmt_aud_out(struct file *file, void *fh, + struct v4l2_format *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct asrc_m2m *m2m = pair_m2m->m2m; + struct device *dev = &m2m->pdev->dev; + + f->fmt.audio.audioformat = asrc_check_format(pair_m2m, IN, f->fmt.audio.audioformat); + f->fmt.audio.channels = asrc_check_channel(pair_m2m, IN, f->fmt.audio.channels); + if (pair_m2m->channels[V4L_OUT] > 0 && + pair_m2m->channels[V4L_OUT] != f->fmt.audio.channels) { + dev_err(dev, "channels don't match for cap and out\n"); + return -EINVAL; + } + + pair_m2m->channels[V4L_OUT] = f->fmt.audio.channels; + pair->channels = f->fmt.audio.channels; + pair->sample_format[V4L_OUT] = find_format(f->fmt.audio.audioformat); + + return 0; +} + +static int asrc_m2m_try_fmt_audio_cap(struct file *file, void *fh, + struct v4l2_format *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + + f->fmt.audio.audioformat = asrc_check_format(pair_m2m, OUT, f->fmt.audio.audioformat); + f->fmt.audio.channels = asrc_check_channel(pair_m2m, OUT, f->fmt.audio.channels); + + return 0; +} + +static int asrc_m2m_try_fmt_audio_out(struct file *file, void *fh, + struct v4l2_format *f) +{ + struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh); + + f->fmt.audio.audioformat = asrc_check_format(pair_m2m, IN, f->fmt.audio.audioformat); + f->fmt.audio.channels = asrc_check_channel(pair_m2m, IN, f->fmt.audio.channels); + + return 0; +} + +static const struct v4l2_ioctl_ops asrc_m2m_ioctl_ops = { + .vidioc_querycap = asrc_m2m_querycap, + + .vidioc_enum_fmt_audio_cap = asrc_m2m_enum_fmt_aud_cap, + .vidioc_enum_fmt_audio_out = asrc_m2m_enum_fmt_aud_out, + + .vidioc_g_fmt_audio_cap = asrc_m2m_g_fmt_aud_cap, + .vidioc_g_fmt_audio_out = asrc_m2m_g_fmt_aud_out, + + .vidioc_s_fmt_audio_cap = asrc_m2m_s_fmt_aud_cap, + .vidioc_s_fmt_audio_out = asrc_m2m_s_fmt_aud_out, + + .vidioc_try_fmt_audio_cap = asrc_m2m_try_fmt_audio_cap, + .vidioc_try_fmt_audio_out = asrc_m2m_try_fmt_audio_out, + + .vidioc_qbuf = v4l2_m2m_ioctl_qbuf, + .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf, + + .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs, + .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf, + .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs, + .vidioc_querybuf = v4l2_m2m_ioctl_querybuf, + .vidioc_streamon = v4l2_m2m_ioctl_streamon, + .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, + .vidioc_subscribe_event = v4l2_ctrl_subscribe_event, + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, +}; + +/* dma complete callback */ +static void asrc_input_dma_callback(void *data) +{ + struct fsl_asrc_pair *pair = (struct fsl_asrc_pair *)data; + + complete(&pair->complete[V4L_OUT]); +} + +/* dma complete callback */ +static void asrc_output_dma_callback(void *data) +{ + struct fsl_asrc_pair *pair = (struct fsl_asrc_pair *)data; + + complete(&pair->complete[V4L_CAP]); +} + +/* config dma channel */ +static int asrc_dmaconfig(struct asrc_pair_m2m *pair_m2m, + struct dma_chan *chan, + u32 dma_addr, dma_addr_t buf_addr, u32 buf_len, + int dir, int width) +{ + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct fsl_asrc *asrc = pair->asrc; + struct asrc_m2m *m2m = pair_m2m->m2m; + struct device *dev = &m2m->pdev->dev; + struct dma_slave_config slave_config; + struct scatterlist sg[ASRC_M2M_SG_NUM]; + enum dma_slave_buswidth buswidth; + unsigned int sg_len, max_period_size; + int ret, i; + + switch (width) { + case 8: + buswidth = DMA_SLAVE_BUSWIDTH_1_BYTE; + break; + case 16: + buswidth = DMA_SLAVE_BUSWIDTH_2_BYTES; + break; + case 24: + buswidth = DMA_SLAVE_BUSWIDTH_3_BYTES; + break; + case 32: + buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES; + break; + default: + dev_err(dev, "invalid word width\n"); + return -EINVAL; + } + + memset(&slave_config, 0, sizeof(slave_config)); + if (dir == V4L_OUT) { + slave_config.direction = DMA_MEM_TO_DEV; + slave_config.dst_addr = dma_addr; + slave_config.dst_addr_width = buswidth; + slave_config.dst_maxburst = asrc->m2m_get_maxburst(IN, pair); + } else { + slave_config.direction = DMA_DEV_TO_MEM; + slave_config.src_addr = dma_addr; + slave_config.src_addr_width = buswidth; + slave_config.src_maxburst = asrc->m2m_get_maxburst(OUT, pair); + } + + ret = dmaengine_slave_config(chan, &slave_config); + if (ret) { + dev_err(dev, "failed to config dmaengine for %s task: %d\n", + DIR_STR(dir), ret); + return -EINVAL; + } + + max_period_size = rounddown(ASRC_M2M_PERIOD_SIZE, width * pair->channels / 8); + /* scatter gather mode */ + sg_len = buf_len / max_period_size; + if (buf_len % max_period_size) + sg_len += 1; + + sg_init_table(sg, sg_len); + for (i = 0; i < (sg_len - 1); i++) { + sg_dma_address(&sg[i]) = buf_addr + i * max_period_size; + sg_dma_len(&sg[i]) = max_period_size; + } + sg_dma_address(&sg[i]) = buf_addr + i * max_period_size; + sg_dma_len(&sg[i]) = buf_len - i * max_period_size; + + pair->desc[dir] = dmaengine_prep_slave_sg(chan, sg, sg_len, + slave_config.direction, + DMA_PREP_INTERRUPT); + if (!pair->desc[dir]) { + dev_err(dev, "failed to prepare dmaengine for %s task\n", DIR_STR(dir)); + return -EINVAL; + } + + pair->desc[dir]->callback = ASRC_xPUT_DMA_CALLBACK(dir); + pair->desc[dir]->callback_param = pair; + + return 0; +} + +/* main function of converter */ +static void asrc_m2m_device_run(void *priv) +{ + struct asrc_pair_m2m *pair_m2m = priv; + struct fsl_asrc_pair *pair = pair_m2m->pair; + struct asrc_m2m *m2m = pair_m2m->m2m; + struct fsl_asrc *asrc = pair->asrc; + struct device *dev = &m2m->pdev->dev; + enum asrc_pair_index index = pair->index; + struct vb2_v4l2_buffer *src_buf, *dst_buf; + unsigned int out_buf_len; + unsigned int cap_dma_len; + unsigned int width; + u32 fifo_addr; + int ret; + + src_buf = v4l2_m2m_next_src_buf(pair_m2m->fh.m2m_ctx); + dst_buf = v4l2_m2m_next_dst_buf(pair_m2m->fh.m2m_ctx); + + width = snd_pcm_format_physical_width(pair->sample_format[V4L_OUT]); + fifo_addr = asrc->paddr + asrc->get_fifo_addr(IN, index); + out_buf_len = vb2_get_plane_payload(&src_buf->vb2_buf, 0); + if (out_buf_len < width * pair->channels / 8 || + out_buf_len > ASRC_M2M_BUFFER_SIZE || + out_buf_len % (width * pair->channels / 8)) { + dev_err(dev, "out buffer size is error: [%d]\n", out_buf_len); + goto end; + } + + /* dma config for output dma channel */ + ret = asrc_dmaconfig(pair_m2m, + pair->dma_chan[V4L_OUT], + fifo_addr, + vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0), + out_buf_len, V4L_OUT, width); + if (ret) { + dev_err(dev, "out dma config error\n"); + goto end; + } + + width = snd_pcm_format_physical_width(pair->sample_format[V4L_CAP]); + fifo_addr = asrc->paddr + asrc->get_fifo_addr(OUT, index); + cap_dma_len = asrc->m2m_calc_out_len(pair, out_buf_len); + if (cap_dma_len > 0 && cap_dma_len <= ASRC_M2M_BUFFER_SIZE) { + /* dma config for capture dma channel */ + ret = asrc_dmaconfig(pair_m2m, + pair->dma_chan[V4L_CAP], + fifo_addr, + vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0), + cap_dma_len, V4L_CAP, width); + if (ret) { + dev_err(dev, "cap dma config error\n"); + goto end; + } + } else if (cap_dma_len > ASRC_M2M_BUFFER_SIZE) { + dev_err(dev, "cap buffer size error\n"); + goto end; + } + + reinit_completion(&pair->complete[V4L_OUT]); + reinit_completion(&pair->complete[V4L_CAP]); + + /* Submit DMA request */ + dmaengine_submit(pair->desc[V4L_OUT]); + dma_async_issue_pending(pair->desc[V4L_OUT]->chan); + if (cap_dma_len > 0) { + dmaengine_submit(pair->desc[V4L_CAP]); + dma_async_issue_pending(pair->desc[V4L_CAP]->chan); + } + + asrc->m2m_start(pair); + + if (!wait_for_completion_interruptible_timeout(&pair->complete[V4L_OUT], 10 * HZ)) { + dev_err(dev, "out DMA task timeout\n"); + goto end; + } + + if (cap_dma_len > 0) { + if (!wait_for_completion_interruptible_timeout(&pair->complete[V4L_CAP], 10 * HZ)) { + dev_err(dev, "cap DMA task timeout\n"); + goto end; + } + } + + /* read the last words from FIFO */ + asrc_read_last_fifo(pair, vb2_plane_vaddr(&dst_buf->vb2_buf, 0), &cap_dma_len); + /* update payload length for capture */ + vb2_set_plane_payload(&dst_buf->vb2_buf, 0, cap_dma_len); + +end: + src_buf = v4l2_m2m_src_buf_remove(pair_m2m->fh.m2m_ctx); + dst_buf = v4l2_m2m_dst_buf_remove(pair_m2m->fh.m2m_ctx); + + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE); + v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE); + + v4l2_m2m_job_finish(m2m->m2m_dev, pair_m2m->fh.m2m_ctx); +} + +static int asrc_m2m_job_ready(void *priv) +{ + struct asrc_pair_m2m *pair_m2m = priv; + + if (v4l2_m2m_num_src_bufs_ready(pair_m2m->fh.m2m_ctx) > 0 && + v4l2_m2m_num_dst_bufs_ready(pair_m2m->fh.m2m_ctx) > 0) { + return 1; + } + + return 0; +} + +static const struct v4l2_m2m_ops asrc_m2m_ops = { + .job_ready = asrc_m2m_job_ready, + .device_run = asrc_m2m_device_run, +}; + +static int asrc_m2m_probe(struct platform_device *pdev) +{ + struct fsl_asrc_m2m_pdata *data = pdev->dev.platform_data; + struct device *dev = &pdev->dev; + struct asrc_m2m *m2m; + int ret; + + m2m = devm_kzalloc(dev, sizeof(struct asrc_m2m), GFP_KERNEL); + if (!m2m) + return -ENOMEM; + + m2m->pdata = *data; + m2m->pdev = pdev; + + ret = v4l2_device_register(dev, &m2m->v4l2_dev); + if (ret) { + dev_err(dev, "failed to register v4l2 device\n"); + goto err_register; + } + + m2m->m2m_dev = v4l2_m2m_init(&asrc_m2m_ops); + if (IS_ERR(m2m->m2m_dev)) { + dev_err(dev, "failed to register v4l2 device\n"); + ret = PTR_ERR(m2m->m2m_dev); + goto err_m2m; + } + + m2m->dec_vdev = video_device_alloc(); + if (!m2m->dec_vdev) { + dev_err(dev, "failed to register v4l2 device\n"); + ret = -ENOMEM; + goto err_vdev_alloc; + } + + mutex_init(&m2m->mlock); + + m2m->dec_vdev->fops = &asrc_m2m_fops; + m2m->dec_vdev->ioctl_ops = &asrc_m2m_ioctl_ops; + m2m->dec_vdev->minor = -1; + m2m->dec_vdev->release = video_device_release; + m2m->dec_vdev->lock = &m2m->mlock; /* lock for ioctl serialization */ + m2m->dec_vdev->v4l2_dev = &m2m->v4l2_dev; + m2m->dec_vdev->vfl_dir = VFL_DIR_M2M; + m2m->dec_vdev->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_AUDIO_M2M; + + ret = video_register_device(m2m->dec_vdev, VFL_TYPE_AUDIO, -1); + if (ret) { + dev_err(dev, "failed to register video device\n"); + goto err_vdev_register; + } + + video_set_drvdata(m2m->dec_vdev, m2m); + platform_set_drvdata(pdev, m2m); + pm_runtime_enable(&pdev->dev); + + return 0; + +err_vdev_register: + video_device_release(m2m->dec_vdev); +err_vdev_alloc: + v4l2_m2m_release(m2m->m2m_dev); +err_m2m: + v4l2_device_unregister(&m2m->v4l2_dev); +err_register: + return ret; +} + +static void asrc_m2m_remove(struct platform_device *pdev) +{ + struct asrc_m2m *m2m = platform_get_drvdata(pdev); + + pm_runtime_disable(&pdev->dev); + video_unregister_device(m2m->dec_vdev); + video_device_release(m2m->dec_vdev); + v4l2_m2m_release(m2m->m2m_dev); + v4l2_device_unregister(&m2m->v4l2_dev); +} + +#ifdef CONFIG_PM_SLEEP +/* suspend callback for m2m */ +static int asrc_m2m_suspend(struct device *dev) +{ + struct asrc_m2m *m2m = dev_get_drvdata(dev); + struct fsl_asrc *asrc = m2m->pdata.asrc; + struct fsl_asrc_pair *pair; + unsigned long lock_flags; + int i; + + for (i = 0; i < PAIR_CTX_NUM; i++) { + spin_lock_irqsave(&asrc->lock, lock_flags); + pair = asrc->pair[i]; + if (!pair || !pair->req_pair) { + spin_unlock_irqrestore(&asrc->lock, lock_flags); + continue; + } + if (!completion_done(&pair->complete[V4L_OUT])) { + if (pair->dma_chan[V4L_OUT]) + dmaengine_terminate_all(pair->dma_chan[V4L_OUT]); + asrc_input_dma_callback((void *)pair); + } + if (!completion_done(&pair->complete[V4L_CAP])) { + if (pair->dma_chan[V4L_CAP]) + dmaengine_terminate_all(pair->dma_chan[V4L_CAP]); + asrc_output_dma_callback((void *)pair); + } + + if (asrc->m2m_pair_suspend) + asrc->m2m_pair_suspend(pair); + + spin_unlock_irqrestore(&asrc->lock, lock_flags); + } + + return 0; +} + +static int asrc_m2m_resume(struct device *dev) +{ + struct asrc_m2m *m2m = dev_get_drvdata(dev); + struct fsl_asrc *asrc = m2m->pdata.asrc; + struct fsl_asrc_pair *pair; + unsigned long lock_flags; + int i; + + for (i = 0; i < PAIR_CTX_NUM; i++) { + spin_lock_irqsave(&asrc->lock, lock_flags); + pair = asrc->pair[i]; + if (!pair || !pair->req_pair) { + spin_unlock_irqrestore(&asrc->lock, lock_flags); + continue; + } + if (asrc->m2m_pair_resume) + asrc->m2m_pair_resume(pair); + + spin_unlock_irqrestore(&asrc->lock, lock_flags); + } + + return 0; +} +#endif + +static const struct dev_pm_ops asrc_m2m_pm_ops = { + SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(asrc_m2m_suspend, + asrc_m2m_resume) +}; + +static struct platform_driver asrc_m2m_driver = { + .probe = asrc_m2m_probe, + .remove_new = asrc_m2m_remove, + .driver = { + .name = "fsl_asrc_m2m", + .pm = &asrc_m2m_pm_ops, + }, +}; +module_platform_driver(asrc_m2m_driver); + +MODULE_DESCRIPTION("Freescale ASRC M2M driver"); +MODULE_LICENSE("GPL");
On 13/10/2023 10:31, Shengjiu Wang wrote:
Implement the ASRC memory to memory function using the v4l2 framework, user can use this function with v4l2 ioctl interface.
User send the output and capture buffer to driver and driver store the converted data to the capture buffer.
This feature can be shared by ASRC and EASRC drivers
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1248 +++++++++++++++++++++++++ 3 files changed, 1261 insertions(+) create mode 100644 drivers/media/platform/nxp/imx-asrc.c
diff --git a/drivers/media/platform/nxp/Kconfig b/drivers/media/platform/nxp/Kconfig index 40e3436669e2..8234644ee341 100644 --- a/drivers/media/platform/nxp/Kconfig +++ b/drivers/media/platform/nxp/Kconfig @@ -67,3 +67,15 @@ config VIDEO_MX2_EMMAPRP
source "drivers/media/platform/nxp/dw100/Kconfig" source "drivers/media/platform/nxp/imx-jpeg/Kconfig"
+config VIDEO_IMX_ASRC
- tristate "NXP i.MX ASRC M2M support"
- depends on V4L_MEM2MEM_DRIVERS
- depends on MEDIA_SUPPORT
- select VIDEOBUF2_DMA_CONTIG
- select V4L2_MEM2MEM_DEV
- help
Say Y if you want to add ASRC M2M support for NXP CPUs.It is a complement for ASRC M2P and ASRC P2M features.This option is only useful for out-of-tree drivers sincein-tree drivers select it automatically.diff --git a/drivers/media/platform/nxp/Makefile b/drivers/media/platform/nxp/Makefile index 4d90eb713652..1325675e34f5 100644 --- a/drivers/media/platform/nxp/Makefile +++ b/drivers/media/platform/nxp/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX8MQ_MIPI_CSI2) += imx8mq-mipi-csi2.o obj-$(CONFIG_VIDEO_IMX_MIPI_CSIS) += imx-mipi-csis.o obj-$(CONFIG_VIDEO_IMX_PXP) += imx-pxp.o obj-$(CONFIG_VIDEO_MX2_EMMAPRP) += mx2_emmaprp.o +obj-$(CONFIG_VIDEO_IMX_ASRC) += imx-asrc.o diff --git a/drivers/media/platform/nxp/imx-asrc.c b/drivers/media/platform/nxp/imx-asrc.c new file mode 100644 index 000000000000..373ca2b5ec90 --- /dev/null +++ b/drivers/media/platform/nxp/imx-asrc.c @@ -0,0 +1,1248 @@ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2014-2016 Freescale Semiconductor, Inc. +// Copyright (C) 2019-2023 NXP +// +// Freescale ASRC Memory to Memory (M2M) driver
+#include <linux/dma/imx-dma.h> +#include <linux/pm_runtime.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-device.h> +#include <media/v4l2-event.h> +#include <media/v4l2-fh.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-mem2mem.h> +#include <media/videobuf2-dma-contig.h> +#include <sound/dmaengine_pcm.h> +#include <sound/fsl_asrc_common.h>
+#define V4L_CAP OUT +#define V4L_OUT IN
+#define ASRC_xPUT_DMA_CALLBACK(dir) \
- (((dir) == V4L_OUT) ? asrc_input_dma_callback \
- : asrc_output_dma_callback)
+#define DIR_STR(dir) (dir) == V4L_OUT ? "out" : "cap"
+#define ASRC_M2M_BUFFER_SIZE (512 * 1024) +#define ASRC_M2M_PERIOD_SIZE (48 * 1024) +#define ASRC_M2M_SG_NUM (20)
Where do all these values come from? How do they relate? Some comments would be welcome.
Esp. ASRC_M2M_SG_NUM is a bit odd.
+struct asrc_fmt {
- u32 fourcc;
- snd_pcm_format_t format;
Do you need this field? If not, then you can drop the whole struct and just use u32 fourcc in the formats[] array.
+};
+struct asrc_pair_m2m {
- struct fsl_asrc_pair *pair;
- struct asrc_m2m *m2m;
- struct v4l2_fh fh;
- struct v4l2_ctrl_handler ctrl_handler;
- int channels[2];
- struct v4l2_ctrl_fixed_point src_rate;
- struct v4l2_ctrl_fixed_point dst_rate;
+};
+struct asrc_m2m {
- struct fsl_asrc_m2m_pdata pdata;
- struct v4l2_device v4l2_dev;
- struct v4l2_m2m_dev *m2m_dev;
- struct video_device *dec_vdev;
- struct mutex mlock; /* v4l2 ioctls serialization */
- struct platform_device *pdev;
+};
+static struct asrc_fmt formats[] = {
- {
.fourcc = V4L2_AUDIO_FMT_S8,- },
- {
.fourcc = V4L2_AUDIO_FMT_S16_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U16_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S24_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S24_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U24_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U24_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S32_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U32_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S20_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U20_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_FLOAT_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE,- },
+};
+#define NUM_FORMATS ARRAY_SIZE(formats)
+static snd_pcm_format_t convert_fourcc(u32 fourcc) {
- return (__force snd_pcm_format_t)v4l2_fourcc_to_audfmt(fourcc);
Is this cast something that should be done in the v4l2_fourcc_to_audfmt define instead?
+}
+static u32 find_fourcc(snd_pcm_format_t format) +{
- struct asrc_fmt *fmt;
- unsigned int k;
- for (k = 0; k < NUM_FORMATS; k++) {
fmt = &formats[k];fmt->format = convert_fourcc(fmt->fourcc);if (fmt->format == format)break;- }
- if (k == NUM_FORMATS)
return 0;- return formats[k].fourcc;
+}
+static snd_pcm_format_t find_format(u32 fourcc) +{
- struct asrc_fmt *fmt;
- unsigned int k;
- for (k = 0; k < NUM_FORMATS; k++) {
fmt = &formats[k];if (fmt->fourcc == fourcc)break;- }
- if (k == NUM_FORMATS)
return 0;- formats[k].format = convert_fourcc(formats[k].fourcc);
- return formats[k].format;
I don't really thing the format field makes any sense. You just keep setting it.
+}
+static int asrc_check_format(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 format) +{
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata;
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- u64 format_bit = 0;
- int i;
- for (i = 0; i < NUM_FORMATS; ++i) {
if (formats[i].fourcc == format) {formats[i].format = convert_fourcc(formats[i].fourcc);format_bit = pcm_format_to_bits(formats[i].format);break;}- }
- if (dir == IN && !(format_bit & pdata->fmt_in))
return find_fourcc(pair->sample_format[V4L_OUT]);- else if (dir == OUT && !(format_bit & pdata->fmt_out))
return find_fourcc(pair->sample_format[V4L_CAP]);- else
return format;+}
+static int asrc_check_channel(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 channels) +{
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata;
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- if (channels < pdata->chan_min || channels > pdata->chan_max)
return pair->channels;- else
return channels;+}
+static int asrc_check_rate(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 rate) +{
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata;
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- if (rate < pdata->rate_min || rate > pdata->rate_max)
return pair->rate[dir];- else
return rate;+}
+static inline struct asrc_pair_m2m *asrc_m2m_fh_to_ctx(struct v4l2_fh *fh) +{
- return container_of(fh, struct asrc_pair_m2m, fh);
+}
+/**
- asrc_read_last_fifo: read all the remaining data from FIFO
- @pair: Structure pointer of fsl_asrc_pair
- @dma_vaddr: virtual address of capture buffer
- @length: payload length of capture buffer
- */
+static void asrc_read_last_fifo(struct fsl_asrc_pair *pair, void *dma_vaddr, u32 *length) +{
- struct fsl_asrc *asrc = pair->asrc;
- enum asrc_pair_index index = pair->index;
- u32 i, reg, size, t_size = 0, width;
- u32 *reg32 = NULL;
- u16 *reg16 = NULL;
- u8 *reg24 = NULL;
- width = snd_pcm_format_physical_width(pair->sample_format[V4L_CAP]);
- if (width == 32)
reg32 = dma_vaddr + *length;- else if (width == 16)
reg16 = dma_vaddr + *length;- else
reg24 = dma_vaddr + *length;+retry:
- size = asrc->get_output_fifo_size(pair);
- if (size + *length > ASRC_M2M_BUFFER_SIZE)
goto end;- for (i = 0; i < size * pair->channels; i++) {
regmap_read(asrc->regmap, asrc->get_fifo_addr(OUT, index), ®);if (reg32) {*(reg32) = reg;reg32++;} else if (reg16) {*(reg16) = (u16)reg;reg16++;} else {*reg24++ = (u8)reg;*reg24++ = (u8)(reg >> 8);*reg24++ = (u8)(reg >> 16);}- }
- t_size += size;
- /* In case there is data left in FIFO */
- if (size)
goto retry;+end:
- /* Update payload length */
- if (reg32)
*length += t_size * pair->channels * 4;- else if (reg16)
*length += t_size * pair->channels * 2;- else
*length += t_size * pair->channels * 3;+}
+static int asrc_m2m_start_streaming(struct vb2_queue *q, unsigned int count) +{
- struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct fsl_asrc *asrc = pair->asrc;
- struct device *dev = &m2m->pdev->dev;
- struct vb2_v4l2_buffer *buf;
- bool request_flag = false;
- int ret;
- dev_dbg(dev, "Start streaming pair=%p, %d\n", pair, q->type);
- ret = pm_runtime_get_sync(dev);
- if (ret < 0) {
dev_err(dev, "Failed to power up asrc\n");goto err_pm_runtime;- }
- /* Request asrc pair/context */
- if (!pair->req_pair) {
/* flag for error handler of this function */request_flag = true;ret = asrc->request_pair(pair->channels, pair);if (ret) {dev_err(dev, "failed to request pair: %d\n", ret);goto err_request_pair;}ret = asrc->m2m_prepare(pair);if (ret) {dev_err(dev, "failed to start pair part one: %d\n", ret);goto err_start_part_one;}pair->req_pair = true;- }
- /* Request dma channels */
- if (V4L2_TYPE_IS_OUTPUT(q->type)) {
pair->dma_chan[V4L_OUT] = asrc->get_dma_channel(pair, IN);if (!pair->dma_chan[V4L_OUT]) {dev_err(dev, "[ctx%d] failed to get input DMA channel\n", pair->index);ret = -EBUSY;goto err_dma_channel;}- } else {
pair->dma_chan[V4L_CAP] = asrc->get_dma_channel(pair, OUT);if (!pair->dma_chan[V4L_CAP]) {dev_err(dev, "[ctx%d] failed to get output DMA channel\n", pair->index);ret = -EBUSY;goto err_dma_channel;}- }
- v4l2_m2m_update_start_streaming_state(pair_m2m->fh.m2m_ctx, q);
- return 0;
+err_dma_channel:
- if (request_flag && asrc->m2m_unprepare)
asrc->m2m_unprepare(pair);+err_start_part_one:
- if (request_flag)
asrc->release_pair(pair);+err_request_pair:
- pm_runtime_put_sync(dev);
+err_pm_runtime:
- /* Release buffers */
- if (V4L2_TYPE_IS_OUTPUT(q->type)) {
while ((buf = v4l2_m2m_src_buf_remove(pair_m2m->fh.m2m_ctx)))v4l2_m2m_buf_done(buf, VB2_BUF_STATE_QUEUED);- } else {
while ((buf = v4l2_m2m_dst_buf_remove(pair_m2m->fh.m2m_ctx)))v4l2_m2m_buf_done(buf, VB2_BUF_STATE_QUEUED);- }
- return ret;
+}
+static void asrc_m2m_stop_streaming(struct vb2_queue *q) +{
- struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q);
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct fsl_asrc *asrc = pair->asrc;
- struct device *dev = &m2m->pdev->dev;
- dev_dbg(dev, "Stop streaming pair=%p, %d\n", pair, q->type);
- v4l2_m2m_update_stop_streaming_state(pair_m2m->fh.m2m_ctx, q);
- /* Stop & release pair/context */
- if (asrc->m2m_stop)
asrc->m2m_stop(pair);- if (pair->req_pair) {
if (asrc->m2m_unprepare)asrc->m2m_unprepare(pair);asrc->release_pair(pair);pair->req_pair = false;- }
- /* Release dma channel */
- if (V4L2_TYPE_IS_OUTPUT(q->type)) {
if (pair->dma_chan[V4L_OUT])dma_release_channel(pair->dma_chan[V4L_OUT]);- } else {
if (pair->dma_chan[V4L_CAP])dma_release_channel(pair->dma_chan[V4L_CAP]);- }
- pm_runtime_put_sync(dev);
+}
+static int asrc_m2m_queue_setup(struct vb2_queue *q,
unsigned int *num_buffers, unsigned int *num_planes,unsigned int sizes[], struct device *alloc_devs[])+{
- struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- /* single buffer */
- *num_planes = 1;
This isn't quite right. This driver supports VIDIOC_CREATE_BUFS, which userspace can use to add new buffers on the fly. Note that there is no corresponding DELETE_BUFS ioctl to delete buffers, but work is in progress for that.
If a driver supports CREATE_BUFS, then queue_setup has to check whether the values passed by VIDIOC_CREATE_BUFS are valid.
That's done through:
u32 size;
if (V4L2_TYPE_IS_OUTPUT(q->type)) size = pair->buf_len[V4L_OUT]; else size = pair->buf_len[V4L_CAP];
if (*nplanes) return sizes[0] < size ? -EINVAL : 0;
*num_planes = 1; sizes[0] = size; return 0;
One of these days this rather ugly construction should be cleaned up.
Oh well...
Regards,
Hans
- /*
* The capture buffer size depends on output buffer size* and the convert ratio.** Here just use a fix length for capture and output buffer.* User need to care about it.*/- if (V4L2_TYPE_IS_OUTPUT(q->type))
sizes[0] = pair->buf_len[V4L_OUT];- else
sizes[0] = pair->buf_len[V4L_CAP];- return 0;
+}
+static void asrc_m2m_buf_queue(struct vb2_buffer *vb) +{
- struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
- struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(vb->vb2_queue);
- /* queue buffer */
- v4l2_m2m_buf_queue(pair_m2m->fh.m2m_ctx, vbuf);
+}
+static const struct vb2_ops asrc_m2m_qops = {
- .wait_prepare = vb2_ops_wait_prepare,
- .wait_finish = vb2_ops_wait_finish,
- .start_streaming = asrc_m2m_start_streaming,
- .stop_streaming = asrc_m2m_stop_streaming,
- .queue_setup = asrc_m2m_queue_setup,
- .buf_queue = asrc_m2m_buf_queue,
+};
+/* Init video buffer queue for src and dst. */ +static int asrc_m2m_queue_init(void *priv, struct vb2_queue *src_vq,
struct vb2_queue *dst_vq)+{
- struct asrc_pair_m2m *pair_m2m = priv;
- struct asrc_m2m *m2m = pair_m2m->m2m;
- int ret;
- src_vq->type = V4L2_BUF_TYPE_AUDIO_OUTPUT;
- src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
- src_vq->drv_priv = pair_m2m;
- src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
- src_vq->ops = &asrc_m2m_qops;
- src_vq->mem_ops = &vb2_dma_contig_memops;
- src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
- src_vq->lock = &m2m->mlock;
- src_vq->dev = &m2m->pdev->dev;
- src_vq->min_buffers_needed = 1;
- ret = vb2_queue_init(src_vq);
- if (ret)
return ret;- dst_vq->type = V4L2_BUF_TYPE_AUDIO_CAPTURE;
- dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;
- dst_vq->drv_priv = pair_m2m;
- dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
- dst_vq->ops = &asrc_m2m_qops;
- dst_vq->mem_ops = &vb2_dma_contig_memops;
- dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
- dst_vq->lock = &m2m->mlock;
- dst_vq->dev = &m2m->pdev->dev;
- dst_vq->min_buffers_needed = 1;
- ret = vb2_queue_init(dst_vq);
- return ret;
+}
+static int asrc_m2m_op_s_ctrl(struct v4l2_ctrl *ctrl) +{
- struct asrc_pair_m2m *pair_m2m =
container_of(ctrl->handler, struct asrc_pair_m2m, ctrl_handler);- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct fsl_asrc *asrc = pair->asrc;
- int src_rate_int, src_rate_frac;
- int dst_rate_int, dst_rate_frac;
- int new_rate, new_frac;
- u64 src_rate, dst_rate;
- u64 ratio_pre, ratio_cur;
- s64 ratio_diff;
- int ret = 0;
- switch (ctrl->id) {
- case V4L2_CID_ASRC_SOURCE_RATE:
new_rate = ctrl->p_new.p_fixed_point->integer;new_frac = ctrl->p_new.p_fixed_point->fractional;src_rate_int = asrc_check_rate(pair_m2m, IN, new_rate);if (src_rate_int != new_rate ||(pair_m2m->src_rate.integer > 0 &&src_rate_int != pair_m2m->src_rate.integer))return -EINVAL;pair->rate[V4L_OUT] = src_rate_int;if (new_frac != pair_m2m->src_rate.fractional &&new_rate == pair_m2m->src_rate.integer &&pair_m2m->dst_rate.integer > 0) {/** use maximum rate 768kHz as limitation, then we can shift right 21 bit for* division*/src_rate_frac = pair_m2m->src_rate.fractional;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate_int = pair_m2m->dst_rate.integer;dst_rate_frac = pair_m2m->dst_rate.fractional;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_pre = src_rate;src_rate_frac = new_frac;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_cur = src_rate;ratio_diff = ratio_cur - ratio_pre;asrc->m2m_set_ratio_mod(pair, ratio_diff << 11);}pair_m2m->src_rate.integer = new_rate;pair_m2m->src_rate.fractional = new_frac;break;- case V4L2_CID_ASRC_DEST_RATE:
new_rate = ctrl->p_new.p_fixed_point->integer;new_frac = ctrl->p_new.p_fixed_point->fractional;dst_rate_int = asrc_check_rate(pair_m2m, OUT, new_rate);if (dst_rate_int != new_rate ||(pair_m2m->dst_rate.integer > 0 &&dst_rate_int != pair_m2m->dst_rate.integer))return -EINVAL;pair->rate[V4L_CAP] = dst_rate_int;if (new_frac != pair_m2m->dst_rate.fractional &&new_rate == pair_m2m->dst_rate.integer &&pair_m2m->src_rate.integer > 0) {/** use maximum rate 768kHz as limitation, then we can shift right 21 bit for* division*/src_rate_int = pair_m2m->src_rate.integer;src_rate_frac = pair_m2m->src_rate.fractional;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate_frac = pair_m2m->dst_rate.fractional;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_pre = src_rate;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate_int = new_rate;dst_rate_frac = new_frac;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_cur = src_rate;ratio_diff = ratio_cur - ratio_pre;/* convert ratio_diff to Q31*/asrc->m2m_set_ratio_mod(pair, ratio_diff << 11);
This is very similar to the other control. You really just want to pass two rates (source and dest) and let that function calculate the ratio mod.
}pair_m2m->dst_rate.integer = new_rate;pair_m2m->dst_rate.fractional = new_frac;break;- default:
ret = -EINVAL;break;- }
- return ret;
+}
+static const struct v4l2_ctrl_ops asrc_m2m_ctrl_ops = {
- .s_ctrl = asrc_m2m_op_s_ctrl,
+};
+static const struct v4l2_ctrl_config asrc_src_rate_control = {
- .ops = &asrc_m2m_ctrl_ops,
- .id = V4L2_CID_ASRC_SOURCE_RATE,
- .name = "asrc source sample rate",
How about "Audio Source Sample Rate"?
But we want these controls as standard types (not driver specific), so this should be added to drivers/media/v4l2-core/v4l2-ctrls-defs.c.
And if we just use an s64 to store the fixed point, then you can just call v4l2_ctrl_new_std().
But you probably want to add some helper defines to split a fixed point value into integer and fractional parts, and to construct one.
- .type = V4L2_CTRL_TYPE_FIXED_POINT,
- .min = 0,
- .max = 0x7fffffff,
- .def = 8000,
- .flags = V4L2_CTRL_FLAG_UPDATE,
+};
+static const struct v4l2_ctrl_config asrc_dst_rate_control = {
- .ops = &asrc_m2m_ctrl_ops,
- .id = V4L2_CID_ASRC_DEST_RATE,
- .name = "asrc dest sample rate",
- .type = V4L2_CTRL_TYPE_FIXED_POINT,
- .min = 0,
- .max = 0x7fffffff,
- .def = 8000,
- .flags = V4L2_CTRL_FLAG_UPDATE,
+};
+/* system callback for open() */ +static int asrc_m2m_open(struct file *file) +{
- struct asrc_m2m *m2m = video_drvdata(file);
- struct fsl_asrc *asrc = m2m->pdata.asrc;
- struct video_device *vdev = video_devdata(file);
- struct fsl_asrc_pair *pair;
- struct asrc_pair_m2m *pair_m2m;
- int ret = 0;
- if (mutex_lock_interruptible(&m2m->mlock))
return -ERESTARTSYS;- pair = kzalloc(sizeof(*pair) + asrc->pair_priv_size, GFP_KERNEL);
- if (!pair) {
ret = -ENOMEM;goto err_alloc_pair;- }
- pair_m2m = kzalloc(sizeof(*pair_m2m), GFP_KERNEL);
- if (!pair_m2m) {
ret = -ENOMEM;goto err_alloc_pair_m2m;- }
- pair->private = (void *)pair + sizeof(struct fsl_asrc_pair);
- pair->asrc = asrc;
- pair->buf_len[V4L_OUT] = ASRC_M2M_BUFFER_SIZE;
- pair->buf_len[V4L_CAP] = ASRC_M2M_BUFFER_SIZE;
- pair->channels = 2;
- pair->rate[V4L_OUT] = 8000;
- pair->rate[V4L_CAP] = 8000;
- pair->sample_format[V4L_OUT] = SNDRV_PCM_FORMAT_S16_LE;
- pair->sample_format[V4L_CAP] = SNDRV_PCM_FORMAT_S16_LE;
- init_completion(&pair->complete[V4L_OUT]);
- init_completion(&pair->complete[V4L_CAP]);
- v4l2_fh_init(&pair_m2m->fh, vdev);
- v4l2_fh_add(&pair_m2m->fh);
- file->private_data = &pair_m2m->fh;
- pair_m2m->pair = pair;
- pair_m2m->m2m = m2m;
- /* m2m context init */
- pair_m2m->fh.m2m_ctx = v4l2_m2m_ctx_init(m2m->m2m_dev, pair_m2m,
asrc_m2m_queue_init);- if (IS_ERR(pair_m2m->fh.m2m_ctx)) {
ret = PTR_ERR(pair_m2m->fh.m2m_ctx);goto err_ctx_init;- }
- v4l2_ctrl_handler_init(&pair_m2m->ctrl_handler, 2);
- /* use V4L2_CID_GAIN for ratio update control */
- v4l2_ctrl_new_custom(&pair_m2m->ctrl_handler, &asrc_src_rate_control, NULL);
- v4l2_ctrl_new_custom(&pair_m2m->ctrl_handler, &asrc_dst_rate_control, NULL);
- if (pair_m2m->ctrl_handler.error) {
ret = pair_m2m->ctrl_handler.error;v4l2_ctrl_handler_free(&pair_m2m->ctrl_handler);goto err_ctrl_handler;- }
- pair_m2m->fh.ctrl_handler = &pair_m2m->ctrl_handler;
- mutex_unlock(&m2m->mlock);
- return 0;
+err_ctrl_handler:
- v4l2_m2m_ctx_release(pair_m2m->fh.m2m_ctx);
+err_ctx_init:
- v4l2_fh_del(&pair_m2m->fh);
- v4l2_fh_exit(&pair_m2m->fh);
- kfree(pair_m2m);
+err_alloc_pair_m2m:
- kfree(pair);
+err_alloc_pair:
- mutex_unlock(&m2m->mlock);
- return ret;
+}
+static int asrc_m2m_release(struct file *file) +{
- struct asrc_m2m *m2m = video_drvdata(file);
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(file->private_data);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- mutex_lock(&m2m->mlock);
- v4l2_ctrl_handler_free(&pair_m2m->ctrl_handler);
- v4l2_m2m_ctx_release(pair_m2m->fh.m2m_ctx);
- v4l2_fh_del(&pair_m2m->fh);
- v4l2_fh_exit(&pair_m2m->fh);
- kfree(pair_m2m);
- kfree(pair);
- mutex_unlock(&m2m->mlock);
- return 0;
+}
+static const struct v4l2_file_operations asrc_m2m_fops = {
- .owner = THIS_MODULE,
- .open = asrc_m2m_open,
- .release = asrc_m2m_release,
- .poll = v4l2_m2m_fop_poll,
- .unlocked_ioctl = video_ioctl2,
- .mmap = v4l2_m2m_fop_mmap,
+};
+static int asrc_m2m_querycap(struct file *file, void *priv,
struct v4l2_capability *cap)+{
- strscpy(cap->driver, "asrc m2m", sizeof(cap->driver));
- strscpy(cap->card, "asrc m2m", sizeof(cap->card));
This is rather ugly. It should at least mention imx, right?
- cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_AUDIO_M2M;
- cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
You can drop this last line, that's done for you.
- return 0;
+}
+static int enum_fmt(struct v4l2_fmtdesc *f, u64 fmtbit) +{
- int i, num;
- struct asrc_fmt *fmt;
- num = 0;
- for (i = 0; i < NUM_FORMATS; ++i) {
formats[i].format = convert_fourcc(formats[i].fourcc);if (pcm_format_to_bits(formats[i].format) & fmtbit) {if (num == f->index)break;/** Correct type but haven't reached our index yet,* just increment per-type index*/++num;}- }
- if (i < NUM_FORMATS) {
/* Format found */fmt = &formats[i];f->pixelformat = fmt->fourcc;return 0;- }
- return -EINVAL;
+}
+static int asrc_m2m_enum_fmt_aud_cap(struct file *file, void *fh,
struct v4l2_fmtdesc *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- struct asrc_m2m *m2m = pair_m2m->m2m;
- return enum_fmt(f, m2m->pdata.fmt_out);
+}
+static int asrc_m2m_enum_fmt_aud_out(struct file *file, void *fh,
struct v4l2_fmtdesc *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- struct asrc_m2m *m2m = pair_m2m->m2m;
- return enum_fmt(f, m2m->pdata.fmt_in);
+}
+static int asrc_m2m_g_fmt_aud_cap(struct file *file, void *fh,
struct v4l2_format *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- f->fmt.audio.channels = pair->channels;
- f->fmt.audio.buffersize = pair->buf_len[V4L_CAP];
- f->fmt.audio.audioformat = find_fourcc(pair->sample_format[V4L_CAP]);
- return 0;
+}
+static int asrc_m2m_g_fmt_aud_out(struct file *file, void *fh,
struct v4l2_format *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- f->fmt.audio.channels = pair->channels;
- f->fmt.audio.buffersize = pair->buf_len[V4L_OUT];
- f->fmt.audio.audioformat = find_fourcc(pair->sample_format[V4L_OUT]);
- return 0;
+}
+/* output for asrc */ +static int asrc_m2m_s_fmt_aud_cap(struct file *file, void *fh,
struct v4l2_format *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct device *dev = &m2m->pdev->dev;
- f->fmt.audio.audioformat = asrc_check_format(pair_m2m, OUT, f->fmt.audio.audioformat);
- f->fmt.audio.channels = asrc_check_channel(pair_m2m, OUT, f->fmt.audio.channels);
- if (pair_m2m->channels[V4L_CAP] > 0 &&
pair_m2m->channels[V4L_CAP] != f->fmt.audio.channels) {dev_err(dev, "channels don't match for cap and out\n");return -EINVAL;- }
- pair_m2m->channels[V4L_CAP] = f->fmt.audio.channels;
- pair->channels = f->fmt.audio.channels;
- pair->sample_format[V4L_CAP] = find_format(f->fmt.audio.audioformat);
- return 0;
+}
+/* input for asrc */ +static int asrc_m2m_s_fmt_aud_out(struct file *file, void *fh,
struct v4l2_format *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct device *dev = &m2m->pdev->dev;
- f->fmt.audio.audioformat = asrc_check_format(pair_m2m, IN, f->fmt.audio.audioformat);
- f->fmt.audio.channels = asrc_check_channel(pair_m2m, IN, f->fmt.audio.channels);
- if (pair_m2m->channels[V4L_OUT] > 0 &&
pair_m2m->channels[V4L_OUT] != f->fmt.audio.channels) {dev_err(dev, "channels don't match for cap and out\n");return -EINVAL;- }
- pair_m2m->channels[V4L_OUT] = f->fmt.audio.channels;
- pair->channels = f->fmt.audio.channels;
- pair->sample_format[V4L_OUT] = find_format(f->fmt.audio.audioformat);
- return 0;
+}
+static int asrc_m2m_try_fmt_audio_cap(struct file *file, void *fh,
struct v4l2_format *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- f->fmt.audio.audioformat = asrc_check_format(pair_m2m, OUT, f->fmt.audio.audioformat);
- f->fmt.audio.channels = asrc_check_channel(pair_m2m, OUT, f->fmt.audio.channels);
- return 0;
+}
+static int asrc_m2m_try_fmt_audio_out(struct file *file, void *fh,
struct v4l2_format *f)+{
- struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);
- f->fmt.audio.audioformat = asrc_check_format(pair_m2m, IN, f->fmt.audio.audioformat);
- f->fmt.audio.channels = asrc_check_channel(pair_m2m, IN, f->fmt.audio.channels);
- return 0;
+}
+static const struct v4l2_ioctl_ops asrc_m2m_ioctl_ops = {
- .vidioc_querycap = asrc_m2m_querycap,
- .vidioc_enum_fmt_audio_cap = asrc_m2m_enum_fmt_aud_cap,
- .vidioc_enum_fmt_audio_out = asrc_m2m_enum_fmt_aud_out,
- .vidioc_g_fmt_audio_cap = asrc_m2m_g_fmt_aud_cap,
- .vidioc_g_fmt_audio_out = asrc_m2m_g_fmt_aud_out,
- .vidioc_s_fmt_audio_cap = asrc_m2m_s_fmt_aud_cap,
- .vidioc_s_fmt_audio_out = asrc_m2m_s_fmt_aud_out,
- .vidioc_try_fmt_audio_cap = asrc_m2m_try_fmt_audio_cap,
- .vidioc_try_fmt_audio_out = asrc_m2m_try_fmt_audio_out,
- .vidioc_qbuf = v4l2_m2m_ioctl_qbuf,
- .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
- .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
- .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,
- .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
- .vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
- .vidioc_streamon = v4l2_m2m_ioctl_streamon,
- .vidioc_streamoff = v4l2_m2m_ioctl_streamoff,
- .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
- .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+/* dma complete callback */ +static void asrc_input_dma_callback(void *data) +{
- struct fsl_asrc_pair *pair = (struct fsl_asrc_pair *)data;
- complete(&pair->complete[V4L_OUT]);
+}
+/* dma complete callback */ +static void asrc_output_dma_callback(void *data) +{
- struct fsl_asrc_pair *pair = (struct fsl_asrc_pair *)data;
- complete(&pair->complete[V4L_CAP]);
+}
+/* config dma channel */ +static int asrc_dmaconfig(struct asrc_pair_m2m *pair_m2m,
struct dma_chan *chan,u32 dma_addr, dma_addr_t buf_addr, u32 buf_len,int dir, int width)+{
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct fsl_asrc *asrc = pair->asrc;
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct device *dev = &m2m->pdev->dev;
- struct dma_slave_config slave_config;
- struct scatterlist sg[ASRC_M2M_SG_NUM];
- enum dma_slave_buswidth buswidth;
- unsigned int sg_len, max_period_size;
- int ret, i;
- switch (width) {
- case 8:
buswidth = DMA_SLAVE_BUSWIDTH_1_BYTE;break;- case 16:
buswidth = DMA_SLAVE_BUSWIDTH_2_BYTES;break;- case 24:
buswidth = DMA_SLAVE_BUSWIDTH_3_BYTES;break;- case 32:
buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES;break;- default:
dev_err(dev, "invalid word width\n");return -EINVAL;- }
- memset(&slave_config, 0, sizeof(slave_config));
- if (dir == V4L_OUT) {
slave_config.direction = DMA_MEM_TO_DEV;slave_config.dst_addr = dma_addr;slave_config.dst_addr_width = buswidth;slave_config.dst_maxburst = asrc->m2m_get_maxburst(IN, pair);- } else {
slave_config.direction = DMA_DEV_TO_MEM;slave_config.src_addr = dma_addr;slave_config.src_addr_width = buswidth;slave_config.src_maxburst = asrc->m2m_get_maxburst(OUT, pair);- }
- ret = dmaengine_slave_config(chan, &slave_config);
- if (ret) {
dev_err(dev, "failed to config dmaengine for %s task: %d\n",DIR_STR(dir), ret);return -EINVAL;- }
- max_period_size = rounddown(ASRC_M2M_PERIOD_SIZE, width * pair->channels / 8);
- /* scatter gather mode */
- sg_len = buf_len / max_period_size;
- if (buf_len % max_period_size)
sg_len += 1;- sg_init_table(sg, sg_len);
- for (i = 0; i < (sg_len - 1); i++) {
sg_dma_address(&sg[i]) = buf_addr + i * max_period_size;sg_dma_len(&sg[i]) = max_period_size;- }
- sg_dma_address(&sg[i]) = buf_addr + i * max_period_size;
- sg_dma_len(&sg[i]) = buf_len - i * max_period_size;
- pair->desc[dir] = dmaengine_prep_slave_sg(chan, sg, sg_len,
slave_config.direction,DMA_PREP_INTERRUPT);- if (!pair->desc[dir]) {
dev_err(dev, "failed to prepare dmaengine for %s task\n", DIR_STR(dir));return -EINVAL;- }
- pair->desc[dir]->callback = ASRC_xPUT_DMA_CALLBACK(dir);
- pair->desc[dir]->callback_param = pair;
- return 0;
+}
+/* main function of converter */ +static void asrc_m2m_device_run(void *priv) +{
- struct asrc_pair_m2m *pair_m2m = priv;
- struct fsl_asrc_pair *pair = pair_m2m->pair;
- struct asrc_m2m *m2m = pair_m2m->m2m;
- struct fsl_asrc *asrc = pair->asrc;
- struct device *dev = &m2m->pdev->dev;
- enum asrc_pair_index index = pair->index;
- struct vb2_v4l2_buffer *src_buf, *dst_buf;
- unsigned int out_buf_len;
- unsigned int cap_dma_len;
- unsigned int width;
- u32 fifo_addr;
- int ret;
- src_buf = v4l2_m2m_next_src_buf(pair_m2m->fh.m2m_ctx);
- dst_buf = v4l2_m2m_next_dst_buf(pair_m2m->fh.m2m_ctx);
- width = snd_pcm_format_physical_width(pair->sample_format[V4L_OUT]);
- fifo_addr = asrc->paddr + asrc->get_fifo_addr(IN, index);
- out_buf_len = vb2_get_plane_payload(&src_buf->vb2_buf, 0);
- if (out_buf_len < width * pair->channels / 8 ||
out_buf_len > ASRC_M2M_BUFFER_SIZE ||out_buf_len % (width * pair->channels / 8)) {dev_err(dev, "out buffer size is error: [%d]\n", out_buf_len);goto end;- }
- /* dma config for output dma channel */
- ret = asrc_dmaconfig(pair_m2m,
pair->dma_chan[V4L_OUT],fifo_addr,vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0),out_buf_len, V4L_OUT, width);- if (ret) {
dev_err(dev, "out dma config error\n");goto end;- }
- width = snd_pcm_format_physical_width(pair->sample_format[V4L_CAP]);
- fifo_addr = asrc->paddr + asrc->get_fifo_addr(OUT, index);
- cap_dma_len = asrc->m2m_calc_out_len(pair, out_buf_len);
- if (cap_dma_len > 0 && cap_dma_len <= ASRC_M2M_BUFFER_SIZE) {
/* dma config for capture dma channel */ret = asrc_dmaconfig(pair_m2m,pair->dma_chan[V4L_CAP],fifo_addr,vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0),cap_dma_len, V4L_CAP, width);if (ret) {dev_err(dev, "cap dma config error\n");goto end;}- } else if (cap_dma_len > ASRC_M2M_BUFFER_SIZE) {
dev_err(dev, "cap buffer size error\n");goto end;- }
- reinit_completion(&pair->complete[V4L_OUT]);
- reinit_completion(&pair->complete[V4L_CAP]);
- /* Submit DMA request */
- dmaengine_submit(pair->desc[V4L_OUT]);
- dma_async_issue_pending(pair->desc[V4L_OUT]->chan);
- if (cap_dma_len > 0) {
dmaengine_submit(pair->desc[V4L_CAP]);dma_async_issue_pending(pair->desc[V4L_CAP]->chan);- }
- asrc->m2m_start(pair);
- if (!wait_for_completion_interruptible_timeout(&pair->complete[V4L_OUT], 10 * HZ)) {
dev_err(dev, "out DMA task timeout\n");goto end;- }
- if (cap_dma_len > 0) {
if (!wait_for_completion_interruptible_timeout(&pair->complete[V4L_CAP], 10 * HZ)) {dev_err(dev, "cap DMA task timeout\n");goto end;}- }
- /* read the last words from FIFO */
- asrc_read_last_fifo(pair, vb2_plane_vaddr(&dst_buf->vb2_buf, 0), &cap_dma_len);
- /* update payload length for capture */
- vb2_set_plane_payload(&dst_buf->vb2_buf, 0, cap_dma_len);
+end:
- src_buf = v4l2_m2m_src_buf_remove(pair_m2m->fh.m2m_ctx);
- dst_buf = v4l2_m2m_dst_buf_remove(pair_m2m->fh.m2m_ctx);
- v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE);
- v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE);
- v4l2_m2m_job_finish(m2m->m2m_dev, pair_m2m->fh.m2m_ctx);
+}
+static int asrc_m2m_job_ready(void *priv) +{
- struct asrc_pair_m2m *pair_m2m = priv;
- if (v4l2_m2m_num_src_bufs_ready(pair_m2m->fh.m2m_ctx) > 0 &&
v4l2_m2m_num_dst_bufs_ready(pair_m2m->fh.m2m_ctx) > 0) {return 1;- }
- return 0;
+}
+static const struct v4l2_m2m_ops asrc_m2m_ops = {
- .job_ready = asrc_m2m_job_ready,
- .device_run = asrc_m2m_device_run,
+};
+static int asrc_m2m_probe(struct platform_device *pdev) +{
- struct fsl_asrc_m2m_pdata *data = pdev->dev.platform_data;
- struct device *dev = &pdev->dev;
- struct asrc_m2m *m2m;
- int ret;
- m2m = devm_kzalloc(dev, sizeof(struct asrc_m2m), GFP_KERNEL);
- if (!m2m)
return -ENOMEM;- m2m->pdata = *data;
- m2m->pdev = pdev;
- ret = v4l2_device_register(dev, &m2m->v4l2_dev);
- if (ret) {
dev_err(dev, "failed to register v4l2 device\n");goto err_register;- }
- m2m->m2m_dev = v4l2_m2m_init(&asrc_m2m_ops);
- if (IS_ERR(m2m->m2m_dev)) {
dev_err(dev, "failed to register v4l2 device\n");ret = PTR_ERR(m2m->m2m_dev);goto err_m2m;- }
- m2m->dec_vdev = video_device_alloc();
- if (!m2m->dec_vdev) {
dev_err(dev, "failed to register v4l2 device\n");ret = -ENOMEM;goto err_vdev_alloc;- }
- mutex_init(&m2m->mlock);
- m2m->dec_vdev->fops = &asrc_m2m_fops;
- m2m->dec_vdev->ioctl_ops = &asrc_m2m_ioctl_ops;
- m2m->dec_vdev->minor = -1;
- m2m->dec_vdev->release = video_device_release;
- m2m->dec_vdev->lock = &m2m->mlock; /* lock for ioctl serialization */
- m2m->dec_vdev->v4l2_dev = &m2m->v4l2_dev;
- m2m->dec_vdev->vfl_dir = VFL_DIR_M2M;
- m2m->dec_vdev->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_AUDIO_M2M;
- ret = video_register_device(m2m->dec_vdev, VFL_TYPE_AUDIO, -1);
- if (ret) {
dev_err(dev, "failed to register video device\n");goto err_vdev_register;- }
- video_set_drvdata(m2m->dec_vdev, m2m);
- platform_set_drvdata(pdev, m2m);
- pm_runtime_enable(&pdev->dev);
- return 0;
+err_vdev_register:
- video_device_release(m2m->dec_vdev);
+err_vdev_alloc:
- v4l2_m2m_release(m2m->m2m_dev);
+err_m2m:
- v4l2_device_unregister(&m2m->v4l2_dev);
+err_register:
- return ret;
+}
+static void asrc_m2m_remove(struct platform_device *pdev) +{
- struct asrc_m2m *m2m = platform_get_drvdata(pdev);
- pm_runtime_disable(&pdev->dev);
- video_unregister_device(m2m->dec_vdev);
- video_device_release(m2m->dec_vdev);
- v4l2_m2m_release(m2m->m2m_dev);
- v4l2_device_unregister(&m2m->v4l2_dev);
+}
+#ifdef CONFIG_PM_SLEEP +/* suspend callback for m2m */ +static int asrc_m2m_suspend(struct device *dev) +{
- struct asrc_m2m *m2m = dev_get_drvdata(dev);
- struct fsl_asrc *asrc = m2m->pdata.asrc;
- struct fsl_asrc_pair *pair;
- unsigned long lock_flags;
- int i;
- for (i = 0; i < PAIR_CTX_NUM; i++) {
spin_lock_irqsave(&asrc->lock, lock_flags);pair = asrc->pair[i];if (!pair || !pair->req_pair) {spin_unlock_irqrestore(&asrc->lock, lock_flags);continue;}if (!completion_done(&pair->complete[V4L_OUT])) {if (pair->dma_chan[V4L_OUT])dmaengine_terminate_all(pair->dma_chan[V4L_OUT]);asrc_input_dma_callback((void *)pair);}if (!completion_done(&pair->complete[V4L_CAP])) {if (pair->dma_chan[V4L_CAP])dmaengine_terminate_all(pair->dma_chan[V4L_CAP]);asrc_output_dma_callback((void *)pair);}if (asrc->m2m_pair_suspend)asrc->m2m_pair_suspend(pair);spin_unlock_irqrestore(&asrc->lock, lock_flags);- }
- return 0;
+}
+static int asrc_m2m_resume(struct device *dev) +{
- struct asrc_m2m *m2m = dev_get_drvdata(dev);
- struct fsl_asrc *asrc = m2m->pdata.asrc;
- struct fsl_asrc_pair *pair;
- unsigned long lock_flags;
- int i;
- for (i = 0; i < PAIR_CTX_NUM; i++) {
spin_lock_irqsave(&asrc->lock, lock_flags);pair = asrc->pair[i];if (!pair || !pair->req_pair) {spin_unlock_irqrestore(&asrc->lock, lock_flags);continue;}if (asrc->m2m_pair_resume)asrc->m2m_pair_resume(pair);spin_unlock_irqrestore(&asrc->lock, lock_flags);- }
- return 0;
+} +#endif
+static const struct dev_pm_ops asrc_m2m_pm_ops = {
- SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(asrc_m2m_suspend,
asrc_m2m_resume)+};
+static struct platform_driver asrc_m2m_driver = {
- .probe = asrc_m2m_probe,
- .remove_new = asrc_m2m_remove,
- .driver = {
.name = "fsl_asrc_m2m",.pm = &asrc_m2m_pm_ops,- },
+}; +module_platform_driver(asrc_m2m_driver);
+MODULE_DESCRIPTION("Freescale ASRC M2M driver"); +MODULE_LICENSE("GPL");
Regards,
Hans
On Mon, Oct 16, 2023 at 10:01 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 13/10/2023 10:31, Shengjiu Wang wrote:
Implement the ASRC memory to memory function using the v4l2 framework, user can use this function with v4l2 ioctl interface.
User send the output and capture buffer to driver and driver store the converted data to the capture buffer.
This feature can be shared by ASRC and EASRC drivers
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1248 +++++++++++++++++++++++++ 3 files changed, 1261 insertions(+) create mode 100644 drivers/media/platform/nxp/imx-asrc.c
diff --git a/drivers/media/platform/nxp/Kconfig b/drivers/media/platform/nxp/Kconfig index 40e3436669e2..8234644ee341 100644 --- a/drivers/media/platform/nxp/Kconfig +++ b/drivers/media/platform/nxp/Kconfig @@ -67,3 +67,15 @@ config VIDEO_MX2_EMMAPRP
source "drivers/media/platform/nxp/dw100/Kconfig" source "drivers/media/platform/nxp/imx-jpeg/Kconfig"
+config VIDEO_IMX_ASRC
tristate "NXP i.MX ASRC M2M support"depends on V4L_MEM2MEM_DRIVERSdepends on MEDIA_SUPPORTselect VIDEOBUF2_DMA_CONTIGselect V4L2_MEM2MEM_DEVhelpSay Y if you want to add ASRC M2M support for NXP CPUs.It is a complement for ASRC M2P and ASRC P2M features.This option is only useful for out-of-tree drivers sincein-tree drivers select it automatically.diff --git a/drivers/media/platform/nxp/Makefile b/drivers/media/platform/nxp/Makefile index 4d90eb713652..1325675e34f5 100644 --- a/drivers/media/platform/nxp/Makefile +++ b/drivers/media/platform/nxp/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX8MQ_MIPI_CSI2) += imx8mq-mipi-csi2.o obj-$(CONFIG_VIDEO_IMX_MIPI_CSIS) += imx-mipi-csis.o obj-$(CONFIG_VIDEO_IMX_PXP) += imx-pxp.o obj-$(CONFIG_VIDEO_MX2_EMMAPRP) += mx2_emmaprp.o +obj-$(CONFIG_VIDEO_IMX_ASRC) += imx-asrc.o diff --git a/drivers/media/platform/nxp/imx-asrc.c b/drivers/media/platform/nxp/imx-asrc.c new file mode 100644 index 000000000000..373ca2b5ec90 --- /dev/null +++ b/drivers/media/platform/nxp/imx-asrc.c @@ -0,0 +1,1248 @@ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2014-2016 Freescale Semiconductor, Inc. +// Copyright (C) 2019-2023 NXP +// +// Freescale ASRC Memory to Memory (M2M) driver
+#include <linux/dma/imx-dma.h> +#include <linux/pm_runtime.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-device.h> +#include <media/v4l2-event.h> +#include <media/v4l2-fh.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-mem2mem.h> +#include <media/videobuf2-dma-contig.h> +#include <sound/dmaengine_pcm.h> +#include <sound/fsl_asrc_common.h>
+#define V4L_CAP OUT +#define V4L_OUT IN
+#define ASRC_xPUT_DMA_CALLBACK(dir) \
(((dir) == V4L_OUT) ? asrc_input_dma_callback \: asrc_output_dma_callback)+#define DIR_STR(dir) (dir) == V4L_OUT ? "out" : "cap"
+#define ASRC_M2M_BUFFER_SIZE (512 * 1024) +#define ASRC_M2M_PERIOD_SIZE (48 * 1024) +#define ASRC_M2M_SG_NUM (20)
Where do all these values come from? How do they relate? Some comments would be welcome.
Esp. ASRC_M2M_SG_NUM is a bit odd.
+struct asrc_fmt {
u32 fourcc;snd_pcm_format_t format;Do you need this field? If not, then you can drop the whole struct and just use u32 fourcc in the formats[] array.
+};
+struct asrc_pair_m2m {
struct fsl_asrc_pair *pair;struct asrc_m2m *m2m;struct v4l2_fh fh;struct v4l2_ctrl_handler ctrl_handler;int channels[2];struct v4l2_ctrl_fixed_point src_rate;struct v4l2_ctrl_fixed_point dst_rate;+};
+struct asrc_m2m {
struct fsl_asrc_m2m_pdata pdata;struct v4l2_device v4l2_dev;struct v4l2_m2m_dev *m2m_dev;struct video_device *dec_vdev;struct mutex mlock; /* v4l2 ioctls serialization */struct platform_device *pdev;+};
+static struct asrc_fmt formats[] = {
{.fourcc = V4L2_AUDIO_FMT_S8,},{.fourcc = V4L2_AUDIO_FMT_S16_LE,},{.fourcc = V4L2_AUDIO_FMT_U16_LE,},{.fourcc = V4L2_AUDIO_FMT_S24_LE,},{.fourcc = V4L2_AUDIO_FMT_S24_3LE,},{.fourcc = V4L2_AUDIO_FMT_U24_LE,},{.fourcc = V4L2_AUDIO_FMT_U24_3LE,},{.fourcc = V4L2_AUDIO_FMT_S32_LE,},{.fourcc = V4L2_AUDIO_FMT_U32_LE,},{.fourcc = V4L2_AUDIO_FMT_S20_3LE,},{.fourcc = V4L2_AUDIO_FMT_U20_3LE,},{.fourcc = V4L2_AUDIO_FMT_FLOAT_LE,},{.fourcc = V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE,},+};
+#define NUM_FORMATS ARRAY_SIZE(formats)
+static snd_pcm_format_t convert_fourcc(u32 fourcc) {
return (__force snd_pcm_format_t)v4l2_fourcc_to_audfmt(fourcc);Is this cast something that should be done in the v4l2_fourcc_to_audfmt define instead?
need to avoid include asound.h in videodev2.h, so add this cast in driver.
best regards wang shengjiu
+}
+static u32 find_fourcc(snd_pcm_format_t format) +{
struct asrc_fmt *fmt;unsigned int k;for (k = 0; k < NUM_FORMATS; k++) {fmt = &formats[k];fmt->format = convert_fourcc(fmt->fourcc);if (fmt->format == format)break;}if (k == NUM_FORMATS)return 0;return formats[k].fourcc;+}
+static snd_pcm_format_t find_format(u32 fourcc) +{
struct asrc_fmt *fmt;unsigned int k;for (k = 0; k < NUM_FORMATS; k++) {fmt = &formats[k];if (fmt->fourcc == fourcc)break;}if (k == NUM_FORMATS)return 0;formats[k].format = convert_fourcc(formats[k].fourcc);return formats[k].format;I don't really thing the format field makes any sense. You just keep setting it.
+}
+static int asrc_check_format(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 format) +{
struct asrc_m2m *m2m = pair_m2m->m2m;struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata;struct fsl_asrc_pair *pair = pair_m2m->pair;u64 format_bit = 0;int i;for (i = 0; i < NUM_FORMATS; ++i) {if (formats[i].fourcc == format) {formats[i].format = convert_fourcc(formats[i].fourcc);format_bit = pcm_format_to_bits(formats[i].format);break;}}if (dir == IN && !(format_bit & pdata->fmt_in))return find_fourcc(pair->sample_format[V4L_OUT]);else if (dir == OUT && !(format_bit & pdata->fmt_out))return find_fourcc(pair->sample_format[V4L_CAP]);elsereturn format;+}
+static int asrc_check_channel(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 channels) +{
struct asrc_m2m *m2m = pair_m2m->m2m;struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata;struct fsl_asrc_pair *pair = pair_m2m->pair;if (channels < pdata->chan_min || channels > pdata->chan_max)return pair->channels;elsereturn channels;+}
+static int asrc_check_rate(struct asrc_pair_m2m *pair_m2m, u8 dir, u32 rate) +{
struct asrc_m2m *m2m = pair_m2m->m2m;struct fsl_asrc_m2m_pdata *pdata = &m2m->pdata;struct fsl_asrc_pair *pair = pair_m2m->pair;if (rate < pdata->rate_min || rate > pdata->rate_max)return pair->rate[dir];elsereturn rate;+}
+static inline struct asrc_pair_m2m *asrc_m2m_fh_to_ctx(struct v4l2_fh *fh) +{
return container_of(fh, struct asrc_pair_m2m, fh);+}
+/**
- asrc_read_last_fifo: read all the remaining data from FIFO
- @pair: Structure pointer of fsl_asrc_pair
- @dma_vaddr: virtual address of capture buffer
- @length: payload length of capture buffer
- */
+static void asrc_read_last_fifo(struct fsl_asrc_pair *pair, void *dma_vaddr, u32 *length) +{
struct fsl_asrc *asrc = pair->asrc;enum asrc_pair_index index = pair->index;u32 i, reg, size, t_size = 0, width;u32 *reg32 = NULL;u16 *reg16 = NULL;u8 *reg24 = NULL;width = snd_pcm_format_physical_width(pair->sample_format[V4L_CAP]);if (width == 32)reg32 = dma_vaddr + *length;else if (width == 16)reg16 = dma_vaddr + *length;elsereg24 = dma_vaddr + *length;+retry:
size = asrc->get_output_fifo_size(pair);if (size + *length > ASRC_M2M_BUFFER_SIZE)goto end;for (i = 0; i < size * pair->channels; i++) {regmap_read(asrc->regmap, asrc->get_fifo_addr(OUT, index), ®);if (reg32) {*(reg32) = reg;reg32++;} else if (reg16) {*(reg16) = (u16)reg;reg16++;} else {*reg24++ = (u8)reg;*reg24++ = (u8)(reg >> 8);*reg24++ = (u8)(reg >> 16);}}t_size += size;/* In case there is data left in FIFO */if (size)goto retry;+end:
/* Update payload length */if (reg32)*length += t_size * pair->channels * 4;else if (reg16)*length += t_size * pair->channels * 2;else*length += t_size * pair->channels * 3;+}
+static int asrc_m2m_start_streaming(struct vb2_queue *q, unsigned int count) +{
struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q);struct fsl_asrc_pair *pair = pair_m2m->pair;struct asrc_m2m *m2m = pair_m2m->m2m;struct fsl_asrc *asrc = pair->asrc;struct device *dev = &m2m->pdev->dev;struct vb2_v4l2_buffer *buf;bool request_flag = false;int ret;dev_dbg(dev, "Start streaming pair=%p, %d\n", pair, q->type);ret = pm_runtime_get_sync(dev);if (ret < 0) {dev_err(dev, "Failed to power up asrc\n");goto err_pm_runtime;}/* Request asrc pair/context */if (!pair->req_pair) {/* flag for error handler of this function */request_flag = true;ret = asrc->request_pair(pair->channels, pair);if (ret) {dev_err(dev, "failed to request pair: %d\n", ret);goto err_request_pair;}ret = asrc->m2m_prepare(pair);if (ret) {dev_err(dev, "failed to start pair part one: %d\n", ret);goto err_start_part_one;}pair->req_pair = true;}/* Request dma channels */if (V4L2_TYPE_IS_OUTPUT(q->type)) {pair->dma_chan[V4L_OUT] = asrc->get_dma_channel(pair, IN);if (!pair->dma_chan[V4L_OUT]) {dev_err(dev, "[ctx%d] failed to get input DMA channel\n", pair->index);ret = -EBUSY;goto err_dma_channel;}} else {pair->dma_chan[V4L_CAP] = asrc->get_dma_channel(pair, OUT);if (!pair->dma_chan[V4L_CAP]) {dev_err(dev, "[ctx%d] failed to get output DMA channel\n", pair->index);ret = -EBUSY;goto err_dma_channel;}}v4l2_m2m_update_start_streaming_state(pair_m2m->fh.m2m_ctx, q);return 0;+err_dma_channel:
if (request_flag && asrc->m2m_unprepare)asrc->m2m_unprepare(pair);+err_start_part_one:
if (request_flag)asrc->release_pair(pair);+err_request_pair:
pm_runtime_put_sync(dev);+err_pm_runtime:
/* Release buffers */if (V4L2_TYPE_IS_OUTPUT(q->type)) {while ((buf = v4l2_m2m_src_buf_remove(pair_m2m->fh.m2m_ctx)))v4l2_m2m_buf_done(buf, VB2_BUF_STATE_QUEUED);} else {while ((buf = v4l2_m2m_dst_buf_remove(pair_m2m->fh.m2m_ctx)))v4l2_m2m_buf_done(buf, VB2_BUF_STATE_QUEUED);}return ret;+}
+static void asrc_m2m_stop_streaming(struct vb2_queue *q) +{
struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q);struct asrc_m2m *m2m = pair_m2m->m2m;struct fsl_asrc_pair *pair = pair_m2m->pair;struct fsl_asrc *asrc = pair->asrc;struct device *dev = &m2m->pdev->dev;dev_dbg(dev, "Stop streaming pair=%p, %d\n", pair, q->type);v4l2_m2m_update_stop_streaming_state(pair_m2m->fh.m2m_ctx, q);/* Stop & release pair/context */if (asrc->m2m_stop)asrc->m2m_stop(pair);if (pair->req_pair) {if (asrc->m2m_unprepare)asrc->m2m_unprepare(pair);asrc->release_pair(pair);pair->req_pair = false;}/* Release dma channel */if (V4L2_TYPE_IS_OUTPUT(q->type)) {if (pair->dma_chan[V4L_OUT])dma_release_channel(pair->dma_chan[V4L_OUT]);} else {if (pair->dma_chan[V4L_CAP])dma_release_channel(pair->dma_chan[V4L_CAP]);}pm_runtime_put_sync(dev);+}
+static int asrc_m2m_queue_setup(struct vb2_queue *q,
unsigned int *num_buffers, unsigned int *num_planes,unsigned int sizes[], struct device *alloc_devs[])+{
struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(q);struct fsl_asrc_pair *pair = pair_m2m->pair;/* single buffer */*num_planes = 1;This isn't quite right. This driver supports VIDIOC_CREATE_BUFS, which userspace can use to add new buffers on the fly. Note that there is no corresponding DELETE_BUFS ioctl to delete buffers, but work is in progress for that.
If a driver supports CREATE_BUFS, then queue_setup has to check whether the values passed by VIDIOC_CREATE_BUFS are valid.
That's done through:
u32 size; if (V4L2_TYPE_IS_OUTPUT(q->type)) size = pair->buf_len[V4L_OUT]; else size = pair->buf_len[V4L_CAP]; if (*nplanes) return sizes[0] < size ? -EINVAL : 0; *num_planes = 1; sizes[0] = size; return 0;One of these days this rather ugly construction should be cleaned up.
Oh well...
Regards,
Hans
/** The capture buffer size depends on output buffer size* and the convert ratio.** Here just use a fix length for capture and output buffer.* User need to care about it.*/if (V4L2_TYPE_IS_OUTPUT(q->type))sizes[0] = pair->buf_len[V4L_OUT];elsesizes[0] = pair->buf_len[V4L_CAP];return 0;+}
+static void asrc_m2m_buf_queue(struct vb2_buffer *vb) +{
struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);struct asrc_pair_m2m *pair_m2m = vb2_get_drv_priv(vb->vb2_queue);/* queue buffer */v4l2_m2m_buf_queue(pair_m2m->fh.m2m_ctx, vbuf);+}
+static const struct vb2_ops asrc_m2m_qops = {
.wait_prepare = vb2_ops_wait_prepare,.wait_finish = vb2_ops_wait_finish,.start_streaming = asrc_m2m_start_streaming,.stop_streaming = asrc_m2m_stop_streaming,.queue_setup = asrc_m2m_queue_setup,.buf_queue = asrc_m2m_buf_queue,+};
+/* Init video buffer queue for src and dst. */ +static int asrc_m2m_queue_init(void *priv, struct vb2_queue *src_vq,
struct vb2_queue *dst_vq)+{
struct asrc_pair_m2m *pair_m2m = priv;struct asrc_m2m *m2m = pair_m2m->m2m;int ret;src_vq->type = V4L2_BUF_TYPE_AUDIO_OUTPUT;src_vq->io_modes = VB2_MMAP | VB2_DMABUF;src_vq->drv_priv = pair_m2m;src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);src_vq->ops = &asrc_m2m_qops;src_vq->mem_ops = &vb2_dma_contig_memops;src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;src_vq->lock = &m2m->mlock;src_vq->dev = &m2m->pdev->dev;src_vq->min_buffers_needed = 1;ret = vb2_queue_init(src_vq);if (ret)return ret;dst_vq->type = V4L2_BUF_TYPE_AUDIO_CAPTURE;dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;dst_vq->drv_priv = pair_m2m;dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);dst_vq->ops = &asrc_m2m_qops;dst_vq->mem_ops = &vb2_dma_contig_memops;dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;dst_vq->lock = &m2m->mlock;dst_vq->dev = &m2m->pdev->dev;dst_vq->min_buffers_needed = 1;ret = vb2_queue_init(dst_vq);return ret;+}
+static int asrc_m2m_op_s_ctrl(struct v4l2_ctrl *ctrl) +{
struct asrc_pair_m2m *pair_m2m =container_of(ctrl->handler, struct asrc_pair_m2m, ctrl_handler);struct fsl_asrc_pair *pair = pair_m2m->pair;struct fsl_asrc *asrc = pair->asrc;int src_rate_int, src_rate_frac;int dst_rate_int, dst_rate_frac;int new_rate, new_frac;u64 src_rate, dst_rate;u64 ratio_pre, ratio_cur;s64 ratio_diff;int ret = 0;switch (ctrl->id) {case V4L2_CID_ASRC_SOURCE_RATE:new_rate = ctrl->p_new.p_fixed_point->integer;new_frac = ctrl->p_new.p_fixed_point->fractional;src_rate_int = asrc_check_rate(pair_m2m, IN, new_rate);if (src_rate_int != new_rate ||(pair_m2m->src_rate.integer > 0 &&src_rate_int != pair_m2m->src_rate.integer))return -EINVAL;pair->rate[V4L_OUT] = src_rate_int;if (new_frac != pair_m2m->src_rate.fractional &&new_rate == pair_m2m->src_rate.integer &&pair_m2m->dst_rate.integer > 0) {/** use maximum rate 768kHz as limitation, then we can shift right 21 bit for* division*/src_rate_frac = pair_m2m->src_rate.fractional;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate_int = pair_m2m->dst_rate.integer;dst_rate_frac = pair_m2m->dst_rate.fractional;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_pre = src_rate;src_rate_frac = new_frac;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_cur = src_rate;ratio_diff = ratio_cur - ratio_pre;asrc->m2m_set_ratio_mod(pair, ratio_diff << 11);}pair_m2m->src_rate.integer = new_rate;pair_m2m->src_rate.fractional = new_frac;break;case V4L2_CID_ASRC_DEST_RATE:new_rate = ctrl->p_new.p_fixed_point->integer;new_frac = ctrl->p_new.p_fixed_point->fractional;dst_rate_int = asrc_check_rate(pair_m2m, OUT, new_rate);if (dst_rate_int != new_rate ||(pair_m2m->dst_rate.integer > 0 &&dst_rate_int != pair_m2m->dst_rate.integer))return -EINVAL;pair->rate[V4L_CAP] = dst_rate_int;if (new_frac != pair_m2m->dst_rate.fractional &&new_rate == pair_m2m->dst_rate.integer &&pair_m2m->src_rate.integer > 0) {/** use maximum rate 768kHz as limitation, then we can shift right 21 bit for* division*/src_rate_int = pair_m2m->src_rate.integer;src_rate_frac = pair_m2m->src_rate.fractional;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate_frac = pair_m2m->dst_rate.fractional;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_pre = src_rate;src_rate = ((s64)src_rate_int << 31) + src_rate_frac;dst_rate_int = new_rate;dst_rate_frac = new_frac;dst_rate = (((s64)dst_rate_int << 31) + dst_rate_frac) >> 20;do_div(src_rate, dst_rate);ratio_cur = src_rate;ratio_diff = ratio_cur - ratio_pre;/* convert ratio_diff to Q31*/asrc->m2m_set_ratio_mod(pair, ratio_diff << 11);This is very similar to the other control. You really just want to pass two rates (source and dest) and let that function calculate the ratio mod.
}pair_m2m->dst_rate.integer = new_rate;pair_m2m->dst_rate.fractional = new_frac;break;default:ret = -EINVAL;break;}return ret;+}
+static const struct v4l2_ctrl_ops asrc_m2m_ctrl_ops = {
.s_ctrl = asrc_m2m_op_s_ctrl,+};
+static const struct v4l2_ctrl_config asrc_src_rate_control = {
.ops = &asrc_m2m_ctrl_ops,.id = V4L2_CID_ASRC_SOURCE_RATE,.name = "asrc source sample rate",How about "Audio Source Sample Rate"?
But we want these controls as standard types (not driver specific), so this should be added to drivers/media/v4l2-core/v4l2-ctrls-defs.c.
And if we just use an s64 to store the fixed point, then you can just call v4l2_ctrl_new_std().
But you probably want to add some helper defines to split a fixed point value into integer and fractional parts, and to construct one.
.type = V4L2_CTRL_TYPE_FIXED_POINT,.min = 0,.max = 0x7fffffff,.def = 8000,.flags = V4L2_CTRL_FLAG_UPDATE,+};
+static const struct v4l2_ctrl_config asrc_dst_rate_control = {
.ops = &asrc_m2m_ctrl_ops,.id = V4L2_CID_ASRC_DEST_RATE,.name = "asrc dest sample rate",.type = V4L2_CTRL_TYPE_FIXED_POINT,.min = 0,.max = 0x7fffffff,.def = 8000,.flags = V4L2_CTRL_FLAG_UPDATE,+};
+/* system callback for open() */ +static int asrc_m2m_open(struct file *file) +{
struct asrc_m2m *m2m = video_drvdata(file);struct fsl_asrc *asrc = m2m->pdata.asrc;struct video_device *vdev = video_devdata(file);struct fsl_asrc_pair *pair;struct asrc_pair_m2m *pair_m2m;int ret = 0;if (mutex_lock_interruptible(&m2m->mlock))return -ERESTARTSYS;pair = kzalloc(sizeof(*pair) + asrc->pair_priv_size, GFP_KERNEL);if (!pair) {ret = -ENOMEM;goto err_alloc_pair;}pair_m2m = kzalloc(sizeof(*pair_m2m), GFP_KERNEL);if (!pair_m2m) {ret = -ENOMEM;goto err_alloc_pair_m2m;}pair->private = (void *)pair + sizeof(struct fsl_asrc_pair);pair->asrc = asrc;pair->buf_len[V4L_OUT] = ASRC_M2M_BUFFER_SIZE;pair->buf_len[V4L_CAP] = ASRC_M2M_BUFFER_SIZE;pair->channels = 2;pair->rate[V4L_OUT] = 8000;pair->rate[V4L_CAP] = 8000;pair->sample_format[V4L_OUT] = SNDRV_PCM_FORMAT_S16_LE;pair->sample_format[V4L_CAP] = SNDRV_PCM_FORMAT_S16_LE;init_completion(&pair->complete[V4L_OUT]);init_completion(&pair->complete[V4L_CAP]);v4l2_fh_init(&pair_m2m->fh, vdev);v4l2_fh_add(&pair_m2m->fh);file->private_data = &pair_m2m->fh;pair_m2m->pair = pair;pair_m2m->m2m = m2m;/* m2m context init */pair_m2m->fh.m2m_ctx = v4l2_m2m_ctx_init(m2m->m2m_dev, pair_m2m,asrc_m2m_queue_init);if (IS_ERR(pair_m2m->fh.m2m_ctx)) {ret = PTR_ERR(pair_m2m->fh.m2m_ctx);goto err_ctx_init;}v4l2_ctrl_handler_init(&pair_m2m->ctrl_handler, 2);/* use V4L2_CID_GAIN for ratio update control */v4l2_ctrl_new_custom(&pair_m2m->ctrl_handler, &asrc_src_rate_control, NULL);v4l2_ctrl_new_custom(&pair_m2m->ctrl_handler, &asrc_dst_rate_control, NULL);if (pair_m2m->ctrl_handler.error) {ret = pair_m2m->ctrl_handler.error;v4l2_ctrl_handler_free(&pair_m2m->ctrl_handler);goto err_ctrl_handler;}pair_m2m->fh.ctrl_handler = &pair_m2m->ctrl_handler;mutex_unlock(&m2m->mlock);return 0;+err_ctrl_handler:
v4l2_m2m_ctx_release(pair_m2m->fh.m2m_ctx);+err_ctx_init:
v4l2_fh_del(&pair_m2m->fh);v4l2_fh_exit(&pair_m2m->fh);kfree(pair_m2m);+err_alloc_pair_m2m:
kfree(pair);+err_alloc_pair:
mutex_unlock(&m2m->mlock);return ret;+}
+static int asrc_m2m_release(struct file *file) +{
struct asrc_m2m *m2m = video_drvdata(file);struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(file->private_data);struct fsl_asrc_pair *pair = pair_m2m->pair;mutex_lock(&m2m->mlock);v4l2_ctrl_handler_free(&pair_m2m->ctrl_handler);v4l2_m2m_ctx_release(pair_m2m->fh.m2m_ctx);v4l2_fh_del(&pair_m2m->fh);v4l2_fh_exit(&pair_m2m->fh);kfree(pair_m2m);kfree(pair);mutex_unlock(&m2m->mlock);return 0;+}
+static const struct v4l2_file_operations asrc_m2m_fops = {
.owner = THIS_MODULE,.open = asrc_m2m_open,.release = asrc_m2m_release,.poll = v4l2_m2m_fop_poll,.unlocked_ioctl = video_ioctl2,.mmap = v4l2_m2m_fop_mmap,+};
+static int asrc_m2m_querycap(struct file *file, void *priv,
struct v4l2_capability *cap)+{
strscpy(cap->driver, "asrc m2m", sizeof(cap->driver));strscpy(cap->card, "asrc m2m", sizeof(cap->card));This is rather ugly. It should at least mention imx, right?
cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_AUDIO_M2M;cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;You can drop this last line, that's done for you.
return 0;+}
+static int enum_fmt(struct v4l2_fmtdesc *f, u64 fmtbit) +{
int i, num;struct asrc_fmt *fmt;num = 0;for (i = 0; i < NUM_FORMATS; ++i) {formats[i].format = convert_fourcc(formats[i].fourcc);if (pcm_format_to_bits(formats[i].format) & fmtbit) {if (num == f->index)break;/** Correct type but haven't reached our index yet,* just increment per-type index*/++num;}}if (i < NUM_FORMATS) {/* Format found */fmt = &formats[i];f->pixelformat = fmt->fourcc;return 0;}return -EINVAL;+}
+static int asrc_m2m_enum_fmt_aud_cap(struct file *file, void *fh,
struct v4l2_fmtdesc *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);struct asrc_m2m *m2m = pair_m2m->m2m;return enum_fmt(f, m2m->pdata.fmt_out);+}
+static int asrc_m2m_enum_fmt_aud_out(struct file *file, void *fh,
struct v4l2_fmtdesc *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);struct asrc_m2m *m2m = pair_m2m->m2m;return enum_fmt(f, m2m->pdata.fmt_in);+}
+static int asrc_m2m_g_fmt_aud_cap(struct file *file, void *fh,
struct v4l2_format *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);struct fsl_asrc_pair *pair = pair_m2m->pair;f->fmt.audio.channels = pair->channels;f->fmt.audio.buffersize = pair->buf_len[V4L_CAP];f->fmt.audio.audioformat = find_fourcc(pair->sample_format[V4L_CAP]);return 0;+}
+static int asrc_m2m_g_fmt_aud_out(struct file *file, void *fh,
struct v4l2_format *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);struct fsl_asrc_pair *pair = pair_m2m->pair;f->fmt.audio.channels = pair->channels;f->fmt.audio.buffersize = pair->buf_len[V4L_OUT];f->fmt.audio.audioformat = find_fourcc(pair->sample_format[V4L_OUT]);return 0;+}
+/* output for asrc */ +static int asrc_m2m_s_fmt_aud_cap(struct file *file, void *fh,
struct v4l2_format *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);struct fsl_asrc_pair *pair = pair_m2m->pair;struct asrc_m2m *m2m = pair_m2m->m2m;struct device *dev = &m2m->pdev->dev;f->fmt.audio.audioformat = asrc_check_format(pair_m2m, OUT, f->fmt.audio.audioformat);f->fmt.audio.channels = asrc_check_channel(pair_m2m, OUT, f->fmt.audio.channels);if (pair_m2m->channels[V4L_CAP] > 0 &&pair_m2m->channels[V4L_CAP] != f->fmt.audio.channels) {dev_err(dev, "channels don't match for cap and out\n");return -EINVAL;}pair_m2m->channels[V4L_CAP] = f->fmt.audio.channels;pair->channels = f->fmt.audio.channels;pair->sample_format[V4L_CAP] = find_format(f->fmt.audio.audioformat);return 0;+}
+/* input for asrc */ +static int asrc_m2m_s_fmt_aud_out(struct file *file, void *fh,
struct v4l2_format *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);struct fsl_asrc_pair *pair = pair_m2m->pair;struct asrc_m2m *m2m = pair_m2m->m2m;struct device *dev = &m2m->pdev->dev;f->fmt.audio.audioformat = asrc_check_format(pair_m2m, IN, f->fmt.audio.audioformat);f->fmt.audio.channels = asrc_check_channel(pair_m2m, IN, f->fmt.audio.channels);if (pair_m2m->channels[V4L_OUT] > 0 &&pair_m2m->channels[V4L_OUT] != f->fmt.audio.channels) {dev_err(dev, "channels don't match for cap and out\n");return -EINVAL;}pair_m2m->channels[V4L_OUT] = f->fmt.audio.channels;pair->channels = f->fmt.audio.channels;pair->sample_format[V4L_OUT] = find_format(f->fmt.audio.audioformat);return 0;+}
+static int asrc_m2m_try_fmt_audio_cap(struct file *file, void *fh,
struct v4l2_format *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);f->fmt.audio.audioformat = asrc_check_format(pair_m2m, OUT, f->fmt.audio.audioformat);f->fmt.audio.channels = asrc_check_channel(pair_m2m, OUT, f->fmt.audio.channels);return 0;+}
+static int asrc_m2m_try_fmt_audio_out(struct file *file, void *fh,
struct v4l2_format *f)+{
struct asrc_pair_m2m *pair_m2m = asrc_m2m_fh_to_ctx(fh);f->fmt.audio.audioformat = asrc_check_format(pair_m2m, IN, f->fmt.audio.audioformat);f->fmt.audio.channels = asrc_check_channel(pair_m2m, IN, f->fmt.audio.channels);return 0;+}
+static const struct v4l2_ioctl_ops asrc_m2m_ioctl_ops = {
.vidioc_querycap = asrc_m2m_querycap,.vidioc_enum_fmt_audio_cap = asrc_m2m_enum_fmt_aud_cap,.vidioc_enum_fmt_audio_out = asrc_m2m_enum_fmt_aud_out,.vidioc_g_fmt_audio_cap = asrc_m2m_g_fmt_aud_cap,.vidioc_g_fmt_audio_out = asrc_m2m_g_fmt_aud_out,.vidioc_s_fmt_audio_cap = asrc_m2m_s_fmt_aud_cap,.vidioc_s_fmt_audio_out = asrc_m2m_s_fmt_aud_out,.vidioc_try_fmt_audio_cap = asrc_m2m_try_fmt_audio_cap,.vidioc_try_fmt_audio_out = asrc_m2m_try_fmt_audio_out,.vidioc_qbuf = v4l2_m2m_ioctl_qbuf,.vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,.vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,.vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,.vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,.vidioc_querybuf = v4l2_m2m_ioctl_querybuf,.vidioc_streamon = v4l2_m2m_ioctl_streamon,.vidioc_streamoff = v4l2_m2m_ioctl_streamoff,.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,.vidioc_unsubscribe_event = v4l2_event_unsubscribe,+};
+/* dma complete callback */ +static void asrc_input_dma_callback(void *data) +{
struct fsl_asrc_pair *pair = (struct fsl_asrc_pair *)data;complete(&pair->complete[V4L_OUT]);+}
+/* dma complete callback */ +static void asrc_output_dma_callback(void *data) +{
struct fsl_asrc_pair *pair = (struct fsl_asrc_pair *)data;complete(&pair->complete[V4L_CAP]);+}
+/* config dma channel */ +static int asrc_dmaconfig(struct asrc_pair_m2m *pair_m2m,
struct dma_chan *chan,u32 dma_addr, dma_addr_t buf_addr, u32 buf_len,int dir, int width)+{
struct fsl_asrc_pair *pair = pair_m2m->pair;struct fsl_asrc *asrc = pair->asrc;struct asrc_m2m *m2m = pair_m2m->m2m;struct device *dev = &m2m->pdev->dev;struct dma_slave_config slave_config;struct scatterlist sg[ASRC_M2M_SG_NUM];enum dma_slave_buswidth buswidth;unsigned int sg_len, max_period_size;int ret, i;switch (width) {case 8:buswidth = DMA_SLAVE_BUSWIDTH_1_BYTE;break;case 16:buswidth = DMA_SLAVE_BUSWIDTH_2_BYTES;break;case 24:buswidth = DMA_SLAVE_BUSWIDTH_3_BYTES;break;case 32:buswidth = DMA_SLAVE_BUSWIDTH_4_BYTES;break;default:dev_err(dev, "invalid word width\n");return -EINVAL;}memset(&slave_config, 0, sizeof(slave_config));if (dir == V4L_OUT) {slave_config.direction = DMA_MEM_TO_DEV;slave_config.dst_addr = dma_addr;slave_config.dst_addr_width = buswidth;slave_config.dst_maxburst = asrc->m2m_get_maxburst(IN, pair);} else {slave_config.direction = DMA_DEV_TO_MEM;slave_config.src_addr = dma_addr;slave_config.src_addr_width = buswidth;slave_config.src_maxburst = asrc->m2m_get_maxburst(OUT, pair);}ret = dmaengine_slave_config(chan, &slave_config);if (ret) {dev_err(dev, "failed to config dmaengine for %s task: %d\n",DIR_STR(dir), ret);return -EINVAL;}max_period_size = rounddown(ASRC_M2M_PERIOD_SIZE, width * pair->channels / 8);/* scatter gather mode */sg_len = buf_len / max_period_size;if (buf_len % max_period_size)sg_len += 1;sg_init_table(sg, sg_len);for (i = 0; i < (sg_len - 1); i++) {sg_dma_address(&sg[i]) = buf_addr + i * max_period_size;sg_dma_len(&sg[i]) = max_period_size;}sg_dma_address(&sg[i]) = buf_addr + i * max_period_size;sg_dma_len(&sg[i]) = buf_len - i * max_period_size;pair->desc[dir] = dmaengine_prep_slave_sg(chan, sg, sg_len,slave_config.direction,DMA_PREP_INTERRUPT);if (!pair->desc[dir]) {dev_err(dev, "failed to prepare dmaengine for %s task\n", DIR_STR(dir));return -EINVAL;}pair->desc[dir]->callback = ASRC_xPUT_DMA_CALLBACK(dir);pair->desc[dir]->callback_param = pair;return 0;+}
+/* main function of converter */ +static void asrc_m2m_device_run(void *priv) +{
struct asrc_pair_m2m *pair_m2m = priv;struct fsl_asrc_pair *pair = pair_m2m->pair;struct asrc_m2m *m2m = pair_m2m->m2m;struct fsl_asrc *asrc = pair->asrc;struct device *dev = &m2m->pdev->dev;enum asrc_pair_index index = pair->index;struct vb2_v4l2_buffer *src_buf, *dst_buf;unsigned int out_buf_len;unsigned int cap_dma_len;unsigned int width;u32 fifo_addr;int ret;src_buf = v4l2_m2m_next_src_buf(pair_m2m->fh.m2m_ctx);dst_buf = v4l2_m2m_next_dst_buf(pair_m2m->fh.m2m_ctx);width = snd_pcm_format_physical_width(pair->sample_format[V4L_OUT]);fifo_addr = asrc->paddr + asrc->get_fifo_addr(IN, index);out_buf_len = vb2_get_plane_payload(&src_buf->vb2_buf, 0);if (out_buf_len < width * pair->channels / 8 ||out_buf_len > ASRC_M2M_BUFFER_SIZE ||out_buf_len % (width * pair->channels / 8)) {dev_err(dev, "out buffer size is error: [%d]\n", out_buf_len);goto end;}/* dma config for output dma channel */ret = asrc_dmaconfig(pair_m2m,pair->dma_chan[V4L_OUT],fifo_addr,vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0),out_buf_len, V4L_OUT, width);if (ret) {dev_err(dev, "out dma config error\n");goto end;}width = snd_pcm_format_physical_width(pair->sample_format[V4L_CAP]);fifo_addr = asrc->paddr + asrc->get_fifo_addr(OUT, index);cap_dma_len = asrc->m2m_calc_out_len(pair, out_buf_len);if (cap_dma_len > 0 && cap_dma_len <= ASRC_M2M_BUFFER_SIZE) {/* dma config for capture dma channel */ret = asrc_dmaconfig(pair_m2m,pair->dma_chan[V4L_CAP],fifo_addr,vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0),cap_dma_len, V4L_CAP, width);if (ret) {dev_err(dev, "cap dma config error\n");goto end;}} else if (cap_dma_len > ASRC_M2M_BUFFER_SIZE) {dev_err(dev, "cap buffer size error\n");goto end;}reinit_completion(&pair->complete[V4L_OUT]);reinit_completion(&pair->complete[V4L_CAP]);/* Submit DMA request */dmaengine_submit(pair->desc[V4L_OUT]);dma_async_issue_pending(pair->desc[V4L_OUT]->chan);if (cap_dma_len > 0) {dmaengine_submit(pair->desc[V4L_CAP]);dma_async_issue_pending(pair->desc[V4L_CAP]->chan);}asrc->m2m_start(pair);if (!wait_for_completion_interruptible_timeout(&pair->complete[V4L_OUT], 10 * HZ)) {dev_err(dev, "out DMA task timeout\n");goto end;}if (cap_dma_len > 0) {if (!wait_for_completion_interruptible_timeout(&pair->complete[V4L_CAP], 10 * HZ)) {dev_err(dev, "cap DMA task timeout\n");goto end;}}/* read the last words from FIFO */asrc_read_last_fifo(pair, vb2_plane_vaddr(&dst_buf->vb2_buf, 0), &cap_dma_len);/* update payload length for capture */vb2_set_plane_payload(&dst_buf->vb2_buf, 0, cap_dma_len);+end:
src_buf = v4l2_m2m_src_buf_remove(pair_m2m->fh.m2m_ctx);dst_buf = v4l2_m2m_dst_buf_remove(pair_m2m->fh.m2m_ctx);v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE);v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE);v4l2_m2m_job_finish(m2m->m2m_dev, pair_m2m->fh.m2m_ctx);+}
+static int asrc_m2m_job_ready(void *priv) +{
struct asrc_pair_m2m *pair_m2m = priv;if (v4l2_m2m_num_src_bufs_ready(pair_m2m->fh.m2m_ctx) > 0 &&v4l2_m2m_num_dst_bufs_ready(pair_m2m->fh.m2m_ctx) > 0) {return 1;}return 0;+}
+static const struct v4l2_m2m_ops asrc_m2m_ops = {
.job_ready = asrc_m2m_job_ready,.device_run = asrc_m2m_device_run,+};
+static int asrc_m2m_probe(struct platform_device *pdev) +{
struct fsl_asrc_m2m_pdata *data = pdev->dev.platform_data;struct device *dev = &pdev->dev;struct asrc_m2m *m2m;int ret;m2m = devm_kzalloc(dev, sizeof(struct asrc_m2m), GFP_KERNEL);if (!m2m)return -ENOMEM;m2m->pdata = *data;m2m->pdev = pdev;ret = v4l2_device_register(dev, &m2m->v4l2_dev);if (ret) {dev_err(dev, "failed to register v4l2 device\n");goto err_register;}m2m->m2m_dev = v4l2_m2m_init(&asrc_m2m_ops);if (IS_ERR(m2m->m2m_dev)) {dev_err(dev, "failed to register v4l2 device\n");ret = PTR_ERR(m2m->m2m_dev);goto err_m2m;}m2m->dec_vdev = video_device_alloc();if (!m2m->dec_vdev) {dev_err(dev, "failed to register v4l2 device\n");ret = -ENOMEM;goto err_vdev_alloc;}mutex_init(&m2m->mlock);m2m->dec_vdev->fops = &asrc_m2m_fops;m2m->dec_vdev->ioctl_ops = &asrc_m2m_ioctl_ops;m2m->dec_vdev->minor = -1;m2m->dec_vdev->release = video_device_release;m2m->dec_vdev->lock = &m2m->mlock; /* lock for ioctl serialization */m2m->dec_vdev->v4l2_dev = &m2m->v4l2_dev;m2m->dec_vdev->vfl_dir = VFL_DIR_M2M;m2m->dec_vdev->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_AUDIO_M2M;ret = video_register_device(m2m->dec_vdev, VFL_TYPE_AUDIO, -1);if (ret) {dev_err(dev, "failed to register video device\n");goto err_vdev_register;}video_set_drvdata(m2m->dec_vdev, m2m);platform_set_drvdata(pdev, m2m);pm_runtime_enable(&pdev->dev);return 0;+err_vdev_register:
video_device_release(m2m->dec_vdev);+err_vdev_alloc:
v4l2_m2m_release(m2m->m2m_dev);+err_m2m:
v4l2_device_unregister(&m2m->v4l2_dev);+err_register:
return ret;+}
+static void asrc_m2m_remove(struct platform_device *pdev) +{
struct asrc_m2m *m2m = platform_get_drvdata(pdev);pm_runtime_disable(&pdev->dev);video_unregister_device(m2m->dec_vdev);video_device_release(m2m->dec_vdev);v4l2_m2m_release(m2m->m2m_dev);v4l2_device_unregister(&m2m->v4l2_dev);+}
+#ifdef CONFIG_PM_SLEEP +/* suspend callback for m2m */ +static int asrc_m2m_suspend(struct device *dev) +{
struct asrc_m2m *m2m = dev_get_drvdata(dev);struct fsl_asrc *asrc = m2m->pdata.asrc;struct fsl_asrc_pair *pair;unsigned long lock_flags;int i;for (i = 0; i < PAIR_CTX_NUM; i++) {spin_lock_irqsave(&asrc->lock, lock_flags);pair = asrc->pair[i];if (!pair || !pair->req_pair) {spin_unlock_irqrestore(&asrc->lock, lock_flags);continue;}if (!completion_done(&pair->complete[V4L_OUT])) {if (pair->dma_chan[V4L_OUT])dmaengine_terminate_all(pair->dma_chan[V4L_OUT]);asrc_input_dma_callback((void *)pair);}if (!completion_done(&pair->complete[V4L_CAP])) {if (pair->dma_chan[V4L_CAP])dmaengine_terminate_all(pair->dma_chan[V4L_CAP]);asrc_output_dma_callback((void *)pair);}if (asrc->m2m_pair_suspend)asrc->m2m_pair_suspend(pair);spin_unlock_irqrestore(&asrc->lock, lock_flags);}return 0;+}
+static int asrc_m2m_resume(struct device *dev) +{
struct asrc_m2m *m2m = dev_get_drvdata(dev);struct fsl_asrc *asrc = m2m->pdata.asrc;struct fsl_asrc_pair *pair;unsigned long lock_flags;int i;for (i = 0; i < PAIR_CTX_NUM; i++) {spin_lock_irqsave(&asrc->lock, lock_flags);pair = asrc->pair[i];if (!pair || !pair->req_pair) {spin_unlock_irqrestore(&asrc->lock, lock_flags);continue;}if (asrc->m2m_pair_resume)asrc->m2m_pair_resume(pair);spin_unlock_irqrestore(&asrc->lock, lock_flags);}return 0;+} +#endif
+static const struct dev_pm_ops asrc_m2m_pm_ops = {
SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(asrc_m2m_suspend,asrc_m2m_resume)+};
+static struct platform_driver asrc_m2m_driver = {
.probe = asrc_m2m_probe,.remove_new = asrc_m2m_remove,.driver = {.name = "fsl_asrc_m2m",.pm = &asrc_m2m_pm_ops,},+}; +module_platform_driver(asrc_m2m_driver);
+MODULE_DESCRIPTION("Freescale ASRC M2M driver"); +MODULE_LICENSE("GPL");
Regards,
Hans
On 18/10/2023 14:53, Shengjiu Wang wrote:
On Mon, Oct 16, 2023 at 10:01 PM Hans Verkuil hverkuil@xs4all.nl wrote:
On 13/10/2023 10:31, Shengjiu Wang wrote:
Implement the ASRC memory to memory function using the v4l2 framework, user can use this function with v4l2 ioctl interface.
User send the output and capture buffer to driver and driver store the converted data to the capture buffer.
This feature can be shared by ASRC and EASRC drivers
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1248 +++++++++++++++++++++++++ 3 files changed, 1261 insertions(+) create mode 100644 drivers/media/platform/nxp/imx-asrc.c
diff --git a/drivers/media/platform/nxp/Kconfig b/drivers/media/platform/nxp/Kconfig index 40e3436669e2..8234644ee341 100644 --- a/drivers/media/platform/nxp/Kconfig +++ b/drivers/media/platform/nxp/Kconfig @@ -67,3 +67,15 @@ config VIDEO_MX2_EMMAPRP
source "drivers/media/platform/nxp/dw100/Kconfig" source "drivers/media/platform/nxp/imx-jpeg/Kconfig"
+config VIDEO_IMX_ASRC
tristate "NXP i.MX ASRC M2M support"depends on V4L_MEM2MEM_DRIVERSdepends on MEDIA_SUPPORTselect VIDEOBUF2_DMA_CONTIGselect V4L2_MEM2MEM_DEVhelpSay Y if you want to add ASRC M2M support for NXP CPUs.It is a complement for ASRC M2P and ASRC P2M features.This option is only useful for out-of-tree drivers sincein-tree drivers select it automatically.diff --git a/drivers/media/platform/nxp/Makefile b/drivers/media/platform/nxp/Makefile index 4d90eb713652..1325675e34f5 100644 --- a/drivers/media/platform/nxp/Makefile +++ b/drivers/media/platform/nxp/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX8MQ_MIPI_CSI2) += imx8mq-mipi-csi2.o obj-$(CONFIG_VIDEO_IMX_MIPI_CSIS) += imx-mipi-csis.o obj-$(CONFIG_VIDEO_IMX_PXP) += imx-pxp.o obj-$(CONFIG_VIDEO_MX2_EMMAPRP) += mx2_emmaprp.o +obj-$(CONFIG_VIDEO_IMX_ASRC) += imx-asrc.o diff --git a/drivers/media/platform/nxp/imx-asrc.c b/drivers/media/platform/nxp/imx-asrc.c new file mode 100644 index 000000000000..373ca2b5ec90 --- /dev/null +++ b/drivers/media/platform/nxp/imx-asrc.c @@ -0,0 +1,1248 @@ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2014-2016 Freescale Semiconductor, Inc. +// Copyright (C) 2019-2023 NXP +// +// Freescale ASRC Memory to Memory (M2M) driver
+#include <linux/dma/imx-dma.h> +#include <linux/pm_runtime.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-device.h> +#include <media/v4l2-event.h> +#include <media/v4l2-fh.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-mem2mem.h> +#include <media/videobuf2-dma-contig.h> +#include <sound/dmaengine_pcm.h> +#include <sound/fsl_asrc_common.h>
+#define V4L_CAP OUT +#define V4L_OUT IN
+#define ASRC_xPUT_DMA_CALLBACK(dir) \
(((dir) == V4L_OUT) ? asrc_input_dma_callback \: asrc_output_dma_callback)+#define DIR_STR(dir) (dir) == V4L_OUT ? "out" : "cap"
+#define ASRC_M2M_BUFFER_SIZE (512 * 1024) +#define ASRC_M2M_PERIOD_SIZE (48 * 1024) +#define ASRC_M2M_SG_NUM (20)
Where do all these values come from? How do they relate? Some comments would be welcome.
Esp. ASRC_M2M_SG_NUM is a bit odd.
+struct asrc_fmt {
u32 fourcc;snd_pcm_format_t format;Do you need this field? If not, then you can drop the whole struct and just use u32 fourcc in the formats[] array.
+};
+struct asrc_pair_m2m {
struct fsl_asrc_pair *pair;struct asrc_m2m *m2m;struct v4l2_fh fh;struct v4l2_ctrl_handler ctrl_handler;int channels[2];struct v4l2_ctrl_fixed_point src_rate;struct v4l2_ctrl_fixed_point dst_rate;+};
+struct asrc_m2m {
struct fsl_asrc_m2m_pdata pdata;struct v4l2_device v4l2_dev;struct v4l2_m2m_dev *m2m_dev;struct video_device *dec_vdev;struct mutex mlock; /* v4l2 ioctls serialization */struct platform_device *pdev;+};
+static struct asrc_fmt formats[] = {
{.fourcc = V4L2_AUDIO_FMT_S8,},{.fourcc = V4L2_AUDIO_FMT_S16_LE,},{.fourcc = V4L2_AUDIO_FMT_U16_LE,},{.fourcc = V4L2_AUDIO_FMT_S24_LE,},{.fourcc = V4L2_AUDIO_FMT_S24_3LE,},{.fourcc = V4L2_AUDIO_FMT_U24_LE,},{.fourcc = V4L2_AUDIO_FMT_U24_3LE,},{.fourcc = V4L2_AUDIO_FMT_S32_LE,},{.fourcc = V4L2_AUDIO_FMT_U32_LE,},{.fourcc = V4L2_AUDIO_FMT_S20_3LE,},{.fourcc = V4L2_AUDIO_FMT_U20_3LE,},{.fourcc = V4L2_AUDIO_FMT_FLOAT_LE,},{.fourcc = V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE,},+};
+#define NUM_FORMATS ARRAY_SIZE(formats)
+static snd_pcm_format_t convert_fourcc(u32 fourcc) {
return (__force snd_pcm_format_t)v4l2_fourcc_to_audfmt(fourcc);Is this cast something that should be done in the v4l2_fourcc_to_audfmt define instead?
need to avoid include asound.h in videodev2.h, so add this cast in driver.
It's a #define, so just including videodev2.h won't require asound.h.
Regards,
Hans
Audio memory to memory virtual driver use video memory to memory virtual driver vim2m.c as example. The main difference is device type is VFL_TYPE_AUDIO and device cap type is V4L2_CAP_AUDIO_M2M.
The device_run function is a dummy function, which is simply copy the data from input buffer to output buffer.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com --- drivers/media/test-drivers/Kconfig | 9 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/viaudm2m.c | 707 ++++++++++++++++++++++++++ 3 files changed, 717 insertions(+) create mode 100644 drivers/media/test-drivers/viaudm2m.c
diff --git a/drivers/media/test-drivers/Kconfig b/drivers/media/test-drivers/Kconfig index 459b433e9fae..143d829e1b67 100644 --- a/drivers/media/test-drivers/Kconfig +++ b/drivers/media/test-drivers/Kconfig @@ -17,6 +17,15 @@ config VIDEO_VIM2M This is a virtual test device for the memory-to-memory driver framework.
+config VIDEO_VIAUDM2M + tristate "Virtual Memory-to-Memory Driver For Audio" + depends on VIDEO_DEV + select VIDEOBUF2_VMALLOC + select V4L2_MEM2MEM_DEV + help + This is a virtual audio test device for the memory-to-memory driver + framework. + source "drivers/media/test-drivers/vicodec/Kconfig" source "drivers/media/test-drivers/vimc/Kconfig" source "drivers/media/test-drivers/vivid/Kconfig" diff --git a/drivers/media/test-drivers/Makefile b/drivers/media/test-drivers/Makefile index 740714a4584d..a39d27eb53fa 100644 --- a/drivers/media/test-drivers/Makefile +++ b/drivers/media/test-drivers/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_DVB_VIDTV) += vidtv/
obj-$(CONFIG_VIDEO_VICODEC) += vicodec/ obj-$(CONFIG_VIDEO_VIM2M) += vim2m.o +obj-$(CONFIG_VIDEO_VIAUDM2M) += viaudm2m.o obj-$(CONFIG_VIDEO_VIMC) += vimc/ obj-$(CONFIG_VIDEO_VIVID) += vivid/ obj-$(CONFIG_VIDEO_VISL) += visl/ diff --git a/drivers/media/test-drivers/viaudm2m.c b/drivers/media/test-drivers/viaudm2m.c new file mode 100644 index 000000000000..5fefd616b6c8 --- /dev/null +++ b/drivers/media/test-drivers/viaudm2m.c @@ -0,0 +1,707 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * A virtual v4l2-mem2mem example for audio device. + */ + +#include <linux/module.h> +#include <linux/delay.h> +#include <linux/fs.h> +#include <linux/sched.h> +#include <linux/slab.h> + +#include <linux/platform_device.h> +#include <media/v4l2-mem2mem.h> +#include <media/v4l2-device.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-event.h> +#include <media/videobuf2-vmalloc.h> +#include <sound/dmaengine_pcm.h> + +MODULE_DESCRIPTION("Virtual device for audio mem2mem testing"); +MODULE_LICENSE("GPL"); +MODULE_VERSION("0.1"); +MODULE_ALIAS("audio_mem2mem_testdev"); + +static unsigned int debug; +module_param(debug, uint, 0644); +MODULE_PARM_DESC(debug, "debug level"); + +#define MEM2MEM_NAME "viaudm2m" + +#define dprintk(dev, lvl, fmt, arg...) \ + v4l2_dbg(lvl, debug, &(dev)->v4l2_dev, "%s: " fmt, __func__, ## arg) + +#define SAMPLE_NUM 4096 + +static void audm2m_dev_release(struct device *dev) +{} + +static struct platform_device audm2m_pdev = { + .name = MEM2MEM_NAME, + .dev.release = audm2m_dev_release, +}; + +struct audm2m_fmt { + u32 fourcc; + snd_pcm_format_t format; +}; + +static struct audm2m_fmt formats[] = { + { + .fourcc = V4L2_AUDIO_FMT_S8, + }, + { + .fourcc = V4L2_AUDIO_FMT_S16_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U16_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S24_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S24_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U24_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U24_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S32_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U32_LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_S20_3LE, + }, + { + .fourcc = V4L2_AUDIO_FMT_U20_3LE, + }, +}; + +#define NUM_FORMATS ARRAY_SIZE(formats) + +/* Per-queue, driver-specific private data */ +struct audm2m_q_data { + unsigned int rate; + unsigned int channels; + unsigned int buffersize; + struct audm2m_fmt *fmt; +}; + +enum { + V4L2_M2M_SRC = 0, + V4L2_M2M_DST = 1, +}; + +static struct audm2m_fmt *find_format(u32 fourcc) +{ + struct audm2m_fmt *fmt; + unsigned int k; + + for (k = 0; k < NUM_FORMATS; k++) { + fmt = &formats[k]; + if (fmt->fourcc == fourcc) + break; + } + + if (k == NUM_FORMATS) + return NULL; + + formats[k].format = (__force snd_pcm_format_t)v4l2_fourcc_to_audfmt(formats[k].fourcc); + + return &formats[k]; +} + +struct audm2m_dev { + struct v4l2_device v4l2_dev; + struct video_device vfd; + + struct mutex dev_mutex; + + struct v4l2_m2m_dev *m2m_dev; +}; + +struct audm2m_ctx { + struct v4l2_fh fh; + struct audm2m_dev *dev; + + struct mutex vb_mutex; + + /* Source and destination queue data */ + struct audm2m_q_data q_data[2]; +}; + +static inline struct audm2m_ctx *file2ctx(struct file *file) +{ + return container_of(file->private_data, struct audm2m_ctx, fh); +} + +static struct audm2m_q_data *get_q_data(struct audm2m_ctx *ctx, + enum v4l2_buf_type type) +{ + if (type == V4L2_BUF_TYPE_AUDIO_OUTPUT) + return &ctx->q_data[V4L2_M2M_SRC]; + return &ctx->q_data[V4L2_M2M_DST]; +} + +static const char *type_name(enum v4l2_buf_type type) +{ + if (type == V4L2_BUF_TYPE_AUDIO_OUTPUT) + return "Output"; + return "Capture"; +} + +/* + * mem2mem callbacks + */ + +/* + * device_run() - prepares and starts the device + */ +static void device_run(void *priv) +{ + struct audm2m_ctx *ctx = priv; + struct audm2m_dev *audm2m_dev; + struct vb2_v4l2_buffer *src_buf, *dst_buf; + struct audm2m_q_data *q_data_src, *q_data_dst; + int src_size, dst_size; + + audm2m_dev = ctx->dev; + + q_data_src = get_q_data(ctx, V4L2_BUF_TYPE_AUDIO_OUTPUT); + if (!q_data_src) + return; + + q_data_dst = get_q_data(ctx, V4L2_BUF_TYPE_AUDIO_CAPTURE); + if (!q_data_dst) + return; + + src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx); + dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx); + + /* Process the conversion */ + src_size = vb2_get_plane_payload(&src_buf->vb2_buf, 0); + + if (src_size > q_data_dst->buffersize) + dst_size = q_data_dst->buffersize; + else + dst_size = src_size; + + memcpy(vb2_plane_vaddr(&dst_buf->vb2_buf, 0), + vb2_plane_vaddr(&src_buf->vb2_buf, 0), + dst_size); + + vb2_set_plane_payload(&dst_buf->vb2_buf, 0, dst_size); + + src_buf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx); + dst_buf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE); + v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE); + v4l2_m2m_job_finish(audm2m_dev->m2m_dev, ctx->fh.m2m_ctx); +} + +static int audm2m_querycap(struct file *file, void *priv, + struct v4l2_capability *cap) +{ + strscpy(cap->driver, MEM2MEM_NAME, sizeof(cap->driver)); + strscpy(cap->card, MEM2MEM_NAME, sizeof(cap->card)); + snprintf(cap->bus_info, sizeof(cap->bus_info), + "platform:%s", MEM2MEM_NAME); + + return 0; +} + +static int enum_fmt(struct v4l2_fmtdesc *f) +{ + int i, num; + struct audm2m_fmt *fmt; + + num = 0; + + for (i = 0; i < NUM_FORMATS; ++i) { + if (num == f->index) + break; + /* + * Correct type but haven't reached our index yet, + * just increment per-type index + */ + ++num; + } + + if (i < NUM_FORMATS) { + /* Format found */ + fmt = &formats[i]; + f->pixelformat = fmt->fourcc; + return 0; + } + + /* Format not found */ + return -EINVAL; +} + +static int audm2m_enum_fmt_audio_cap(struct file *file, void *priv, + struct v4l2_fmtdesc *f) +{ + return enum_fmt(f); +} + +static int audm2m_enum_fmt_audio_out(struct file *file, void *priv, + struct v4l2_fmtdesc *f) +{ + return enum_fmt(f); +} + +static int audm2m_g_fmt(struct audm2m_ctx *ctx, struct v4l2_format *f) +{ + struct vb2_queue *vq; + struct audm2m_q_data *q_data; + + vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type); + if (!vq) + return -EINVAL; + + q_data = get_q_data(ctx, f->type); + if (!q_data) + return -EINVAL; + + f->fmt.audio.audioformat = q_data->fmt->fourcc; + f->fmt.audio.channels = q_data->channels; + f->fmt.audio.buffersize = q_data->buffersize; + + return 0; +} + +static int audm2m_g_fmt_audio_out(struct file *file, void *priv, + struct v4l2_format *f) +{ + return audm2m_g_fmt(file2ctx(file), f); +} + +static int audm2m_g_fmt_audio_cap(struct file *file, void *priv, + struct v4l2_format *f) +{ + return audm2m_g_fmt(file2ctx(file), f); +} + +static int audm2m_try_fmt(struct v4l2_format *f, struct audm2m_fmt *fmt) +{ + if (f->fmt.audio.channels < 1) + f->fmt.audio.channels = 1; + else if (f->fmt.audio.channels > 8) + f->fmt.audio.channels = 8; + + f->fmt.audio.buffersize = f->fmt.audio.channels * + snd_pcm_format_physical_width(fmt->format) * + SAMPLE_NUM; + return 0; +} + +static int audm2m_try_fmt_audio_cap(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct audm2m_fmt *fmt; + + fmt = find_format(f->fmt.audio.audioformat); + if (!fmt) { + f->fmt.audio.audioformat = formats[0].fourcc; + fmt = find_format(f->fmt.audio.audioformat); + } + + return audm2m_try_fmt(f, fmt); +} + +static int audm2m_try_fmt_audio_out(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct audm2m_fmt *fmt; + + fmt = find_format(f->fmt.audio.audioformat); + if (!fmt) { + f->fmt.audio.audioformat = formats[0].fourcc; + fmt = find_format(f->fmt.audio.audioformat); + } + + return audm2m_try_fmt(f, fmt); +} + +static int audm2m_s_fmt(struct audm2m_ctx *ctx, struct v4l2_format *f) +{ + struct audm2m_q_data *q_data; + struct vb2_queue *vq; + + vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type); + if (!vq) + return -EINVAL; + + q_data = get_q_data(ctx, f->type); + if (!q_data) + return -EINVAL; + + if (vb2_is_busy(vq)) { + v4l2_err(&ctx->dev->v4l2_dev, "%s queue busy\n", __func__); + return -EBUSY; + } + + q_data->fmt = find_format(f->fmt.audio.audioformat); + q_data->channels = f->fmt.audio.channels; + q_data->buffersize = q_data->channels * + snd_pcm_format_physical_width(q_data->fmt->format) * + SAMPLE_NUM; + + dprintk(ctx->dev, 1, + "Format for type %s: %d/%d, fmt: %c%c%c%c\n", + type_name(f->type), q_data->rate, + q_data->channels, + (q_data->fmt->fourcc & 0xff), + (q_data->fmt->fourcc >> 8) & 0xff, + (q_data->fmt->fourcc >> 16) & 0xff, + (q_data->fmt->fourcc >> 24) & 0xff); + + return 0; +} + +static int audm2m_s_fmt_audio_cap(struct file *file, void *priv, + struct v4l2_format *f) +{ + int ret; + + ret = audm2m_try_fmt_audio_cap(file, priv, f); + if (ret) + return ret; + + return audm2m_s_fmt(file2ctx(file), f); +} + +static int audm2m_s_fmt_audio_out(struct file *file, void *priv, + struct v4l2_format *f) +{ + int ret; + + ret = audm2m_try_fmt_audio_out(file, priv, f); + if (ret) + return ret; + + return audm2m_s_fmt(file2ctx(file), f); +} + +static const struct v4l2_ioctl_ops audm2m_ioctl_ops = { + .vidioc_querycap = audm2m_querycap, + + .vidioc_enum_fmt_audio_cap = audm2m_enum_fmt_audio_cap, + .vidioc_g_fmt_audio_cap = audm2m_g_fmt_audio_cap, + .vidioc_try_fmt_audio_cap = audm2m_try_fmt_audio_cap, + .vidioc_s_fmt_audio_cap = audm2m_s_fmt_audio_cap, + + .vidioc_enum_fmt_audio_out = audm2m_enum_fmt_audio_out, + .vidioc_g_fmt_audio_out = audm2m_g_fmt_audio_out, + .vidioc_try_fmt_audio_out = audm2m_try_fmt_audio_out, + .vidioc_s_fmt_audio_out = audm2m_s_fmt_audio_out, + + .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs, + .vidioc_querybuf = v4l2_m2m_ioctl_querybuf, + .vidioc_qbuf = v4l2_m2m_ioctl_qbuf, + .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf, + .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf, + .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs, + .vidioc_expbuf = v4l2_m2m_ioctl_expbuf, + + .vidioc_streamon = v4l2_m2m_ioctl_streamon, + .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, + + .vidioc_subscribe_event = v4l2_ctrl_subscribe_event, + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, +}; + +/* + * Queue operations + */ +static int audm2m_queue_setup(struct vb2_queue *vq, + unsigned int *nbuffers, + unsigned int *nplanes, + unsigned int sizes[], + struct device *alloc_devs[]) +{ + struct audm2m_ctx *ctx = vb2_get_drv_priv(vq); + struct audm2m_q_data *q_data; + + q_data = get_q_data(ctx, vq->type); + + if (*nplanes) + return sizes[0] < q_data->buffersize ? -EINVAL : 0; + + *nplanes = 1; + sizes[0] = q_data->buffersize; + + dprintk(ctx->dev, 1, "%s: get %d buffer(s) of size %d each.\n", + type_name(vq->type), *nplanes, sizes[0]); + + return 0; +} + +static void audm2m_buf_queue(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); + struct audm2m_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + + v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, vbuf); +} + +static void audm2m_stop_streaming(struct vb2_queue *q) +{ + struct audm2m_ctx *ctx = vb2_get_drv_priv(q); + struct vb2_v4l2_buffer *vbuf; + + for (;;) { + if (V4L2_TYPE_IS_OUTPUT(q->type)) + vbuf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx); + else + vbuf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx); + if (!vbuf) + return; + v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR); + } +} + +static const struct vb2_ops audm2m_qops = { + .queue_setup = audm2m_queue_setup, + .buf_queue = audm2m_buf_queue, + .stop_streaming = audm2m_stop_streaming, + .wait_prepare = vb2_ops_wait_prepare, + .wait_finish = vb2_ops_wait_finish, +}; + +static int queue_init(void *priv, struct vb2_queue *src_vq, + struct vb2_queue *dst_vq) +{ + struct audm2m_ctx *ctx = priv; + int ret; + + src_vq->type = V4L2_BUF_TYPE_AUDIO_OUTPUT; + src_vq->io_modes = VB2_MMAP | VB2_DMABUF; + src_vq->drv_priv = ctx; + src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + src_vq->ops = &audm2m_qops; + src_vq->mem_ops = &vb2_vmalloc_memops; + src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + src_vq->lock = &ctx->vb_mutex; + src_vq->min_buffers_needed = 1; + + ret = vb2_queue_init(src_vq); + if (ret) + return ret; + + dst_vq->type = V4L2_BUF_TYPE_AUDIO_CAPTURE; + dst_vq->io_modes = VB2_MMAP | VB2_DMABUF; + dst_vq->drv_priv = ctx; + dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer); + dst_vq->ops = &audm2m_qops; + dst_vq->mem_ops = &vb2_vmalloc_memops; + dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + dst_vq->lock = &ctx->vb_mutex; + dst_vq->min_buffers_needed = 1; + + return vb2_queue_init(dst_vq); +} + +/* + * File operations + */ +static int audm2m_open(struct file *file) +{ + struct audm2m_dev *dev = video_drvdata(file); + struct audm2m_ctx *ctx = NULL; + int width; + int rc = 0; + + if (mutex_lock_interruptible(&dev->dev_mutex)) + return -ERESTARTSYS; + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (!ctx) { + rc = -ENOMEM; + goto open_unlock; + } + + v4l2_fh_init(&ctx->fh, video_devdata(file)); + file->private_data = &ctx->fh; + ctx->dev = dev; + + ctx->q_data[V4L2_M2M_SRC].fmt = &formats[0]; + ctx->q_data[V4L2_M2M_SRC].rate = 8000; + ctx->q_data[V4L2_M2M_SRC].channels = 2; + + /* Fix to 4096 samples */ + width = snd_pcm_format_physical_width(formats[0].format); + ctx->q_data[V4L2_M2M_SRC].buffersize = SAMPLE_NUM * 2 * width; + ctx->q_data[V4L2_M2M_DST] = ctx->q_data[V4L2_M2M_SRC]; + + ctx->fh.m2m_ctx = v4l2_m2m_ctx_init(dev->m2m_dev, ctx, &queue_init); + + mutex_init(&ctx->vb_mutex); + + if (IS_ERR(ctx->fh.m2m_ctx)) { + rc = PTR_ERR(ctx->fh.m2m_ctx); + + v4l2_fh_exit(&ctx->fh); + kfree(ctx); + goto open_unlock; + } + + v4l2_fh_add(&ctx->fh); + + dprintk(dev, 1, "Created instance: %p, m2m_ctx: %p\n", + ctx, ctx->fh.m2m_ctx); + +open_unlock: + mutex_unlock(&dev->dev_mutex); + return rc; +} + +static int audm2m_release(struct file *file) +{ + struct audm2m_dev *dev = video_drvdata(file); + struct audm2m_ctx *ctx = file2ctx(file); + + dprintk(dev, 1, "Releasing instance %p\n", ctx); + + v4l2_fh_del(&ctx->fh); + v4l2_fh_exit(&ctx->fh); + mutex_lock(&dev->dev_mutex); + v4l2_m2m_ctx_release(ctx->fh.m2m_ctx); + mutex_unlock(&dev->dev_mutex); + kfree(ctx); + + return 0; +} + +static void audm2m_device_release(struct video_device *vdev) +{ + struct audm2m_dev *dev = container_of(vdev, struct audm2m_dev, vfd); + + v4l2_device_unregister(&dev->v4l2_dev); + v4l2_m2m_release(dev->m2m_dev); + + kfree(dev); +} + +static const struct v4l2_file_operations audm2m_fops = { + .owner = THIS_MODULE, + .open = audm2m_open, + .release = audm2m_release, + .poll = v4l2_m2m_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = v4l2_m2m_fop_mmap, +}; + +static const struct video_device audm2m_videodev = { + .name = MEM2MEM_NAME, + .vfl_dir = VFL_DIR_M2M, + .fops = &audm2m_fops, + .ioctl_ops = &audm2m_ioctl_ops, + .minor = -1, + .release = audm2m_device_release, + .device_caps = V4L2_CAP_AUDIO_M2M | V4L2_CAP_STREAMING, +}; + +static const struct v4l2_m2m_ops m2m_ops = { + .device_run = device_run, +}; + +static int audm2m_probe(struct platform_device *pdev) +{ + struct audm2m_dev *dev; + struct video_device *vfd; + int ret; + + dev = kzalloc(sizeof(*dev), GFP_KERNEL); + if (!dev) + return -ENOMEM; + + ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev); + if (ret) + goto error_free; + + mutex_init(&dev->dev_mutex); + + dev->vfd = audm2m_videodev; + vfd = &dev->vfd; + vfd->lock = &dev->dev_mutex; + vfd->v4l2_dev = &dev->v4l2_dev; + + video_set_drvdata(vfd, dev); + v4l2_info(&dev->v4l2_dev, + "Device registered as /dev/v4l-audio%d\n", vfd->num); + + platform_set_drvdata(pdev, dev); + + dev->m2m_dev = v4l2_m2m_init(&m2m_ops); + if (IS_ERR(dev->m2m_dev)) { + v4l2_err(&dev->v4l2_dev, "Failed to init mem2mem device\n"); + ret = PTR_ERR(dev->m2m_dev); + dev->m2m_dev = NULL; + goto error_dev; + } + + ret = video_register_device(vfd, VFL_TYPE_AUDIO, 0); + if (ret) { + v4l2_err(&dev->v4l2_dev, "Failed to register video device\n"); + goto error_m2m; + } + + return 0; + +error_m2m: + v4l2_m2m_release(dev->m2m_dev); +error_dev: + v4l2_device_unregister(&dev->v4l2_dev); +error_free: + kfree(dev); + + return ret; +} + +static void audm2m_remove(struct platform_device *pdev) +{ + struct audm2m_dev *dev = platform_get_drvdata(pdev); + + v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME); + + video_unregister_device(&dev->vfd); +} + +static struct platform_driver audm2m_pdrv = { + .probe = audm2m_probe, + .remove_new = audm2m_remove, + .driver = { + .name = MEM2MEM_NAME, + }, +}; + +static void __exit audm2m_exit(void) +{ + platform_driver_unregister(&audm2m_pdrv); + platform_device_unregister(&audm2m_pdev); +} + +static int __init audm2m_init(void) +{ + int ret; + + ret = platform_device_register(&audm2m_pdev); + if (ret) + return ret; + + ret = platform_driver_register(&audm2m_pdrv); + if (ret) + platform_device_unregister(&audm2m_pdev); + + return ret; +} + +module_init(audm2m_init); +module_exit(audm2m_exit);
On 13/10/2023 10:31, Shengjiu Wang wrote:
Audio memory to memory virtual driver use video memory to memory virtual driver vim2m.c as example. The main difference is device type is VFL_TYPE_AUDIO and device cap type is V4L2_CAP_AUDIO_M2M.
The device_run function is a dummy function, which is simply copy the data from input buffer to output buffer.
Signed-off-by: Shengjiu Wang shengjiu.wang@nxp.com
drivers/media/test-drivers/Kconfig | 9 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/viaudm2m.c | 707 ++++++++++++++++++++++++++ 3 files changed, 717 insertions(+) create mode 100644 drivers/media/test-drivers/viaudm2m.c
diff --git a/drivers/media/test-drivers/Kconfig b/drivers/media/test-drivers/Kconfig index 459b433e9fae..143d829e1b67 100644 --- a/drivers/media/test-drivers/Kconfig +++ b/drivers/media/test-drivers/Kconfig @@ -17,6 +17,15 @@ config VIDEO_VIM2M This is a virtual test device for the memory-to-memory driver framework.
+config VIDEO_VIAUDM2M
- tristate "Virtual Memory-to-Memory Driver For Audio"
I think this is better:
VIDEO_VIM2M_AUDIO
and name the source: vim2m_audio.c.
- depends on VIDEO_DEV
- select VIDEOBUF2_VMALLOC
- select V4L2_MEM2MEM_DEV
- help
This is a virtual audio test device for the memory-to-memory driverframework.source "drivers/media/test-drivers/vicodec/Kconfig" source "drivers/media/test-drivers/vimc/Kconfig" source "drivers/media/test-drivers/vivid/Kconfig" diff --git a/drivers/media/test-drivers/Makefile b/drivers/media/test-drivers/Makefile index 740714a4584d..a39d27eb53fa 100644 --- a/drivers/media/test-drivers/Makefile +++ b/drivers/media/test-drivers/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_DVB_VIDTV) += vidtv/
obj-$(CONFIG_VIDEO_VICODEC) += vicodec/ obj-$(CONFIG_VIDEO_VIM2M) += vim2m.o +obj-$(CONFIG_VIDEO_VIAUDM2M) += viaudm2m.o obj-$(CONFIG_VIDEO_VIMC) += vimc/ obj-$(CONFIG_VIDEO_VIVID) += vivid/ obj-$(CONFIG_VIDEO_VISL) += visl/ diff --git a/drivers/media/test-drivers/viaudm2m.c b/drivers/media/test-drivers/viaudm2m.c new file mode 100644 index 000000000000..5fefd616b6c8 --- /dev/null +++ b/drivers/media/test-drivers/viaudm2m.c @@ -0,0 +1,707 @@ +// SPDX-License-Identifier: GPL-2.0+ +/*
- A virtual v4l2-mem2mem example for audio device.
- */
+#include <linux/module.h> +#include <linux/delay.h> +#include <linux/fs.h> +#include <linux/sched.h> +#include <linux/slab.h>
+#include <linux/platform_device.h> +#include <media/v4l2-mem2mem.h> +#include <media/v4l2-device.h> +#include <media/v4l2-ioctl.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-event.h> +#include <media/videobuf2-vmalloc.h> +#include <sound/dmaengine_pcm.h>
+MODULE_DESCRIPTION("Virtual device for audio mem2mem testing"); +MODULE_LICENSE("GPL"); +MODULE_VERSION("0.1");
Drop this, rarely needed.
+MODULE_ALIAS("audio_mem2mem_testdev");
Drop this as well. I'm not sure why it is in the original vim2m.c source.
+static unsigned int debug; +module_param(debug, uint, 0644); +MODULE_PARM_DESC(debug, "debug level");
+#define MEM2MEM_NAME "viaudm2m"
"vim2m-audio"
+#define dprintk(dev, lvl, fmt, arg...) \
- v4l2_dbg(lvl, debug, &(dev)->v4l2_dev, "%s: " fmt, __func__, ## arg)
+#define SAMPLE_NUM 4096
+static void audm2m_dev_release(struct device *dev) +{}
+static struct platform_device audm2m_pdev = {
- .name = MEM2MEM_NAME,
- .dev.release = audm2m_dev_release,
+};
+struct audm2m_fmt {
- u32 fourcc;
- snd_pcm_format_t format;
Same as for the previous driver: I think you can just drop this field and thus the whole struct.
+};
+static struct audm2m_fmt formats[] = {
And this is just a u32 array.
- {
.fourcc = V4L2_AUDIO_FMT_S8,- },
- {
.fourcc = V4L2_AUDIO_FMT_S16_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U16_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S24_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S24_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U24_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U24_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S32_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U32_LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_S20_3LE,- },
- {
.fourcc = V4L2_AUDIO_FMT_U20_3LE,- },
+};
+#define NUM_FORMATS ARRAY_SIZE(formats)
+/* Per-queue, driver-specific private data */ +struct audm2m_q_data {
- unsigned int rate;
- unsigned int channels;
- unsigned int buffersize;
- struct audm2m_fmt *fmt;
+};
+enum {
- V4L2_M2M_SRC = 0,
- V4L2_M2M_DST = 1,
+};
+static struct audm2m_fmt *find_format(u32 fourcc) +{
- struct audm2m_fmt *fmt;
- unsigned int k;
- for (k = 0; k < NUM_FORMATS; k++) {
fmt = &formats[k];if (fmt->fourcc == fourcc)break;- }
- if (k == NUM_FORMATS)
return NULL;- formats[k].format = (__force snd_pcm_format_t)v4l2_fourcc_to_audfmt(formats[k].fourcc);
- return &formats[k];
+}
+struct audm2m_dev {
- struct v4l2_device v4l2_dev;
- struct video_device vfd;
- struct mutex dev_mutex;
- struct v4l2_m2m_dev *m2m_dev;
+};
+struct audm2m_ctx {
- struct v4l2_fh fh;
- struct audm2m_dev *dev;
- struct mutex vb_mutex;
- /* Source and destination queue data */
- struct audm2m_q_data q_data[2];
+};
+static inline struct audm2m_ctx *file2ctx(struct file *file) +{
- return container_of(file->private_data, struct audm2m_ctx, fh);
+}
+static struct audm2m_q_data *get_q_data(struct audm2m_ctx *ctx,
enum v4l2_buf_type type)+{
- if (type == V4L2_BUF_TYPE_AUDIO_OUTPUT)
return &ctx->q_data[V4L2_M2M_SRC];- return &ctx->q_data[V4L2_M2M_DST];
+}
+static const char *type_name(enum v4l2_buf_type type) +{
- if (type == V4L2_BUF_TYPE_AUDIO_OUTPUT)
return "Output";- return "Capture";
+}
+/*
- mem2mem callbacks
- */
+/*
- device_run() - prepares and starts the device
- */
+static void device_run(void *priv) +{
- struct audm2m_ctx *ctx = priv;
- struct audm2m_dev *audm2m_dev;
- struct vb2_v4l2_buffer *src_buf, *dst_buf;
- struct audm2m_q_data *q_data_src, *q_data_dst;
- int src_size, dst_size;
- audm2m_dev = ctx->dev;
- q_data_src = get_q_data(ctx, V4L2_BUF_TYPE_AUDIO_OUTPUT);
- if (!q_data_src)
return;- q_data_dst = get_q_data(ctx, V4L2_BUF_TYPE_AUDIO_CAPTURE);
- if (!q_data_dst)
return;- src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
- dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
- /* Process the conversion */
- src_size = vb2_get_plane_payload(&src_buf->vb2_buf, 0);
- if (src_size > q_data_dst->buffersize)
dst_size = q_data_dst->buffersize;- else
dst_size = src_size;- memcpy(vb2_plane_vaddr(&dst_buf->vb2_buf, 0),
vb2_plane_vaddr(&src_buf->vb2_buf, 0),dst_size);- vb2_set_plane_payload(&dst_buf->vb2_buf, 0, dst_size);
- src_buf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx);
- dst_buf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx);
- v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE);
- v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE);
- v4l2_m2m_job_finish(audm2m_dev->m2m_dev, ctx->fh.m2m_ctx);
+}
+static int audm2m_querycap(struct file *file, void *priv,
struct v4l2_capability *cap)+{
- strscpy(cap->driver, MEM2MEM_NAME, sizeof(cap->driver));
- strscpy(cap->card, MEM2MEM_NAME, sizeof(cap->card));
- snprintf(cap->bus_info, sizeof(cap->bus_info),
"platform:%s", MEM2MEM_NAME);- return 0;
+}
+static int enum_fmt(struct v4l2_fmtdesc *f) +{
- int i, num;
- struct audm2m_fmt *fmt;
- num = 0;
- for (i = 0; i < NUM_FORMATS; ++i) {
if (num == f->index)break;/** Correct type but haven't reached our index yet,* just increment per-type index*/++num;- }
- if (i < NUM_FORMATS) {
/* Format found */fmt = &formats[i];f->pixelformat = fmt->fourcc;return 0;- }
- /* Format not found */
- return -EINVAL;
+}
+static int audm2m_enum_fmt_audio_cap(struct file *file, void *priv,
struct v4l2_fmtdesc *f)+{
- return enum_fmt(f);
+}
+static int audm2m_enum_fmt_audio_out(struct file *file, void *priv,
struct v4l2_fmtdesc *f)+{
- return enum_fmt(f);
+}
+static int audm2m_g_fmt(struct audm2m_ctx *ctx, struct v4l2_format *f) +{
- struct vb2_queue *vq;
- struct audm2m_q_data *q_data;
- vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type);
- if (!vq)
return -EINVAL;- q_data = get_q_data(ctx, f->type);
- if (!q_data)
return -EINVAL;- f->fmt.audio.audioformat = q_data->fmt->fourcc;
- f->fmt.audio.channels = q_data->channels;
- f->fmt.audio.buffersize = q_data->buffersize;
- return 0;
+}
+static int audm2m_g_fmt_audio_out(struct file *file, void *priv,
struct v4l2_format *f)+{
- return audm2m_g_fmt(file2ctx(file), f);
+}
+static int audm2m_g_fmt_audio_cap(struct file *file, void *priv,
struct v4l2_format *f)+{
- return audm2m_g_fmt(file2ctx(file), f);
+}
+static int audm2m_try_fmt(struct v4l2_format *f, struct audm2m_fmt *fmt) +{
- if (f->fmt.audio.channels < 1)
f->fmt.audio.channels = 1;- else if (f->fmt.audio.channels > 8)
f->fmt.audio.channels = 8;- f->fmt.audio.buffersize = f->fmt.audio.channels *
snd_pcm_format_physical_width(fmt->format) *SAMPLE_NUM;- return 0;
+}
+static int audm2m_try_fmt_audio_cap(struct file *file, void *priv,
struct v4l2_format *f)+{
- struct audm2m_fmt *fmt;
- fmt = find_format(f->fmt.audio.audioformat);
- if (!fmt) {
f->fmt.audio.audioformat = formats[0].fourcc;fmt = find_format(f->fmt.audio.audioformat);- }
- return audm2m_try_fmt(f, fmt);
+}
+static int audm2m_try_fmt_audio_out(struct file *file, void *priv,
struct v4l2_format *f)+{
- struct audm2m_fmt *fmt;
- fmt = find_format(f->fmt.audio.audioformat);
- if (!fmt) {
f->fmt.audio.audioformat = formats[0].fourcc;fmt = find_format(f->fmt.audio.audioformat);- }
- return audm2m_try_fmt(f, fmt);
+}
+static int audm2m_s_fmt(struct audm2m_ctx *ctx, struct v4l2_format *f) +{
- struct audm2m_q_data *q_data;
- struct vb2_queue *vq;
- vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type);
- if (!vq)
return -EINVAL;- q_data = get_q_data(ctx, f->type);
- if (!q_data)
return -EINVAL;- if (vb2_is_busy(vq)) {
v4l2_err(&ctx->dev->v4l2_dev, "%s queue busy\n", __func__);return -EBUSY;- }
- q_data->fmt = find_format(f->fmt.audio.audioformat);
- q_data->channels = f->fmt.audio.channels;
- q_data->buffersize = q_data->channels *
snd_pcm_format_physical_width(q_data->fmt->format) *SAMPLE_NUM;- dprintk(ctx->dev, 1,
"Format for type %s: %d/%d, fmt: %c%c%c%c\n",type_name(f->type), q_data->rate,q_data->channels,(q_data->fmt->fourcc & 0xff),(q_data->fmt->fourcc >> 8) & 0xff,(q_data->fmt->fourcc >> 16) & 0xff,(q_data->fmt->fourcc >> 24) & 0xff);- return 0;
+}
+static int audm2m_s_fmt_audio_cap(struct file *file, void *priv,
struct v4l2_format *f)+{
- int ret;
- ret = audm2m_try_fmt_audio_cap(file, priv, f);
- if (ret)
return ret;- return audm2m_s_fmt(file2ctx(file), f);
+}
+static int audm2m_s_fmt_audio_out(struct file *file, void *priv,
struct v4l2_format *f)+{
- int ret;
- ret = audm2m_try_fmt_audio_out(file, priv, f);
- if (ret)
return ret;- return audm2m_s_fmt(file2ctx(file), f);
+}
+static const struct v4l2_ioctl_ops audm2m_ioctl_ops = {
- .vidioc_querycap = audm2m_querycap,
- .vidioc_enum_fmt_audio_cap = audm2m_enum_fmt_audio_cap,
- .vidioc_g_fmt_audio_cap = audm2m_g_fmt_audio_cap,
- .vidioc_try_fmt_audio_cap = audm2m_try_fmt_audio_cap,
- .vidioc_s_fmt_audio_cap = audm2m_s_fmt_audio_cap,
- .vidioc_enum_fmt_audio_out = audm2m_enum_fmt_audio_out,
- .vidioc_g_fmt_audio_out = audm2m_g_fmt_audio_out,
- .vidioc_try_fmt_audio_out = audm2m_try_fmt_audio_out,
- .vidioc_s_fmt_audio_out = audm2m_s_fmt_audio_out,
- .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
- .vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
- .vidioc_qbuf = v4l2_m2m_ioctl_qbuf,
- .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
- .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,
- .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
- .vidioc_expbuf = v4l2_m2m_ioctl_expbuf,
- .vidioc_streamon = v4l2_m2m_ioctl_streamon,
- .vidioc_streamoff = v4l2_m2m_ioctl_streamoff,
- .vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
- .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+/*
- Queue operations
- */
+static int audm2m_queue_setup(struct vb2_queue *vq,
unsigned int *nbuffers,unsigned int *nplanes,unsigned int sizes[],struct device *alloc_devs[])+{
- struct audm2m_ctx *ctx = vb2_get_drv_priv(vq);
- struct audm2m_q_data *q_data;
- q_data = get_q_data(ctx, vq->type);
- if (*nplanes)
return sizes[0] < q_data->buffersize ? -EINVAL : 0;- *nplanes = 1;
- sizes[0] = q_data->buffersize;
- dprintk(ctx->dev, 1, "%s: get %d buffer(s) of size %d each.\n",
type_name(vq->type), *nplanes, sizes[0]);- return 0;
+}
+static void audm2m_buf_queue(struct vb2_buffer *vb) +{
- struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
- struct audm2m_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
- v4l2_m2m_buf_queue(ctx->fh.m2m_ctx, vbuf);
+}
+static void audm2m_stop_streaming(struct vb2_queue *q) +{
- struct audm2m_ctx *ctx = vb2_get_drv_priv(q);
- struct vb2_v4l2_buffer *vbuf;
- for (;;) {
if (V4L2_TYPE_IS_OUTPUT(q->type))vbuf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx);elsevbuf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx);if (!vbuf)return;v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR);- }
+}
+static const struct vb2_ops audm2m_qops = {
- .queue_setup = audm2m_queue_setup,
- .buf_queue = audm2m_buf_queue,
- .stop_streaming = audm2m_stop_streaming,
- .wait_prepare = vb2_ops_wait_prepare,
- .wait_finish = vb2_ops_wait_finish,
+};
+static int queue_init(void *priv, struct vb2_queue *src_vq,
struct vb2_queue *dst_vq)+{
- struct audm2m_ctx *ctx = priv;
- int ret;
- src_vq->type = V4L2_BUF_TYPE_AUDIO_OUTPUT;
- src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
- src_vq->drv_priv = ctx;
- src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
- src_vq->ops = &audm2m_qops;
- src_vq->mem_ops = &vb2_vmalloc_memops;
- src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
- src_vq->lock = &ctx->vb_mutex;
- src_vq->min_buffers_needed = 1;
- ret = vb2_queue_init(src_vq);
- if (ret)
return ret;- dst_vq->type = V4L2_BUF_TYPE_AUDIO_CAPTURE;
- dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;
- dst_vq->drv_priv = ctx;
- dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
- dst_vq->ops = &audm2m_qops;
- dst_vq->mem_ops = &vb2_vmalloc_memops;
- dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
- dst_vq->lock = &ctx->vb_mutex;
- dst_vq->min_buffers_needed = 1;
- return vb2_queue_init(dst_vq);
+}
+/*
- File operations
- */
+static int audm2m_open(struct file *file) +{
- struct audm2m_dev *dev = video_drvdata(file);
- struct audm2m_ctx *ctx = NULL;
- int width;
- int rc = 0;
- if (mutex_lock_interruptible(&dev->dev_mutex))
return -ERESTARTSYS;- ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
- if (!ctx) {
rc = -ENOMEM;goto open_unlock;- }
- v4l2_fh_init(&ctx->fh, video_devdata(file));
- file->private_data = &ctx->fh;
- ctx->dev = dev;
- ctx->q_data[V4L2_M2M_SRC].fmt = &formats[0];
- ctx->q_data[V4L2_M2M_SRC].rate = 8000;
- ctx->q_data[V4L2_M2M_SRC].channels = 2;
- /* Fix to 4096 samples */
- width = snd_pcm_format_physical_width(formats[0].format);
- ctx->q_data[V4L2_M2M_SRC].buffersize = SAMPLE_NUM * 2 * width;
- ctx->q_data[V4L2_M2M_DST] = ctx->q_data[V4L2_M2M_SRC];
- ctx->fh.m2m_ctx = v4l2_m2m_ctx_init(dev->m2m_dev, ctx, &queue_init);
- mutex_init(&ctx->vb_mutex);
- if (IS_ERR(ctx->fh.m2m_ctx)) {
rc = PTR_ERR(ctx->fh.m2m_ctx);v4l2_fh_exit(&ctx->fh);kfree(ctx);goto open_unlock;- }
- v4l2_fh_add(&ctx->fh);
- dprintk(dev, 1, "Created instance: %p, m2m_ctx: %p\n",
ctx, ctx->fh.m2m_ctx);+open_unlock:
- mutex_unlock(&dev->dev_mutex);
- return rc;
+}
+static int audm2m_release(struct file *file) +{
- struct audm2m_dev *dev = video_drvdata(file);
- struct audm2m_ctx *ctx = file2ctx(file);
- dprintk(dev, 1, "Releasing instance %p\n", ctx);
- v4l2_fh_del(&ctx->fh);
- v4l2_fh_exit(&ctx->fh);
- mutex_lock(&dev->dev_mutex);
- v4l2_m2m_ctx_release(ctx->fh.m2m_ctx);
- mutex_unlock(&dev->dev_mutex);
- kfree(ctx);
- return 0;
+}
+static void audm2m_device_release(struct video_device *vdev) +{
- struct audm2m_dev *dev = container_of(vdev, struct audm2m_dev, vfd);
- v4l2_device_unregister(&dev->v4l2_dev);
- v4l2_m2m_release(dev->m2m_dev);
- kfree(dev);
+}
+static const struct v4l2_file_operations audm2m_fops = {
- .owner = THIS_MODULE,
- .open = audm2m_open,
- .release = audm2m_release,
- .poll = v4l2_m2m_fop_poll,
- .unlocked_ioctl = video_ioctl2,
- .mmap = v4l2_m2m_fop_mmap,
+};
+static const struct video_device audm2m_videodev = {
- .name = MEM2MEM_NAME,
- .vfl_dir = VFL_DIR_M2M,
- .fops = &audm2m_fops,
- .ioctl_ops = &audm2m_ioctl_ops,
- .minor = -1,
- .release = audm2m_device_release,
- .device_caps = V4L2_CAP_AUDIO_M2M | V4L2_CAP_STREAMING,
+};
+static const struct v4l2_m2m_ops m2m_ops = {
- .device_run = device_run,
+};
+static int audm2m_probe(struct platform_device *pdev) +{
- struct audm2m_dev *dev;
- struct video_device *vfd;
- int ret;
- dev = kzalloc(sizeof(*dev), GFP_KERNEL);
- if (!dev)
return -ENOMEM;- ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
- if (ret)
goto error_free;- mutex_init(&dev->dev_mutex);
- dev->vfd = audm2m_videodev;
- vfd = &dev->vfd;
- vfd->lock = &dev->dev_mutex;
- vfd->v4l2_dev = &dev->v4l2_dev;
- video_set_drvdata(vfd, dev);
- v4l2_info(&dev->v4l2_dev,
"Device registered as /dev/v4l-audio%d\n", vfd->num);- platform_set_drvdata(pdev, dev);
- dev->m2m_dev = v4l2_m2m_init(&m2m_ops);
- if (IS_ERR(dev->m2m_dev)) {
v4l2_err(&dev->v4l2_dev, "Failed to init mem2mem device\n");ret = PTR_ERR(dev->m2m_dev);dev->m2m_dev = NULL;goto error_dev;- }
- ret = video_register_device(vfd, VFL_TYPE_AUDIO, 0);
- if (ret) {
v4l2_err(&dev->v4l2_dev, "Failed to register video device\n");goto error_m2m;- }
- return 0;
+error_m2m:
- v4l2_m2m_release(dev->m2m_dev);
+error_dev:
- v4l2_device_unregister(&dev->v4l2_dev);
+error_free:
- kfree(dev);
- return ret;
+}
+static void audm2m_remove(struct platform_device *pdev) +{
- struct audm2m_dev *dev = platform_get_drvdata(pdev);
- v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME);
- video_unregister_device(&dev->vfd);
+}
+static struct platform_driver audm2m_pdrv = {
- .probe = audm2m_probe,
- .remove_new = audm2m_remove,
- .driver = {
.name = MEM2MEM_NAME,- },
+};
+static void __exit audm2m_exit(void) +{
- platform_driver_unregister(&audm2m_pdrv);
- platform_device_unregister(&audm2m_pdev);
+}
+static int __init audm2m_init(void) +{
- int ret;
- ret = platform_device_register(&audm2m_pdev);
- if (ret)
return ret;- ret = platform_driver_register(&audm2m_pdrv);
- if (ret)
platform_device_unregister(&audm2m_pdev);- return ret;
+}
+module_init(audm2m_init); +module_exit(audm2m_exit);
Regards,
Hans
participants (4)
-
Hans Verkuil -
Mark Brown -
Shengjiu Wang -
Shengjiu Wang