[alsa-utils][PATCH 00/14] axfer: reduce test time
Hi,
The axfer in alsa-utils has test programs for internal modules and overall time to execute them takes much. The issue is filing in github repository[1]. It comes from three causes:
1. file I/O operations are done in file system for actual storage 2. Some cases operate much audio data frames (over 4MB, approx.) 3. much test iteration count (so simple...)
This patchset uses shm by memfd_create(2) for the cause 1. In addition, the number of maximum audio data frame to test is reduced as well as the number of samples per frame.
I got benchmark with berow machine environment.
* AMD Ryzen 5 2400G * memory total 30823852 KB * linux-image-5.8.0-44-generic on Ubuntu 20.04 amd64 * ext4 on SATA SSD
As a result: |container| mapper | | test | test | ================== | ======= | ======= | Current | 112 min | 5 min | +shm | 58 min | 50 sec | +maximum reduction | 38 min | - | +iter reduction | 4 min | - |
In my opinion, the issue comes from package build server in each distribution. 5 min for test execution is not so worse time.
Finally, test programs run on shm and commit c3f2344b7209 is reverted[1] since it's useless now.
(I have free time at last from reverse-engineering work to analyze protocols of RME Fireface series[1], and start to solve stacked issues.)
[1] https://github.com/alsa-project/snd-firewire-ctl-services/pull/37 [2] https://github.com/alsa-project/alsa-utils/issues/19 [3] https://github.com/alsa-project/alsa-utils/commit/c3f2344b7209
Regards
Takashi Sakamoto (14): axfer: minor code arrangement for container module in a point of nonblocking flag axfer: minor code arrangement in a point of stdio detection axfer: minor code arrangement in a point of opened file descriptor axfer: minor code arrangement to allocate containers axfer: open file descriptor outside of container module axfer: maintain lifetime of file descriptor outside of container module autotools: preparation to use memfd_create(2) axfer: test: minor code arrangement to use the same file descriptor for container-test axfer: test: use memfd_create() for container-test axfer: test: minor code arrangement to use the same file descriptor for mappter-test axfer: test: use memfd_create() for mapper-test axfer: test: reduce test case for maximum number of frame count axfer: test: reduce test case for maximum number of samples per frame Revert "axfer: test - add run-test-in-tmpdir.sh script"
axfer/container.c | 57 +++++++++++----------------- axfer/container.h | 10 ++--- axfer/subcmd-transfer.c | 64 ++++++++++++++++++++++++++------ axfer/test/container-test.c | 53 +++++++++++++++++++------- axfer/test/mapper-test.c | 55 ++++++++++++++++++++++----- axfer/test/run-test-in-tmpdir.sh | 19 ---------- configure.ac | 5 +++ 7 files changed, 167 insertions(+), 96 deletions(-) delete mode 100755 axfer/test/run-test-in-tmpdir.sh
In internal container module, any file descriptor is expected as non-blocking mode. Current implementation distinguish the case of standard input and output from the case to open actual file since O_NONBLOCK is used for the latter case. However, in both cases, fcntl(2) is available to set non-blocking mode to the file descriptor.
This commit arranges to use fcntl(2) for both cases.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/container.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/axfer/container.c b/axfer/container.c index 566acd0..8733ff7 100644 --- a/axfer/container.c +++ b/axfer/container.c @@ -176,16 +176,17 @@ int container_parser_init(struct container_context *cntr, "should be referred instead.\n"); return -EIO; } - err = set_nonblock_flag(cntr->fd); - if (err < 0) - return err; cntr->stdio = true; } else { - cntr->fd = open(path, O_RDONLY | O_NONBLOCK); + cntr->fd = open(path, O_RDONLY); if (cntr->fd < 0) return -errno; }
+ err = set_nonblock_flag(cntr->fd); + if (err < 0) + return err; + // 4 bytes are enough to detect supported containers. err = container_recursive_read(cntr, cntr->magic, sizeof(cntr->magic)); if (err < 0) @@ -260,17 +261,17 @@ int container_builder_init(struct container_context *cntr, "should be referred instead.\n"); return -EIO; } - err = set_nonblock_flag(cntr->fd); - if (err < 0) - return err; cntr->stdio = true; } else { - cntr->fd = open(path, O_RDWR | O_NONBLOCK | O_CREAT | O_TRUNC, - 0644); + cntr->fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); if (cntr->fd < 0) return -errno; }
+ err = set_nonblock_flag(cntr->fd); + if (err < 0) + return err; + builder = builders[format];
// Allocate private data for the builder.
Current implementation sets stdio member in a condition branch, however it's convenient to set it always regardless of any condition.
This commit arranges assignment to the member.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/container.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/axfer/container.c b/axfer/container.c index 8733ff7..fb35eba 100644 --- a/axfer/container.c +++ b/axfer/container.c @@ -169,6 +169,14 @@ int container_parser_init(struct container_context *cntr, // Open a target descriptor. if (!strcmp(path, "-")) { cntr->fd = fileno(stdin); + } else { + cntr->fd = open(path, O_RDONLY); + if (cntr->fd < 0) + return -errno; + } + + cntr->stdio = (cntr->fd == fileno(stdin)); + if (cntr->stdio) { if (isatty(cntr->fd)) { fprintf(stderr, "A terminal is referred for standard input. " @@ -176,11 +184,6 @@ int container_parser_init(struct container_context *cntr, "should be referred instead.\n"); return -EIO; } - cntr->stdio = true; - } else { - cntr->fd = open(path, O_RDONLY); - if (cntr->fd < 0) - return -errno; }
err = set_nonblock_flag(cntr->fd); @@ -254,6 +257,14 @@ int container_builder_init(struct container_context *cntr, return -EINVAL; if (!strcmp(path, "-")) { cntr->fd = fileno(stdout); + } else { + cntr->fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (cntr->fd < 0) + return -errno; + } + + cntr->stdio = (cntr->fd == fileno(stdout)); + if (cntr->stdio) { if (isatty(cntr->fd)) { fprintf(stderr, "A terminal is referred for standard output. " @@ -261,11 +272,6 @@ int container_builder_init(struct container_context *cntr, "should be referred instead.\n"); return -EIO; } - cntr->stdio = true; - } else { - cntr->fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); - if (cntr->fd < 0) - return -errno; }
err = set_nonblock_flag(cntr->fd);
This commit arranges assignment to fd member after checking file descriptor.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/container.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/axfer/container.c b/axfer/container.c index fb35eba..b4646b9 100644 --- a/axfer/container.c +++ b/axfer/container.c @@ -151,6 +151,7 @@ int container_parser_init(struct container_context *cntr, [CONTAINER_FORMAT_AU] = &container_parser_au, [CONTAINER_FORMAT_VOC] = &container_parser_voc, }; + int fd; const struct container_parser *parser; unsigned int size; int i; @@ -168,12 +169,13 @@ int container_parser_init(struct container_context *cntr,
// Open a target descriptor. if (!strcmp(path, "-")) { - cntr->fd = fileno(stdin); + fd = fileno(stdin); } else { - cntr->fd = open(path, O_RDONLY); - if (cntr->fd < 0) + fd = open(path, O_RDONLY); + if (fd < 0) return -errno; } + cntr->fd = fd;
cntr->stdio = (cntr->fd == fileno(stdin)); if (cntr->stdio) { @@ -239,6 +241,7 @@ int container_builder_init(struct container_context *cntr, [CONTAINER_FORMAT_VOC] = &container_builder_voc, [CONTAINER_FORMAT_RAW] = &container_builder_raw, }; + int fd; const struct container_builder *builder; int err;
@@ -256,12 +259,13 @@ int container_builder_init(struct container_context *cntr, if (path == NULL || *path == '\0') return -EINVAL; if (!strcmp(path, "-")) { - cntr->fd = fileno(stdout); + fd = fileno(stdout); } else { - cntr->fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); - if (cntr->fd < 0) + fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (fd < 0) return -errno; } + cntr->fd = fd;
cntr->stdio = (cntr->fd == fileno(stdout)); if (cntr->stdio) {
This commit unifies duplicated code to allocate for container structure.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/subcmd-transfer.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/axfer/subcmd-transfer.c b/axfer/subcmd-transfer.c index 8746e6f..6962208 100644 --- a/axfer/subcmd-transfer.c +++ b/axfer/subcmd-transfer.c @@ -146,6 +146,16 @@ static int context_init(struct context *ctx, snd_pcm_stream_t direction, return xfer_context_init(&ctx->xfer, xfer_type, direction, argc, argv); }
+static int allocate_containers(struct context *ctx, unsigned int count) +{ + ctx->cntrs = calloc(count, sizeof(*ctx->cntrs)); + if (ctx->cntrs == NULL) + return -ENOMEM; + ctx->cntr_count = count; + + return 0; +} + static int capture_pre_process(struct context *ctx, snd_pcm_access_t *access, snd_pcm_uframes_t *frames_per_buffer, uint64_t *total_frame_count) @@ -164,10 +174,9 @@ static int capture_pre_process(struct context *ctx, snd_pcm_access_t *access, return err;
// Prepare for containers. - ctx->cntrs = calloc(ctx->xfer.path_count, sizeof(*ctx->cntrs)); - if (ctx->cntrs == NULL) - return -ENOMEM; - ctx->cntr_count = ctx->xfer.path_count; + err = allocate_containers(ctx, ctx->xfer.path_count); + if (err < 0) + return err;
if (ctx->cntr_count > 1) channels = 1; @@ -212,10 +221,9 @@ static int playback_pre_process(struct context *ctx, snd_pcm_access_t *access, int err;
// Prepare for containers. - ctx->cntrs = calloc(ctx->xfer.path_count, sizeof(*ctx->cntrs)); - if (ctx->cntrs == NULL) - return -ENOMEM; - ctx->cntr_count = ctx->xfer.path_count; + err = allocate_containers(ctx, ctx->xfer.path_count); + if (err < 0) + return err;
for (i = 0; i < ctx->cntr_count; ++i) { snd_pcm_format_t format;
Internal container module operates file descriptor to media file. For this purpose, the structure has fd member and any file operation is done internally. However, the case to use special file descriptor such as memfd requires to maintain file descriptor externally.
This commit opens file descriptor outside of container module. The internal APIs to initialize container get an argument for the file descriptor instead of file path.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/container.c | 35 ++++++----------------------------- axfer/container.h | 9 ++++----- axfer/subcmd-transfer.c | 26 ++++++++++++++++++++++---- axfer/test/container-test.c | 12 ++++++++++-- axfer/test/mapper-test.c | 19 ++++++++++++++----- 5 files changed, 56 insertions(+), 45 deletions(-)
diff --git a/axfer/container.c b/axfer/container.c index b4646b9..255f12c 100644 --- a/axfer/container.c +++ b/axfer/container.c @@ -143,23 +143,21 @@ static int set_nonblock_flag(int fd) return 0; }
-int container_parser_init(struct container_context *cntr, - const char *const path, unsigned int verbose) +int container_parser_init(struct container_context *cntr, int fd, + unsigned int verbose) { const struct container_parser *parsers[] = { [CONTAINER_FORMAT_RIFF_WAVE] = &container_parser_riff_wave, [CONTAINER_FORMAT_AU] = &container_parser_au, [CONTAINER_FORMAT_VOC] = &container_parser_voc, }; - int fd; const struct container_parser *parser; unsigned int size; int i; int err;
assert(cntr); - assert(path); - assert(path[0] != '\0'); + assert(fd >= 0);
// Detect forgotten to destruct. assert(cntr->fd == 0); @@ -167,14 +165,6 @@ int container_parser_init(struct container_context *cntr,
memset(cntr, 0, sizeof(*cntr));
- // Open a target descriptor. - if (!strcmp(path, "-")) { - fd = fileno(stdin); - } else { - fd = open(path, O_RDONLY); - if (fd < 0) - return -errno; - } cntr->fd = fd;
cntr->stdio = (cntr->fd == fileno(stdin)); @@ -231,9 +221,8 @@ int container_parser_init(struct container_context *cntr, return 0; }
-int container_builder_init(struct container_context *cntr, - const char *const path, enum container_format format, - unsigned int verbose) +int container_builder_init(struct container_context *cntr, int fd, + enum container_format format, unsigned int verbose) { const struct container_builder *builders[] = { [CONTAINER_FORMAT_RIFF_WAVE] = &container_builder_riff_wave, @@ -241,13 +230,11 @@ int container_builder_init(struct container_context *cntr, [CONTAINER_FORMAT_VOC] = &container_builder_voc, [CONTAINER_FORMAT_RAW] = &container_builder_raw, }; - int fd; const struct container_builder *builder; int err;
assert(cntr); - assert(path); - assert(path[0] != '\0'); + assert(fd >= 0);
// Detect forgotten to destruct. assert(cntr->fd == 0); @@ -255,16 +242,6 @@ int container_builder_init(struct container_context *cntr,
memset(cntr, 0, sizeof(*cntr));
- // Open a target descriptor. - if (path == NULL || *path == '\0') - return -EINVAL; - if (!strcmp(path, "-")) { - fd = fileno(stdout); - } else { - fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); - if (fd < 0) - return -errno; - } cntr->fd = fd;
cntr->stdio = (cntr->fd == fileno(stdout)); diff --git a/axfer/container.h b/axfer/container.h index cb64816..0840369 100644 --- a/axfer/container.h +++ b/axfer/container.h @@ -61,11 +61,10 @@ struct container_context {
const char *const container_suffix_from_format(enum container_format format); enum container_format container_format_from_path(const char *path); -int container_parser_init(struct container_context *cntr, - const char *const path, unsigned int verbose); -int container_builder_init(struct container_context *cntr, - const char *const path, enum container_format format, - unsigned int verbose); +int container_parser_init(struct container_context *cntr, int fd, + unsigned int verbose); +int container_builder_init(struct container_context *cntr, int fd, + enum container_format format, unsigned int verbose); void container_context_destroy(struct container_context *cntr); int container_context_pre_process(struct container_context *cntr, snd_pcm_format_t *format, diff --git a/axfer/subcmd-transfer.c b/axfer/subcmd-transfer.c index 6962208..52c32d5 100644 --- a/axfer/subcmd-transfer.c +++ b/axfer/subcmd-transfer.c @@ -185,10 +185,19 @@ static int capture_pre_process(struct context *ctx, snd_pcm_access_t *access,
*total_frame_count = 0; for (i = 0; i < ctx->cntr_count; ++i) { + const char *path = ctx->xfer.paths[i]; + int fd; uint64_t frame_count;
- err = container_builder_init(ctx->cntrs + i, - ctx->xfer.paths[i], + if (!strcmp(path, "-")) { + fd = fileno(stdout); + } else { + fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (fd < 0) + return -errno; + } + + err = container_builder_init(ctx->cntrs + i, fd, ctx->xfer.cntr_format, ctx->xfer.verbose > 1); if (err < 0) @@ -226,13 +235,22 @@ static int playback_pre_process(struct context *ctx, snd_pcm_access_t *access, return err;
for (i = 0; i < ctx->cntr_count; ++i) { + const char *path = ctx->xfer.paths[i]; + int fd; snd_pcm_format_t format; unsigned int channels; unsigned int rate; uint64_t frame_count;
- err = container_parser_init(ctx->cntrs + i, - ctx->xfer.paths[i], + if (!strcmp(path, "-")) { + fd = fileno(stdin); + } else { + fd = open(path, O_RDONLY); + if (fd < 0) + return -errno; + } + + err = container_parser_init(ctx->cntrs + i, fd, ctx->xfer.verbose > 1); if (err < 0) return err; diff --git a/axfer/test/container-test.c b/axfer/test/container-test.c index 9b30ae3..fbef3a4 100644 --- a/axfer/test/container-test.c +++ b/axfer/test/container-test.c @@ -33,6 +33,7 @@ static void test_builder(struct container_context *cntr, void *frame_buffer, unsigned int frame_count, bool verbose) { + int fd; snd_pcm_format_t sample; unsigned int channels; unsigned int rate; @@ -41,7 +42,10 @@ static void test_builder(struct container_context *cntr, uint64_t total_frame_count; int err;
- err = container_builder_init(cntr, name, format, verbose); + fd = open(name, O_RDWR | O_CREAT | O_TRUNC, 0644); + assert(fd >= 0); + + err = container_builder_init(cntr, fd, format, verbose); assert(err == 0);
sample = sample_format; @@ -79,6 +83,7 @@ static void test_parser(struct container_context *cntr, void *frame_buffer, unsigned int frame_count, bool verbose) { + int fd; snd_pcm_format_t sample; unsigned int channels; unsigned int rate; @@ -86,7 +91,10 @@ static void test_parser(struct container_context *cntr, unsigned int handled_frame_count; int err;
- err = container_parser_init(cntr, name, verbose); + fd = open(name, O_RDONLY); + assert(fd >= 0); + + err = container_parser_init(cntr, fd, verbose); assert(err == 0);
sample = sample_format; diff --git a/axfer/test/mapper-test.c b/axfer/test/mapper-test.c index f0376c7..6252900 100644 --- a/axfer/test/mapper-test.c +++ b/axfer/test/mapper-test.c @@ -66,7 +66,6 @@ static int test_demux(struct mapper_trial *trial, snd_pcm_access_t access, unsigned int cntr_count) { struct container_context *cntrs = trial->cntrs; - char **paths = trial->paths; enum container_format cntr_format = trial->cntr_format; unsigned int bytes_per_sample; uint64_t total_frame_count; @@ -74,12 +73,17 @@ static int test_demux(struct mapper_trial *trial, snd_pcm_access_t access, int err = 0;
for (i = 0; i < cntr_count; ++i) { + const char *path = trial->paths[i]; + int fd; snd_pcm_format_t format; unsigned int channels; unsigned int rate;
- err = container_builder_init(cntrs + i, paths[i], cntr_format, - 0); + fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (fd < 0) + return -errno; + + err = container_builder_init(cntrs + i, fd, cntr_format, 0); if (err < 0) goto end;
@@ -159,18 +163,23 @@ static int test_mux(struct mapper_trial *trial, snd_pcm_access_t access, unsigned int cntr_count) { struct container_context *cntrs = trial->cntrs; - char **paths = trial->paths; unsigned int bytes_per_sample; uint64_t total_frame_count; int i; int err = 0;
for (i = 0; i < cntr_count; ++i) { + const char *path = trial->paths[i]; + int fd; snd_pcm_format_t format; unsigned int channels; unsigned int rate;
- err = container_parser_init(cntrs + i, paths[i], 0); + fd = open(path, O_RDONLY); + if (fd < 0) + return -errno; + + err = container_parser_init(cntrs + i, fd, 0); if (err < 0) goto end;
This commit closes file descriptor outside of container module so that maintenance of lifetime for the descriptor is delegated to container user.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/container.c | 1 - axfer/container.h | 1 - axfer/subcmd-transfer.c | 18 +++++++++++++-- axfer/test/container-test.c | 2 ++ axfer/test/mapper-test.c | 44 +++++++++++++++++++++++++++---------- 5 files changed, 50 insertions(+), 16 deletions(-)
diff --git a/axfer/container.c b/axfer/container.c index 255f12c..8c88d5c 100644 --- a/axfer/container.c +++ b/axfer/container.c @@ -451,7 +451,6 @@ void container_context_destroy(struct container_context *cntr) { assert(cntr);
- close(cntr->fd); if (cntr->private_data) free(cntr->private_data);
diff --git a/axfer/container.h b/axfer/container.h index 0840369..71017a6 100644 --- a/axfer/container.h +++ b/axfer/container.h @@ -11,7 +11,6 @@
#define _LARGEFILE64_SOURCE #include <sys/types.h> -#include <unistd.h>
#include <stdbool.h> #include <stdint.h> diff --git a/axfer/subcmd-transfer.c b/axfer/subcmd-transfer.c index 52c32d5..27d2cc5 100644 --- a/axfer/subcmd-transfer.c +++ b/axfer/subcmd-transfer.c @@ -19,6 +19,8 @@ struct context { struct container_context *cntrs; unsigned int cntr_count;
+ int *cntr_fds; + // NOTE: To handling Unix signal. bool interrupted; int signal; @@ -153,6 +155,10 @@ static int allocate_containers(struct context *ctx, unsigned int count) return -ENOMEM; ctx->cntr_count = count;
+ ctx->cntr_fds = calloc(count, sizeof(*ctx->cntrs)); + if (ctx->cntr_fds == NULL) + return -ENOMEM; + return 0; }
@@ -196,8 +202,9 @@ static int capture_pre_process(struct context *ctx, snd_pcm_access_t *access, if (fd < 0) return -errno; } + ctx->cntr_fds[i] = fd;
- err = container_builder_init(ctx->cntrs + i, fd, + err = container_builder_init(ctx->cntrs + i, ctx->cntr_fds[i], ctx->xfer.cntr_format, ctx->xfer.verbose > 1); if (err < 0) @@ -249,8 +256,9 @@ static int playback_pre_process(struct context *ctx, snd_pcm_access_t *access, if (fd < 0) return -errno; } + ctx->cntr_fds[i] = fd;
- err = container_parser_init(ctx->cntrs + i, fd, + err = container_parser_init(ctx->cntrs + i, ctx->cntr_fds[i], ctx->xfer.verbose > 1); if (err < 0) return err; @@ -447,6 +455,12 @@ static void context_post_process(struct context *ctx, free(ctx->cntrs); }
+ if (ctx->cntr_fds) { + for (i = 0; i < ctx->cntr_count; ++i) + close(ctx->cntr_fds[i]); + free(ctx->cntr_fds); + } + mapper_context_post_process(&ctx->mapper); mapper_context_destroy(&ctx->mapper); } diff --git a/axfer/test/container-test.c b/axfer/test/container-test.c index fbef3a4..d89852a 100644 --- a/axfer/test/container-test.c +++ b/axfer/test/container-test.c @@ -73,6 +73,7 @@ static void test_builder(struct container_context *cntr, assert(total_frame_count == frame_count);
container_context_destroy(cntr); + close(fd); }
static void test_parser(struct container_context *cntr, @@ -121,6 +122,7 @@ static void test_parser(struct container_context *cntr, assert(total_frame_count == handled_frame_count);
container_context_destroy(cntr); + close(fd); }
static int callback(struct test_generator *gen, snd_pcm_access_t access, diff --git a/axfer/test/mapper-test.c b/axfer/test/mapper-test.c index 6252900..78a063a 100644 --- a/axfer/test/mapper-test.c +++ b/axfer/test/mapper-test.c @@ -67,23 +67,29 @@ static int test_demux(struct mapper_trial *trial, snd_pcm_access_t access, { struct container_context *cntrs = trial->cntrs; enum container_format cntr_format = trial->cntr_format; + int *cntr_fds; unsigned int bytes_per_sample; uint64_t total_frame_count; int i; int err = 0;
+ cntr_fds = calloc(cntr_count, sizeof(*cntr_fds)); + if (cntr_fds == NULL) + return -ENOMEM; + for (i = 0; i < cntr_count; ++i) { const char *path = trial->paths[i]; - int fd; snd_pcm_format_t format; unsigned int channels; unsigned int rate;
- fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); - if (fd < 0) - return -errno; + cntr_fds[i] = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (cntr_fds[i] < 0) { + err = -errno; + goto end; + }
- err = container_builder_init(cntrs + i, fd, cntr_format, 0); + err = container_builder_init(cntrs + i, cntr_fds[i], cntr_format, 0); if (err < 0) goto end;
@@ -118,8 +124,12 @@ static int test_demux(struct mapper_trial *trial, snd_pcm_access_t access, assert(total_frame_count == frame_count); } end: - for (i = 0; i < cntr_count; ++i) + for (i = 0; i < cntr_count; ++i) { container_context_destroy(cntrs + i); + close(cntr_fds[i]); + } + + free(cntr_fds);
return err; } @@ -163,23 +173,29 @@ static int test_mux(struct mapper_trial *trial, snd_pcm_access_t access, unsigned int cntr_count) { struct container_context *cntrs = trial->cntrs; + int *cntr_fds; unsigned int bytes_per_sample; uint64_t total_frame_count; int i; int err = 0;
+ cntr_fds = calloc(cntr_count, sizeof(*cntr_fds)); + if (cntr_fds == NULL) + return -ENOMEM; + for (i = 0; i < cntr_count; ++i) { const char *path = trial->paths[i]; - int fd; snd_pcm_format_t format; unsigned int channels; unsigned int rate;
- fd = open(path, O_RDONLY); - if (fd < 0) - return -errno; + cntr_fds[i] = open(path, O_RDONLY); + if (cntr_fds[i] < 0) { + err = -errno; + goto end; + }
- err = container_parser_init(cntrs + i, fd, 0); + err = container_parser_init(cntrs + i, cntr_fds[i], 0); if (err < 0) goto end;
@@ -214,8 +230,12 @@ static int test_mux(struct mapper_trial *trial, snd_pcm_access_t access, assert(total_frame_count == frame_count); } end: - for (i = 0; i < cntr_count; ++i) + for (i = 0; i < cntr_count; ++i) { container_context_destroy(cntrs + i); + close(cntr_fds[i]); + } + + free(cntr_fds);
return err; }
This is a preparation to use memfd_create(2) system call for test programs of axfer. The system call was introduced at Linux kernel v3.17 and relatively new.
For safe, this commit adds detection of memfd_create() in autotools side so that application can handle the case not to detect.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- configure.ac | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/configure.ac b/configure.ac index 7005ccc..ff3e1f6 100644 --- a/configure.ac +++ b/configure.ac @@ -62,6 +62,11 @@ AC_CHECK_LIB([ffado], [ffado_streaming_init], [have_ffado="yes"], [have_ffado="n AS_IF([test x"$have_ffado" = xyes], [AC_DEFINE([WITH_FFADO], [1], [Define if FFADO library is available])])
+# Test programs for axfer use shm by memfd_create(2). If not supported, open(2) is used alternatively. +AC_CHECK_FUNC([memfd_create], [have_memfd_create="yes"], [have_memfd_create="no"]) +AS_IF([test x$have_memfd_create = xyes], + [AC_DEFINE([HAVE_MEMFD_CREATE], [1], [Define if Linux kernel supports memfd_create system call])]) + AM_CONDITIONAL(HAVE_PCM, test "$have_pcm" = "yes") AM_CONDITIONAL(HAVE_MIXER, test "$have_mixer" = "yes") AM_CONDITIONAL(HAVE_RAWMIDI, test "$have_rawmidi" = "yes")
In container test program, two file descriptors open to the same file for builder and parser contexts, however the same file descriptor is available for the case.
This commit arranges to use the same file descriptor for the contexts.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/container-test.c | 40 +++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 17 deletions(-)
diff --git a/axfer/test/container-test.c b/axfer/test/container-test.c index d89852a..4d825b6 100644 --- a/axfer/test/container-test.c +++ b/axfer/test/container-test.c @@ -24,8 +24,8 @@ struct container_trial { bool verbose; };
-static void test_builder(struct container_context *cntr, - enum container_format format, const char *const name, +static void test_builder(struct container_context *cntr, int fd, + enum container_format format, snd_pcm_access_t access, snd_pcm_format_t sample_format, unsigned int samples_per_frame, @@ -33,7 +33,6 @@ static void test_builder(struct container_context *cntr, void *frame_buffer, unsigned int frame_count, bool verbose) { - int fd; snd_pcm_format_t sample; unsigned int channels; unsigned int rate; @@ -42,9 +41,6 @@ static void test_builder(struct container_context *cntr, uint64_t total_frame_count; int err;
- fd = open(name, O_RDWR | O_CREAT | O_TRUNC, 0644); - assert(fd >= 0); - err = container_builder_init(cntr, fd, format, verbose); assert(err == 0);
@@ -73,18 +69,16 @@ static void test_builder(struct container_context *cntr, assert(total_frame_count == frame_count);
container_context_destroy(cntr); - close(fd); }
-static void test_parser(struct container_context *cntr, - enum container_format format, const char *const name, +static void test_parser(struct container_context *cntr, int fd, + enum container_format format, snd_pcm_access_t access, snd_pcm_format_t sample_format, unsigned int samples_per_frame, unsigned int frames_per_second, void *frame_buffer, unsigned int frame_count, bool verbose) { - int fd; snd_pcm_format_t sample; unsigned int channels; unsigned int rate; @@ -92,9 +86,6 @@ static void test_parser(struct container_context *cntr, unsigned int handled_frame_count; int err;
- fd = open(name, O_RDONLY); - assert(fd >= 0); - err = container_parser_init(cntr, fd, verbose); assert(err == 0);
@@ -122,7 +113,6 @@ static void test_parser(struct container_context *cntr, assert(total_frame_count == handled_frame_count);
container_context_destroy(cntr); - close(fd); }
static int callback(struct test_generator *gen, snd_pcm_access_t access, @@ -156,26 +146,42 @@ static int callback(struct test_generator *gen, snd_pcm_access_t access, unlink(name);
for (i = 0; i < ARRAY_SIZE(entries); ++i) { + int fd; + off64_t pos; + frames_per_second = entries[i];
- test_builder(&trial->cntr, trial->format, name, access, + fd = open(name, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (fd < 0) { + err = -errno; + break; + } + + test_builder(&trial->cntr, fd, trial->format, access, sample_format, samples_per_frame, frames_per_second, frame_buffer, frame_count, trial->verbose);
- test_parser(&trial->cntr, trial->format, name, access, + pos = lseek64(fd, 0, SEEK_SET); + if (pos < 0) { + err = -errno; + break; + } + + test_parser(&trial->cntr, fd, trial->format, access, sample_format, samples_per_frame, frames_per_second, buf, frame_count, trial->verbose);
err = memcmp(buf, frame_buffer, size); assert(err == 0);
+ close(fd); unlink(name); }
free(buf);
- return 0; + return err; }
int main(int argc, const char *argv[])
The container test program writes audio data frame to file, and read them from the file, then validate them. For the operations, usage of any in-memory file is good to shorten time of overall operations.
This commit uses shm via memfd_create(). As a result, overall time to run is shorten one half of before, depending on machine environment.
I note that we can achieve the same result by using O_TMPFILE flag in open(2) system call, however the implementation of O_TMPFILE is to add i-node without name on underling file system, thus it has overhead depending on implementation in each file system. On the other hand, memfd_create() is directly relevant to shm and expected to be less overhead.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/container-test.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/axfer/test/container-test.c b/axfer/test/container-test.c index 4d825b6..4e2dcc2 100644 --- a/axfer/test/container-test.c +++ b/axfer/test/container-test.c @@ -6,11 +6,20 @@ // // Licensed under the terms of the GNU General Public License, version 2.
+#include <aconfig.h> +#ifdef HAVE_MEMFD_CREATE +#define _GNU_SOURCE +#endif + #include "../container.h" #include "../misc.h"
#include "generator.h"
+#ifdef HAVE_MEMFD_CREATE +#include <sys/mman.h> +#endif + #include <stdlib.h> #include <unistd.h> #include <stdbool.h> @@ -142,16 +151,17 @@ static int callback(struct test_generator *gen, snd_pcm_access_t access, if (buf == NULL) return -ENOMEM;
- // Remove a result of a previous trial. - unlink(name); - for (i = 0; i < ARRAY_SIZE(entries); ++i) { int fd; off64_t pos;
frames_per_second = entries[i];
+#ifdef HAVE_MEMFD_CREATE + fd = memfd_create(name, 0); +#else fd = open(name, O_RDWR | O_CREAT | O_TRUNC, 0644); +#endif if (fd < 0) { err = -errno; break; @@ -176,7 +186,6 @@ static int callback(struct test_generator *gen, snd_pcm_access_t access, assert(err == 0);
close(fd); - unlink(name); }
free(buf);
In mapper test program, two set of file descriptors open to the same files for container builder and parser contexts, however the same file descriptor is available for the case.
This commit arranges to use the same file descriptor for the contexts.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/mapper-test.c | 73 +++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 39 deletions(-)
diff --git a/axfer/test/mapper-test.c b/axfer/test/mapper-test.c index 78a063a..0bed4bb 100644 --- a/axfer/test/mapper-test.c +++ b/axfer/test/mapper-test.c @@ -63,32 +63,20 @@ static int test_demux(struct mapper_trial *trial, snd_pcm_access_t access, unsigned int frames_per_second, unsigned int frames_per_buffer, void *frame_buffer, unsigned int frame_count, - unsigned int cntr_count) + int *cntr_fds, unsigned int cntr_count) { struct container_context *cntrs = trial->cntrs; enum container_format cntr_format = trial->cntr_format; - int *cntr_fds; unsigned int bytes_per_sample; uint64_t total_frame_count; int i; int err = 0;
- cntr_fds = calloc(cntr_count, sizeof(*cntr_fds)); - if (cntr_fds == NULL) - return -ENOMEM; - for (i = 0; i < cntr_count; ++i) { - const char *path = trial->paths[i]; snd_pcm_format_t format; unsigned int channels; unsigned int rate;
- cntr_fds[i] = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); - if (cntr_fds[i] < 0) { - err = -errno; - goto end; - } - err = container_builder_init(cntrs + i, cntr_fds[i], cntr_format, 0); if (err < 0) goto end; @@ -124,12 +112,8 @@ static int test_demux(struct mapper_trial *trial, snd_pcm_access_t access, assert(total_frame_count == frame_count); } end: - for (i = 0; i < cntr_count; ++i) { + for (i = 0; i < cntr_count; ++i) container_context_destroy(cntrs + i); - close(cntr_fds[i]); - } - - free(cntr_fds);
return err; } @@ -170,31 +154,19 @@ static int test_mux(struct mapper_trial *trial, snd_pcm_access_t access, unsigned int frames_per_second, unsigned int frames_per_buffer, void *frame_buffer, unsigned int frame_count, - unsigned int cntr_count) + int *cntr_fds, unsigned int cntr_count) { struct container_context *cntrs = trial->cntrs; - int *cntr_fds; unsigned int bytes_per_sample; uint64_t total_frame_count; int i; int err = 0;
- cntr_fds = calloc(cntr_count, sizeof(*cntr_fds)); - if (cntr_fds == NULL) - return -ENOMEM; - for (i = 0; i < cntr_count; ++i) { - const char *path = trial->paths[i]; snd_pcm_format_t format; unsigned int channels; unsigned int rate;
- cntr_fds[i] = open(path, O_RDONLY); - if (cntr_fds[i] < 0) { - err = -errno; - goto end; - } - err = container_parser_init(cntrs + i, cntr_fds[i], 0); if (err < 0) goto end; @@ -230,12 +202,8 @@ static int test_mux(struct mapper_trial *trial, snd_pcm_access_t access, assert(total_frame_count == frame_count); } end: - for (i = 0; i < cntr_count; ++i) { + for (i = 0; i < cntr_count; ++i) container_context_destroy(cntrs + i); - close(cntr_fds[i]); - } - - free(cntr_fds);
return err; } @@ -247,6 +215,7 @@ static int test_mapper(struct mapper_trial *trial, snd_pcm_access_t access, void *check_buffer, unsigned int frame_count, unsigned int cntr_count) { + int *cntr_fds; unsigned int frames_per_buffer; int i; int err; @@ -254,18 +223,44 @@ static int test_mapper(struct mapper_trial *trial, snd_pcm_access_t access, // Use a buffer aligned by typical size of page frame. frames_per_buffer = ((frame_count + 4096) / 4096) * 4096;
+ cntr_fds = calloc(cntr_count, sizeof(*cntr_fds)); + if (cntr_fds == NULL) + return -ENOMEM; + + for (i = 0; i < cntr_count; ++i) { + const char *path = trial->paths[i]; + + cntr_fds[i] = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + if (cntr_fds[i] < 0) { + err = -errno; + goto end; + } + } + err = test_demux(trial, access, sample_format, samples_per_frame, frames_per_second, frames_per_buffer, frame_buffer, - frame_count, cntr_count); + frame_count, cntr_fds, cntr_count); if (err < 0) goto end;
+ for (i = 0; i < cntr_count; ++i) { + off64_t pos = lseek64(cntr_fds[i], 0, SEEK_SET); + if (pos != 0) { + err = -EIO; + goto end; + } + } + err = test_mux(trial, access, sample_format, samples_per_frame, frames_per_second, frames_per_buffer, check_buffer, - frame_count, cntr_count); + frame_count, cntr_fds, cntr_count); end: - for (i = 0; i < cntr_count; ++i) + for (i = 0; i < cntr_count; ++i) { unlink(trial->paths[i]); + close(cntr_fds[i]); + } + + free(cntr_fds);
return err; }
The mapper test program writes audio data frame to files, and read them from the files, then validate them. For the operations, usage of any in-memory file is good to shorten time of overall operations.
This commit uses shm by memfd_create().
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/mapper-test.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/axfer/test/mapper-test.c b/axfer/test/mapper-test.c index 0bed4bb..477871d 100644 --- a/axfer/test/mapper-test.c +++ b/axfer/test/mapper-test.c @@ -6,11 +6,20 @@ // // Licensed under the terms of the GNU General Public License, version 2.
+#include <aconfig.h> +#ifdef HAVE_MEMFD_CREATE +#define _GNU_SOURCE +#endif + #include "../mapper.h" #include "../misc.h"
#include "generator.h"
+#ifdef HAVE_MEMFD_CREATE +#include <sys/mman.h> +#endif + #include <stdlib.h> #include <unistd.h> #include <stdbool.h> @@ -230,7 +239,11 @@ static int test_mapper(struct mapper_trial *trial, snd_pcm_access_t access, for (i = 0; i < cntr_count; ++i) { const char *path = trial->paths[i];
+#ifdef HAVE_MEMFD_CREATE + cntr_fds[i] = memfd_create(path, 0); +#else cntr_fds[i] = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); +#endif if (cntr_fds[i] < 0) { err = -errno; goto end; @@ -255,10 +268,8 @@ static int test_mapper(struct mapper_trial *trial, snd_pcm_access_t access, frames_per_second, frames_per_buffer, check_buffer, frame_count, cntr_fds, cntr_count); end: - for (i = 0; i < cntr_count; ++i) { - unlink(trial->paths[i]); + for (i = 0; i < cntr_count; ++i) close(cntr_fds[i]); - }
free(cntr_fds);
This commit reduces test case for maximum number of frame count so that overall time is shortened. The count of total iteration is still the same.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/container-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/axfer/test/container-test.c b/axfer/test/container-test.c index 4e2dcc2..788507b 100644 --- a/axfer/test/container-test.c +++ b/axfer/test/container-test.c @@ -300,7 +300,7 @@ int main(int argc, const char *argv[]) for (i = begin; i < end; ++i) { err = generator_context_init(&gen, access_mask, sample_format_masks[i], - 1, 128, 23, 4500, 1024, + 1, 128, 23, 3000, 512, sizeof(struct container_trial)); if (err >= 0) { trial = gen.private_data;
This reverts commit e1551de8dd28c3a63f8d7c146952a8d2649ac9de since tests run for in-memory files now.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/run-test-in-tmpdir.sh | 19 ------------------- 1 file changed, 19 deletions(-) delete mode 100755 axfer/test/run-test-in-tmpdir.sh
diff --git a/axfer/test/run-test-in-tmpdir.sh b/axfer/test/run-test-in-tmpdir.sh deleted file mode 100755 index e66fa73..0000000 --- a/axfer/test/run-test-in-tmpdir.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/sh - -bin="$1" - -test -z ${bin} && exit 90 -test ! -x ${bin} && exit 91 -test -z ${TMPDIR} && exit 92 -test ! -d ${TMPDIR} && exit 93 - -tmp_dir=$(mktemp -d ${TMPDIR}/container-test.XXXXX) -cur_dir=$(pwd) - -echo ${tmp_dir} -cd ${tmp_dir} -${cur_dir}/${bin} -retval=$? -cd ${cur_dir} -rm -rf ${tmp_dir} -exit $retval
Oops. I missed delete this old-versioned patch. Please abandon this when applying the others.
Regards
Takashi Sakamoto
On Thu, Mar 11, 2021 at 02:21:44PM +0900, Takashi Sakamoto wrote:
This reverts commit e1551de8dd28c3a63f8d7c146952a8d2649ac9de since tests run for in-memory files now.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp
axfer/test/run-test-in-tmpdir.sh | 19 ------------------- 1 file changed, 19 deletions(-) delete mode 100755 axfer/test/run-test-in-tmpdir.sh
diff --git a/axfer/test/run-test-in-tmpdir.sh b/axfer/test/run-test-in-tmpdir.sh deleted file mode 100755 index e66fa73..0000000 --- a/axfer/test/run-test-in-tmpdir.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/sh
-bin="$1"
-test -z ${bin} && exit 90 -test ! -x ${bin} && exit 91 -test -z ${TMPDIR} && exit 92 -test ! -d ${TMPDIR} && exit 93
-tmp_dir=$(mktemp -d ${TMPDIR}/container-test.XXXXX) -cur_dir=$(pwd)
-echo ${tmp_dir} -cd ${tmp_dir} -${cur_dir}/${bin} -retval=$? -cd ${cur_dir} -rm -rf ${tmp_dir}
-exit $retval
2.27.0
This commit reduces test case for maximum number of samples per frame so that overall time is shortened. The count of total iteration is also reduced by one quarter.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/container-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/axfer/test/container-test.c b/axfer/test/container-test.c index 788507b..2a3dcff 100644 --- a/axfer/test/container-test.c +++ b/axfer/test/container-test.c @@ -300,7 +300,7 @@ int main(int argc, const char *argv[]) for (i = begin; i < end; ++i) { err = generator_context_init(&gen, access_mask, sample_format_masks[i], - 1, 128, 23, 3000, 512, + 1, 32, 23, 3000, 512, sizeof(struct container_trial)); if (err >= 0) { trial = gen.private_data;
This reverts commit e1551de8dd28c3a63f8d7c146952a8d2649ac9de since tests run for in-memory files now.
Signed-off-by: Takashi Sakamoto o-takashi@sakamocchi.jp --- axfer/test/run-test-in-tmpdir.sh | 19 ------------------- 1 file changed, 19 deletions(-) delete mode 100755 axfer/test/run-test-in-tmpdir.sh
diff --git a/axfer/test/run-test-in-tmpdir.sh b/axfer/test/run-test-in-tmpdir.sh deleted file mode 100755 index e66fa73..0000000 --- a/axfer/test/run-test-in-tmpdir.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/sh - -bin="$1" - -test -z ${bin} && exit 90 -test ! -x ${bin} && exit 91 -test -z ${TMPDIR} && exit 92 -test ! -d ${TMPDIR} && exit 93 - -tmp_dir=$(mktemp -d ${TMPDIR}/container-test.XXXXX) -cur_dir=$(pwd) - -echo ${tmp_dir} -cd ${tmp_dir} -${cur_dir}/${bin} -retval=$? -cd ${cur_dir} -rm -rf ${tmp_dir} -exit $retval
Dne 11. 03. 21 v 6:21 Takashi Sakamoto napsal(a):
Hi,
The axfer in alsa-utils has test programs for internal modules and overall time to execute them takes much. The issue is filing in github repository[1]. It comes from three causes:
- file I/O operations are done in file system for actual storage
- Some cases operate much audio data frames (over 4MB, approx.)
- much test iteration count (so simple...)
This patchset uses shm by memfd_create(2) for the cause 1. In addition, the number of maximum audio data frame to test is reduced as well as the number of samples per frame.
I got benchmark with berow machine environment.
- AMD Ryzen 5 2400G
- memory total 30823852 KB
- linux-image-5.8.0-44-generic on Ubuntu 20.04 amd64
- ext4 on SATA SSD
As a result: |container| mapper | | test | test | ================== | ======= | ======= | Current | 112 min | 5 min | +shm | 58 min | 50 sec | +maximum reduction | 38 min | - | +iter reduction | 4 min | - |
In my opinion, the issue comes from package build server in each distribution. 5 min for test execution is not so worse time.
Finally, test programs run on shm and commit c3f2344b7209 is reverted[1] since it's useless now.
Applied all patches except the double revert. Thank you.
Jaroslav
participants (2)
-
Jaroslav Kysela -
Takashi Sakamoto