From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 007D246EA8 for ; Fri, 18 Aug 2023 16:17:17 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8687068C5DD; Fri, 18 Aug 2023 19:17:14 +0300 (EEST) Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4255C68BD32 for ; Fri, 18 Aug 2023 19:17:08 +0300 (EEST) Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-1c4de3b9072so609133fac.3 for ; Fri, 18 Aug 2023 09:17:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692375426; x=1692980226; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id:from :to:cc:subject:date:message-id:reply-to; bh=3D9CwQr568rvml4opRCfYnJoRriOhlhTh8xiG8Xuvsk=; b=rma4VatRskrFVx/92NZP1BzH0GXK9PFPYWa0C8hBSLdMDRZtopipzY853Bt+NIhPbD jyJ2t0VENjfM0e5teUBAAfzp/eMQLp9mpLzBtUnNhCf9lXjkGv1mZLbj4O9IHurb6aXg afv5dbUqg+qBtV8vdwu80h/9DKG6bS8AefXltpnTx4DV8fFd/k0M+6KhDAADP1lxHedC 8pZTn772SO3o0SISg4F+Vv1wNjE9ndVapVVBfsPmSZw8XUvz9HH62S9zn40H1SNL2p/B pDnkBpWGZrwN++1o9XipHMafs/pntK7DvgsmXHwfnepNVNYe0QJx5S7ohsFyHKQDvKpj xsAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692375426; x=1692980226; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3D9CwQr568rvml4opRCfYnJoRriOhlhTh8xiG8Xuvsk=; b=YfLXfgP1IY3rGSRvNN62NfcT40JGwIFj5t3AwKxQlh7XvE+WvdcZuIsoqECQ5Gk6nF gCMhFmCfKc3jEFeotmynMv7RY0Nie3wa8zGYRrZeaRA1Hnj0quHAm3VceenpN/VeqJeM wy1DmeXaJDvQ7hbwOF+ai8VlT7X4R7SA1x1ewwU2Oc20ozy+dchVv/4PoFxJv+ADaMRl IA2AXbtAM/ZsO28u7EfIPb8851Wz5p2N8dkRYzSjnSGHh8xWhBBS7oWMsWlql3a0tqxc rIzNW6RUufKYlcELAfGC/gMumr53ua4ixUF140Cznz6hHlbr2zCGgjNtj5nLh1G8bKp5 havw== X-Gm-Message-State: AOJu0YyafOarkGWncU62I+gvRO9/EfWYIC2fzK8U+P/nvPygqUtyTmMT Qm0IIxTnLQ2fCX0nZcM4Q2UF7nurhS8= X-Google-Smtp-Source: AGHT+IEDmdVA6vUXvvT4Ifmh7GXal7EAL1MWDuv170NYybtVjy8EqShFCHLnTeZg864GkE2xkrTyaA== X-Received: by 2002:a05:6870:2184:b0:1c4:e139:331 with SMTP id l4-20020a056870218400b001c4e1390331mr2898119oae.55.1692375426412; Fri, 18 Aug 2023 09:17:06 -0700 (PDT) Received: from [192.168.0.16] (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id z14-20020a4a984e000000b0056360466d3esm948494ooi.48.2023.08.18.09.17.05 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 18 Aug 2023 09:17:06 -0700 (PDT) Message-ID: Date: Fri, 18 Aug 2023 13:17:11 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 To: ffmpeg-devel@ffmpeg.org References: Content-Language: en-US From: James Almer In-Reply-To: Subject: Re: [FFmpeg-devel] [PATCH v4] vvcdec: add thread executor X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: On 8/15/2023 11:50 AM, Nuo Mi wrote: > The executor design pattern was inroduced by java > > it also adapted by python > > Compared to handcrafted thread pool management, it greatly simplifies the thread code. > --- > libavutil/Makefile | 2 + > libavutil/executor.c | 201 +++++++++++++++++++++++++++++++++++++++++++ > libavutil/executor.h | 67 +++++++++++++++ > 3 files changed, 270 insertions(+) > create mode 100644 libavutil/executor.c > create mode 100644 libavutil/executor.h > > diff --git a/libavutil/Makefile b/libavutil/Makefile > index 7828c94dc5..4711f8cde8 100644 > --- a/libavutil/Makefile > +++ b/libavutil/Makefile > @@ -31,6 +31,7 @@ HEADERS = adler32.h \ > encryption_info.h \ > error.h \ > eval.h \ > + executor.h \ > fifo.h \ > file.h \ > frame.h \ > @@ -127,6 +128,7 @@ OBJS = adler32.o \ > encryption_info.o \ > error.o \ > eval.o \ > + executor.o \ > fifo.o \ > file.o \ > file_open.o \ > diff --git a/libavutil/executor.c b/libavutil/executor.c > new file mode 100644 > index 0000000000..6b47ba0bf2 > --- /dev/null > +++ b/libavutil/executor.c > @@ -0,0 +1,201 @@ > +/* > + * Copyright (C) 2023 Nuo Mi > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > +#include "internal.h" > +#include "mem.h" > +#include "thread.h" > + > +#include "executor.h" > + > +#if !HAVE_THREADS > +#define pthread_create(t, a, s, ar) 0 > +#define pthread_join(t, r) do {} while(0) > + > +#define pthread_cond_init(c, a) 0 > +#define pthread_cond_broadcast(c) do {} while(0) > +#define pthread_cond_signal(c) do {} while(0) > +#define pthread_cond_wait(c, m) do {} while(0) > +#define pthread_cond_destroy(c) do {} while(0) > + > +#define pthread_mutex_init(m, a) 0 > +#define pthread_mutex_lock(l) do {} while(0) > +#define pthread_mutex_unlock(l) do {} while(0) > +#define pthread_mutex_destroy(l) do {} while(0) You have ff_mutex_* and AVMutex for this. I sent a patch adding ff_cond_* and AVCond, as well as ff_thread_create/join and AVThread to thread.h to simplify things. > +#endif > + > +typedef struct ThreadInfo { > + AVExecutor *e; > + pthread_t thread; You will need to use AVThread, else this will not be defined when !HAVE_THREADS. > +} ThreadInfo; > + > +struct AVExecutor { > + AVTaskCallbacks cb; > + int thread_count; > + > + ThreadInfo *threads; > + uint8_t *local_contexts; > + > + pthread_mutex_t lock; > + pthread_cond_t cond; Same here. AVMutex and AVCond. > + int die; > + > + AVTask *tasks; > +}; > + > +static AVTask* remove_task(AVTask **prev, AVTask *t) > +{ > + *prev = t->next; > + t->next = NULL; > + return t; > +} > + > +static void add_task(AVTask **prev, AVTask *t) > +{ > + t->next = *prev; > + *prev = t; > +} > + > +static int run_one_task(AVExecutor *e, void *lc) > +{ > + AVTaskCallbacks *cb = &e->cb; > + AVTask **prev; > + > + for (prev = &e->tasks; *prev && !cb->ready(*prev, cb->user_data); prev = &(*prev)->next) > + /* nothing */; > + if (*prev) { > + AVTask *t = remove_task(prev, *prev); > + pthread_mutex_unlock(&e->lock); > + cb->run(t, lc, cb->user_data); > + pthread_mutex_lock(&e->lock); > + return 1; > + } > + return 0; > +} > + > +#if HAVE_THREADS > +static void *executor_worker_task(void *data) > +{ > + ThreadInfo *ti = (ThreadInfo*)data; > + AVExecutor *e = ti->e; > + void *lc = e->local_contexts + (ti - e->threads) * e->cb.local_context_size; > + > + pthread_mutex_lock(&e->lock); > + while (1) { > + if (e->die) break; > + > + if (!run_one_task(e, lc)) { > + //no task in one loop > + pthread_cond_wait(&e->cond, &e->lock); > + } > + } > + pthread_mutex_unlock(&e->lock); > + return NULL; > +} > +#endif > + > +static void executor_free(AVExecutor *e, const int has_lock, const int has_cond) > +{ > + if (e->thread_count) { > + //signal die > + pthread_mutex_lock(&e->lock); > + e->die = 1; > + pthread_cond_broadcast(&e->cond); > + pthread_mutex_unlock(&e->lock); > + > + for (int i = 0; i < e->thread_count; i++) > + pthread_join(e->threads[i].thread, NULL); > + } > + if (has_cond) > + pthread_cond_destroy(&e->cond); > + if (has_lock) > + pthread_mutex_destroy(&e->lock); > + > + av_free(e->threads); > + av_free(e->local_contexts); > + > + av_free(e); > +} > + > +AVExecutor* av_executor_alloc(const AVTaskCallbacks *cb, int thread_count) > +{ > + AVExecutor *e; > + int has_lock = 0, has_cond = 0; > + if (!cb || !cb->user_data || !cb->ready || !cb->run || !cb->priority_higher) > + return NULL; > + > + e = av_calloc(1, sizeof(*e)); av_mallocz() > + if (!e) > + return NULL; > + e->cb = *cb; > + > + e->local_contexts = av_calloc(thread_count, e->cb.local_context_size); > + if (!e->local_contexts) > + goto free_executor; > + > + e->threads = av_calloc(thread_count, sizeof(*e->threads)); > + if (!e->threads) > + goto free_executor; > + > + has_lock = !pthread_mutex_init(&e->lock, NULL); > + has_cond = !pthread_cond_init(&e->cond, NULL); > + > + if (!has_lock || !has_cond) > + goto free_executor; > + > + for (/* nothing */; e->thread_count < thread_count; e->thread_count++) { > + ThreadInfo *ti = e->threads + e->thread_count; > + ti->e = e; > + if (pthread_create(&ti->thread, NULL, executor_worker_task, ti)) > + goto free_executor; > + } > + return e; > + > +free_executor: > + executor_free(e, has_lock, has_cond); > + return NULL; > +} > + > +void av_executor_free(AVExecutor **executor) > +{ > + if (!executor || !*executor) > + return; > + executor_free(*executor, 1, 1); > + *executor = NULL; > +} > + > +void av_executor_execute(AVExecutor *e, AVTask *t) > +{ > + AVTaskCallbacks *cb = &e->cb; > + AVTask **prev; > + > + pthread_mutex_lock(&e->lock); > + if (t) { > + for (prev = &e->tasks; *prev && cb->priority_higher(*prev, t); prev = &(*prev)->next) > + /* nothing */; > + add_task(prev, t); > + } > + pthread_cond_signal(&e->cond); > + pthread_mutex_unlock(&e->lock); > + > +#if !HAVE_THREADS > + // We are running in a single-threaded environment, so we must handle all tasks ourselves > + while (run_one_task(e, e->local_contexts)) > + /* nothing */; > +#endif > +} > diff --git a/libavutil/executor.h b/libavutil/executor.h > new file mode 100644 > index 0000000000..c3fe44bd0e > --- /dev/null > +++ b/libavutil/executor.h > @@ -0,0 +1,67 @@ > +/* > + * Copyright (C) 2023 Nuo Mi > + * > + * This file is part of FFmpeg. > + * > + * FFmpeg is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public > + * License as published by the Free Software Foundation; either > + * version 2.1 of the License, or (at your option) any later version. > + * > + * FFmpeg is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + * Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public > + * License along with FFmpeg; if not, write to the Free Software > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef AVUTIL_EXECUTOR_H > +#define AVUTIL_EXECUTOR_H > + > +typedef struct AVExecutor AVExecutor; > +typedef struct AVTask AVTask; > + > +struct AVTask { > + AVTask *next; > +}; > + > +typedef struct AVTaskCallbacks { > + void *user_data; > + > + int local_context_size; > + > + // return 1 if a's priority > b's priority > + int (*priority_higher)(const AVTask *a, const AVTask *b); > + > + // task is ready for run > + int (*ready)(const AVTask *t, void *user_data); > + > + // run the task > + int (*run)(AVTask *t, void *local_context, void *user_data); > +} AVTaskCallbacks; > + > +/** > + * Alloc executor > + * @param callbacks callback strucutre for executor > + * @param thread_count worker thread number > + * @return return the executor > + */ > +AVExecutor* av_executor_alloc(const AVTaskCallbacks *callbacks, int thread_count); > + > +/** > + * Free executor > + * @param e pointer to executor > + */ > +void av_executor_free(AVExecutor **e); > + > +/** > + * Add task to executor > + * @param e pointer to executor > + * @param t pointer to task. If NULL, it will wakeup one work thread > + */ > +void av_executor_execute(AVExecutor *e, AVTask *t); > + > +#endif //AVUTIL_EXECUTOR_H _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".