From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <ffmpeg-devel-bounces@ffmpeg.org> Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 854CF4DA7D for <ffmpegdev@gitmailbox.com>; Tue, 22 Apr 2025 07:43:21 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4CE8968A2D8; Tue, 22 Apr 2025 10:43:17 +0300 (EEST) Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6B2DB687DCB for <ffmpeg-devel@ffmpeg.org>; Tue, 22 Apr 2025 10:43:11 +0300 (EEST) Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-736ad42dfd6so4056847b3a.3 for <ffmpeg-devel@ffmpeg.org>; Tue, 22 Apr 2025 00:43:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=emma-gg.20230601.gappssmtp.com; s=20230601; t=1745307789; x=1745912589; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=XQJ5nS0QWscihwWIbXpcnT3hNUdxhoOMFv9QgRPWxJ8=; b=Uc+JKzD2XcYsZWtQjU7vVUKMNGiM0TZ842AAsQW9RSzVIBALb/w6wG0TUGmc1lFT4H djBh9C/auoPJTI/vwVXN8nEa3EBvbEGDi0oSZzoWfeDHpeD4TAthFmQ1GBh4a7WxzZxC qcn0KipNox+gdGKChACJI/GjwsZhodjmMqNShh3y9R7RgQj53kk6Su/otBqWUO5V4jFC /9iMG/9kOTZ78Z+iLtoRV8kCntzgRkoAqyQSeyBZHYCXQcGKUutMarINMbT9vHik0z/6 IHaGb23m9O/JnLP//7Kw6/sFpeRbqelXQdKJSN0iVlwih833XUuB/MGw9iSb/QBCmyIY BsPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745307789; x=1745912589; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XQJ5nS0QWscihwWIbXpcnT3hNUdxhoOMFv9QgRPWxJ8=; b=eEf07kLPY1u74lq4V+fCz62/uZ4RIY34kczreZpnpvXuozX9UncKYLxJTHOp2ZthxP FgY+P8PHmhEQLqLQQnoCUBm8U0f1AqBoEBppCH4BFbV0Qff1fSftpDLaxaUg+M92AB/d qjjFi0z4xA+wuA9/TuV7XnfFrCmoVZjcUGVddyq17B1YtnckOtseAqr2ZAGuTu8jtBcv X2lyhVfJ2bCIjHfdX+x6YIjofn6WhyFH0fjAmVUd4+YHbfdoW9zkJuBNS40O5551BIxZ RuOHKwOiNmyslowyy+zGIeYkxlQ5KUhMh4SXP6iEavXobx3E40qCabi6e9Wi8dSjv4Hf 6dGA== X-Gm-Message-State: AOJu0YxITa2ZTq7jSlWhqjfWeUtuSeHQZAVm84DUAA2YPyhd/om6gbZD J3S+X1GwFeCDllzuq/L+2IMtbZXQJhFNgJXmZIF/wZPXBitiVYogu+DoLTE3F3fU5N1WjsM2oUL 9 X-Gm-Gg: ASbGncvwC1hUOHB0Qk/wyb6IusxRGychd0NgJU2wsIccgiPF0THGptT/RAzZFf407FU PBpLVeij1KSjcCloEsiuQFNK2iQHEFYh6S6WcfuZyukeYhsN9Fjk+VCwcexMwG/NmLWgyQbswhj Tpb+tIlF1A2n+Okf7IvX3YtcktjGQfpTrL7sJL2A7ymbaKUSamrQQYiNbjSFNBAxSgJqoHDmTPi cbs5Kmf61c5h0zSTZVeqxB5TbEGHRN56ywke42UhyJziZdskmGI/EAL28h1caUVhpQGNvV8TfDs UeEZYOsUQGIJgtBQudxHaj8mpAXSCI2NKCW40DZtftYE+wQTpg== X-Google-Smtp-Source: AGHT+IE1d8ofKh/Z5l3SEw2O/D7+jsrckuKPHIxuOuGkFeHX4PHL/qWZx61kNLCvBnOE9O0wf335JQ== X-Received: by 2002:a05:6a00:14d2:b0:730:9801:d3e2 with SMTP id d2e1a72fcca58-73dc14ad1a8mr23122155b3a.8.1745307788942; Tue, 22 Apr 2025 00:43:08 -0700 (PDT) Received: from localhost ([2600:1700:65a7:3600:e43f:b3c9:c97a:8ef3]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73dbf8e481asm8235044b3a.64.2025.04.22.00.43.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Apr 2025 00:43:08 -0700 (PDT) From: Emma Worley <emma@emma.gg> To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Apr 2025 00:42:52 -0700 Message-ID: <20250422074254.77328-1-emma@emma.gg> X-Mailer: git-send-email 2.48.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 1/3] lavc/hashtable: create generic robin hood hash table X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches <ffmpeg-devel.ffmpeg.org> List-Unsubscribe: <https://ffmpeg.org/mailman/options/ffmpeg-devel>, <mailto:ffmpeg-devel-request@ffmpeg.org?subject=unsubscribe> List-Archive: <https://ffmpeg.org/pipermail/ffmpeg-devel> List-Post: <mailto:ffmpeg-devel@ffmpeg.org> List-Help: <mailto:ffmpeg-devel-request@ffmpeg.org?subject=help> List-Subscribe: <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>, <mailto:ffmpeg-devel-request@ffmpeg.org?subject=subscribe> Reply-To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> Cc: Emma Worley <emma@emma.gg> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" <ffmpeg-devel-bounces@ffmpeg.org> Archived-At: <https://master.gitmailbox.com/ffmpegdev/20250422074254.77328-1-emma@emma.gg/> List-Archive: <https://master.gitmailbox.com/ffmpegdev/> List-Post: <mailto:ffmpegdev@gitmailbox.com> Adds a generic hash table with the DXV encoder as an initial use case. Signed-off-by: Emma Worley <emma@emma.gg> --- libavcodec/Makefile | 2 + libavcodec/hashtable.c | 192 +++++++++++++++++++++++++++++++++++ libavcodec/hashtable.h | 91 +++++++++++++++++ libavcodec/tests/hashtable.c | 110 ++++++++++++++++++++ 4 files changed, 395 insertions(+) create mode 100644 libavcodec/hashtable.c create mode 100644 libavcodec/hashtable.h create mode 100644 libavcodec/tests/hashtable.c diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 7bd1dbec9a..8071c59378 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -42,6 +42,7 @@ OBJS = ac3_parser.o \ dv_profile.o \ encode.o \ get_buffer.o \ + hashtable.o \ imgconvert.o \ jni.o \ lcevcdec.o \ @@ -1321,6 +1322,7 @@ TESTPROGS = avcodec \ bitstream_le \ celp_math \ codec_desc \ + hashtable \ htmlsubtitles \ jpeg2000dwt \ mathops \ diff --git a/libavcodec/hashtable.c b/libavcodec/hashtable.c new file mode 100644 index 0000000000..4d6d4a9506 --- /dev/null +++ b/libavcodec/hashtable.c @@ -0,0 +1,192 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley <connorbworley@gmail.com> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <stdint.h> +#include <string.h> + +#include "libavutil/crc.h" +#include "libavutil/error.h" +#include "libavutil/mem.h" +#include "hashtable.h" + +#define ALIGN _Alignof(size_t) + +struct FFHashtableContext { + size_t key_size; + size_t key_size_aligned; + size_t val_size; + size_t val_size_aligned; + size_t entry_size; + size_t max_entries; + size_t utilization; + const AVCRC *crc; + uint8_t *table; + uint8_t *swapbuf; +}; + +#define ENTRY_PSL(entry) (entry) +#define ENTRY_OCC(entry) (ENTRY_PSL(entry) + FFALIGN(sizeof(size_t), ALIGN)) +#define ENTRY_KEY(entry) (ENTRY_OCC(entry) + FFALIGN(sizeof(size_t), ALIGN)) +#define ENTRY_VAL(entry) (ENTRY_KEY(entry) + ctx->key_size_aligned) + +#define KEYS_EQUAL(k1, k2) !memcmp(k1, k2, ctx->key_size) + +int ff_hashtable_alloc(struct FFHashtableContext **ctx, size_t key_size, size_t val_size, size_t max_entries) +{ + struct FFHashtableContext *res = av_malloc(sizeof(struct FFHashtableContext)); + if (!res) + return AVERROR(ENOMEM); + res->key_size = key_size; + res->key_size_aligned = FFALIGN(key_size, ALIGN); + res->val_size = val_size; + res->val_size_aligned = FFALIGN(val_size, ALIGN); + res->entry_size = FFALIGN(sizeof(size_t), ALIGN) + + FFALIGN(sizeof(size_t), ALIGN) + + res->key_size_aligned + + res->val_size_aligned; + res->max_entries = max_entries; + res->utilization = 0; + res->crc = av_crc_get_table(AV_CRC_32_IEEE); + if (!res->crc) { + ff_hashtable_freep(&res); + return AVERROR_BUG; + } + res->table = av_calloc(res->max_entries, res->entry_size); + if (!res->table) { + ff_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + res->swapbuf = av_calloc(2, res->key_size_aligned + res->val_size_aligned); + if (!res->swapbuf) { + ff_hashtable_freep(&res); + return AVERROR(ENOMEM); + } + *ctx = res; + return 0; +} + +static size_t hash_key(const struct FFHashtableContext *ctx, const void *key) +{ + return av_crc(ctx->crc, 0, key, ctx->key_size) % ctx->max_entries; +} + +int ff_hashtable_get(const struct FFHashtableContext *ctx, const void *key, void *val) +{ + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + memcpy(val, ENTRY_VAL(entry), ctx->val_size); + return 1; + } + } + return 0; +} + +int ff_hashtable_set(struct FFHashtableContext *ctx, const void *key, const void *val) +{ + int swapping = 0; + size_t psl = 0; + size_t hash = hash_key(ctx, key); + uint8_t *set = ctx->swapbuf; + uint8_t *tmp = ctx->swapbuf + ctx->key_size_aligned + ctx->val_size_aligned; + + memcpy(set, key, ctx->key_size); + memcpy(set + ctx->key_size_aligned, val, ctx->val_size); + + for (size_t i = 0; i < ctx->max_entries; i++) { + size_t wrapped_index = (hash + i) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || (!swapping && KEYS_EQUAL(ENTRY_KEY(entry), set))) { + if (!*ENTRY_OCC(entry)) + ctx->utilization++; + *(size_t*)ENTRY_PSL(entry) = psl; + *ENTRY_OCC(entry) = 1; + memcpy(ENTRY_KEY(entry), set, ctx->key_size_aligned + ctx->val_size); + return 1; + } + if (*(size_t*)ENTRY_PSL(entry) < psl) { + if (ctx->utilization == ctx->max_entries) + return 0; + swapping = 1; + // set needs to swap with entry + memcpy(tmp, ENTRY_KEY(entry), ctx->key_size_aligned + ctx->val_size); + memcpy(ENTRY_KEY(entry), set, ctx->key_size_aligned + ctx->val_size); + FFSWAP(uint8_t*, set, tmp); + FFSWAP(size_t, psl, *(size_t*)ENTRY_PSL(entry)); + } + psl++; + } + return 0; +} + +int ff_hashtable_delete(struct FFHashtableContext *ctx, const void *key) +{ + uint8_t *next_entry; + size_t hash = hash_key(ctx, key); + + if (!ctx->utilization) + return 0; + + for (size_t psl = 0; psl < ctx->max_entries; psl++) { + size_t wrapped_index = (hash + psl) % ctx->max_entries; + uint8_t *entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(entry) || *(size_t*)ENTRY_PSL(entry) < psl) + return 0; + if (KEYS_EQUAL(ENTRY_KEY(entry), key)) { + *ENTRY_OCC(entry) = 0; + for (psl++; psl < ctx->max_entries; psl++) { + wrapped_index = (hash + psl) % ctx->max_entries; + next_entry = ctx->table + wrapped_index * ctx->entry_size; + if (!*ENTRY_OCC(next_entry) || !*(size_t*)ENTRY_PSL(next_entry)) { + ctx->utilization--; + return 1; + } + memcpy(entry, next_entry, ctx->entry_size); + (*(size_t*)ENTRY_PSL(entry))--; + *ENTRY_OCC(next_entry) = 0; + entry = next_entry; + } + } + }; + return 0; +} + +void ff_hashtable_clear(struct FFHashtableContext *ctx) +{ + memset(ctx->table, 0, ctx->entry_size * ctx->max_entries); +} + +void ff_hashtable_freep(struct FFHashtableContext **ctx) +{ + if (*ctx) { + av_freep(&(*ctx)->table); + av_freep(&(*ctx)->swapbuf); + } + av_freep(ctx); +} diff --git a/libavcodec/hashtable.h b/libavcodec/hashtable.h new file mode 100644 index 0000000000..fd28d3bdd7 --- /dev/null +++ b/libavcodec/hashtable.h @@ -0,0 +1,91 @@ +/* + * Generic hashtable + * Copyright (C) 2024 Connor Worley <connorbworley@gmail.com> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_HASHTABLE_H +#define AVCODEC_HASHTABLE_H + +#include <stddef.h> + +/* Implements a hash table using Robin Hood open addressing. + * See: https://cs.uwaterloo.ca/research/tr/1986/CS-86-14.pdf + */ + +struct FFHashtableContext; + +/** + * Creates a fixed-sized Robin Hood hash table. + * + * @param ctx context to allocate and initialize + * @param key_size size of key type in bytes + * @param val_size size of value type in bytes + * @param max_entries maximum number of key-value pairs to store + * + * @return zero on success, nonzero on error + */ +int ff_hashtable_alloc(struct FFHashtableContext **ctx, size_t key_size, size_t val_size, size_t max_entries); + +/** + * Looks up a value from a hash table given a key. + * + * @param ctx hash table context + * @param key pointer to key data + * @param val destination pointer for value data + * + * @return 1 if the key is found, zero if the key is not found + */ +int ff_hashtable_get(const struct FFHashtableContext *ctx, const void *key, void *val); + +/** + * Stores a value in a hash table given a key. + * + * @param ctx hash table context + * @param key pointer to key data + * @param val pointer for value data + * + * @return 1 if the key is written, zero if the key is not written due to the hash table reaching max capacity + */ +int ff_hashtable_set(struct FFHashtableContext *ctx, const void *key, const void *val); + +/** + * Deletes a value from a hash table given a key. + * + * @param ctx hash table context + * @param key pointer to key data + * + * @return 1 if the key is deleted, zero if the key is not deleted due to not being found + */ +int ff_hashtable_delete(struct FFHashtableContext *ctx, const void *key); + +/** + * Deletes all values from a hash table. + * + * @param ctx hash table context + */ +void ff_hashtable_clear(struct FFHashtableContext *ctx); + +/** + * Frees a hash table. + * + * @param ctx hash table context + */ +void ff_hashtable_freep(struct FFHashtableContext **ctx); + +#endif diff --git a/libavcodec/tests/hashtable.c b/libavcodec/tests/hashtable.c new file mode 100644 index 0000000000..367baf4f96 --- /dev/null +++ b/libavcodec/tests/hashtable.c @@ -0,0 +1,110 @@ +/* + * Generic hashtable tests + * Copyright (C) 2024 Connor Worley <connorbworley@gmail.com> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <stdint.h> + +#include "libavutil/avassert.h" +#include "libavcodec/hashtable.h" + +int main(void) +{ + struct FFHashtableContext *ctx; + uint8_t k; + uint64_t v; + + // impossibly large allocation should fail gracefully + av_assert0(ff_hashtable_alloc(&ctx, -1, -1, -1) < 0); + + // hashtable can store up to 3 uint8_t->uint64_t entries + av_assert0(!ff_hashtable_alloc(&ctx, sizeof(k), sizeof(v), 3)); + + // unsuccessful deletes return 0 + k = 1; + av_assert0(!ff_hashtable_delete(ctx, &k)); + + // unsuccessful gets return 0 + k = 1; + av_assert0(!ff_hashtable_get(ctx, &k, &v)); + + // successful sets returns 1 + k = 1; + v = 1; + av_assert0(ff_hashtable_set(ctx, &k, &v)); + + // get should now contain 1 + k = 1; + v = 0; + av_assert0(ff_hashtable_get(ctx, &k, &v)); + av_assert0(v == 1); + + // updating sets should return 1 + k = 1; + v = 2; + av_assert0(ff_hashtable_set(ctx, &k, &v)); + + // get should now contain 2 + k = 1; + v = 0; + av_assert0(ff_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + + // fill the table + k = 2; + v = 2; + av_assert0(ff_hashtable_set(ctx, &k, &v)); + k = 3; + v = 3; + av_assert0(ff_hashtable_set(ctx, &k, &v)); + + // inserting sets on a full table should return 0 + k = 4; + v = 4; + av_assert0(!ff_hashtable_set(ctx, &k, &v)); + + // updating sets on a full table should return 1 + k = 1; + v = 4; + av_assert0(ff_hashtable_set(ctx, &k, &v)); + v = 0; + av_assert0(ff_hashtable_get(ctx, &k, &v)); + av_assert0(v == 4); + + // successful deletes should return 1 + k = 1; + av_assert0(ff_hashtable_delete(ctx, &k)); + + // get should now return 0 + av_assert0(!ff_hashtable_get(ctx, &k, &v)); + + // sanity check remaining keys + k = 2; + v = 0; + av_assert0(ff_hashtable_get(ctx, &k, &v)); + av_assert0(v == 2); + k = 3; + v = 0; + av_assert0(ff_hashtable_get(ctx, &k, &v)); + av_assert0(v == 3); + + ff_hashtable_freep(&ctx); + + return 0; +} -- 2.48.0 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".