From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 48F7F45F8D for ; Fri, 21 Apr 2023 20:21:18 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AB0CD68BEA9; Fri, 21 Apr 2023 23:21:15 +0300 (EEST) Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7566F68AF28 for ; Fri, 21 Apr 2023 23:21:09 +0300 (EEST) Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-5ef8aaf12bdso10061466d6.3 for ; Fri, 21 Apr 2023 13:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20221208.gappssmtp.com; s=20221208; t=1682108468; x=1684700468; h=message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JNShM7wQeRxZqSKDNx4y6/X2Nq+Ih8U9QGNwD2DZa3I=; b=TaS8cdOxSH1apGHC0LUpS/aCCS75hXLybQl77JZN4GsYWD6oSmOd5s+xZQbN4g+659 WMPh1bVAwv1HqyJkTkolad/5kDQ/4867KK4Gt0WwTEQc0ZWwincG/lKhgFudMSr0EJON cejl4AcpIdu5iuTmGhmHDDN4HkXxdtrjxqXXq1BM13ske966Lgk3TEXk7mZyyo4cHMa1 9QTYDVU8JBYh/KORTR5xS+tvYi/TQBRmdzeAYlStVsZEqc8rHMixnrxE+FsZ4RdlZ9lS ZaconQySf99WIi1Ut+4PcFT0x6lc+BFOnzAzaGG3/lkenswHwsWngPjBEReoc4POFx3o 39jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682108468; x=1684700468; h=message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JNShM7wQeRxZqSKDNx4y6/X2Nq+Ih8U9QGNwD2DZa3I=; b=DtRVOGeo2Svo4AnMGABWkS14WX3A2s0Krrbbv+Q6HX8UbC2VFrf3cUYT6GVF1jXw0F k3/JsiZ5aGfj2Chw4mEPaTWi2m4tFbVgIMeP5mBQ0le6LhQzga3N8pCt/nO1ydUsAWx3 J1jSMpBpHQxEmfPw7I6zYXSWPuhD2Aq72gVW8GQG5z7DWy0Ac+b60mS906If3XqE/Lrs hryNHmKFaH9YDyfOxQ8Gg6xkDd+KtmAUReYH0w6sFGARI7L44wOTqqufm/1R61RZ0a6Y xT6gnGr8wvow0eqFColevHZJ8m6u14axYMU8OAIxJYBNuaVaBBj3P6SVOhbtLE6l0MWk ERzQ== X-Gm-Message-State: AAQBX9dZJ0xUWpi776V4FkZvKfTyzZ7kBTwPSzZTJtT37u0zuK46qDVi G3roTCT58Hvj6VjEA+T8g/BW4u/z64XATB+LJes= X-Google-Smtp-Source: AKy350brbsjF7hvKxz7sg7OUPMY7w0sYFZWXOVWLuiDwWTqYfZDtz/2/XSdpCzqyD7rXSNEqTfzzTw== X-Received: by 2002:a05:6214:da7:b0:56e:9986:4fa9 with SMTP id h7-20020a0562140da700b0056e99864fa9mr10836987qvh.7.1682108468042; Fri, 21 Apr 2023 13:21:08 -0700 (PDT) Received: from ltnt-nyc-580testdevin.livetimenet.com (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id x11-20020a0ce0cb000000b005ef6bcbb99fsm1394397qvk.22.2023.04.21.13.21.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Apr 2023 13:21:07 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Fri, 21 Apr 2023 17:16:45 -0400 Message-Id: <1682111805-31679-1-git-send-email-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 1.8.3.1 Subject: [FFmpeg-devel] [RFC/PATCH] decklink_enc: Add support for playout of EIA-608 caption codec X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Today the decklink SDI output supports putting out captions over VANC when the caption data is packet side data. However some container formats such as MP4 support a dedicated subtitle stream, and we end up receiving packets containing the 608 tuples separate from the video frames. Make use of a simple intermediate queue to stash the 608 tuples, and then attach them to the video frame side data such that it can be inserted into VANC via the subsequent call to construct_cc(). Unlike with packet side data that carries the captions associated with that one frame, individual c608 packets can contain tuples for entire batches of frames, so we need to pace out how quickly they are inserted into the video based on the framerate. Example usage: ./ffmpeg -i file.mp4 -map 0:0 -map 0:1 -map 0:3 -codec:2 copy -vcodec v210 -f decklink 'DeckLink Duo (4)' Note that the user needs to specify -codec copy for the c608 subtitle track, since there is no decoding/encoding happening. The CC fifo code here overlaps with some of the functionality found in the libavfilter/ccfifo.[c/h], but that code can't be referenced as it is currently private to libavfilter. If the ccfifo stuff gets moved to libavutil then the routine here can be simplified. Signed-off-by: Devin Heitmueller --- libavdevice/decklink_common.h | 2 + libavdevice/decklink_enc.cpp | 94 ++++++++++++++++++++++++++++++++++++++++++- libavdevice/decklink_enc_c.c | 2 +- 3 files changed, 95 insertions(+), 3 deletions(-) diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 99afcd4..51ad5da 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -31,6 +31,7 @@ extern "C" { #include "libavcodec/packet_internal.h" +#include "libavutil/fifo.h" } #include "libavutil/thread.h" #include "decklink_common_c.h" @@ -114,6 +115,7 @@ struct decklink_ctx { /* Output VANC queue */ AVPacketQueue vanc_queue; + AVFifo *cc_fifo; /* Streams present */ int audio; diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp index d7357a0..23f00c8 100644 --- a/libavdevice/decklink_enc.cpp +++ b/libavdevice/decklink_enc.cpp @@ -345,6 +345,30 @@ static int decklink_setup_data(AVFormatContext *avctx, AVStream *st) return ret; } +static int decklink_setup_subtitle(AVFormatContext *avctx, AVStream *st) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + + int ret = -1; + + switch(st->codecpar->codec_id) { +#if CONFIG_LIBKLVANC + case AV_CODEC_ID_EIA_608: + if (!ctx->cc_fifo) + ctx->cc_fifo = av_fifo_alloc2(128, 3, 0); + if (ctx->cc_fifo) + ret = 0; + break; +#endif + default: + av_log(avctx, AV_LOG_ERROR, "Unsupported subtitle codec specified\n"); + break; + } + + return ret; +} + av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) { struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; @@ -371,6 +395,7 @@ av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) klvanc_context_destroy(ctx->vanc_ctx); #endif avpacket_queue_end(&ctx->vanc_queue); + av_fifo_freep2(&ctx->cc_fifo); av_freep(&cctx->ctx); @@ -523,6 +548,54 @@ out: free(afd_words); } +/* Parse any EIA-608 subtitles sitting on the queue, and write packet side data + that will later be handled by construct_cc... */ +static void parse_608subs(AVFormatContext *avctx, struct decklink_ctx *ctx, AVPacket *pkt) +{ + int cc_count = 0; + int cc_filled = 0; + + if (!ctx->cc_fifo) + return; + + /* 29.97 and 59.94 are really the only two framerates we care about */ + if (ctx->bmd_tb_den == 30000 && ctx->bmd_tb_num == 1001) { + cc_count = 20; + } else if (ctx->bmd_tb_den == 60000 && ctx->bmd_tb_num == 1001) { + cc_count = 10; + } + if (cc_count > 0) { + uint8_t *cc_buf = av_packet_new_side_data(pkt, AV_PKT_DATA_A53_CC, cc_count * 3); + if (!cc_buf) + return; + int len = av_fifo_can_read(ctx->cc_fifo); + if (len > 0) { + uint8_t buf[3]; + av_fifo_read(ctx->cc_fifo, buf, 1); + av_log(avctx, AV_LOG_ERROR, "Got a 608 packet! size=%d\n", len); + memcpy(cc_buf, buf, 3); + cc_filled++; + if (cc_count == 20 && len > 1) { + /* If we're at 30 FPS, we need to insert a second tuple if one is + available and it's for CC3/CC4) */ + av_fifo_peek(ctx->cc_fifo, buf, 1, 0); + if (buf[0] == 0xfd) { + av_fifo_read(ctx->cc_fifo, buf, 1); + av_log(avctx, AV_LOG_ERROR, "Got a 608 packet! size=%d\n", len); + memcpy(cc_buf, buf, 3); + cc_filled++; + } + } + } + /* Pad out the rest of the field */ + for(int i = cc_filled; i < cc_count; i++) { + cc_buf[i*3] = 0xfa; + cc_buf[i*3+1] = 0x00; + cc_buf[i*3+2] = 0x00; + } + } +} + static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx *ctx, AVPacket *pkt, decklink_frame *frame, AVStream *st) @@ -533,6 +606,7 @@ static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx * if (!ctx->supports_vanc) return 0; + parse_608subs(avctx, ctx, pkt); construct_cc(avctx, ctx, pkt, &vanc_lines); construct_afd(avctx, ctx, pkt, &vanc_lines, st); @@ -793,6 +867,16 @@ static int decklink_write_data_packet(AVFormatContext *avctx, AVPacket *pkt) return 0; } +static int decklink_write_subtitle_packet(AVFormatContext *avctx, AVPacket *pkt) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + + av_fifo_write(ctx->cc_fifo, pkt->data, pkt->size / 3); + + return 0; +} + extern "C" { av_cold int ff_decklink_write_header(AVFormatContext *avctx) @@ -860,6 +944,9 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) } else if (c->codec_type == AVMEDIA_TYPE_DATA) { if (decklink_setup_data(avctx, st)) goto error; + } else if (c->codec_type == AVMEDIA_TYPE_SUBTITLE) { + if (decklink_setup_subtitle(avctx, st)) + goto error; } else { av_log(avctx, AV_LOG_ERROR, "Unsupported stream type.\n"); goto error; @@ -870,7 +957,8 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) for (n = 0; n < avctx->nb_streams; n++) { AVStream *st = avctx->streams[n]; AVCodecParameters *c = st->codecpar; - if (c->codec_type == AVMEDIA_TYPE_DATA) + if (c->codec_type == AVMEDIA_TYPE_DATA || + c->codec_type == AVMEDIA_TYPE_SUBTITLE) avpriv_set_pts_info(st, 64, ctx->bmd_tb_num, ctx->bmd_tb_den); } avpacket_queue_init (avctx, &ctx->vanc_queue); @@ -890,8 +978,10 @@ int ff_decklink_write_packet(AVFormatContext *avctx, AVPacket *pkt) return decklink_write_video_packet(avctx, pkt); else if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) return decklink_write_audio_packet(avctx, pkt); - else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) { + else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) return decklink_write_data_packet(avctx, pkt); + else if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) { + return decklink_write_subtitle_packet(avctx, pkt); } return AVERROR(EIO); diff --git a/libavdevice/decklink_enc_c.c b/libavdevice/decklink_enc_c.c index 7aab660..d593cca 100644 --- a/libavdevice/decklink_enc_c.c +++ b/libavdevice/decklink_enc_c.c @@ -78,7 +78,7 @@ const FFOutputFormat ff_decklink_muxer = { .p.long_name = NULL_IF_CONFIG_SMALL("Blackmagic DeckLink output"), .p.audio_codec = AV_CODEC_ID_PCM_S16LE, .p.video_codec = AV_CODEC_ID_WRAPPED_AVFRAME, - .p.subtitle_codec = AV_CODEC_ID_NONE, + .p.subtitle_codec = AV_CODEC_ID_EIA_608, .p.flags = AVFMT_NOFILE, .p.priv_class = &decklink_muxer_class, .get_device_list = ff_decklink_list_output_devices, -- 1.8.3.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".