From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id A7B7A4BC5C for ; Wed, 2 Jul 2025 08:11:50 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 8534B68CD14; Wed, 2 Jul 2025 11:11:25 +0300 (EEST) Received: from CAN01-YQB-obe.outbound.protection.outlook.com (mail-yqbcan01on2118.outbound.protection.outlook.com [40.107.116.118]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTPS id 3274268CD20 for ; Wed, 2 Jul 2025 11:11:22 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tM95kGDFi2O0Q/jKnHXSDkJ8zO0PhK6veRYJj3Yg9s/GMGrN04jsv5WEqj+Rz5aW8ACJdCC+CKdut789Y2ZVczf5BMEktus0ZFuVByou7qMCVwxqWKGHvz4W0LLt9LUWMIbXR4uQtQ+v+cyDrMT47s0C3O9p5jVhOmuPIrUvSKYrG8YpIV3O4DzJG8mz2Y7foWd4NHtBl8pq4UI9nbN8Mx6hMqM29W/+3ziq9u9yRkEr2QS2r7/j/0It11Rq2FO1y5eU9moUngWJjbw1gg3cO+gJ1+GlAQxg8kdKJ9LRMhFiVvVtkrYWFI0EtJPCIHNp03dyytUDl0Zo+cM04xVQfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZjQuATvb7WMwAEqv4MRItqRipwMc/nrnyGxDINJonZw=; b=wpsbc6gK67Kd634SkBoIi4WJLDUWR+uSQeSSpRgoe5tsdE/vVNteF7BV02UJCUjuDQtskosTD8A5VZM2wy9D67SKdlx9MDaoN64HzA97ahHxlG4wYOEj+gym2WcjP7p3wdx/f5QY0EnAEijHCnxcKg46G5kooBDMcdHGjw/AeZVyimp+i2myCHBsIo99DzT4KwgCTxJDvQjXRo5WSP7BzdZ3deM3+xkRv3ajRN7bhOGsv6t9imbVZmyL9ZhsmYjoQE1/2BXWxOKLs6TVpWh/YpNAO2Or6GAaqrPJkCqv9d5XeXth4GSUO2cOGjOYblVvO78Bc84RvMbgaZ3MtaxPPQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netint.ca; dmarc=pass action=none header.from=netint.ca; dkim=pass header.d=netint.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netint.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZjQuATvb7WMwAEqv4MRItqRipwMc/nrnyGxDINJonZw=; b=RG6P53DjO67qXoFdH6ZXMkc+52jwTC+LiDy1me6u4IGFkjTmZ24UV9TJGy2gFF92R5VRclbtzZmqEsVuidpy70J//ye7R0eOWC47t4GWpLP6X92KqCIU4O6g2mB/JyPKnhSXkQW3dmeH4awvjNgaf0LqXEEiN4URSXkVoFZMunU= Received: from YT2PR01MB4701.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:3c::6) by YQBPR0101MB5441.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:44::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.19; Wed, 2 Jul 2025 08:11:10 +0000 Received: from YT2PR01MB4701.CANPRD01.PROD.OUTLOOK.COM ([fe80::6697:95af:2af:a8bb]) by YT2PR01MB4701.CANPRD01.PROD.OUTLOOK.COM ([fe80::6697:95af:2af:a8bb%6]) with mapi id 15.20.8901.021; Wed, 2 Jul 2025 08:11:10 +0000 From: Steven Zhou To: "ffmpeg-devel@ffmpeg.org" Thread-Topic: [PATCH 3/3] libavfilter: add NETINT Quadra HW video filters Thread-Index: AdvqQqJYatRICeJjSnmQHctHA66thg== Date: Wed, 2 Jul 2025 08:11:09 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=netint.ca; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: YT2PR01MB4701:EE_|YQBPR0101MB5441:EE_ x-ms-office365-filtering-correlation-id: fbfe7c07-0b02-423f-c26f-08ddb94000ce x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?iso-8859-1?Q?cfCQAwJcG28WzyorhOA4a6vUa/Y52h7ZWllblupAmYjJo4ds5rkCJ/cemc?= =?iso-8859-1?Q?kl1H+gTtwE0rus/UfWXIhebhzP2dMpoVXnJ8QpqtI3I2iNtmCJJz7lOtP/?= =?iso-8859-1?Q?s8nG4OeCUBs+SthAt6Xv4IaDshysEFj16MriS2rP4JZ8PhARWMkzer12wL?= =?iso-8859-1?Q?kogtF7OaUM1w53L8H9lUG0gJU/5XShJSKnrADSdm7t6dOMPrwWiq7EwGLK?= =?iso-8859-1?Q?XwcsjYHAkWHjK7a+9IiDB5ldCWnU2rGlsaeWer7Iplu/1a/OmOVCDcPZQ/?= =?iso-8859-1?Q?DTHgcuva8uWe6ogDqb2RMxs7RWs9jOwKtS5MrvYaY+PSQboDBUyNcFtSz4?= =?iso-8859-1?Q?eDYejQt0HfdP4ArWbIMCTBOln4JE1FSeYt2Awn0RPFPBsZQGW4X8ATdXih?= =?iso-8859-1?Q?P8AV6KTHoj1sa0DkQUZN/Z7axWdomFbxb1sT34YgY7eANGUD2eqcMvcnIJ?= =?iso-8859-1?Q?+ApDsl1/lJsv6OgTAVrTqJVhlhDdHkx4LKxeZyJEVzfrbZwkwRWjggGvfy?= =?iso-8859-1?Q?1X0lBtwlJPtfO5PjWHcpSgdHTsSaAYkB0aXsLjUZvbuGa4OddXzKLS8zV4?= =?iso-8859-1?Q?H+KCfvqU8qL09Ukns7oDk5X5ZlUfu4gMM22rePATaVnaUNU/uKnB8E1JIZ?= =?iso-8859-1?Q?50n6pXKvlGky1n2A90lvnMPcMPEfsta7w6rupPZb/ekN8B7UypgCC9a6GB?= =?iso-8859-1?Q?lf6cUkooBNu1DLnfDT2NAQoer4dQHbQgtybey/SA1AcxnVofGCBmM60mOU?= =?iso-8859-1?Q?2TUMfLe5doqjeWCrEQMPim7c6qYk9kuUNbUoWIFqwgNQfMmHFstwCYaAcP?= =?iso-8859-1?Q?y6+TRr58a6PNY/VmvX6pEFuX0xjo0SyptitlZD56t2QE5fwOUvGnTHuk6L?= =?iso-8859-1?Q?wauzyUpynKr4P9EsiKIrGkS/yetpValL7Q37muCasJIyix03Z37QPxtDXm?= =?iso-8859-1?Q?4tU4NG8tA9BPu1NUHsG4HTw3OH2S/p6HX8GgSR28Cw/J6/zVGW+ts9NFF0?= =?iso-8859-1?Q?a68nS329qgjrXL5r+5sxeYf+wYfqsfetBgBjNPSPhn7brFGbjfL40DKP6l?= =?iso-8859-1?Q?NreQgkIjKmhD+/7H5iy2LBg0uVBcPaqvyrC3oywb/e4Q8umpqyR4J0nHCb?= =?iso-8859-1?Q?YEqehcHdxPuUY502hk9HcvFMc6B4M5T+TuOs+hQFpXl9w1w9ZZ2LgHP4NK?= =?iso-8859-1?Q?E4cmd1jTWhl9SHmGbGj2dN4sh93gdfpX6KQtsG8zgCFtIIgwv9gu1Q6wzD?= =?iso-8859-1?Q?CqcLTKq+vo8w46pG5MkfPRGFxzo96sRQOCu/8ebfJtRy/6L6/vy565lb1A?= =?iso-8859-1?Q?+MZOuX8YMP+TzrSN2Zrx6mGTA/Ay6oSNF0nCOlULMpMSk931yj5jwh0qvy?= =?iso-8859-1?Q?E0CR9Fm9JrijcknQV+3MGKjvN6clZYmbk52E078jmCmNnkCkKDo71DVW/n?= =?iso-8859-1?Q?3+PYS9+k1dYmwxBNiCVAen57kw5bNUM3zSE89cySC9DRlpvYAi/BacPUWf?= =?iso-8859-1?Q?mCzPpINU0OST/WtHLtxzdeWixkOdWi0lvZXpId8G2IC4r5hMjq2aT+V9dn?= =?iso-8859-1?Q?JHOz3fo=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YT2PR01MB4701.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(38070700018); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?Ym89CDKnOrmzKL+vjgPfOIslrS0KwpPchsCENTWufv9xmFjvYncqSLrE7l?= =?iso-8859-1?Q?QtrpybUQiERxM4tuOkq4c0z779rPcztlZKtPXYMbhCnOxGq+TlJYjvlVEA?= =?iso-8859-1?Q?Yq7iBBvWkwp0B52KGHX1YJ15wQwTYZ2Rb2TuDbqXa4RG9kU+7d71RuGx1i?= =?iso-8859-1?Q?RFF/7aNN6iZOcgMigz2YjpjVA/OPqAVAmdkl18bTi8aQh/wt09yBdqbGUa?= =?iso-8859-1?Q?VVd0kZRPS0GR2P89LPdiux6gFbG6p8WyPNm66mBX2E2kn9PSeSwSw4I7lC?= =?iso-8859-1?Q?QXIBFT3UebrTUMth6wA4oWAEEzEN8x6JrFRTpVMHBSeuzXdUUkRp2XJDSS?= =?iso-8859-1?Q?S5sO09yt9xNUv+vt9L00gbYNZ6NAagToLl3FhU0lzegS8SWD2Ho2vwBYEm?= =?iso-8859-1?Q?QLQmGmnzN3HakA9fLhuJUwY8JaBfbAAUrJeegqNtd730u/O1eiQ6AYGI5h?= =?iso-8859-1?Q?0pUbcbPSgQrDSTQ7cGMIz1i0RtLx9mVzwlcV3LZW+vq/BthgXYryu/AKV8?= =?iso-8859-1?Q?ci1SruqbeKH/8fyUtVe2WJ2ewsg7ueqFS9Fs9VapofILPv3ztg81MNC2Iy?= =?iso-8859-1?Q?+VtoIJMd3XCTAP698kXll6T5Ah05c2gxbkLXQvcv2jQcqVAzO6A9NUlDYY?= =?iso-8859-1?Q?wgPW9Pvhl5e13SESreDZtqQBwOFTEtjjTOHs9SSGzOCWTxh6OOuJcvhjse?= =?iso-8859-1?Q?r+3L4geG+tw4OjavW1JFIwYgmEOgepKRQU/w+ptns9YZRQybCIT1nI0dgB?= =?iso-8859-1?Q?q+CcimSMiqvThddcJkYQYkT9ml5vXn3Uys5xjjbUCla3CxonHFtSLOUFF7?= =?iso-8859-1?Q?uan8wbFzTdyAfFSPra3QOc4NRbmCzn/dpX6OjyWeePzPt6xIjcnwqN0GHJ?= =?iso-8859-1?Q?e5G6lUIzBEQYzC2irsPpzSmI4/HdpRWmV1s7VL/pmPyCc7Js439l5ZFTHr?= =?iso-8859-1?Q?HaM0eJz3bFe+CPcdtTFOC6ZQEu75w+lT/0CH7LtoX4okjDDK/C6ykcGVNJ?= =?iso-8859-1?Q?U08PO5mn388SNrwBUpjES87l2PcgLf+ZOjj+yx7SQntOCul8yGTEkg9CPc?= =?iso-8859-1?Q?y8kBuVwVF26t+Mpu9dkiBk4uW77BrKwgAE0Xp9ktwzpEyz+FFeCr2LcbRO?= =?iso-8859-1?Q?P37SncRKH4Oyx0d21KCM28OX2K+KqMQJzmpCVZCE9ChTEArvogrcXPRGl1?= =?iso-8859-1?Q?YH6/mfY9GZzPX2k7hPkqeVWxcFyEuwOmm6ziZu6B828U/6aHa+xr7eWu4a?= =?iso-8859-1?Q?v/deTLJ2evFf/okfkS9lYImjLkoxWqSqLK+bAKQLysg5CsNKNdBJbh2xBQ?= =?iso-8859-1?Q?sHirWJwtukqFe8E48no6S6OAJEtMZ2ncg70FMuUoTkYUvJlnFrhVbXhwGM?= =?iso-8859-1?Q?gN7mFf1/LM1OxYzx7gAE0k6VcqwE8HTragpHC4KvCMzYRafuCZfTaR4X6n?= =?iso-8859-1?Q?Bk41nEhkYMFM9MUsst61rnCU7io47TzHtGd6AoQscpWgg+8QCYv6ZnB4f4?= =?iso-8859-1?Q?ZXw+hcCFusRffp43u6GKsQwhGnVBeLxVVDCQ4Tz+4Hr5iqGg1MdRuwcppE?= =?iso-8859-1?Q?NrVF8yIRPhz9vhXtp07lGF23+3nFs2jvVU004KPoS7logtm+4iabGKJhvs?= =?iso-8859-1?Q?1HxuJaItFJSLdwR3NDD4b1Peqww1j2Dlny?= MIME-Version: 1.0 X-OriginatorOrg: netint.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB4701.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: fbfe7c07-0b02-423f-c26f-08ddb94000ce X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Jul 2025 08:11:09.8664 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 76291bb4-3d04-48ef-9eb8-3e33475af3cb X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3MnDgvGuSW/6o6Lt88BS6AuMc+skkJl0GlCYzBpfRZGMPokf2t+0OYHgcvpYaJyUsAZWjBug25RYaRgQzX3Y5Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB5441 Subject: [FFmpeg-devel] [PATCH 3/3] libavfilter: add NETINT Quadra HW video filters X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Add NETINT Quadra hardware video filters ni_quadra_crop, ni_quadra_drawbox, ni_quadra_drawtext, ni_quadra_flip, ni_quadra_hvsplus, ni_quadra_hwupload, ni_quadra_overlay, ni_quadra_pad, ni_quadra_rotate, ni_quadra_scale, and ni_quadra_split More information: https://netint.com/products/quadra-t1a-video-processing-unit/ https://docs.netint.com/vpu/quadra/ Signed-off-by: Steven Zhou --- configure | 13 + libavfilter/Makefile | 12 + libavfilter/allfilters.c | 11 + libavfilter/nifilter.c | 361 +++ libavfilter/nifilter.h | 69 + libavfilter/vf_crop_ni.c | 719 ++++++ libavfilter/vf_drawbox_ni.c | 831 +++++++ libavfilter/vf_drawtext_ni.c | 3401 +++++++++++++++++++++++++++ libavfilter/vf_flip_ni.c | 469 ++++ libavfilter/vf_hvsplus_ni.c | 1792 ++++++++++++++ libavfilter/vf_hwupload_ni_quadra.c | 297 +++ libavfilter/vf_overlay_ni.c | 1397 +++++++++++ libavfilter/vf_pad_ni.c | 703 ++++++ libavfilter/vf_rotate_ni.c | 764 ++++++ libavfilter/vf_scale_ni.c | 958 ++++++++ libavfilter/vf_split_ni.c | 529 +++++ 16 files changed, 12326 insertions(+) create mode 100644 libavfilter/nifilter.c create mode 100644 libavfilter/nifilter.h create mode 100644 libavfilter/vf_crop_ni.c create mode 100644 libavfilter/vf_drawbox_ni.c create mode 100644 libavfilter/vf_drawtext_ni.c create mode 100644 libavfilter/vf_flip_ni.c create mode 100644 libavfilter/vf_hvsplus_ni.c create mode 100644 libavfilter/vf_hwupload_ni_quadra.c create mode 100644 libavfilter/vf_overlay_ni.c create mode 100644 libavfilter/vf_pad_ni.c create mode 100644 libavfilter/vf_rotate_ni.c create mode 100644 libavfilter/vf_scale_ni.c create mode 100644 libavfilter/vf_split_ni.c diff --git a/configure b/configure index 0a2dda84c9..4d8f438f6c 100755 --- a/configure +++ b/configure @@ -3372,6 +3372,19 @@ amf_deps_any=3D"libdl LoadLibrary" nvenc_deps=3D"ffnvcodec" nvenc_deps_any=3D"libdl LoadLibrary" = +hwupload_ni_quadra_filter_deps=3D"ni_quadra" +drawbox_ni_quadra_filter_deps=3D"ni_quadra" +drawtext_ni_quadra_filter_deps=3D"ni_quadra libfreetype" +drawtext_ni_quadra_filter_suggest=3D"libfontconfig libfribidi" +crop_ni_quadra_filter_deps=3D"ni_quadra" +overlay_ni_quadra_filter_deps=3D"ni_quadra" +pad_ni_quadra_filter_deps=3D"ni_quadra" +rotate_ni_quadra_filter_deps=3D"ni_quadra" +scale_ni_quadra_filter_deps=3D"ni_quadra" +split_ni_quadra_filter_deps=3D"ni_quadra" +hvsplus_ni_quadra_filter_deps=3D"ni_quadra" +flip_ni_quadra_filter_deps=3D"ni_quadra" + aac_mediacodec_decoder_deps=3D"mediacodec" aac_mediacodec_decoder_select=3D"aac_adtstoasc_bsf aac_parser" aac_mf_encoder_deps=3D"mediafoundation" diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 97f8f17272..e8f278edfe 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -259,6 +259,7 @@ OBJS-$(CONFIG_COREIMAGE_FILTER) +=3D vf_co= reimage.o OBJS-$(CONFIG_CORR_FILTER) +=3D vf_corr.o framesync.o OBJS-$(CONFIG_COVER_RECT_FILTER) +=3D vf_cover_rect.o lavfutil= s.o OBJS-$(CONFIG_CROP_FILTER) +=3D vf_crop.o +OBJS-$(CONFIG_CROP_NI_QUADRA_FILTER) +=3D vf_crop_ni.o nifilter.o OBJS-$(CONFIG_CROPDETECT_FILTER) +=3D vf_cropdetect.o edge_com= mon.o OBJS-$(CONFIG_CUE_FILTER) +=3D f_cue.o OBJS-$(CONFIG_CURVES_FILTER) +=3D vf_curves.o @@ -292,9 +293,11 @@ OBJS-$(CONFIG_DNN_DETECT_FILTER) +=3D vf_d= nn_detect.o OBJS-$(CONFIG_DNN_PROCESSING_FILTER) +=3D vf_dnn_processing.o OBJS-$(CONFIG_DOUBLEWEAVE_FILTER) +=3D vf_weave.o OBJS-$(CONFIG_DRAWBOX_FILTER) +=3D vf_drawbox.o +OBJS-$(CONFIG_DRAWBOX_NI_QUADRA_FILTER) +=3D vf_drawbox_ni.o OBJS-$(CONFIG_DRAWGRAPH_FILTER) +=3D f_drawgraph.o OBJS-$(CONFIG_DRAWGRID_FILTER) +=3D vf_drawbox.o OBJS-$(CONFIG_DRAWTEXT_FILTER) +=3D vf_drawtext.o textutils.o +OBJS-$(CONFIG_DRAWTEXT_NI_QUADRA_FILTER) +=3D vf_drawtext_ni.o OBJS-$(CONFIG_EDGEDETECT_FILTER) +=3D vf_edgedetect.o edge_com= mon.o OBJS-$(CONFIG_ELBG_FILTER) +=3D vf_elbg.o OBJS-$(CONFIG_ENTROPY_FILTER) +=3D vf_entropy.o @@ -346,10 +349,12 @@ OBJS-$(CONFIG_HSTACK_FILTER) +=3D vf_= stack.o framesync.o OBJS-$(CONFIG_HSVHOLD_FILTER) +=3D vf_hsvkey.o OBJS-$(CONFIG_HSVKEY_FILTER) +=3D vf_hsvkey.o OBJS-$(CONFIG_HUE_FILTER) +=3D vf_hue.o +OBJS-$(CONFIG_HVSPLUS_NI_QUADRA_FILTER) +=3D vf_hvsplus_ni.o OBJS-$(CONFIG_HUESATURATION_FILTER) +=3D vf_huesaturation.o OBJS-$(CONFIG_HWDOWNLOAD_FILTER) +=3D vf_hwdownload.o OBJS-$(CONFIG_HWMAP_FILTER) +=3D vf_hwmap.o OBJS-$(CONFIG_HWUPLOAD_CUDA_FILTER) +=3D vf_hwupload_cuda.o +OBJS-$(CONFIG_HWUPLOAD_NI_QUADRA_FILTER) +=3D vf_hwupload_ni_quadra.o OBJS-$(CONFIG_HWUPLOAD_FILTER) +=3D vf_hwupload.o OBJS-$(CONFIG_HYSTERESIS_FILTER) +=3D vf_hysteresis.o framesyn= c.o OBJS-$(CONFIG_ICCDETECT_FILTER) +=3D vf_iccdetect.o fflcms2.o @@ -413,6 +418,7 @@ OBJS-$(CONFIG_OCR_FILTER) +=3D vf_oc= r.o OBJS-$(CONFIG_OCV_FILTER) +=3D vf_libopencv.o OBJS-$(CONFIG_OSCILLOSCOPE_FILTER) +=3D vf_datascope.o OBJS-$(CONFIG_OVERLAY_FILTER) +=3D vf_overlay.o framesync.o +OBJS-$(CONFIG_OVERLAY_NI_QUADRA_FILTER) +=3D vf_overlay_ni.o framesyn= c.o OBJS-$(CONFIG_OVERLAY_CUDA_FILTER) +=3D vf_overlay_cuda.o frames= ync.o vf_overlay_cuda.ptx.o \ cuda/load_helper.o OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER) +=3D vf_overlay_opencl.o open= cl.o \ @@ -422,6 +428,7 @@ OBJS-$(CONFIG_OVERLAY_VAAPI_FILTER) +=3D vf_ov= erlay_vaapi.o framesync.o v OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) +=3D vf_overlay_vulkan.o vulk= an.o vulkan_filter.o OBJS-$(CONFIG_OWDENOISE_FILTER) +=3D vf_owdenoise.o OBJS-$(CONFIG_PAD_FILTER) +=3D vf_pad.o +OBJS-$(CONFIG_PAD_NI_QUADRA_FILTER) +=3D vf_pad_ni.o nifilter.o OBJS-$(CONFIG_PAD_OPENCL_FILTER) +=3D vf_pad_opencl.o opencl.o= opencl/pad.o OBJS-$(CONFIG_PALETTEGEN_FILTER) +=3D vf_palettegen.o palette.o OBJS-$(CONFIG_PALETTEUSE_FILTER) +=3D vf_paletteuse.o framesyn= c.o palette.o @@ -460,10 +467,12 @@ OBJS-$(CONFIG_ROBERTS_FILTER) +=3D vf_= convolution.o OBJS-$(CONFIG_ROBERTS_OPENCL_FILTER) +=3D vf_convolution_opencl.o = opencl.o \ opencl/convolution.o OBJS-$(CONFIG_ROTATE_FILTER) +=3D vf_rotate.o +OBJS-$(CONFIG_ROTATE_NI_QUADRA_FILTER) +=3D vf_rotate_ni.o nifilter.o OBJS-$(CONFIG_SAB_FILTER) +=3D vf_sab.o OBJS-$(CONFIG_SCALE_FILTER) +=3D vf_scale.o scale_eval.o = framesync.o OBJS-$(CONFIG_SCALE_CUDA_FILTER) +=3D vf_scale_cuda.o scale_ev= al.o \ vf_scale_cuda.ptx.o cuda/l= oad_helper.o +OBJS-$(CONFIG_SCALE_NI_QUADRA_FILTER) +=3D vf_scale_ni.o nifilter.o OBJS-$(CONFIG_SCALE_NPP_FILTER) +=3D vf_scale_npp.o scale_eva= l.o OBJS-$(CONFIG_SCALE_QSV_FILTER) +=3D vf_vpp_qsv.o OBJS-$(CONFIG_SCALE_VAAPI_FILTER) +=3D vf_scale_vaapi.o scale_e= val.o vaapi_vpp.o @@ -504,6 +513,7 @@ OBJS-$(CONFIG_SOBEL_OPENCL_FILTER) +=3D vf_co= nvolution_opencl.o opencl.o opencl/convolution.o OBJS-$(CONFIG_SITI_FILTER) +=3D vf_siti.o OBJS-$(CONFIG_SPLIT_FILTER) +=3D split.o +OBJS-$(CONFIG_SPLIT_NI_QUADRA_FILTER) +=3D vf_split_ni.o OBJS-$(CONFIG_SPP_FILTER) +=3D vf_spp.o qp_table.o OBJS-$(CONFIG_SR_FILTER) +=3D vf_sr.o OBJS-$(CONFIG_SR_AMF_FILTER) +=3D vf_sr_amf.o scale_eval.o= vf_amf_common.o @@ -552,6 +562,7 @@ OBJS-$(CONFIG_VAGUEDENOISER_FILTER) +=3D vf_va= guedenoiser.o OBJS-$(CONFIG_VARBLUR_FILTER) +=3D vf_varblur.o framesync.o OBJS-$(CONFIG_VECTORSCOPE_FILTER) +=3D vf_vectorscope.o OBJS-$(CONFIG_VFLIP_FILTER) +=3D vf_vflip.o +OBJS-$(CONFIG_FLIP_NI_QUADRA_FILTER) +=3D vf_flip_ni.o OBJS-$(CONFIG_VFLIP_VULKAN_FILTER) +=3D vf_flip_vulkan.o vulkan.o OBJS-$(CONFIG_VFRDET_FILTER) +=3D vf_vfrdet.o OBJS-$(CONFIG_VIBRANCE_FILTER) +=3D vf_vibrance.o @@ -667,6 +678,7 @@ SKIPHEADERS-$(CONFIG_LIBVIDSTAB) +=3D vidst= abutils.h = SKIPHEADERS-$(CONFIG_AMF) +=3D vf_amf_common.h SKIPHEADERS-$(CONFIG_QSVVPP) +=3D qsvvpp.h stack_internal.h +SKIPHEADERS-$(CONFIG_NI_QUADRA) +=3D nifilter.h SKIPHEADERS-$(CONFIG_OPENCL) +=3D opencl.h SKIPHEADERS-$(CONFIG_VAAPI) +=3D vaapi_vpp.h stack_intern= al.h SKIPHEADERS-$(CONFIG_VULKAN) +=3D vulkan_filter.h diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 3bc045b28f..f013238cdd 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -236,6 +236,7 @@ extern const FFFilter ff_vf_coreimage; extern const FFFilter ff_vf_corr; extern const FFFilter ff_vf_cover_rect; extern const FFFilter ff_vf_crop; +extern const FFFilter ff_vf_crop_ni_quadra; extern const FFFilter ff_vf_cropdetect; extern const FFFilter ff_vf_cue; extern const FFFilter ff_vf_curves; @@ -267,9 +268,11 @@ extern const FFFilter ff_vf_dnn_detect; extern const FFFilter ff_vf_dnn_processing; extern const FFFilter ff_vf_doubleweave; extern const FFFilter ff_vf_drawbox; +extern const FFFilter ff_vf_drawbox_ni_quadra; extern const FFFilter ff_vf_drawgraph; extern const FFFilter ff_vf_drawgrid; extern const FFFilter ff_vf_drawtext; +extern const FFFilter ff_vf_drawtext_ni_quadra; extern const FFFilter ff_vf_edgedetect; extern const FFFilter ff_vf_elbg; extern const FFFilter ff_vf_entropy; @@ -321,10 +324,12 @@ extern const FFFilter ff_vf_hstack; extern const FFFilter ff_vf_hsvhold; extern const FFFilter ff_vf_hsvkey; extern const FFFilter ff_vf_hue; +extern const FFFilter ff_vf_hvsplus_ni_quadra; extern const FFFilter ff_vf_huesaturation; extern const FFFilter ff_vf_hwdownload; extern const FFFilter ff_vf_hwmap; extern const FFFilter ff_vf_hwupload; +extern const FFFilter ff_vf_hwupload_ni_quadra; extern const FFFilter ff_vf_hwupload_cuda; extern const FFFilter ff_vf_hysteresis; extern const FFFilter ff_vf_iccdetect; @@ -388,6 +393,7 @@ extern const FFFilter ff_vf_ocr; extern const FFFilter ff_vf_ocv; extern const FFFilter ff_vf_oscilloscope; extern const FFFilter ff_vf_overlay; +extern const FFFilter ff_vf_overlay_ni_quadra; extern const FFFilter ff_vf_overlay_opencl; extern const FFFilter ff_vf_overlay_qsv; extern const FFFilter ff_vf_overlay_vaapi; @@ -395,6 +401,7 @@ extern const FFFilter ff_vf_overlay_vulkan; extern const FFFilter ff_vf_overlay_cuda; extern const FFFilter ff_vf_owdenoise; extern const FFFilter ff_vf_pad; +extern const FFFilter ff_vf_pad_ni_quadra; extern const FFFilter ff_vf_pad_opencl; extern const FFFilter ff_vf_palettegen; extern const FFFilter ff_vf_paletteuse; @@ -431,11 +438,13 @@ extern const FFFilter ff_vf_rgbashift; extern const FFFilter ff_vf_roberts; extern const FFFilter ff_vf_roberts_opencl; extern const FFFilter ff_vf_rotate; +extern const FFFilter ff_vf_rotate_ni_quadra; extern const FFFilter ff_vf_sab; extern const FFFilter ff_vf_scale; extern const FFFilter ff_vf_vpp_amf; extern const FFFilter ff_vf_sr_amf; extern const FFFilter ff_vf_scale_cuda; +extern const FFFilter ff_vf_scale_ni_quadra; extern const FFFilter ff_vf_scale_npp; extern const FFFilter ff_vf_scale_qsv; extern const FFFilter ff_vf_scale_vaapi; @@ -475,6 +484,7 @@ extern const FFFilter ff_vf_smartblur; extern const FFFilter ff_vf_sobel; extern const FFFilter ff_vf_sobel_opencl; extern const FFFilter ff_vf_split; +extern const FFFilter ff_vf_split_ni_quadra; extern const FFFilter ff_vf_spp; extern const FFFilter ff_vf_sr; extern const FFFilter ff_vf_ssim; @@ -519,6 +529,7 @@ extern const FFFilter ff_vf_vaguedenoiser; extern const FFFilter ff_vf_varblur; extern const FFFilter ff_vf_vectorscope; extern const FFFilter ff_vf_vflip; +extern const FFFilter ff_vf_flip_ni_quadra; extern const FFFilter ff_vf_vflip_vulkan; extern const FFFilter ff_vf_vfrdet; extern const FFFilter ff_vf_vibrance; diff --git a/libavfilter/nifilter.c b/libavfilter/nifilter.c new file mode 100644 index 0000000000..2008af6c6f --- /dev/null +++ b/libavfilter/nifilter.c @@ -0,0 +1,361 @@ +/* + * Copyright (c) 2020 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * video common filter routines + */ + +#include + +#include + +#include "avfilter.h" +#include "nifilter.h" +#include "formats.h" +#include "filters.h" +#include "libavutil/mem.h" +#include "video.h" +#include "libavutil/eval.h" +#include "libavutil/avstring.h" +#include "libavutil/internal.h" +#include "libavutil/libm.h" +#include "libavutil/imgutils.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" +#include "libavutil/time.h" + +#if HAVE_SYS_RESOURCE_H +#include +#include +#include +#elif HAVE_GETPROCESSTIMES +#include +#endif + +typedef struct BenchmarkTimeStamps { + int64_t real_usec; + int64_t user_usec; + int64_t sys_usec; +} BenchmarkTimeStamps; + +typedef struct gc620_pixel_fmts { + enum AVPixelFormat pix_fmt_ffmpeg; + int pix_fmt_gc620; + ni_pix_fmt_t pix_fmt_libxcoder; +} gc620_pixel_fmts_t; + +static struct gc620_pixel_fmts gc620_pixel_fmt_list[] =3D { + {AV_PIX_FMT_NV12, GC620_NV12, NI_PIX_FMT_NV12}, + {AV_PIX_FMT_NV21, GC620_NV21, NI_PIX_FMT_NONE}, + {AV_PIX_FMT_YUV420P, GC620_I420, NI_PIX_FMT_YUV420P}, + {AV_PIX_FMT_P010LE, GC620_P010_MSB, NI_PIX_FMT_P010LE}, + {AV_PIX_FMT_YUV420P10LE, GC620_I010, NI_PIX_FMT_YUV420P10LE}, + {AV_PIX_FMT_YUYV422, GC620_YUYV, NI_PIX_FMT_YUYV422}, + {AV_PIX_FMT_UYVY422, GC620_UYVY, NI_PIX_FMT_UYVY422}, + {AV_PIX_FMT_NV16, GC620_NV16, NI_PIX_FMT_NONE}, + {AV_PIX_FMT_RGBA, GC620_RGBA8888, NI_PIX_FMT_RGBA}, + {AV_PIX_FMT_BGR0, GC620_BGRX8888, NI_PIX_FMT_BGR0}, + {AV_PIX_FMT_BGRA, GC620_BGRA8888, NI_PIX_FMT_BGRA}, + {AV_PIX_FMT_ABGR, GC620_ABGR8888, NI_PIX_FMT_ABGR}, + {AV_PIX_FMT_ARGB, GC620_ARGB8888, NI_PIX_FMT_ARGB}, + {AV_PIX_FMT_BGR565LE, GC620_RGB565, NI_PIX_FMT_NONE}, + {AV_PIX_FMT_RGB565LE, GC620_BGR565, NI_PIX_FMT_NONE}, + {AV_PIX_FMT_RGB555LE, GC620_B5G5R5X1, NI_PIX_FMT_NONE}, + {AV_PIX_FMT_NI_QUAD_8_TILE_4X4, GC620_NV12, NI_PIX_FMT_NV12} +}; + +static BenchmarkTimeStamps get_benchmark_time_stamps(void) +{ + BenchmarkTimeStamps time_stamps =3D { av_gettime_relative() }; +#if HAVE_GETRUSAGE + struct rusage rusage; + + getrusage(RUSAGE_SELF, &rusage); + time_stamps.user_usec =3D + (rusage.ru_utime.tv_sec * 1000000LL) + rusage.ru_utime.tv_usec; + time_stamps.sys_usec =3D + (rusage.ru_stime.tv_sec * 1000000LL) + rusage.ru_stime.tv_usec; +#elif HAVE_GETPROCESSTIMES + HANDLE proc; + FILETIME c, e, k, u; + proc =3D GetCurrentProcess(); + GetProcessTimes(proc, &c, &e, &k, &u); + time_stamps.user_usec =3D + ((int64_t)u.dwHighDateTime << 32 | u.dwLowDateTime) / 10; + time_stamps.sys_usec =3D + ((int64_t)k.dwHighDateTime << 32 | k.dwLowDateTime) / 10; +#else + time_stamps.user_usec =3D time_stamps.sys_usec =3D 0; +#endif + return time_stamps; +} + +void ff_ni_update_benchmark(const char *fmt, ...) +{ + static BenchmarkTimeStamps current_time; + va_list va; + char buf[1024]; + BenchmarkTimeStamps t =3D get_benchmark_time_stamps(); + + if (fmt) { + va_start(va, fmt); + vsnprintf(buf, sizeof(buf), fmt, va); + va_end(va); + av_log(NULL, AV_LOG_INFO, "bench: %8" PRIu64 " user %8" PRIu64 " " + "sys %8" PRIu64 " real %s \n", + t.user_usec - current_time.user_usec, + t.sys_usec - current_time.sys_usec, + t.real_usec - current_time.real_usec, buf); + } + current_time =3D t; +} + +int ff_ni_ffmpeg_to_gc620_pix_fmt(enum AVPixelFormat pix_fmt) +{ + int i, tablesz; + + tablesz =3D sizeof(gc620_pixel_fmt_list)/sizeof(struct gc620_pixel_fmt= s); + + /* linear search through table to find if the pixel format is supporte= d */ + for (i =3D 0; i < tablesz; i++) { + if (gc620_pixel_fmt_list[i].pix_fmt_ffmpeg =3D=3D pix_fmt) { + return gc620_pixel_fmt_list[i].pix_fmt_gc620; + } + } + return -1; +} + +int ff_ni_ffmpeg_to_libxcoder_pix_fmt(enum AVPixelFormat pix_fmt) +{ + int i, tablesz; + + tablesz =3D sizeof(gc620_pixel_fmt_list)/sizeof(struct gc620_pixel_fmt= s); + + /* linear search through table to find if the pixel format is supporte= d */ + for (i =3D 0; i < tablesz; i++) { + if (gc620_pixel_fmt_list[i].pix_fmt_ffmpeg =3D=3D pix_fmt) { + return gc620_pixel_fmt_list[i].pix_fmt_libxcoder; + } + } + + return -1; +} + +int ff_ni_copy_device_to_host_frame(AVFrame *dst, const ni_frame_t *src, i= nt pix_fmt) +{ + switch (pix_fmt) + { + /* packed */ + case GC620_RGBA8888: + case GC620_BGRA8888: + case GC620_ABGR8888: + case GC620_ARGB8888: + case GC620_RGB565: + case GC620_BGR565: + case GC620_B5G5R5X1: + case GC620_YUYV: + memcpy(dst->data[0],src->p_data[0],src->data_len[0]); + break; + + /* semi-planar */ + case GC620_NV12: + case GC620_NV21: + case GC620_P010_MSB: + case GC620_NV16: + memcpy(dst->data[0], src->p_data[0], src->data_len[0]); + memcpy(dst->data[1], src->p_data[1], src->data_len[1]); + break; + + /* planar */ + case GC620_I420: + case GC620_I010: + memcpy(dst->data[0], src->p_data[0], src->data_len[0]); + memcpy(dst->data[1], src->p_data[1], src->data_len[1]); + memcpy(dst->data[2], src->p_data[2], src->data_len[2]); + break; + + default: + return -1; + } + + return 0; +} + +int ff_ni_copy_host_to_device_frame(ni_frame_t *dst, const AVFrame *src, i= nt pix_fmt) +{ + switch (pix_fmt) + { + /* packed */ + case GC620_RGBA8888: + case GC620_BGRA8888: + case GC620_ABGR8888: + case GC620_ARGB8888: + case GC620_RGB565: + case GC620_BGR565: + case GC620_B5G5R5X1: + case GC620_YUYV: + memcpy(dst->p_data[0], src->data[0], dst->data_len[0]); + dst->pixel_format =3D pix_fmt; + break; + + /* planar */ + case GC620_I420: + case GC620_I010: + memcpy(dst->p_data[0], src->data[0], dst->data_len[0]); + memcpy(dst->p_data[1], src->data[1], dst->data_len[1]); + memcpy(dst->p_data[2], src->data[2], dst->data_len[2]); + dst->pixel_format =3D pix_fmt; + break; + + /* semi-planar */ + case GC620_NV12: + case GC620_NV21: + case GC620_P010_MSB: + case GC620_NV16: + memcpy(dst->p_data[0], src->data[0], dst->data_len[0]); + memcpy(dst->p_data[0], src->data[0], dst->data_len[0]); + dst->pixel_format =3D pix_fmt; + break; + + default: + dst->pixel_format =3D -1; + return -1; + } + + return 0; +} + +void ff_ni_frame_free(void *opaque, uint8_t *data) +{ + int ret; + + if (data) + { + niFrameSurface1_t* p_data3 =3D (niFrameSurface1_t*)((uint8_t*)data); + if (p_data3->ui16FrameIdx !=3D 0) + { + av_log(NULL, AV_LOG_DEBUG, "Recycle trace ui16FrameIdx =3D [%d] DevH= andle %d\n", p_data3->ui16FrameIdx, p_data3->device_handle); + ret =3D ni_hwframe_buffer_recycle(p_data3, p_data3->device_handle); + if (ret !=3D NI_RETCODE_SUCCESS) + { + av_log(NULL, AV_LOG_ERROR, "ERROR Failed to recycle trace ui16Fram= eIdx =3D [%d] DevHandle %d\n", p_data3->ui16FrameIdx, p_data3->device_handl= e); + } + } + // buffer is created by av_malloc, so use av_free to release. + av_free(data); + } +}; + +int ff_ni_build_frame_pool(ni_session_context_t *ctx, + int width, int height, + enum AVPixelFormat out_format, + int pool_size, + int buffer_limit) +{ + int rc; + int scaler_format; + int options; + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(out_format); + options =3D NI_SCALER_FLAG_IO | NI_SCALER_FLAG_PC; + if (buffer_limit) + options |=3D NI_SCALER_FLAG_LM; + + /* Allocate a pool of frames by the scaler */ + rc =3D ni_device_alloc_frame(ctx, + FFALIGN(width,2), + FFALIGN(height,2), + scaler_format, + options, + 0, // rec width + 0, // rec height + 0, // rec X pos + 0, // rec Y pos + pool_size, // rgba color/pool size + 0, // frame index + NI_DEVICE_TYPE_SCALER); + + return rc; +} + +void ff_ni_set_bit_depth_and_encoding_type(int8_t *p_bit_depth, + int8_t *p_enc_type, + enum AVPixelFormat pix_fmt) +{ + + // bit depth is 1 for 8-bit format, 2 for 10-bit format + // encoding type should be 1 for planar or packed, 0 for semi-planar + + switch (pix_fmt) + { + case AV_PIX_FMT_YUV420P: + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D 1; // planar + break; + + case AV_PIX_FMT_YUV420P10LE: + *p_bit_depth =3D 2; // 10-bits per component + *p_enc_type =3D 1; // planar + break; + + case AV_PIX_FMT_NV12: + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D 0; // semi-planar + break; + case AV_PIX_FMT_NI_QUAD_8_TILE_4X4: + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D NI_PIXEL_PLANAR_FORMAT_TILED4X4; // semi-pla= nar + break; + case AV_PIX_FMT_P010LE: + *p_bit_depth =3D 2; // 10-bits per component + *p_enc_type =3D 0; // semi-planar + break; + + case AV_PIX_FMT_YUYV422: + case AV_PIX_FMT_UYVY422: + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D 1; // packed + break; + + case AV_PIX_FMT_NV16: + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D 0; // semi-planar + break; + + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_BGR0: + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D 1; // packed or planar + break; + + default: + av_log(NULL, AV_LOG_WARNING, "WARNING: unexpected pix format %= s\n", + av_get_pix_fmt_name(pix_fmt)); + + // use default values if we've supported a new pixel format + *p_bit_depth =3D 1; // 8-bits per component + *p_enc_type =3D 1; // planar or packed + break; + } +} diff --git a/libavfilter/nifilter.h b/libavfilter/nifilter.h new file mode 100644 index 0000000000..e92ef3d6a6 --- /dev/null +++ b/libavfilter/nifilter.h @@ -0,0 +1,69 @@ +/* + * XCoder Filter Lib Wrapper + * + * Copyright (c) 2020 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * XCoder codec lib wrapper. + */ + +#ifndef AVFILTER_NIFILTER_H +#define AVFILTER_NIFILTER_H + +#include "version.h" +#include "libavutil/pixdesc.h" +#include "libavutil/imgutils.h" +#include "libavutil/hwcontext.h" +#include "libavutil/hwcontext_ni_quad.h" +#include + +#define DEFAULT_NI_FILTER_POOL_SIZE 4 + +#define NI_FILT_OPTION_KEEPALIVE = \ + { "keep_alive_timeout", "Specify a custom session keep alive timeout i= n seconds.", \ + OFFSET(keep_alive_timeout), AV_OPT_TYPE_INT, {.i64 =3D NI_DEFAULT_KE= EP_ALIVE_TIMEOUT}, \ + NI_MIN_KEEP_ALIVE_TIMEOUT, NI_MAX_KEEP_ALIVE_TIMEOUT, FLAGS } + +#define NI_FILT_OPTION_KEEPALIVE10 = \ + { "keep_alive_timeout", "Specify a custom session keep alive timeout i= n seconds.", \ + OFFSET(keep_alive_timeout), AV_OPT_TYPE_INT, {.i64 =3D 10}, NI_MIN_K= EEP_ALIVE_TIMEOUT, \ + NI_MAX_KEEP_ALIVE_TIMEOUT, FLAGS } + +#define NI_FILT_OPTION_BUFFER_LIMIT = \ + { "buffer_limit", "Limit output buffering", OFFSET(buffer_limit), AV_O= PT_TYPE_BOOL, \ + {.i64 =3D 0}, 0, 1, FLAGS } + +#define NI_FILT_OPTION_AUTO_SKIP = \ + { "auto_skip", "skip processing when output would be same as input", O= FFSET(auto_skip), \ + AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS} + +void ff_ni_update_benchmark(const char *fmt, ...); +int ff_ni_ffmpeg_to_gc620_pix_fmt(enum AVPixelFormat pix_fmt); +int ff_ni_ffmpeg_to_libxcoder_pix_fmt(enum AVPixelFormat pix_fmt); +int ff_ni_copy_device_to_host_frame(AVFrame *dst, const ni_frame_t *src, i= nt pix_fmt); +int ff_ni_copy_host_to_device_frame(ni_frame_t *dst, const AVFrame *src, i= nt pix_fmt); +int ff_ni_build_frame_pool(ni_session_context_t *ctx,int width,int height,= enum AVPixelFormat out_format, int pool_size, int buffer_limit); +void ff_ni_frame_free(void *opaque, uint8_t *data); +void ff_ni_set_bit_depth_and_encoding_type(int8_t *p_bit_depth, + int8_t *p_enc_type, + enum AVPixelFormat pix_fmt); + +#endif diff --git a/libavfilter/vf_crop_ni.c b/libavfilter/vf_crop_ni.c new file mode 100644 index 0000000000..2d8683fb45 --- /dev/null +++ b/libavfilter/vf_crop_ni.c @@ -0,0 +1,719 @@ +/* + * Copyright (c) 2007 Bobby Bingham + * Copyright (c) 2020 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * video crop filter + */ + +#include + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "video.h" +#include "libavutil/eval.h" +#include "libavutil/avstring.h" +#include "libavutil/internal.h" +#include "libavutil/libm.h" +#include "libavutil/imgutils.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" + +#include + +static const char *const var_names[] =3D { + "in_w", "iw", ///< width of the input video + "in_h", "ih", ///< height of the input video + "out_w", "ow", ///< width of the cropped video + "out_h", "oh", ///< height of the cropped video + "a", + "sar", + "dar", + "hsub", + "vsub", + "x", + "y", + "n", ///< number of frame +#if FF_API_FRAME_PKT + "pos", ///< position in the file +#endif + "t", ///< timestamp expressed in seconds + NULL +}; + +enum var_name { + VAR_IN_W, VAR_IW, + VAR_IN_H, VAR_IH, + VAR_OUT_W, VAR_OW, + VAR_OUT_H, VAR_OH, + VAR_A, + VAR_SAR, + VAR_DAR, + VAR_HSUB, + VAR_VSUB, + VAR_X, + VAR_Y, + VAR_N, +#if FF_API_FRAME_PKT + VAR_POS, +#endif + VAR_T, + VAR_VARS_NB +}; + +typedef struct NetIntCropContext { + const AVClass *class; + int x; ///< x offset of the non-cropped area with respect= to the input area + int y; ///< y offset of the non-cropped area with respect= to the input area + int w; ///< width of the cropped area + int h; ///< height of the cropped area + + AVRational out_sar; ///< output sample aspect ratio + int keep_aspect; ///< keep display aspect ratio when cropping + + int max_step[4]; ///< max pixel step for each plane, expressed as a= number of bytes + int hsub, vsub; ///< chroma subsampling + char *x_expr, *y_expr, *w_expr, *h_expr; + AVExpr *x_pexpr, *y_pexpr; /* parsed expressions for x and y */ + double var_values[VAR_VARS_NB]; + + AVBufferRef *out_frames_ref; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + + int initialized; + int session_opened; + int keep_alive_timeout; /* keep alive timeout setting */ + + int auto_skip; + int skip_filter; + int buffer_limit; +} NetIntCropContext; + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D + {AV_PIX_FMT_NI_QUAD, AV_PIX_FMT_NONE}; + AVFilterFormats *formats; + + formats =3D ff_make_format_list(pix_fmts); + + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntCropContext *s =3D ctx->priv; + + av_expr_free(s->x_pexpr); + s->x_pexpr =3D NULL; + av_expr_free(s->y_pexpr); + s->y_pexpr =3D NULL; + + if (s->api_dst_frame.data.frame.p_buffer) + ni_frame_buffer_free(&s->api_dst_frame.data.frame); + + if (s->session_opened) { + /* Close operation will free the device frames */ + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + } + + av_buffer_unref(&s->out_frames_ref); +} + +static inline int normalize_double(int *n, double d) +{ + int ret =3D 0; + + if (isnan(d)) { + ret =3D AVERROR(EINVAL); + } else if (d > INT_MAX || d < INT_MIN) { + *n =3D d > INT_MAX ? INT_MAX : INT_MIN; + ret =3D AVERROR(EINVAL); + } else { + *n =3D (int)lrint(d); + } + + return ret; +} + +static int config_input(AVFilterLink *link) +{ + AVFilterContext *ctx =3D link->dst; + AVHWFramesContext *hwctx; + NetIntCropContext *s; + const AVPixFmtDescriptor *pix_desc; + int ret; + const char *expr; + double res; + FilterLink *li =3D ff_filter_link(link); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + hwctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + s =3D ctx->priv; + pix_desc =3D av_pix_fmt_desc_get(hwctx->sw_format); + + s->var_values[VAR_IN_W] =3D s->var_values[VAR_IW] =3D ctx->inputs[0]-= >w; + s->var_values[VAR_IN_H] =3D s->var_values[VAR_IH] =3D ctx->inputs[0]-= >h; + s->var_values[VAR_A] =3D (double)link->w / (double)link->h; + s->var_values[VAR_SAR] =3D link->sample_aspect_ratio.num ? av_q2d(li= nk->sample_aspect_ratio) : 1; + s->var_values[VAR_DAR] =3D s->var_values[VAR_A] * s->var_values[VAR_= SAR]; + s->var_values[VAR_HSUB] =3D 1<log2_chroma_w; + s->var_values[VAR_VSUB] =3D 1<log2_chroma_h; + s->var_values[VAR_X] =3D NAN; + s->var_values[VAR_Y] =3D NAN; + s->var_values[VAR_OUT_W] =3D s->var_values[VAR_OW] =3D NAN; + s->var_values[VAR_OUT_H] =3D s->var_values[VAR_OH] =3D NAN; + s->var_values[VAR_N] =3D 0; + s->var_values[VAR_T] =3D NAN; +#if FF_API_FRAME_PKT + s->var_values[VAR_POS] =3D NAN; +#endif + + av_image_fill_max_pixsteps(s->max_step, NULL, pix_desc); + s->hsub =3D pix_desc->log2_chroma_w; + s->vsub =3D pix_desc->log2_chroma_h; + + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->w_expr), + var_names, s->var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto fail_expr; + s->var_values[VAR_OUT_W] =3D s->var_values[VAR_OW] =3D res; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->h_expr), + var_names, s->var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto fail_expr; + s->var_values[VAR_OUT_H] =3D s->var_values[VAR_OH] =3D res; + /* evaluate again ow as it may depend on oh */ + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->w_expr), + var_names, s->var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto fail_expr; + + s->var_values[VAR_OUT_W] =3D s->var_values[VAR_OW] =3D res; + if (normalize_double(&s->w, s->var_values[VAR_OUT_W]) < 0 || + normalize_double(&s->h, s->var_values[VAR_OUT_H]) < 0) { + av_log(ctx, AV_LOG_ERROR, + "Too big value or invalid expression for out_w/ow or out_h/= oh. " + "Maybe the expression for out_w:'%s' or for out_h:'%s' is s= elf-referencing.\n", + s->w_expr, s->h_expr); + return AVERROR(EINVAL); + } + + s->w &=3D ~((1 << s->hsub) - 1); + s->h &=3D ~((1 << s->vsub) - 1); + + av_expr_free(s->x_pexpr); + s->x_pexpr =3D NULL; + av_expr_free(s->y_pexpr); + s->y_pexpr =3D NULL; + if ((av_expr_parse(&s->x_pexpr, s->x_expr, var_names, NULL, NULL, NULL, + NULL, 0, ctx) < 0) || + (av_expr_parse(&s->y_pexpr, s->y_expr, var_names, NULL, NULL, NULL, + NULL, 0, ctx) < 0)) + return AVERROR(EINVAL); + + if (s->keep_aspect) { + AVRational dar =3D av_mul_q(link->sample_aspect_ratio, + (AVRational){ link->w, link->h }); + av_reduce(&s->out_sar.num, &s->out_sar.den, + dar.num * s->h, dar.den * s->w, INT_MAX); + } else { + s->out_sar =3D link->sample_aspect_ratio; + } + + av_log(ctx, AV_LOG_VERBOSE, "w:%d h:%d sar:%d/%d -> w:%d h:%d sar:%d/%= d\n", + link->w, link->h, link->sample_aspect_ratio.num, link->sample_a= spect_ratio.den, + s->w, s->h, s->out_sar.num, s->out_sar.den); + + if (s->w <=3D 0 || s->h <=3D 0 || + s->w > link->w || s->h > link->h) { + av_log(ctx, AV_LOG_ERROR, + "Invalid too big or non positive size for width '%d' or hei= ght '%d'\n", + s->w, s->h); + return AVERROR(EINVAL); + } + + /* set default, required in the case the first computed value for x/y = is NAN */ + s->x =3D (link->w - s->w) / 2; + s->y =3D (link->h - s->h) / 2; + + s->x &=3D ~((1 << s->hsub) - 1); + s->y &=3D ~((1 << s->vsub) - 1); + + return 0; + +fail_expr: + av_log(NULL, AV_LOG_ERROR, "Error when evaluating the expression '%s'\= n", expr); + return ret; +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntCropContext *s =3D ctx->priv; + AVHWFramesContext *out_frames_ctx; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_ctx =3D (AVHWFramesContext*)s->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + s->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&s->api_ctx, out_frames_ctx->width, + out_frames_ctx->height, + out_frames_ctx->sw_format, pool_size, + s->buffer_limit); +} + +static int config_output(AVFilterLink *link) +{ + NetIntCropContext *s =3D link->src->priv; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + AVFilterContext *ctx =3D (AVFilterContext *)link->src;; + + link->w =3D s->w; + link->h =3D s->h; + link->sample_aspect_ratio =3D s->out_sar; + + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X4 || + in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 not supported\n"); + return AVERROR(EINVAL); + } + + //skip the color range check + if (s->auto_skip && + (s->x_expr && strcmp(s->x_expr, "0") =3D=3D 0 && s->y_expr && strc= mp(s->y_expr, "0") =3D=3D 0) && + (in_frames_ctx->width =3D=3D link->w && in_frames_ctx->height =3D= =3D link->h) + ) { + //skip hardware crop + s->skip_filter =3D 1; + + FilterLink *lo =3D ff_filter_link(link); + s->out_frames_ref =3D av_buffer_ref(li->hw_frames_ctx); + if (!s->out_frames_ref) { + return AVERROR(ENOMEM); + } + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + return 0; + } + + s->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_ref); + if (!s->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D s->w; + out_frames_ctx->height =3D s->h; + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + out_frames_ctx->initial_pool_size =3D + NI_CROP_ID; // Repurposed as identity code + + av_hwframe_ctx_init(s->out_frames_ref); + + FilterLink *lo =3D ff_filter_link(link); + av_buffer_unref(&lo->hw_frames_ctx); + + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + return 0; +} + +static int filter_frame(AVFilterLink *link, AVFrame *frame) +{ + AVFilterContext *ctx =3D link->dst; + NetIntCropContext *s =3D ctx->priv; + AVFilterLink *outlink =3D link->dst->outputs[0]; + AVFrame *out =3D NULL; + niFrameSurface1_t* frame_surface,*new_frame_surface; + AVHWFramesContext *pAVHFWCtx; + AVNIDeviceContext *pAVNIDevCtx; + ni_retcode_t retcode; + uint32_t scaler_format; + int cardno; + uint16_t tempFID; + + pAVHFWCtx =3D (AVHWFramesContext *)frame->hw_frames_ctx->data; + if (!pAVHFWCtx) { + return AVERROR(EINVAL); + } + + pAVNIDevCtx =3D (AVNIDeviceContext *)pAVHFWCtx->device_ctx->hwctx; + if (!pAVNIDevCtx) { + return AVERROR(EINVAL); + } + + cardno =3D ni_get_cardno(frame); + + if (s->skip_filter) { + //skip hardware crop + return ff_filter_frame(link->dst->outputs[0], frame); + } + + if (!s->initialized) { + retcode =3D ni_device_session_context_init(&s->api_ctx); + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "ni crop filter session context init failure\n"); + goto fail; + } + + s->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + s->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + + s->api_ctx.hw_id =3D cardno; + s->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_CROP; + s->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retcode =3D ni_device_session_open(&s->api_ctx, NI_DEVICE_TYPE_SCA= LER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't open device session on card %= d\n", + cardno); + + /* Close operation will free the device frames */ + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + goto fail; + } + + s->session_opened =3D 1; + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + ctx->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SIZE > 1) = ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + retcode =3D init_out_pool(ctx); + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", retcod= e); + goto fail; + } + + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)s->out_= frames_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_frames_= ctx->hwctx; + ni_cpy_hwframe_ctx(pAVHFWCtx, out_frames_ctx); + ni_device_session_copy(&s->api_ctx, &out_ni_ctx->api_ctx); + + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(pAVHFWCtx->= sw_format); + + if ((frame->color_range =3D=3D AVCOL_RANGE_JPEG) && !(desc->flags = & AV_PIX_FMT_FLAG_RGB)) { + av_log(link->dst, AV_LOG_WARNING, + "WARNING: Full color range input, limited color range o= utput\n"); + } + + s->initialized =3D 1; + } + + FilterLink *l =3D ff_filter_link(link); + s->var_values[VAR_N] =3D l->frame_count_out; + s->var_values[VAR_T] =3D frame->pts =3D=3D AV_NOPTS_VALUE ? + NAN : frame->pts * av_q2d(link->time_base); + s->var_values[VAR_X] =3D av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] =3D av_expr_eval(s->y_pexpr, s->var_values, NULL); + s->var_values[VAR_X] =3D av_expr_eval(s->x_pexpr, s->var_values, NULL); + + normalize_double(&s->x, s->var_values[VAR_X]); + normalize_double(&s->y, s->var_values[VAR_Y]); + + if (s->x < 0) + s->x =3D 0; + if (s->y < 0) + s->y =3D 0; + if ((unsigned)s->x + (unsigned)s->w > link->w) + s->x =3D link->w - s->w; + if ((unsigned)s->y + (unsigned)s->h > link->h) + s->y =3D link->h - s->h; + + s->x &=3D ~((1 << s->hsub) - 1); + s->y &=3D ~((1 << s->vsub) - 1); + + av_log(ctx, AV_LOG_TRACE, "n:%d t:%f x:%d y:%d x+w:%d y+h:%d\n", + (int)s->var_values[VAR_N], s->var_values[VAR_T], s->x, s->y, + s->x+s->w, s->y+s->h); + + frame_surface =3D (niFrameSurface1_t *) frame->data[3]; + if (frame_surface =3D=3D NULL) { + retcode =3D AVERROR(EINVAL); + goto fail; + } + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + /* + * Allocate device input frame. This call won't actually allocate a fr= ame, + * but sends the incoming hardware frame index to the scaler manager + */ + retcode =3D ni_device_alloc_frame(&s->api_ctx, + FFALIGN(frame->width, 2), + FFALIGN(frame->height, 2), + scaler_format, + 0, // input frame + s->w, // src rectangle width + s->h, // src rectangle height + s->x, // src rectangle x + s->y, // src rectangle y + frame_surface->ui32nodeAddress, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't assign input frame %d\n", + retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Allocate device destination frame This will acquire a frame from th= e pool */ + retcode =3D ni_device_alloc_frame(&s->api_ctx, + FFALIGN(outlink->w,2), + FFALIGN(outlink->h,2), + scaler_format, + NI_SCALER_FLAG_IO, + 0, + 0, + 0, + 0, + 0, + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't allocate device output frame %d\n= ", + retcode); + + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + out =3D av_frame_alloc(); + if (!out) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + av_frame_copy_props(out,frame); + + out->width =3D s->w; + out->height =3D s->h; + + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + + if (!out->data[3]) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Copy the frame surface from the incoming frame */ + memcpy(out->data[3], frame->data[3], sizeof(niFrameSurface1_t)); + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, &s->api_dst_fra= me, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't acquire output frame %d\n",retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_crop"); +#endif + + tempFID =3D frame_surface->ui16FrameIdx; + frame_surface =3D (niFrameSurface1_t *) out->data[3]; + new_frame_surface =3D (niFrameSurface1_t *) s->api_dst_frame.data.fram= e.p_data[3]; + frame_surface->ui16FrameIdx =3D new_frame_surface->ui16FrameIdx; + frame_surface->ui16session_ID =3D new_frame_surface->ui16session_ID; + frame_surface->device_handle =3D (int32_t)pAVNIDevCtx->cards[cardno]; + frame_surface->output_idx =3D new_frame_surface->output_idx; + frame_surface->src_cpu =3D new_frame_surface->src_cpu; + frame_surface->dma_buf_fd =3D 0; + + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + pAVHFWCtx->sw_format); + + /* Remove ni-split specific assets */ + frame_surface->ui32nodeAddress =3D 0; + + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + av_log(ctx, AV_LOG_DEBUG, + "vf_crop_ni.c:IN trace ui16FrameIdx =3D [%d] --> out =3D [%d] \= n", + tempFID, frame_surface->ui16FrameIdx); + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurface1_= t), ff_ni_frame_free, NULL, 0); + + av_frame_free(&frame); + + return ff_filter_frame(link->dst->outputs[0], out); + +fail: + av_frame_free(&frame); + if (out) + av_frame_free(&out); + return retcode; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntCropContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %u available_frame %d outlink framequeue %u frame_wanted %d - r= eturn NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntCropContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption ni_crop_options[] =3D { + { "out_w", "set the width crop area expression", OFFSET(w_expr)= , AV_OPT_TYPE_STRING, {.str =3D "iw"}, .flags =3D FLAGS }, + { "w", "set the width crop area expression", OFFSET(w_expr)= , AV_OPT_TYPE_STRING, {.str =3D "iw"}, .flags =3D FLAGS }, + { "out_h", "set the height crop area expression", OFFSET(h_expr)= , AV_OPT_TYPE_STRING, {.str =3D "ih"}, .flags =3D FLAGS }, + { "h", "set the height crop area expression", OFFSET(h_expr)= , AV_OPT_TYPE_STRING, {.str =3D "ih"}, .flags =3D FLAGS }, + { "x", "set the x crop area expression", OFFSET(x_expr)= , AV_OPT_TYPE_STRING, {.str =3D "(in_w-out_w)/2"}, .flags =3D FLAGS }, + { "y", "set the y crop area expression", OFFSET(y_expr)= , AV_OPT_TYPE_STRING, {.str =3D "(in_h-out_h)/2"}, .flags =3D FLAGS }, + { "keep_aspect", "keep aspect ratio", OFFSET(keep_as= pect), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + NI_FILT_OPTION_AUTO_SKIP, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_crop); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + .config_props =3D config_input, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_output, + }, +}; + +FFFilter ff_vf_crop_ni_quadra =3D { + .p.name =3D "ni_quadra_crop", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra crop the input video v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_crop_class, + .priv_size =3D sizeof(NetIntCropContext), + .uninit =3D uninit, + .activate =3D activate, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +}; diff --git a/libavfilter/vf_drawbox_ni.c b/libavfilter/vf_drawbox_ni.c new file mode 100644 index 0000000000..601e3a49d3 --- /dev/null +++ b/libavfilter/vf_drawbox_ni.c @@ -0,0 +1,831 @@ +/* + * Copyright (c) 2007 Bobby Bingham + * Copyright (c) 2020 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * drawbox video filter + */ + +#include +#include + +#include "nifilter.h" +#include "version.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "scale_eval.h" +#include "video.h" +#include "libavutil/avstring.h" +#include "libavutil/internal.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" +#include "libavutil/eval.h" +#include "libavutil/parseutils.h" +#include "libavutil/avassert.h" +#include "libswscale/swscale.h" + +enum OutputFormat { + OUTPUT_FORMAT_YUV420P, + OUTPUT_FORMAT_YUYV422, + OUTPUT_FORMAT_UYVY422, + OUTPUT_FORMAT_NV12, + OUTPUT_FORMAT_ARGB, + OUTPUT_FORMAT_RGBA, + OUTPUT_FORMAT_ABGR, + OUTPUT_FORMAT_BGRA, + OUTPUT_FORMAT_YUV420P10LE, + OUTPUT_FORMAT_NV16, + OUTPUT_FORMAT_BGR0, + OUTPUT_FORMAT_P010LE, + OUTPUT_FORMAT_AUTO, + OUTPUT_FORMAT_NB +}; + +static const char *const var_names[] =3D { + "dar", + "in_h", "ih", ///< height of the input video + "in_w", "iw", ///< width of the input video + "sar", + "x", + "y", + "h", ///< height of the rendered box + "w", ///< width of the rendered box + "fill", + NULL +}; + +enum { R, G, B, A }; + +enum var_name { + VAR_DAR, + VAR_IN_H, VAR_IH, + VAR_IN_W, VAR_IW, + VAR_SAR, + VAR_X, + VAR_Y, + VAR_H, + VAR_W, + VAR_MAX, + VARS_NB +}; + +typedef struct NetIntDrawBoxContext { + const AVClass *class; + + /** + * New dimensions. Special values are: + * 0 =3D original width/height + * -1 =3D keep original aspect + * -N =3D try to keep aspect but make sure it is divisible by N + */ + int w, h; + int box_x[NI_MAX_SUPPORT_DRAWBOX_NUM], box_y[NI_MAX_SUPPORT_DRAWBOX_NU= M], box_w[NI_MAX_SUPPORT_DRAWBOX_NUM], box_h[NI_MAX_SUPPORT_DRAWBOX_NUM]; + unsigned char box_rgba_color[NI_MAX_SUPPORT_DRAWBOX_NUM][4]; + ni_scaler_multi_drawbox_params_t scaler_drawbox_paras; + char *size_str; + + char *box_x_expr[NI_MAX_SUPPORT_DRAWBOX_NUM]; + char *box_y_expr[NI_MAX_SUPPORT_DRAWBOX_NUM]; + char *box_w_expr[NI_MAX_SUPPORT_DRAWBOX_NUM]; + char *box_h_expr[NI_MAX_SUPPORT_DRAWBOX_NUM]; + char *box_color_str[NI_MAX_SUPPORT_DRAWBOX_NUM]; + + int format; + + enum AVPixelFormat out_format; + AVBufferRef *out_frames_ref; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + ni_scaler_params_t params; + + int initialized; + int session_opened; + int keep_alive_timeout; /* keep alive timeout setting */ + int inplace; + int buffer_limit; + + ni_frame_config_t frame_in; + ni_frame_config_t frame_out; +} NetIntDrawBoxContext; + +FFFilter ff_vf_drawbox_ni; + +static const int NUM_EXPR_EVALS =3D 4; + +static av_cold int init(AVFilterContext *ctx) +{ + NetIntDrawBoxContext *drawbox =3D ctx->priv; + + uint8_t rgba_color[4]; + + if (av_parse_color(rgba_color, drawbox->box_color_str[0], -1, ctx) < 0) + return AVERROR(EINVAL); + + drawbox->box_rgba_color[0][R] =3D rgba_color[0]; + drawbox->box_rgba_color[0][G] =3D rgba_color[1]; + drawbox->box_rgba_color[0][B] =3D rgba_color[2]; + drawbox->box_rgba_color[0][A] =3D rgba_color[3]; + + return 0; +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx =3D inlink->dst; + NetIntDrawBoxContext *s =3D ctx->priv; + double var_values[VARS_NB], res; + char *expr; + int ret; + int i; + + var_values[VAR_IN_H] =3D var_values[VAR_IH] =3D inlink->h; + var_values[VAR_IN_W] =3D var_values[VAR_IW] =3D inlink->w; + var_values[VAR_SAR] =3D inlink->sample_aspect_ratio.num ? av_q2d(inli= nk->sample_aspect_ratio) : 1; + var_values[VAR_DAR] =3D (double)inlink->w / inlink->h * var_values[VA= R_SAR]; + var_values[VAR_X] =3D NAN; + var_values[VAR_Y] =3D NAN; + var_values[VAR_H] =3D NAN; + var_values[VAR_W] =3D NAN; + + for (i =3D 0; i < NI_MAX_SUPPORT_DRAWBOX_NUM; i++) { + /* evaluate expressions, fail on last iteration */ + var_values[VAR_MAX] =3D inlink->w; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->box_x_expr[= i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= ctx)) < 0) + goto fail; + s->box_x[i] =3D var_values[VAR_X] =3D ((res < var_values[VAR_MAX])= ? res : (var_values[VAR_MAX] - 1)); + + var_values[VAR_MAX] =3D inlink->h; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->box_y_expr[= i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= ctx)) < 0) + goto fail; + s->box_y[i] =3D var_values[VAR_Y] =3D ((res < var_values[VAR_MAX])= ? res : (var_values[VAR_MAX] - 1)); + + var_values[VAR_MAX] =3D inlink->w - s->box_x[i]; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->box_w_expr[= i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= ctx)) < 0) + goto fail; + s->box_w[i] =3D var_values[VAR_W] =3D ((res < var_values[VAR_MAX])= ? res : var_values[VAR_MAX]); + + var_values[VAR_MAX] =3D inlink->h - s->box_y[i]; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->box_h_expr[= i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= ctx)) < 0) + goto fail; + s->box_h[i] =3D var_values[VAR_H] =3D ((res < var_values[VAR_MAX])= ? res : var_values[VAR_MAX]); + + /* if w or h are zero, use the input w/h */ + s->box_w[i] =3D (s->box_w[i] > 0) ? s->box_w[i] : inlink->w; + s->box_h[i] =3D (s->box_h[i] > 0) ? s->box_h[i] : inlink->h; + + /* sanity check width and height */ + if (s->box_w[i] < 0 || s->box_h[i] < 0) { + av_log(ctx, AV_LOG_ERROR, "Size values less than 0 are not acc= eptable.\n"); + return AVERROR(EINVAL); + } + av_log(ctx, AV_LOG_VERBOSE, "%d: x:%d y:%d w:%d h:%d color:0x%02X%= 02X%02X%02X\n", + i, s->box_x[i], s->box_y[i], s->box_w[i], s->box_h[i], + s->box_rgba_color[0][R], s->box_rgba_color[0][G], s->box_rgba_= color[0][B], s->box_rgba_color[0][A]); + } + + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + + return 0; + +fail: + av_log(ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s'.\n", + expr); + return ret; +} + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D + {AV_PIX_FMT_NI_QUAD, AV_PIX_FMT_NONE}; + AVFilterFormats *formats; + + formats =3D ff_make_format_list(pix_fmts); + + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntDrawBoxContext *drawbox =3D ctx->priv; + + if (drawbox->api_dst_frame.data.frame.p_buffer) + ni_frame_buffer_free(&drawbox->api_dst_frame.data.frame); + + if (drawbox->session_opened) { + /* Close operation will free the device frames */ + ni_device_session_close(&drawbox->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&drawbox->api_ctx); + } + + av_buffer_unref(&drawbox->out_frames_ref); +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntDrawBoxContext *s =3D ctx->priv; + AVHWFramesContext *out_frames_ctx; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_ctx =3D (AVHWFramesContext*)s->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + s->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&s->api_ctx, out_frames_ctx->width, + out_frames_ctx->height, s->out_format, + pool_size, + s->buffer_limit); +} + +static int config_props(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink0 =3D outlink->src->inputs[0]; + AVFilterLink *inlink =3D outlink->src->inputs[0]; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + NetIntDrawBoxContext *drawbox =3D ctx->priv; + int w, h, ret, h_shift, v_shift; + + if ((ret =3D ff_scale_eval_dimensions(ctx, + "iw", "ih", + inlink, outlink, + &w, &h)) < 0) + goto fail; + + /* Note that force_original_aspect_ratio may overwrite the previous set + * dimensions so that it is not divisible by the set factors anymore + * unless force_divisible_by is defined as well */ + + if (w > NI_MAX_RESOLUTION_WIDTH || h > NI_MAX_RESOLUTION_HEIGHT) { + av_log(ctx, AV_LOG_ERROR, "DrawBox value (%dx%d) > 8192 not allowe= d\n", w, h); + return AVERROR(EINVAL); + } + + if ((w <=3D 0) || (h <=3D 0)) { + av_log(ctx, AV_LOG_ERROR, "DrawBox value (%dx%d) not allowed\n", w= , h); + return AVERROR(EINVAL); + } + + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + /* Set the output format */ + drawbox->out_format =3D in_frames_ctx->sw_format; + + av_pix_fmt_get_chroma_sub_sample(drawbox->out_format, &h_shift, &v_shi= ft); + + outlink->w =3D FFALIGN(w, (1 << h_shift)); + outlink->h =3D FFALIGN(h, (1 << v_shift)); + + if (inlink0->sample_aspect_ratio.num) { + outlink->sample_aspect_ratio =3D av_mul_q((AVRational){outlink->h = * inlink0->w, outlink->w * inlink0->h}, inlink0->sample_aspect_ratio); + } else { + outlink->sample_aspect_ratio =3D inlink0->sample_aspect_ratio; + + } + + av_log(ctx, AV_LOG_VERBOSE, + "w:%d h:%d fmt:%s sar:%d/%d -> w:%d h:%d fmt:%s sar:%d/%d\n", + inlink->w, inlink->h, av_get_pix_fmt_name(inlink->format), + inlink->sample_aspect_ratio.num, inlink->sample_aspect_ratio.de= n, + outlink->w, outlink->h, av_get_pix_fmt_name(outlink->format), + outlink->sample_aspect_ratio.num, outlink->sample_aspect_ratio.= den); + + drawbox->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device= _ref); + if (!drawbox->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)drawbox->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D outlink->w; + out_frames_ctx->height =3D outlink->h; + out_frames_ctx->sw_format =3D drawbox->out_format; + out_frames_ctx->initial_pool_size =3D + NI_DRAWBOX_ID; // Repurposed as identity code + + av_hwframe_ctx_init(drawbox->out_frames_ref); + + FilterLink *lo =3D ff_filter_link(ctx->outputs[0]); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(drawbox->out_frames_ref); + + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + return 0; + +fail: + return ret; +} + +/* Process a received frame */ +static int filter_frame(AVFilterLink *link, AVFrame *in) +{ + NetIntDrawBoxContext *drawbox =3D link->dst->priv; + AVFilterLink *outlink =3D link->dst->outputs[0]; + AVFrame *out =3D NULL; + niFrameSurface1_t* frame_surface,*new_frame_surface; + AVHWFramesContext *pAVHFWCtx,*out_frames_ctx; + AVNIDeviceContext *pAVNIDevCtx; + AVNIFramesContext *out_ni_ctx; + ni_retcode_t retcode; + int drawbox_format, cardno; + uint16_t tempFID; + double var_values[VARS_NB], res; + char *expr; + int ret; + int i; + uint32_t box_count =3D 0; + const AVPixFmtDescriptor *desc; + + frame_surface =3D (niFrameSurface1_t *) in->data[3]; + if (frame_surface =3D=3D NULL) { + return AVERROR(EINVAL); + } + + pAVHFWCtx =3D (AVHWFramesContext *) in->hw_frames_ctx->data; + pAVNIDevCtx =3D (AVNIDeviceContext *)pAVHFWCtx->device_ctx->hwct= x; + cardno =3D ni_get_cardno(in); + + if (!drawbox->initialized) { + retcode =3D ni_device_session_context_init(&drawbox->api_ctx); + if (retcode < 0) { + av_log(link->dst, AV_LOG_ERROR, + "ni drawbox filter session context init failure\n"); + goto fail; + } + + drawbox->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + drawbox->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + + drawbox->api_ctx.hw_id =3D cardno; + drawbox->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + drawbox->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_DRAWBOX; + drawbox->api_ctx.keep_alive_timeout =3D drawbox->keep_alive_timeou= t; + + av_log(link->dst, AV_LOG_INFO, + "Open drawbox session to card %d, hdl %d, blk_hdl %d\n", ca= rdno, + drawbox->api_ctx.device_handle, drawbox->api_ctx.blk_io_han= dle); + + retcode =3D + ni_device_session_open(&drawbox->api_ctx, NI_DEVICE_TYPE_SCALE= R); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_ERROR, + "Can't open device session on card %d\n", cardno); + + /* Close operation will free the device frames */ + ni_device_session_close(&drawbox->api_ctx, 1, NI_DEVICE_TYPE_S= CALER); + ni_device_session_context_clear(&drawbox->api_ctx); + goto fail; + } + + drawbox->session_opened =3D 1; + + if (drawbox->params.filterblit) { + retcode =3D ni_scaler_set_params(&drawbox->api_ctx, &(drawbox-= >params)); + if (retcode < 0) + goto fail; + } + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + link->dst->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SIZE= > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + retcode =3D init_out_pool(link->dst); + + if (retcode < 0) { + av_log(link->dst, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", retcod= e); + goto fail; + } + + out_frames_ctx =3D (AVHWFramesContext *)drawbox->out_frames_ref->d= ata; + out_ni_ctx =3D (AVNIFramesContext *)out_frames_ctx->hwctx; + ni_cpy_hwframe_ctx(pAVHFWCtx, out_frames_ctx); + ni_device_session_copy(&drawbox->api_ctx, &out_ni_ctx->api_ctx); + + desc =3D av_pix_fmt_desc_get(pAVHFWCtx->sw_format); + + if ((in->color_range =3D=3D AVCOL_RANGE_JPEG) && !(desc->flags & A= V_PIX_FMT_FLAG_RGB)) { + av_log(link->dst, AV_LOG_WARNING, + "WARNING: Full color range input, limited color range o= utput\n"); + } + + drawbox->initialized =3D 1; + } + + drawbox_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + + retcode =3D ni_frame_buffer_alloc_hwenc(&drawbox->api_dst_frame.data.f= rame, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + var_values[VAR_IN_H] =3D var_values[VAR_IH] =3D link->h; + var_values[VAR_IN_W] =3D var_values[VAR_IW] =3D link->w; + var_values[VAR_X] =3D NAN; + var_values[VAR_Y] =3D NAN; + var_values[VAR_H] =3D NAN; + var_values[VAR_W] =3D NAN; + + memset(&drawbox->scaler_drawbox_paras, 0, sizeof(drawbox->scaler_drawb= ox_paras)); + for (i =3D 0; i < NI_MAX_SUPPORT_DRAWBOX_NUM; i++) { + /* evaluate expressions, fail on last iteration */ + var_values[VAR_MAX] =3D link->w; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D drawbox->box_x= _expr[i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= link->dst)) < 0) + goto fail; + drawbox->box_x[i] =3D var_values[VAR_X] =3D ((res < var_values[VAR= _MAX]) ? ((res < 0) ? 0 : res) : (var_values[VAR_MAX] - 1)); + + var_values[VAR_MAX] =3D link->h; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D drawbox->box_y= _expr[i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= link->dst)) < 0) + goto fail; + drawbox->box_y[i] =3D var_values[VAR_Y] =3D ((res < var_values[VAR= _MAX]) ? ((res < 0) ? 0 : res) : (var_values[VAR_MAX] - 1)); + + var_values[VAR_MAX] =3D link->w - drawbox->box_x[i]; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D drawbox->box_w= _expr[i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= link->dst)) < 0) + goto fail; + drawbox->box_w[i] =3D var_values[VAR_W] =3D ((res < var_values[VAR= _MAX]) ? res : var_values[VAR_MAX]); + drawbox->box_w[i] =3D (drawbox->box_w[i] >=3D 0) ? drawbox->box_w[= i] : var_values[VAR_MAX]; + + var_values[VAR_MAX] =3D link->h - drawbox->box_y[i]; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D drawbox->box_h= _expr[i]), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0,= link->dst)) < 0) + goto fail; + drawbox->box_h[i] =3D var_values[VAR_H] =3D ((res < var_values[VAR= _MAX]) ? res : var_values[VAR_MAX]); + + drawbox->box_h[i] =3D (drawbox->box_h[i] >=3D 0) ? drawbox->box_h[= i] : var_values[VAR_MAX]; + /* sanity check width and height */ + if (drawbox->box_w[i] < 0 || drawbox->box_h[i] < 0) { + av_log(link->dst, AV_LOG_ERROR, "Size values less than 0 are n= ot acceptable.\n"); + return AVERROR(EINVAL); + } + + // please use drawbox->scaler_drawbox_paras to pass draw param= eters + av_log(link->dst, AV_LOG_DEBUG,"%d: x %d, y %d, w %d, h %d, color = %x\n", \ + i, drawbox->box_x[i], drawbox->box_y[i], drawbox->box_w[i], dr= awbox->box_h[i], \ + drawbox->box_rgba_color[i][0] + (drawbox->box_rgba_color[i][1]= << 8) + (drawbox->box_rgba_color[i][2] << 16) + (drawbox->box_rgba_color[i= ][3] << 24)); + + if ((drawbox->box_w[i] > 0) && (drawbox->box_h[i] > 0)) { + drawbox->scaler_drawbox_paras.multi_drawbox_params[box_count].= start_x =3D drawbox->box_x[i]; + drawbox->scaler_drawbox_paras.multi_drawbox_params[box_count].= start_y =3D drawbox->box_y[i]; + drawbox->scaler_drawbox_paras.multi_drawbox_params[box_count].= end_x =3D drawbox->box_x[i] + drawbox->box_w[i] - 1; + drawbox->scaler_drawbox_paras.multi_drawbox_params[box_count].= end_y =3D drawbox->box_y[i] + drawbox->box_h[i] - 1; + drawbox->scaler_drawbox_paras.multi_drawbox_params[box_count].= rgba_c =3D drawbox->box_rgba_color[0][B] + (drawbox->box_rgba_color[0][G] <= < 8) + (drawbox->box_rgba_color[0][R] << 16) + (drawbox->box_rgba_color[0][= A] << 24); + if ((drawbox->box_w[i] > 0) && (drawbox->box_h[i] > 0)) + box_count++; + } + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + retcode =3D ni_scaler_set_drawbox_params(&drawbox->api_ctx, + &drawbox->scaler_drawbox_paras.multi_drawbox_params[0]= ); + if (retcode !=3D NI_RETCODE_SUCCESS) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + drawbox->frame_in.picture_width =3D FFALIGN(in->width, 2); + drawbox->frame_in.picture_height =3D FFALIGN(in->height, 2); + drawbox->frame_in.picture_format =3D drawbox_format; + drawbox->frame_in.session_id =3D frame_surface->ui16session_ID; + drawbox->frame_in.output_index =3D frame_surface->output_idx; + drawbox->frame_in.frame_index =3D frame_surface->ui16FrameIdx; + + /* + * Config device input frame parameters + */ + retcode =3D ni_device_config_frame(&drawbox->api_ctx, &drawbox->frame_= in); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_DEBUG, + "Can't allocate device input frame %d\n", retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + drawbox_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(drawbox->out_format); + + drawbox->frame_out.picture_width =3D outlink->w; + drawbox->frame_out.picture_height =3D outlink->h; + drawbox->frame_out.picture_format =3D drawbox_format; + + /* Allocate hardware device destination frame. This acquires a frame + * from the pool + */ + retcode =3D ni_device_alloc_frame(&drawbox->api_ctx, + FFALIGN(outlink->w, 2), + FFALIGN(outlink->h, 2), + drawbox_format, + NI_SCALER_FLAG_IO, + 0, + 0, + 0, + 0, + 0, + drawbox->inplace ? frame_surface->ui16= FrameIdx : -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_DEBUG, + "Can't allocate device output frame %d\n", retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc(&drawbox->api_ctx, &drawbox-= >api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_ERROR, + "Can't acquire output frame %d\n",retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_drawbox"); +#endif + + /* + * For an in-place drawbox, we have modified the input + * frame so just pass it along to the downstream. + */ + if (drawbox->inplace) { + av_log(link->dst, AV_LOG_DEBUG, + "vf_drawbox_ni.c:IN trace ui16FrameIdx =3D [%d] --> out [%d= ] \n", + frame_surface->ui16FrameIdx, frame_surface->ui16FrameIdx); + return ff_filter_frame(link->dst->outputs[0], in); + } + + out =3D av_frame_alloc(); + if (!out) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + av_frame_copy_props(out,in); + + out->width =3D outlink->w; + out->height =3D outlink->h; + + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(drawbox->out_frames_ref); + + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + + if (!out->data[3]) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Copy the frame surface from the incoming frame */ + memcpy(out->data[3], in->data[3], sizeof(niFrameSurface1_t)); + + tempFID =3D frame_surface->ui16FrameIdx; + frame_surface =3D (niFrameSurface1_t *)out->data[3]; + new_frame_surface =3D (niFrameSurface1_t *)drawbox->api_dst_frame.data= .frame.p_data[3]; + frame_surface->ui16FrameIdx =3D new_frame_surface->ui16FrameIdx; + frame_surface->ui16session_ID =3D new_frame_surface->ui16session_ID; + frame_surface->device_handle =3D new_frame_surface->device_handle; + frame_surface->output_idx =3D new_frame_surface->output_idx; + frame_surface->src_cpu =3D new_frame_surface->src_cpu; + frame_surface->dma_buf_fd =3D 0; + + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + pAVHFWCtx->sw_format); + + /* Remove ni-split specific assets */ + frame_surface->ui32nodeAddress =3D 0; + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + av_log(link->dst, AV_LOG_DEBUG, + "vf_drawbox_ni.c:IN trace ui16FrameIdx =3D [%d] --> out [%d] \n= ", + tempFID, frame_surface->ui16FrameIdx); + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurface1_= t), + ff_ni_frame_free, NULL, 0); + + av_frame_free(&in); + + return ff_filter_frame(link->dst->outputs[0], out); + +fail: + av_frame_free(&in); + if (out) + av_frame_free(&out); + return retcode; +} + +static int process_command(AVFilterContext *ctx, const char *cmd, const ch= ar *args, char *res, int res_len, int flags) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + NetIntDrawBoxContext *s =3D ctx->priv; + int old_x =3D s->box_x[0]; + int old_y =3D s->box_y[0]; + int old_w =3D s->box_w[0]; + int old_h =3D s->box_h[0]; + char *old_color =3D av_strdup(s->box_color_str[0]); + int ret; + + ret =3D ff_filter_process_command(ctx, cmd, args, res, res_len, flags); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Bad command/arguments (%d)\n", ret); + return ret; + } + + ret =3D init(ctx); + if (ret < 0) + goto end; + ret =3D config_input(inlink); +end: + if (ret < 0) { + s->box_x[0] =3D old_x; + s->box_y[0] =3D old_y; + s->box_w[0] =3D old_w; + s->box_h[0] =3D old_h; + memcpy(s->box_color_str[0], old_color, strlen(old_color)); + } + + av_free(old_color); + return ret; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntDrawBoxContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized && !s->inplace) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %lu available_frame %d outlink framequeue %lu frame_wanted %d -= return NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntDrawBoxContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) +#define RFLAGS (FLAGS | AV_OPT_FLAG_RUNTIME_PARAM) + +static const AVOption ni_drawbox_options[] =3D { + { "x", "set horizontal position of the left box edge", OFFSET(= box_x_expr[0]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "y", "set vertical position of the top box edge", OFFSET(= box_y_expr[0]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "width", "set width of the box", OFFSET(= box_w_expr[0]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "w", "set width of the box", OFFSET(= box_w_expr[0]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "height", "set height of the box", OFFSET(= box_h_expr[0]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "h", "set height of the box", OFFSET(= box_h_expr[0]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "color", "set color of the box", OFFSET(= box_color_str[0]), AV_OPT_TYPE_STRING, {.str=3D"black"}, 0, 0, RFLAGS }, + { "c", "set color of the box", OFFSET(= box_color_str[0]), AV_OPT_TYPE_STRING, {.str=3D"black"}, 0, 0, RFLAGS }, + { "x1", "", OFFSET(= box_x_expr[1]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "y1", "", OFFSET(= box_y_expr[1]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "w1", "", OFFSET(= box_w_expr[1]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "h1", "", OFFSET(= box_h_expr[1]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "x2", "", OFFSET(= box_x_expr[2]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "y2", "", OFFSET(= box_y_expr[2]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "w2", "", OFFSET(= box_w_expr[2]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "h2", "", OFFSET(= box_h_expr[2]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "x3", "", OFFSET(= box_x_expr[3]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "y3", "", OFFSET(= box_y_expr[3]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "w3", "", OFFSET(= box_w_expr[3]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "h3", "", OFFSET(= box_h_expr[3]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "x4", "", OFFSET(= box_x_expr[4]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "y4", "", OFFSET(= box_y_expr[4]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "w4", "", OFFSET(= box_w_expr[4]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "h4", "", OFFSET(= box_h_expr[4]), AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, RFLAGS }, + { "filterblit", "filterblit enable", OFFSET(= params.filterblit), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + { "inplace", "draw boxes in-place", OFFSET(= inplace), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_drawbox); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_input, + .filter_frame =3D filter_frame, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_props, + }, +}; + +FFFilter ff_vf_drawbox_ni_quadra =3D { + .p.name =3D "ni_quadra_drawbox", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra video drawbox v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_drawbox_class, + .priv_size =3D sizeof(NetIntDrawBoxContext), + .init =3D init, + .uninit =3D uninit, + .activate =3D activate, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), + .process_command =3D process_command, +}; diff --git a/libavfilter/vf_drawtext_ni.c b/libavfilter/vf_drawtext_ni.c new file mode 100644 index 0000000000..c7f56c9bcd --- /dev/null +++ b/libavfilter/vf_drawtext_ni.c @@ -0,0 +1,3401 @@ +/* + * Copyright (c) 2011 Stefano Sabatini + * Copyright (c) 2010 S.N. Hemanth Meenakshisundaram + * Copyright (c) 2003 Gustavo Sverzut Barbieri + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * NETINT drawtext filter, based on the original vf_drawtext.c + * + */ + +#include "config.h" + +#if HAVE_SYS_TIME_H +#include +#endif +#include +#include +#include +#if HAVE_UNISTD_H +#include +#endif +#include + +#if CONFIG_LIBFONTCONFIG +#include +#endif + +#include "libavutil/avstring.h" +#include "libavutil/bprint.h" +#include "libavutil/common.h" +#include "libavutil/file.h" +#include "libavutil/eval.h" +#include "libavutil/opt.h" +#include "libavutil/random_seed.h" +#include "libavutil/parseutils.h" +#include "libavutil/timecode.h" +#include "libavutil/time_internal.h" +#include "libavutil/tree.h" +#include "libavutil/lfg.h" +#include "libavutil/version.h" +#include "nifilter.h" +#include "filters.h" +#include "drawutils.h" +#include "formats.h" +#include "libavutil/mem.h" + +#include "video.h" + +#if CONFIG_LIBFRIBIDI +#include +#endif + +#include +#include FT_FREETYPE_H +#include FT_GLYPH_H +#include FT_STROKER_H + +#define MAX_TEXT_NUM 32 + +static const char *const var_names[] =3D { + "dar", + "hsub", "vsub", + "line_h", "lh", ///< line height, same as max_glyph_h + "main_h", "h", "H", ///< height of the input video + "main_w", "w", "W", ///< width of the input video + "max_glyph_a", "ascent", ///< max glyph ascent + "max_glyph_d", "descent", ///< min glyph descent + "max_glyph_h", ///< max glyph height + "max_glyph_w", ///< max glyph width + "n", ///< number of frame + "sar", + "t", ///< timestamp expressed in seconds + "text_h", "th", ///< height of the rendered text + "text_w", "tw", ///< width of the rendered text + "x", + "y", + "pict_type", +#if FF_API_FRAME_PKT + "pkt_pos", + "pkt_size", +#endif + "pkt_duration", + NULL +}; + +static const char *const fun2_names[] =3D { + "rand" +}; + +static double drand(void *opaque, double min, double max) +{ + return min + (max-min) / UINT_MAX * av_lfg_get(opaque); +} + +typedef double (*eval_func2)(void *, double a, double b); + +static const eval_func2 fun2[] =3D { + drand, + NULL +}; + +enum var_name { + VAR_DAR, + VAR_HSUB, VAR_VSUB, + VAR_LINE_H, VAR_LH, + VAR_MAIN_H, VAR_h, VAR_H, + VAR_MAIN_W, VAR_w, VAR_W, + VAR_MAX_GLYPH_A, VAR_ASCENT, + VAR_MAX_GLYPH_D, VAR_DESCENT, + VAR_MAX_GLYPH_H, + VAR_MAX_GLYPH_W, + VAR_N, + VAR_SAR, + VAR_T, + VAR_TEXT_H, VAR_TH, + VAR_TEXT_W, VAR_TW, + VAR_X, + VAR_Y, + VAR_PICT_TYPE, +#if FF_API_FRAME_PKT + VAR_PKT_POS, + VAR_PKT_SIZE, +#endif + VAR_PKT_DURATION, + VAR_VARS_NB +}; + +enum expansion_mode { + EXP_NONE, + EXP_NORMAL, + EXP_STRFTIME, +}; + +typedef struct NetIntDrawTextContext { + const AVClass *class; + int exp_mode; ///< expansion mode to use for the text + int reinit; ///< tells if the filter is being rein= ited + int text_num; ///< number of the text +#if CONFIG_LIBFONTCONFIG + uint8_t *font[MAX_TEXT_NUM]; ///< font to be used +#endif + uint8_t *fontfile[MAX_TEXT_NUM];///< font to be used + uint8_t *text[MAX_TEXT_NUM]; ///< text to be drawn + uint8_t *text_last_updated[MAX_TEXT_NUM]; + AVBPrint expanded_text; ///< used to contain the expanded text + uint8_t *fontcolor_expr[MAX_TEXT_NUM]; ///< fontcolor expressio= n to evaluate + AVBPrint expanded_fontcolor; ///< used to contain the expanded font= color spec + int ft_load_flags; ///< flags used for loading fonts, see= FT_LOAD_* + FT_Vector *positions; ///< positions for each element in the= text + size_t nb_positions; ///< number of elements of positions a= rray + char *textfile; ///< file with text to be drawn + int x[MAX_TEXT_NUM]; ///< x position to start drawing one t= ext + int y[MAX_TEXT_NUM]; ///< y position to start drawing one t= ext + int x_bak[MAX_TEXT_NUM]; ///< x position of last uploaded overl= ay frame + int y_bak[MAX_TEXT_NUM]; ///< y position of last uploaded overl= ay frame + int x_start; ///< x position for text canvas start = in one frame + int y_start; ///< y position for text canvas start = in one frame + int x_end; ///< x position for text canvas end in= one frame + int y_end; ///< y position for text canvas end in= one frame + int max_glyph_w; ///< max glyph width + int max_glyph_h; ///< max glyph height + int shadowx, shadowy; + int borderw; ///< border width + char *fontsize_expr[MAX_TEXT_NUM]; ///< expression for font= size + AVExpr *fontsize_pexpr[MAX_TEXT_NUM]; ///< parsed expressions = for fontsize + unsigned int fontsize[MAX_TEXT_NUM]; ///< font size to use + unsigned int default_fontsize; ///< default font size to use + + int line_spacing; ///< lines spacing in pixels + short int draw_box; ///< draw box around text - true or fa= lse + char *boxborderw[MAX_TEXT_NUM]; ///< box border width (padding) + /// allowed formats: "all", "vert|ori= z", "top|right|bottom|left" + int bb_top[MAX_TEXT_NUM]; ///< the size of the top box border + int bb_right[MAX_TEXT_NUM]; ///< the size of the right box border + int bb_bottom[MAX_TEXT_NUM]; ///< the size of the bottom box border + int bb_left[MAX_TEXT_NUM]; ///< the size of the left box border + int use_kerning[MAX_TEXT_NUM]; ///< font kerning is used - true/false + int tabsize[MAX_TEXT_NUM]; ///< tab size + int fix_bounds; ///< do we let it go out of frame boun= ds - t/f + int optimize_upload; + FFDrawContext dc; + FFDrawColor fontcolor[MAX_TEXT_NUM]; ///< foreground color + FFDrawColor shadowcolor; ///< shadow color + FFDrawColor bordercolor; ///< border color + FFDrawColor boxcolor[MAX_TEXT_NUM]; ///< background color + + FT_Library library; ///< freetype font library handle + FT_Face face[MAX_TEXT_NUM]; ///< freetype font face handle + FT_Stroker stroker; ///< freetype stroker handle + struct AVTreeNode *glyphs; ///< rendered glyphs, stored using the= UTF-32 char code + char *x_expr[MAX_TEXT_NUM]; ///< expression for x position + char *y_expr[MAX_TEXT_NUM]; ///< expression for y position + AVExpr *x_pexpr[MAX_TEXT_NUM]; //< parsed expressions for x + AVExpr *y_pexpr[MAX_TEXT_NUM]; ///< parsed expressions for y + int64_t basetime; ///< base pts time in the real world f= or display + double var_values[VAR_VARS_NB]; + char *a_expr; + AVExpr *a_pexpr; + int alpha; + AVLFG prng; ///< random + char *tc_opt_string; ///< specified timecode option string + AVRational tc_rate; ///< frame rate for timecode + AVTimecode tc; ///< timecode context + int tc24hmax; ///< 1 if timecode is wrapped to 24 ho= urs, 0 otherwise + int reload; ///< reload text file for each frame + int start_number; ///< starting frame number for n/frame= _num var +#if CONFIG_LIBFRIBIDI + int text_shaping; ///< 1 to shape the text before drawin= g it +#endif + AVDictionary *metadata; + + // NI overlay related + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + int session_opened; + + // NI HW frame upload related + AVBufferRef *hwdevice; + AVBufferRef *hwframe; + AVBufferRef *hw_frames_ctx; + + // NI watermark related + ni_scaler_multi_watermark_params_t scaler_watermark_paras; + int watermark_width0; + int watermark_width1; + int watermark_height0; + int watermark_height1; + + // NI ovly inplace crop related + ni_session_context_t crop_api_ctx; + ni_session_data_io_t crop_api_dst_frame; + uint16_t ui16CropFrameIdx; + int crop_session_opened; + + int keep_alive_timeout; /* keep alive timeout setting */ + int buffer_limit; + + int initialized; + int main_has_alpha; + int use_watermark; + AVBufferRef *out_frames_ref; + + // contains data downloaded from the input HW frame + ni_session_data_io_t dl_frame; + // contains text portion of overlaying frame + ni_session_data_io_t txt_frame; + + AVFrame *up_frame; + AVFrame *keep_overlay; + int upload_drawtext_frame; + int filtered_frame_count; + int framerate; + int initiated_upload_width; + int initiated_upload_height; +} NetIntDrawTextContext; + +static const enum AVPixelFormat alpha_pix_fmts[] =3D { + AV_PIX_FMT_RGBA, AV_PIX_FMT_ARGB, AV_PIX_FMT_ABGR, + AV_PIX_FMT_BGRA, AV_PIX_FMT_NONE +}; + +#define OFFSET(x) offsetof(NetIntDrawTextContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM + +static const AVOption ni_drawtext_options[] =3D { + { "fontfile", "set font file", OFFSET(fontfile[0]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff0", "set font file", OFFSET(fontfile[0]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff1", "set font file", OFFSET(fontfile[1]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff2", "set font file", OFFSET(fontfile[2]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff3", "set font file", OFFSET(fontfile[3]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff4", "set font file", OFFSET(fontfile[4]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff5", "set font file", OFFSET(fontfile[5]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff6", "set font file", OFFSET(fontfile[6]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff7", "set font file", OFFSET(fontfile[7]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff8", "set font file", OFFSET(fontfile[8]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff9", "set font file", OFFSET(fontfile[9]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff10", "set font file", OFFSET(fontfile[10]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff11", "set font file", OFFSET(fontfile[11]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff12", "set font file", OFFSET(fontfile[12]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff13", "set font file", OFFSET(fontfile[13]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff14", "set font file", OFFSET(fontfile[14]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff15", "set font file", OFFSET(fontfile[15]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff16", "set font file", OFFSET(fontfile[16]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff17", "set font file", OFFSET(fontfile[17]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff18", "set font file", OFFSET(fontfile[18]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff19", "set font file", OFFSET(fontfile[19]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff20", "set font file", OFFSET(fontfile[20]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff21", "set font file", OFFSET(fontfile[21]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff22", "set font file", OFFSET(fontfile[22]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff23", "set font file", OFFSET(fontfile[23]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff24", "set font file", OFFSET(fontfile[24]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff25", "set font file", OFFSET(fontfile[25]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff26", "set font file", OFFSET(fontfile[26]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff27", "set font file", OFFSET(fontfile[27]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff28", "set font file", OFFSET(fontfile[28]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff29", "set font file", OFFSET(fontfile[29]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff30", "set font file", OFFSET(fontfile[30]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "ff31", "set font file", OFFSET(fontfile[31]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "text", "set text", OFFSET(text[0]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t0", "set text", OFFSET(text[0]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t1", "set text", OFFSET(text[1]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t2", "set text", OFFSET(text[2]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t3", "set text", OFFSET(text[3]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t4", "set text", OFFSET(text[4]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t5", "set text", OFFSET(text[5]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t6", "set text", OFFSET(text[6]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t7", "set text", OFFSET(text[7]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t8", "set text", OFFSET(text[8]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t9", "set text", OFFSET(text[9]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t10", "set text", OFFSET(text[10]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t11", "set text", OFFSET(text[11]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t12", "set text", OFFSET(text[12]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t13", "set text", OFFSET(text[13]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t14", "set text", OFFSET(text[14]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t15", "set text", OFFSET(text[15]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t16", "set text", OFFSET(text[16]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t17", "set text", OFFSET(text[17]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t18", "set text", OFFSET(text[18]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t19", "set text", OFFSET(text[19]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t20", "set text", OFFSET(text[20]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t21", "set text", OFFSET(text[21]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t22", "set text", OFFSET(text[22]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t23", "set text", OFFSET(text[23]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t24", "set text", OFFSET(text[24]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t25", "set text", OFFSET(text[25]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t26", "set text", OFFSET(text[26]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t27", "set text", OFFSET(text[27]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t28", "set text", OFFSET(text[28]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t29", "set text", OFFSET(text[29]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t30", "set text", OFFSET(text[30]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "t31", "set text", OFFSET(text[31]), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "textfile", "set text file", OFFSET(textfile), = AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fontcolor", "set foreground color", OFFSET(fontcolor[0].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc0", "set foreground color", OFFSET(fontcolor[0].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc1", "set foreground color", OFFSET(fontcolor[1].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc2", "set foreground color", OFFSET(fontcolor[2].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc3", "set foreground color", OFFSET(fontcolor[3].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc4", "set foreground color", OFFSET(fontcolor[4].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc5", "set foreground color", OFFSET(fontcolor[5].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc6", "set foreground color", OFFSET(fontcolor[6].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc7", "set foreground color", OFFSET(fontcolor[7].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc8", "set foreground color", OFFSET(fontcolor[8].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc9", "set foreground color", OFFSET(fontcolor[9].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc10", "set foreground color", OFFSET(fontcolor[10].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc11", "set foreground color", OFFSET(fontcolor[11].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc12", "set foreground color", OFFSET(fontcolor[12].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc13", "set foreground color", OFFSET(fontcolor[13].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc14", "set foreground color", OFFSET(fontcolor[14].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc15", "set foreground color", OFFSET(fontcolor[15].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc16", "set foreground color", OFFSET(fontcolor[16].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc17", "set foreground color", OFFSET(fontcolor[17].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc18", "set foreground color", OFFSET(fontcolor[18].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc19", "set foreground color", OFFSET(fontcolor[19].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc20", "set foreground color", OFFSET(fontcolor[20].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc21", "set foreground color", OFFSET(fontcolor[21].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc22", "set foreground color", OFFSET(fontcolor[22].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc23", "set foreground color", OFFSET(fontcolor[23].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc24", "set foreground color", OFFSET(fontcolor[24].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc25", "set foreground color", OFFSET(fontcolor[25].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc26", "set foreground color", OFFSET(fontcolor[26].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc27", "set foreground color", OFFSET(fontcolor[27].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc28", "set foreground color", OFFSET(fontcolor[28].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc29", "set foreground color", OFFSET(fontcolor[29].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc30", "set foreground color", OFFSET(fontcolor[30].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fc31", "set foreground color", OFFSET(fontcolor[31].rgba), = AV_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "fontcolor_expr", "set foreground color expression", OFFSET(fontcolo= r_expr[0]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr0", "set foreground color expression", OFFSET(fontcolor_e= xpr[0]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr1", "set foreground color expression", OFFSET(fontcolor_e= xpr[1]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr2", "set foreground color expression", OFFSET(fontcolor_e= xpr[2]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr3", "set foreground color expression", OFFSET(fontcolor_e= xpr[3]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr4", "set foreground color expression", OFFSET(fontcolor_e= xpr[4]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr5", "set foreground color expression", OFFSET(fontcolor_e= xpr[5]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr6", "set foreground color expression", OFFSET(fontcolor_e= xpr[6]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr7", "set foreground color expression", OFFSET(fontcolor_e= xpr[7]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr8", "set foreground color expression", OFFSET(fontcolor_e= xpr[8]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr9", "set foreground color expression", OFFSET(fontcolor_e= xpr[9]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr10", "set foreground color expression", OFFSET(fontcolor_e= xpr[10]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr11", "set foreground color expression", OFFSET(fontcolor_e= xpr[11]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr12", "set foreground color expression", OFFSET(fontcolor_e= xpr[12]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr13", "set foreground color expression", OFFSET(fontcolor_e= xpr[13]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr14", "set foreground color expression", OFFSET(fontcolor_e= xpr[14]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr15", "set foreground color expression", OFFSET(fontcolor_e= xpr[15]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr16", "set foreground color expression", OFFSET(fontcolor_e= xpr[16]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr17", "set foreground color expression", OFFSET(fontcolor_e= xpr[17]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr18", "set foreground color expression", OFFSET(fontcolor_e= xpr[18]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr19", "set foreground color expression", OFFSET(fontcolor_e= xpr[19]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr20", "set foreground color expression", OFFSET(fontcolor_e= xpr[20]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr21", "set foreground color expression", OFFSET(fontcolor_e= xpr[21]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr22", "set foreground color expression", OFFSET(fontcolor_e= xpr[22]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr23", "set foreground color expression", OFFSET(fontcolor_e= xpr[23]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr24", "set foreground color expression", OFFSET(fontcolor_e= xpr[24]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr25", "set foreground color expression", OFFSET(fontcolor_e= xpr[25]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr26", "set foreground color expression", OFFSET(fontcolor_e= xpr[26]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr27", "set foreground color expression", OFFSET(fontcolor_e= xpr[27]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr28", "set foreground color expression", OFFSET(fontcolor_e= xpr[28]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr29", "set foreground color expression", OFFSET(fontcolor_e= xpr[29]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr30", "set foreground color expression", OFFSET(fontcolor_e= xpr[30]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "fc_expr31", "set foreground color expression", OFFSET(fontcolor_e= xpr[31]), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "boxcolor", "set box color", OFFSET(boxcolor[0].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc0", "set box color", OFFSET(boxcolor[0].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc1", "set box color", OFFSET(boxcolor[1].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc2", "set box color", OFFSET(boxcolor[2].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc3", "set box color", OFFSET(boxcolor[3].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc4", "set box color", OFFSET(boxcolor[4].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc5", "set box color", OFFSET(boxcolor[5].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc6", "set box color", OFFSET(boxcolor[6].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc7", "set box color", OFFSET(boxcolor[7].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc8", "set box color", OFFSET(boxcolor[8].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc9", "set box color", OFFSET(boxcolor[9].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc10", "set box color", OFFSET(boxcolor[10].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc11", "set box color", OFFSET(boxcolor[11].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc12", "set box color", OFFSET(boxcolor[12].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc13", "set box color", OFFSET(boxcolor[13].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc14", "set box color", OFFSET(boxcolor[14].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc15", "set box color", OFFSET(boxcolor[15].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc16", "set box color", OFFSET(boxcolor[16].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc17", "set box color", OFFSET(boxcolor[17].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc18", "set box color", OFFSET(boxcolor[18].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc19", "set box color", OFFSET(boxcolor[19].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc20", "set box color", OFFSET(boxcolor[20].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc21", "set box color", OFFSET(boxcolor[21].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc22", "set box color", OFFSET(boxcolor[22].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc23", "set box color", OFFSET(boxcolor[23].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc24", "set box color", OFFSET(boxcolor[24].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc25", "set box color", OFFSET(boxcolor[25].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc26", "set box color", OFFSET(boxcolor[26].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc27", "set box color", OFFSET(boxcolor[27].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc28", "set box color", OFFSET(boxcolor[28].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc29", "set box color", OFFSET(boxcolor[29].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc30", "set box color", OFFSET(boxcolor[30].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bc31", "set box color", OFFSET(boxcolor[31].rgba), A= V_OPT_TYPE_COLOR, {.str=3D"white"}, 0, 0, FLAGS }, + { "bordercolor", "set border color", OFFSET(bordercolor.rgba), A= V_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "shadowcolor", "set shadow color", OFFSET(shadowcolor.rgba), A= V_OPT_TYPE_COLOR, {.str=3D"black"}, 0, 0, FLAGS }, + { "box", "set box", OFFSET(draw_box), A= V_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1 , FLAGS }, + { "boxborderw", "set box borders width", OFFSET(boxborderw[0]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb0", "set box borders width", OFFSET(boxborderw[0]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb1", "set box borders width", OFFSET(boxborderw[1]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb2", "set box borders width", OFFSET(boxborderw[2]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb3", "set box borders width", OFFSET(boxborderw[3]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb4", "set box borders width", OFFSET(boxborderw[4]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb5", "set box borders width", OFFSET(boxborderw[5]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb6", "set box borders width", OFFSET(boxborderw[6]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb7", "set box borders width", OFFSET(boxborderw[7]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb8", "set box borders width", OFFSET(boxborderw[8]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb9", "set box borders width", OFFSET(boxborderw[9]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb10", "set box borders width", OFFSET(boxborderw[10]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb11", "set box borders width", OFFSET(boxborderw[11]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb12", "set box borders width", OFFSET(boxborderw[12]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb13", "set box borders width", OFFSET(boxborderw[13]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb14", "set box borders width", OFFSET(boxborderw[14]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb15", "set box borders width", OFFSET(boxborderw[15]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb16", "set box borders width", OFFSET(boxborderw[16]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb17", "set box borders width", OFFSET(boxborderw[17]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb18", "set box borders width", OFFSET(boxborderw[18]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb19", "set box borders width", OFFSET(boxborderw[19]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb20", "set box borders width", OFFSET(boxborderw[20]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb21", "set box borders width", OFFSET(boxborderw[21]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb22", "set box borders width", OFFSET(boxborderw[22]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb23", "set box borders width", OFFSET(boxborderw[23]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb24", "set box borders width", OFFSET(boxborderw[24]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb25", "set box borders width", OFFSET(boxborderw[25]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb26", "set box borders width", OFFSET(boxborderw[26]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb27", "set box borders width", OFFSET(boxborderw[27]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb28", "set box borders width", OFFSET(boxborderw[28]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb29", "set box borders width", OFFSET(boxborderw[29]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb30", "set box borders width", OFFSET(boxborderw[30]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "bb31", "set box borders width", OFFSET(boxborderw[31]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "line_spacing", "set line spacing in pixels", OFFSET(line_spacing),= AV_OPT_TYPE_INT, {.i64=3D0}, INT_MIN, INT_MAX,FLAGS }, + { "fontsize", "set font size", OFFSET(fontsize_expr[0]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs0", "set font size", OFFSET(fontsize_expr[0]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs1", "set font size", OFFSET(fontsize_expr[1]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs2", "set font size", OFFSET(fontsize_expr[2]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs3", "set font size", OFFSET(fontsize_expr[3]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs4", "set font size", OFFSET(fontsize_expr[4]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs5", "set font size", OFFSET(fontsize_expr[5]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs6", "set font size", OFFSET(fontsize_expr[6]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs7", "set font size", OFFSET(fontsize_expr[7]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs8", "set font size", OFFSET(fontsize_expr[8]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs9", "set font size", OFFSET(fontsize_expr[9]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs10", "set font size", OFFSET(fontsize_expr[10]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs11", "set font size", OFFSET(fontsize_expr[11]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs12", "set font size", OFFSET(fontsize_expr[12]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs13", "set font size", OFFSET(fontsize_expr[13]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs14", "set font size", OFFSET(fontsize_expr[14]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs15", "set font size", OFFSET(fontsize_expr[15]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs16", "set font size", OFFSET(fontsize_expr[16]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs17", "set font size", OFFSET(fontsize_expr[17]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs18", "set font size", OFFSET(fontsize_expr[18]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs19", "set font size", OFFSET(fontsize_expr[19]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs20", "set font size", OFFSET(fontsize_expr[20]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs21", "set font size", OFFSET(fontsize_expr[21]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs22", "set font size", OFFSET(fontsize_expr[22]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs23", "set font size", OFFSET(fontsize_expr[23]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs24", "set font size", OFFSET(fontsize_expr[24]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs25", "set font size", OFFSET(fontsize_expr[25]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs26", "set font size", OFFSET(fontsize_expr[26]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs27", "set font size", OFFSET(fontsize_expr[27]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs28", "set font size", OFFSET(fontsize_expr[28]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs29", "set font size", OFFSET(fontsize_expr[29]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs30", "set font size", OFFSET(fontsize_expr[30]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "fs31", "set font size", OFFSET(fontsize_expr[31]), = AV_OPT_TYPE_STRING, {.str=3D"36"}, 0, 0 , FLAGS }, + { "x", "set x expression", OFFSET(x_expr[0]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y", "set y expression", OFFSET(y_expr[0]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x0", "set x expression", OFFSET(x_expr[0]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y0", "set y expression", OFFSET(y_expr[0]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x1", "set x expression", OFFSET(x_expr[1]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y1", "set y expression", OFFSET(y_expr[1]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x2", "set x expression", OFFSET(x_expr[2]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y2", "set y expression", OFFSET(y_expr[2]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x3", "set x expression", OFFSET(x_expr[3]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y3", "set y expression", OFFSET(y_expr[3]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x4", "set x expression", OFFSET(x_expr[4]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y4", "set y expression", OFFSET(y_expr[4]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x5", "set x expression", OFFSET(x_expr[5]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y5", "set y expression", OFFSET(y_expr[5]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x6", "set x expression", OFFSET(x_expr[6]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y6", "set y expression", OFFSET(y_expr[6]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x7", "set x expression", OFFSET(x_expr[7]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y7", "set y expression", OFFSET(y_expr[7]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x8", "set x expression", OFFSET(x_expr[8]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y8", "set y expression", OFFSET(y_expr[8]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x9", "set x expression", OFFSET(x_expr[9]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y9", "set y expression", OFFSET(y_expr[9]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x10", "set x expression", OFFSET(x_expr[10]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y10", "set y expression", OFFSET(y_expr[10]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x11", "set x expression", OFFSET(x_expr[11]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y11", "set y expression", OFFSET(y_expr[11]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x12", "set x expression", OFFSET(x_expr[12]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y12", "set y expression", OFFSET(y_expr[12]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x13", "set x expression", OFFSET(x_expr[13]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y13", "set y expression", OFFSET(y_expr[13]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x14", "set x expression", OFFSET(x_expr[14]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y14", "set y expression", OFFSET(y_expr[14]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x15", "set x expression", OFFSET(x_expr[15]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y15", "set y expression", OFFSET(y_expr[15]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x16", "set x expression", OFFSET(x_expr[16]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y16", "set y expression", OFFSET(y_expr[16]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x17", "set x expression", OFFSET(x_expr[17]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y17", "set y expression", OFFSET(y_expr[17]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x18", "set x expression", OFFSET(x_expr[18]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y18", "set y expression", OFFSET(y_expr[18]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x19", "set x expression", OFFSET(x_expr[19]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y19", "set y expression", OFFSET(y_expr[19]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x20", "set x expression", OFFSET(x_expr[20]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y20", "set y expression", OFFSET(y_expr[20]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x21", "set x expression", OFFSET(x_expr[21]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y21", "set y expression", OFFSET(y_expr[21]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x22", "set x expression", OFFSET(x_expr[22]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y22", "set y expression", OFFSET(y_expr[22]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x23", "set x expression", OFFSET(x_expr[23]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y23", "set y expression", OFFSET(y_expr[23]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x24", "set x expression", OFFSET(x_expr[24]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y24", "set y expression", OFFSET(y_expr[24]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x25", "set x expression", OFFSET(x_expr[25]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y25", "set y expression", OFFSET(y_expr[25]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x26", "set x expression", OFFSET(x_expr[26]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y26", "set y expression", OFFSET(y_expr[26]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x27", "set x expression", OFFSET(x_expr[27]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y27", "set y expression", OFFSET(y_expr[27]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x28", "set x expression", OFFSET(x_expr[28]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y28", "set y expression", OFFSET(y_expr[28]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x29", "set x expression", OFFSET(x_expr[29]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y29", "set y expression", OFFSET(y_expr[29]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x30", "set x expression", OFFSET(x_expr[30]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y30", "set y expression", OFFSET(y_expr[30]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "x31", "set x expression", OFFSET(x_expr[31]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "y31", "set y expression", OFFSET(y_expr[31]), = AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "shadowx", "set shadow x offset", OFFSET(shadowx), A= V_OPT_TYPE_INT, {.i64=3D0}, INT_MIN, INT_MAX , FLAGS }, + { "shadowy", "set shadow y offset", OFFSET(shadowy), A= V_OPT_TYPE_INT, {.i64=3D0}, INT_MIN, INT_MAX , FLAGS }, + { "borderw", "set border width", OFFSET(borderw), A= V_OPT_TYPE_INT, {.i64=3D0}, INT_MIN, INT_MAX , FLAGS }, + { "tabsize", "set tab size", OFFSET(tabsize[0]), A= V_OPT_TYPE_INT, {.i64=3D4}, 0, INT_MAX , FLAGS }, + { "basetime", "set base time", OFFSET(basetime), A= V_OPT_TYPE_INT64, {.i64=3DAV_NOPTS_VALUE}, INT64_MIN, INT64_MAX , FLAGS }, +#if CONFIG_LIBFONTCONFIG + { "font", "Font name", OFFSET(font[0]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f0", "Font name", OFFSET(font[0]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f1", "Font name", OFFSET(font[1]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f2", "Font name", OFFSET(font[2]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f3", "Font name", OFFSET(font[3]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f4", "Font name", OFFSET(font[4]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f5", "Font name", OFFSET(font[5]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f6", "Font name", OFFSET(font[6]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f7", "Font name", OFFSET(font[7]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f8", "Font name", OFFSET(font[8]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f9", "Font name", OFFSET(font[9]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f10", "Font name", OFFSET(font[10]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f11", "Font name", OFFSET(font[11]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f12", "Font name", OFFSET(font[12]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f13", "Font name", OFFSET(font[13]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f14", "Font name", OFFSET(font[14]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f15", "Font name", OFFSET(font[15]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f16", "Font name", OFFSET(font[16]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f17", "Font name", OFFSET(font[17]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f18", "Font name", OFFSET(font[18]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f19", "Font name", OFFSET(font[19]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f20", "Font name", OFFSET(font[20]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f21", "Font name", OFFSET(font[21]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f22", "Font name", OFFSET(font[22]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f23", "Font name", OFFSET(font[23]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f24", "Font name", OFFSET(font[24]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f25", "Font name", OFFSET(font[25]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f26", "Font name", OFFSET(font[26]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f27", "Font name", OFFSET(font[27]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f28", "Font name", OFFSET(font[28]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f29", "Font name", OFFSET(font[29]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f30", "Font name", OFFSET(font[30]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, + { "f31", "Font name", OFFSET(font[31]), = AV_OPT_TYPE_STRING, { .str =3D "Sans" }, .flags =3D FLAGS }, +#endif + { "expansion", "set the expansion mode", OFFSET(exp_mode), AV_OPT_TYPE= _INT, {.i64=3DEXP_NORMAL}, 0, 2, FLAGS, "expansion" }, + { "none", "set no expansion", OFFSET(exp_mo= de), AV_OPT_TYPE_CONST, {.i64=3DEXP_NONE}, 0, 0, FLAGS, "expansion" }, + { "normal", "set normal expansion", OFFSET(exp_mo= de), AV_OPT_TYPE_CONST, {.i64=3DEXP_NORMAL}, 0, 0, FLAGS, "expansion" }, + { "strftime", "set strftime expansion (deprecated)", OFFSET(exp_mo= de), AV_OPT_TYPE_CONST, {.i64=3DEXP_STRFTIME}, 0, 0, FLAGS, "expansion" }, + { "timecode", "set initial timecode", OFFSET(tc_opt= _string), AV_OPT_TYPE_STRING, {.str=3DNULL}, 0, 0, FLAGS }, + { "tc24hmax", "set 24 hours max (timecode only)", OFFSET(tc24hm= ax), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + { "timecode_rate", "set rate (timecode only)", OFFSET(tc_rat= e), AV_OPT_TYPE_RATIONAL, {.dbl=3D0}, 0, INT_MAX, FLAGS }, + { "r", "set rate (timecode only)", OFFSET(tc_rat= e), AV_OPT_TYPE_RATIONAL, {.dbl=3D0}, 0, INT_MAX, FLAGS }, + { "rate", "set rate (timecode only)", OFFSET(tc_rat= e), AV_OPT_TYPE_RATIONAL, {.dbl=3D0}, 0, INT_MAX, FLAGS }, + { "reload", "reload text file for each frame", = OFFSET(reload), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + { "alpha", "apply alpha while rendering", OFFSET(a_expr), = AV_OPT_TYPE_STRING, { .str =3D "1" }, .flags =3D FLAGS }, + { "fix_bounds", "check and fix text coords to avoid clipping", OFFSET(= fix_bounds), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + { "start_number", "start frame number for n/frame_num variable", OFFSE= T(start_number), AV_OPT_TYPE_INT, {.i64=3D0}, 0, INT_MAX, FLAGS }, + +#if CONFIG_LIBFRIBIDI + { "text_shaping", "attempt to shape text before drawing", OFFSET(text_= shaping), AV_OPT_TYPE_BOOL, {.i64=3D1}, 0, 1, FLAGS }, +#endif + + /* FT_LOAD_* flags */ + { "ft_load_flags", "set font loading flags for libfreetype", OFFSET(ft= _load_flags), AV_OPT_TYPE_FLAGS, { .i64 =3D FT_LOAD_DEFAULT }, 0, INT_MAX, = FLAGS, "ft_load_flags" }, + { "default", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_DEFAULT }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "no_scale", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_NO_SCALE }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "no_hinting", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_NO_HINTING }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "render", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_RENDER }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "no_bitmap", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_NO_BITMAP }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "vertical_layout", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_VERTICAL_LAYOUT }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "force_autohint", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_FORCE_AUTOHINT }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "crop_bitmap", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_CROP_BITMAP }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "pedantic", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_PEDANTIC }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "ignore_global_advance_width", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "no_recurse", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_NO_RECURSE }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "ignore_transform", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_IGNORE_TRANSFORM }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "monochrome", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_MONOCHROME }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "linear_design", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_LINEAR_DESIGN }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "no_autohint", NULL, 0, AV_OPT_TYPE_CONST, { .i6= 4 =3D FT_LOAD_NO_AUTOHINT }, .flags =3D FLAGS, .unit =3D "f= t_load_flags" }, + { "optimize_upload", "Decrease the drawtext frame uploading frequency"= , OFFSET(optimize_upload), AV_OPT_TYPE_BOOL, {.i64=3D1}, 0, 1, FLAGS}, + { "use_watermark", "Use performance optimizations", OFFSET(use_waterma= rk), AV_OPT_TYPE_BOOL, {.i64 =3D 1}, 0, 1, FLAGS }, // Deprecated language + { "perf_optimization", "Use performance optimizations", OFFSET(use_wat= ermark), AV_OPT_TYPE_BOOL, {.i64 =3D 1}, 0, 1, FLAGS }, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_drawtext); + +#undef __FTERRORS_H__ +#define FT_ERROR_START_LIST { +#define FT_ERRORDEF(e, v, s) { (e), (s) }, +#define FT_ERROR_END_LIST { 0, NULL } }; + +static const struct ft_error { + int err; + const char *err_msg; +} ft_errors[] =3D +#include FT_ERRORS_H + +#define FT_ERRMSG(e) ft_errors[e].err_msg + +typedef struct Glyph { + FT_Glyph glyph; + FT_Glyph border_glyph; + uint32_t code; + unsigned int fontsize; + FT_Bitmap bitmap; ///< array holding bitmaps of font + FT_Bitmap border_bitmap; ///< array holding bitmaps of font border + FT_BBox bbox; + int advance; + int bitmap_left; + int bitmap_top; +} Glyph; + +static int glyph_cmp(const void *key, const void *b) +{ + const Glyph *a =3D key, *bb =3D b; + int64_t diff =3D (int64_t)a->code - (int64_t)bb->code; + + if (diff !=3D 0) + return diff > 0 ? 1 : -1; + else + return FFDIFFSIGN((int64_t)a->fontsize, (int64_t)bb->fontsize); +} + +/** + * Load glyphs corresponding to the UTF-32 codepoint code. + */ +static int load_glyph(AVFilterContext *ctx, Glyph **glyph_ptr, uint32_t co= de, int index) +{ + NetIntDrawTextContext *s =3D ctx->priv; + FT_BitmapGlyph bitmapglyph; + Glyph *glyph, dummy =3D { 0 }; + struct AVTreeNode *node =3D NULL; + int ret; + + /* load glyph into s->face->glyph */ + if (FT_Load_Char(s->face[index], code, s->ft_load_flags)) + return AVERROR(EINVAL); + + /* if glyph has already insert into s->glyphs, return directly */ + dummy.code =3D code; + dummy.fontsize =3D s->fontsize[index]; + glyph =3D av_tree_find(s->glyphs, &dummy, glyph_cmp, NULL); + if (glyph) { + if (glyph_ptr) + *glyph_ptr =3D glyph; + return 0; + } + + glyph =3D av_mallocz(sizeof(*glyph)); + if (!glyph) { + ret =3D AVERROR(ENOMEM); + goto error; + } + glyph->code =3D code; + glyph->fontsize =3D s->fontsize[index]; + + if (FT_Get_Glyph(s->face[index]->glyph, &glyph->glyph)) { + ret =3D AVERROR(EINVAL); + goto error; + } + if (s->borderw) { + glyph->border_glyph =3D glyph->glyph; + if (FT_Glyph_StrokeBorder(&glyph->border_glyph, s->stroker, 0, 0) = || + FT_Glyph_To_Bitmap(&glyph->border_glyph, FT_RENDER_MODE_NORMAL= , 0, 1)) { + ret =3D AVERROR_EXTERNAL; + goto error; + } + bitmapglyph =3D (FT_BitmapGlyph) glyph->border_glyph; + glyph->border_bitmap =3D bitmapglyph->bitmap; + } + if (FT_Glyph_To_Bitmap(&glyph->glyph, FT_RENDER_MODE_NORMAL, 0, 1)) { + ret =3D AVERROR_EXTERNAL; + goto error; + } + bitmapglyph =3D (FT_BitmapGlyph) glyph->glyph; + + glyph->bitmap =3D bitmapglyph->bitmap; + glyph->bitmap_left =3D bitmapglyph->left; + glyph->bitmap_top =3D bitmapglyph->top; + glyph->advance =3D s->face[index]->glyph->advance.x >> 6; + + /* measure text height to calculate text_height (or the maximum text h= eight) */ + FT_Glyph_Get_CBox(glyph->glyph, ft_glyph_bbox_pixels, &glyph->bbox); + + /* cache the newly created glyph */ + if (!(node =3D av_tree_node_alloc())) { + ret =3D AVERROR(ENOMEM); + goto error; + } + av_tree_insert(&s->glyphs, glyph, glyph_cmp, &node); + + if (glyph_ptr) + *glyph_ptr =3D glyph; + return 0; + +error: + if (glyph) + av_freep(&glyph->glyph); + + av_freep(&glyph); + av_freep(&node); + return ret; +} + +// Convert a string formatted as "n1|n2|...|nN" into an integer array +static int string_to_array(const char *source, int *result, int result_siz= e) +{ + int counter =3D 0, size =3D strlen(source) + 1; + char *saveptr, *curval, *dup =3D av_malloc(size); + if (!dup) + return 0; + av_strlcpy(dup, source, size); + if (result_size > 0 && (curval =3D av_strtok(dup, "|", &saveptr))) { + do { + result[counter++] =3D atoi(curval); + } while ((curval =3D av_strtok(NULL, "|", &saveptr)) && counter < = result_size); + } + av_free(dup); + return counter; +} + +// convert FFmpeg AV_PIX_FMT_ to NI_PIX_FMT_ +static int ff_to_ni_pix_fmt(int ff_av_pix_fmt) +{ + int pixel_format; + + switch (ff_av_pix_fmt) { + case AV_PIX_FMT_YUV420P: + pixel_format =3D NI_PIX_FMT_YUV420P; + break; + case AV_PIX_FMT_YUV420P10LE: + pixel_format =3D NI_PIX_FMT_YUV420P10LE; + break; + case AV_PIX_FMT_NV12: + pixel_format =3D NI_PIX_FMT_NV12; + break; + case AV_PIX_FMT_NV16: + pixel_format =3D NI_PIX_FMT_NV16; + break; + case AV_PIX_FMT_YUYV422: + pixel_format =3D NI_PIX_FMT_YUYV422; + break; + case AV_PIX_FMT_UYVY422: + pixel_format =3D NI_PIX_FMT_UYVY422; + break; + case AV_PIX_FMT_P010LE: + pixel_format =3D NI_PIX_FMT_P010LE; + break; + case AV_PIX_FMT_RGBA: + pixel_format =3D NI_PIX_FMT_RGBA; + break; + case AV_PIX_FMT_BGRA: + pixel_format =3D NI_PIX_FMT_BGRA; + break; + case AV_PIX_FMT_ABGR: + pixel_format =3D NI_PIX_FMT_ABGR; + break; + case AV_PIX_FMT_ARGB: + pixel_format =3D NI_PIX_FMT_ARGB; + break; + case AV_PIX_FMT_BGR0: + pixel_format =3D NI_PIX_FMT_BGR0; + break; + default: + av_log(NULL, AV_LOG_ERROR, "Pixel %d format not supported.\n", + ff_av_pix_fmt); + return AVERROR(EINVAL); + } + return pixel_format; +} + +static av_cold int set_fontsize(AVFilterContext *ctx, unsigned int fontsiz= e, int index) +{ + int err; + NetIntDrawTextContext *s =3D ctx->priv; + + if ((err =3D FT_Set_Pixel_Sizes(s->face[index], 0, fontsize))) { + av_log(ctx, AV_LOG_ERROR, "Could not set font size to %d pixels: %= s\n", + fontsize, FT_ERRMSG(err)); + return AVERROR(EINVAL); + } + + s->fontsize[index] =3D fontsize; + + return 0; +} + +static av_cold int parse_fontsize(AVFilterContext *ctx, int index) +{ + NetIntDrawTextContext *s =3D ctx->priv; + int err; + + if (s->fontsize_pexpr[index]) + return 0; + + if (s->fontsize_expr[index] =3D=3D NULL) + return AVERROR(EINVAL); + + if ((err =3D av_expr_parse(&s->fontsize_pexpr[index], s->fontsize_expr= [index], var_names, + NULL, NULL, fun2_names, fun2, 0, ctx)) < 0) + return err; + + return 0; +} + +static av_cold int update_fontsize(AVFilterContext *ctx, int index) +{ + NetIntDrawTextContext *s =3D ctx->priv; + unsigned int fontsize =3D s->default_fontsize; + int err; + double size, roundedsize; + + // if no fontsize specified use the default + if (s->fontsize_expr[index] !=3D NULL) { + if ((err =3D parse_fontsize(ctx, index)) < 0) + return err; + + size =3D av_expr_eval(s->fontsize_pexpr[index], s->var_values, &s-= >prng); + + if (!isnan(size)) { + roundedsize =3D round(size); + // test for overflow before cast + if (!(roundedsize > INT_MIN && roundedsize < INT_MAX)) { + av_log(ctx, AV_LOG_ERROR, "fontsize overflow\n"); + return AVERROR(EINVAL); + } + + fontsize =3D (int)roundedsize; + } + } + + if (fontsize =3D=3D 0) + fontsize =3D 1; + + // no change + if (fontsize =3D=3D s->fontsize[index]) + return 0; + + return set_fontsize(ctx, fontsize, index); +} + +static int load_font_file(AVFilterContext *ctx, const char *path, int inde= x, int text_index) +{ + NetIntDrawTextContext *s =3D ctx->priv; + int err; + + err =3D FT_New_Face(s->library, path, index, &s->face[text_index]); + if (err) { +#if !CONFIG_LIBFONTCONFIG + av_log(ctx, AV_LOG_ERROR, "Could not load font \"%s\": %s\n", + s->fontfile[text_index], FT_ERRMSG(err)); +#endif + return AVERROR(EINVAL); + } + return 0; +} + +#if CONFIG_LIBFONTCONFIG +static int load_font_fontconfig(AVFilterContext *ctx, int text_index) +{ + NetIntDrawTextContext *s =3D ctx->priv; + FcConfig *fontconfig; + FcPattern *pat, *best; + FcResult result =3D FcResultMatch; + FcChar8 *filename; + int index; + double size; + int err =3D AVERROR(ENOENT); + int parse_err; + + fontconfig =3D FcInitLoadConfigAndFonts(); + if (!fontconfig) { + av_log(ctx, AV_LOG_ERROR, "impossible to init fontconfig\n"); + return AVERROR_UNKNOWN; + } + pat =3D FcNameParse(s->fontfile[text_index] ? s->fontfile[text_index] : + (uint8_t *)(intptr_t)"default"); + if (!pat) { + av_log(ctx, AV_LOG_ERROR, "could not parse fontconfig pat"); + return AVERROR(EINVAL); + } + + FcPatternAddString(pat, FC_FAMILY, s->font[text_index]); + + parse_err =3D parse_fontsize(ctx, text_index); + if (!parse_err) { + double size =3D av_expr_eval(s->fontsize_pexpr[text_index], s->var= _values, &s->prng); + + if (isnan(size)) { + av_log(ctx, AV_LOG_ERROR, "impossible to find font information= "); + return AVERROR(EINVAL); + } + + FcPatternAddDouble(pat, FC_SIZE, size); + } + + FcDefaultSubstitute(pat); + + if (!FcConfigSubstitute(fontconfig, pat, FcMatchPattern)) { + av_log(ctx, AV_LOG_ERROR, "could not substitute fontconfig options= "); /* very unlikely */ + FcPatternDestroy(pat); + return AVERROR(ENOMEM); + } + + best =3D FcFontMatch(fontconfig, pat, &result); + FcPatternDestroy(pat); + + if (!best || result !=3D FcResultMatch) { + av_log(ctx, AV_LOG_ERROR, + "Cannot find a valid font for the family %s\n", + s->font[text_index]); + goto fail; + } + + if ( + FcPatternGetInteger(best, FC_INDEX, 0, &index ) !=3D FcResultMat= ch || + FcPatternGetDouble (best, FC_SIZE, 0, &size ) !=3D FcResultMat= ch) { + av_log(ctx, AV_LOG_ERROR, "impossible to find font information"); + return AVERROR(EINVAL); + } + + if (FcPatternGetString(best, FC_FILE, 0, &filename) !=3D FcResultMatch= ) { + av_log(ctx, AV_LOG_ERROR, "No file path for %s\n", + s->font[text_index]); + goto fail; + } + + av_log(ctx, AV_LOG_INFO, "Using \"%s\"\n", filename); + if (parse_err) + s->default_fontsize =3D size + 0.5; + + err =3D load_font_file(ctx, filename, index, text_index); + if (err) + return err; + FcConfigDestroy(fontconfig); +fail: + FcPatternDestroy(best); + return err; +} +#endif + +static int load_font(AVFilterContext *ctx, int index) +{ + NetIntDrawTextContext *s =3D ctx->priv; + int err; + + /* load the face, and set up the encoding, which is by default UTF-8 */ + if (s->fontfile[index]) { + err =3D load_font_file(ctx, s->fontfile[index], 0, index); + if (!err) + return 0; + } +#if CONFIG_LIBFONTCONFIG + err =3D load_font_fontconfig(ctx, index); + if (!err) + return 0; +#endif + return err; +} + +static int load_textfile(AVFilterContext *ctx) +{ + NetIntDrawTextContext *s =3D ctx->priv; + int err; + uint8_t *textbuf; + uint8_t *tmp; + size_t textbuf_size; + + if ((err =3D av_file_map(s->textfile, &textbuf, &textbuf_size, 0, ctx)= ) < 0) { + av_log(ctx, AV_LOG_ERROR, + "The text file '%s' could not be read or is empty\n", + s->textfile); + return err; + } + + if (textbuf_size > SIZE_MAX - 1 || !(tmp =3D av_realloc(s->text[0], te= xtbuf_size + 1))) { + av_file_unmap(textbuf, textbuf_size); + return AVERROR(ENOMEM); + } + s->text[0] =3D tmp; + memcpy(s->text[0], textbuf, textbuf_size); + s->text[0][textbuf_size] =3D 0; + av_file_unmap(textbuf, textbuf_size); + + return 0; +} + +static inline int is_newline(uint32_t c) +{ + return c =3D=3D '\n' || c =3D=3D '\r' || c =3D=3D '\f' || c =3D=3D '\v= '; +} + +#if CONFIG_LIBFRIBIDI +static int shape_text(AVFilterContext *ctx) +{ + NetIntDrawTextContext *s =3D ctx->priv; + uint8_t *tmp; + int ret =3D AVERROR(ENOMEM); + static const FriBidiFlags flags =3D FRIBIDI_FLAGS_DEFAULT | + FRIBIDI_FLAGS_ARABIC; + FriBidiChar *unicodestr =3D NULL; + FriBidiStrIndex len; + FriBidiParType direction =3D FRIBIDI_PAR_LTR; + FriBidiStrIndex line_start =3D 0; + FriBidiStrIndex line_end =3D 0; + FriBidiLevel *embedding_levels =3D NULL; + FriBidiArabicProp *ar_props =3D NULL; + FriBidiCharType *bidi_types =3D NULL; + FriBidiStrIndex i,j; + + len =3D strlen(s->text[0]); + if (!(unicodestr =3D av_malloc_array(len, sizeof(*unicodestr)))) { + goto out; + } + len =3D fribidi_charset_to_unicode(FRIBIDI_CHAR_SET_UTF8, + s->text[0], len, unicodestr); + + bidi_types =3D av_malloc_array(len, sizeof(*bidi_types)); + if (!bidi_types) { + goto out; + } + + fribidi_get_bidi_types(unicodestr, len, bidi_types); + + embedding_levels =3D av_malloc_array(len, sizeof(*embedding_levels)); + if (!embedding_levels) { + goto out; + } + + if (!fribidi_get_par_embedding_levels(bidi_types, len, &direction, + embedding_levels)) { + goto out; + } + + ar_props =3D av_malloc_array(len, sizeof(*ar_props)); + if (!ar_props) { + goto out; + } + + fribidi_get_joining_types(unicodestr, len, ar_props); + fribidi_join_arabic(bidi_types, len, embedding_levels, ar_props); + fribidi_shape(flags, embedding_levels, len, ar_props, unicodestr); + + for (line_end =3D 0, line_start =3D 0; line_end < len; line_end++) { + if (is_newline(unicodestr[line_end]) || line_end =3D=3D len - 1) { + if (!fribidi_reorder_line(flags, bidi_types, + line_end - line_start + 1, line_star= t, + direction, embedding_levels, unicode= str, + NULL)) { + goto out; + } + line_start =3D line_end + 1; + } + } + + /* Remove zero-width fill chars put in by libfribidi */ + for (i =3D 0, j =3D 0; i < len; i++) + if (unicodestr[i] !=3D FRIBIDI_CHAR_FILL) + unicodestr[j++] =3D unicodestr[i]; + len =3D j; + + if (!(tmp =3D av_realloc(s->text[0], (len * 4 + 1) * sizeof(*s->text[0= ])))) { + /* Use len * 4, as a unicode character can be up to 4 bytes in UTF= -8 */ + goto out; + } + + s->text[0] =3D tmp; + len =3D fribidi_unicode_to_charset(FRIBIDI_CHAR_SET_UTF8, + unicodestr, len, s->text[0]); + ret =3D 0; + +out: + av_free(unicodestr); + av_free(embedding_levels); + av_free(ar_props); + av_free(bidi_types); + return ret; +} +#endif + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D + {AV_PIX_FMT_NI_QUAD, AV_PIX_FMT_NONE}; + AVFilterFormats *formats; + + formats =3D ff_make_format_list(pix_fmts); + + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink =3D ctx->inputs[0]; + NetIntDrawTextContext *s =3D ctx->priv; + + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + int ni_pix_fmt; + + av_log(ctx, AV_LOG_DEBUG, "%s inlink wxh %dx%d\n", __func__, + inlink->w, inlink->h); + + outlink->w =3D inlink->w; + outlink->h =3D inlink->h; + + FilterLink *li =3D ff_filter_link(inlink); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + av_log(ctx, AV_LOG_INFO, "vf_drawtext_ni.c %s in_frames_ctx->sw_format= : %d " + "%s\n", __func__, in_frames_ctx->sw_format, + av_get_pix_fmt_name(in_frames_ctx->sw_format)); + if ((ni_pix_fmt =3D ff_to_ni_pix_fmt(in_frames_ctx->sw_format)) < 0) { + return AVERROR(EINVAL); + } + + s->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_ref); + if (!s->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D outlink->w; + out_frames_ctx->height =3D outlink->h; + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + out_frames_ctx->initial_pool_size =3D NI_DRAWTEXT_ID; + + av_hwframe_ctx_init(s->out_frames_ref); + FilterLink *lo =3D ff_filter_link(ctx->outputs[0]); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + //The upload will be per frame if frame rate is not specified/determin= ed + if (li->frame_rate.den) + s->framerate =3D (li->frame_rate.num + li->frame_rate.den - 1) / l= i->frame_rate.den; + + if (s->framerate =3D=3D 0) + s->framerate =3D 1; + av_log(ctx, AV_LOG_INFO, "overlay frame upload frequency %d\n", s->fra= merate); + + return 0; +} + +static int glyph_enu_free(void *opaque, void *elem) +{ + Glyph *glyph =3D elem; + + FT_Done_Glyph(glyph->glyph); + FT_Done_Glyph(glyph->border_glyph); + av_free(elem); + return 0; +} + + +static av_cold int init(AVFilterContext *ctx) +{ + int i, err; + NetIntDrawTextContext *s =3D ctx->priv; + Glyph *glyph; + + for (i =3D 0; i < s->text_num; i++) { + av_expr_free(s->fontsize_pexpr[i]); + s->fontsize_pexpr[i] =3D NULL; + + s->fontsize[i] =3D 0; + } + for (i =3D 0; i < MAX_TEXT_NUM; i++) { + s->text_last_updated[0] =3D NULL; + s->x_bak[i] =3D 0; + s->y_bak[i] =3D 0; + } + s->default_fontsize =3D 16; + s->upload_drawtext_frame =3D 1; + s->keep_overlay =3D NULL; + s->filtered_frame_count=3D0; + s->framerate =3D 0; + + if (!s->fontfile && !CONFIG_LIBFONTCONFIG) { + av_log(ctx, AV_LOG_ERROR, "No font filename provided\n"); + return AVERROR(EINVAL); + } + + if (s->textfile) { + if (s->text[0]) { + av_log(ctx, AV_LOG_ERROR, + "Both text and text file provided. Please provide only = one\n"); + return AVERROR(EINVAL); + } + if ((err =3D load_textfile(ctx)) < 0) + return err; + } + + s->text_num =3D 0; + for (i =3D 0; i < MAX_TEXT_NUM; i++) { + if (!s->text[i]) { + break; + } + s->text_num++; + } + + if (s->reload && !s->textfile) + av_log(ctx, AV_LOG_WARNING, "No file to reload\n"); + + if (s->tc_opt_string) { + int ret =3D av_timecode_init_from_string(&s->tc, s->tc_rate, + s->tc_opt_string, ctx); + if (ret < 0) + return ret; + if (s->tc24hmax) + s->tc.flags |=3D AV_TIMECODE_FLAG_24HOURSMAX; + if (!s->text[0]) + s->text[0] =3D av_strdup(""); + } + + if (!s->text_num) { + av_log(ctx, AV_LOG_ERROR, + "Either text, a valid file or a timecode must be provided\n= "); + return AVERROR(EINVAL); + } + +#if CONFIG_LIBFRIBIDI + if (s->text_shaping) + if ((err =3D shape_text(ctx)) < 0) + return err; +#endif + + if ((err =3D FT_Init_FreeType(&(s->library)))) { + av_log(ctx, AV_LOG_ERROR, + "Could not load FreeType: %s\n", FT_ERRMSG(err)); + return AVERROR(EINVAL); + } + + for (i =3D 0; i < s->text_num; i++) { + if ((err =3D load_font(ctx, i)) < 0) + return err; + + if ((err =3D update_fontsize(ctx, i)) < 0) + return err; + } + + if (s->borderw) { + if (FT_Stroker_New(s->library, &s->stroker)) { + av_log(ctx, AV_LOG_ERROR, "Coult not init FT stroker\n"); + return AVERROR_EXTERNAL; + } + FT_Stroker_Set(s->stroker, s->borderw << 6, FT_STROKER_LINECAP_ROU= ND, + FT_STROKER_LINEJOIN_ROUND, 0); + } + + for (i =3D 0; i < s->text_num; i++) { + s->use_kerning[i] =3D FT_HAS_KERNING(s->face[i]); + + /* load the fallback glyph with code 0 */ + load_glyph(ctx, NULL, 0, i); + + /* set the tabsize in pixels */ + if ((err =3D load_glyph(ctx, &glyph, ' ', i)) < 0) { + av_log(ctx, AV_LOG_ERROR, "Could not set tabsize.\n"); + return err; + } + if (i > 0) { + s->tabsize[i] =3D s->tabsize[0]; + } + s->tabsize[i] *=3D glyph->advance; + + if (s->exp_mode =3D=3D EXP_STRFTIME && + (strchr(s->text[i], '%') || strchr(s->text[i], '\\'))) + av_log(ctx, AV_LOG_WARNING, "expansion=3Dstrftime is deprecate= d.\n"); + } + + av_bprint_init(&s->expanded_text, 0, AV_BPRINT_SIZE_UNLIMITED); + av_bprint_init(&s->expanded_fontcolor, 0, AV_BPRINT_SIZE_UNLIMITED); + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntDrawTextContext *s =3D ctx->priv; + int i; + // NI HW frame related uninit + av_frame_free(&s->keep_overlay); + ni_frame_buffer_free(&s->dl_frame.data.frame); + ni_frame_buffer_free(&s->txt_frame.data.frame); + av_frame_free(&s->up_frame); + + if (s->api_dst_frame.data.frame.p_buffer) { + ni_frame_buffer_free(&s->api_dst_frame.data.frame); + } + + if (s->crop_api_dst_frame.data.frame.p_buffer) { + ni_frame_buffer_free(&s->crop_api_dst_frame.data.frame); + } + + if (s->session_opened) { + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + } + + if (s->crop_session_opened) { + ni_device_session_close(&s->crop_api_ctx, 1, NI_DEVICE_TYPE_SCALER= ); + ni_device_session_context_clear(&s->crop_api_ctx); + } + + av_buffer_unref(&s->hwframe); + av_buffer_unref(&s->hwdevice); + av_buffer_unref(&s->hw_frames_ctx); + + av_buffer_unref(&s->out_frames_ref); + + for (i =3D 0; i < s->text_num; i++) { + av_expr_free(s->x_pexpr[i]); + av_expr_free(s->y_pexpr[i]); + av_expr_free(s->fontsize_pexpr[i]); + av_free(s->text_last_updated[i]); + s->text_last_updated[i] =3D NULL; + + s->x_pexpr[i] =3D s->y_pexpr[i] =3D s->fontsize_pexpr[i] =3D NULL; + } + av_expr_free(s->a_pexpr); + s->a_pexpr =3D NULL; + + av_freep(&s->positions); + s->nb_positions =3D 0; + + av_tree_enumerate(s->glyphs, NULL, NULL, glyph_enu_free); + av_tree_destroy(s->glyphs); + s->glyphs =3D NULL; + + for (i =3D 0; i < s->text_num; i++) { + FT_Done_Face(s->face[i]); + } + FT_Stroker_Done(s->stroker); + FT_Done_FreeType(s->library); + + av_bprint_finalize(&s->expanded_text, NULL); + av_bprint_finalize(&s->expanded_fontcolor, NULL); +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx =3D inlink->dst; + NetIntDrawTextContext *s =3D ctx->priv; + char *expr; + int i, ret, flags; + AVHWFramesContext *in_frames_ctx; + + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + av_log(ctx, AV_LOG_INFO, "vf_drawtext_ni.c: inlink->format %d " + "in_frames_ctx->sw_format %d %s\n", + inlink->format, in_frames_ctx->sw_format, + av_get_pix_fmt_name(in_frames_ctx->sw_format)); + + switch (in_frames_ctx->sw_format) { + case AV_PIX_FMT_NI_QUAD_8_TILE_4X4: + case AV_PIX_FMT_NI_QUAD_10_TILE_4X4: + av_log(ctx, AV_LOG_ERROR, "Error vf_drawtext_ni.c: frame pixel for= mat " + "not supported !\n"); + return AVERROR(EINVAL); + default: + break; + } + + s->main_has_alpha =3D ff_fmt_is_in(in_frames_ctx->sw_format, alpha_pix= _fmts); + + flags =3D FF_DRAW_PROCESS_ALPHA; + if (ff_draw_init(&s->dc, AV_PIX_FMT_RGBA, flags) + < 0) { + av_log(ctx, AV_LOG_ERROR, "Error vf_drawtext_ni.c: frame pixel for= mat " + "not supported !\n"); + return AVERROR(EINVAL); + } else { + av_log(ctx, AV_LOG_INFO, "%s ff_draw_init success main_has_alpha: = %d\n", + __func__, s->main_has_alpha); + } + + for (i =3D 0; i < s->text_num; i++) { + ff_draw_color(&s->dc, &s->fontcolor[i], s->fontcolor[i].rgba); + ff_draw_color(&s->dc, &s->boxcolor[i], s->boxcolor[i].rgba); + } + ff_draw_color(&s->dc, &s->shadowcolor, s->shadowcolor.rgba); + ff_draw_color(&s->dc, &s->bordercolor, s->bordercolor.rgba); + + s->var_values[VAR_w] =3D s->var_values[VAR_W] =3D s->var_value= s[VAR_MAIN_W] =3D inlink->w; + s->var_values[VAR_h] =3D s->var_values[VAR_H] =3D s->var_value= s[VAR_MAIN_H] =3D inlink->h; + s->var_values[VAR_SAR] =3D inlink->sample_aspect_ratio.num ? av_q2d(= inlink->sample_aspect_ratio) : 1; + s->var_values[VAR_DAR] =3D (double)inlink->w / inlink->h * s->var_va= lues[VAR_SAR]; + s->var_values[VAR_HSUB] =3D 1 << s->dc.hsub_max; + s->var_values[VAR_VSUB] =3D 1 << s->dc.vsub_max; + s->var_values[VAR_X] =3D NAN; + s->var_values[VAR_Y] =3D NAN; + s->var_values[VAR_T] =3D NAN; + + av_lfg_init(&s->prng, av_get_random_seed()); + + for (i =3D 0; i < s->text_num; i++) { + av_expr_free(s->x_pexpr[i]); + av_expr_free(s->y_pexpr[i]); + + s->x_pexpr[i] =3D s->y_pexpr[i] =3D NULL; + + if ((ret =3D av_expr_parse(&s->x_pexpr[i], expr =3D s->x_expr[i], = var_names, + NULL, NULL, fun2_names, fun2, 0, ctx)) < 0= || + (ret =3D av_expr_parse(&s->y_pexpr[i], expr =3D s->y_expr[i], = var_names, + NULL, NULL, fun2_names, fun2, 0, ctx)) < 0= ) { + av_log(ctx, AV_LOG_ERROR, "Failed to parse expression: %s \n",= expr); + return AVERROR(EINVAL); + } + } + + av_expr_free(s->a_pexpr); + s->a_pexpr =3D NULL; + + if (ret =3D av_expr_parse(&s->a_pexpr, expr =3D s->a_expr, var_names, + NULL, NULL, fun2_names, fun2, 0, ctx) < 0)= { + av_log(ctx, AV_LOG_ERROR, "Failed to parse expression: %s \n", exp= r); + return AVERROR(EINVAL); + } + + // prep download/upload buffer + if (ni_frame_buffer_alloc_dl(&(s->dl_frame.data.frame), + inlink->w, inlink->h, NI_PIX_FMT_RGBA)) { + return AVERROR(ENOMEM); + } + s->up_frame =3D av_frame_alloc(); + + return 0; +} + +static int command(AVFilterContext *ctx, const char *cmd, const char *arg,= char *res, int res_len, int flags) +{ + NetIntDrawTextContext *old =3D ctx->priv; + NetIntDrawTextContext *new =3D NULL; + int ret; + + if (!strcmp(cmd, "reinit")) { + new =3D av_mallocz(sizeof(NetIntDrawTextContext)); + if (!new) + return AVERROR(ENOMEM); + + new->class =3D &ni_drawtext_class; + ret =3D av_opt_copy(new, old); + if (ret < 0) + goto fail; + + ctx->priv =3D new; + ret =3D av_set_options_string(ctx, arg, "=3D", ":"); + if (ret < 0) { + ctx->priv =3D old; + goto fail; + } + + ret =3D init(ctx); + if (ret < 0) { + uninit(ctx); + ctx->priv =3D old; + goto fail; + } + + new->reinit =3D 1; + new->initialized =3D 0; + new->out_frames_ref =3D av_buffer_ref(old->out_frames_ref); + + ctx->priv =3D old; + // NETINT/FFmpeg-patch: + // fix memory leak for ffmpeg while using this function + av_opt_free(old); + uninit(ctx); + av_freep(&old); + + ctx->priv =3D new; + return config_input(ctx->inputs[0]); + } else { + return AVERROR(ENOSYS); + } + +fail: + av_log(ctx, AV_LOG_ERROR, "Failed to process command. Continuing with = existing parameters.\n"); + av_freep(&new); + return ret; +} + +static int func_pict_type(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + NetIntDrawTextContext *s =3D ctx->priv; + + av_bprintf(bp, "%c", av_get_picture_type_char(s->var_values[VAR_PICT_T= YPE])); + return 0; +} + +static int func_pts(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + NetIntDrawTextContext *s =3D ctx->priv; + const char *fmt; + double pts =3D s->var_values[VAR_T]; + int ret; + + fmt =3D argc >=3D 1 ? argv[0] : "flt"; + if (argc >=3D 2) { + int64_t delta; + if ((ret =3D av_parse_time(&delta, argv[1], 1)) < 0) { + av_log(ctx, AV_LOG_ERROR, "Invalid delta '%s'\n", argv[1]); + return ret; + } + pts +=3D (double)delta / AV_TIME_BASE; + } + if (!strcmp(fmt, "flt")) { + av_bprintf(bp, "%.6f", pts); + } else if (!strcmp(fmt, "hms")) { + if (isnan(pts)) { + av_bprintf(bp, " ??:??:??.???"); + } else { + int64_t ms =3D llrint(pts * 1000); + char sign =3D ' '; + if (ms < 0) { + sign =3D '-'; + ms =3D -ms; + } + if (argc >=3D 3) { + if (!strcmp(argv[2], "24HH")) { + ms %=3D 24 * 60 * 60 * 1000; + } else { + av_log(ctx, AV_LOG_ERROR, "Invalid argument '%s'\n", a= rgv[2]); + return AVERROR(EINVAL); + } + } + av_bprintf(bp, "%c%02d:%02d:%02d.%03d", sign, + (int)(ms / (60 * 60 * 1000)), + (int)(ms / (60 * 1000)) % 60, + (int)(ms / 1000) % 60, + (int)(ms % 1000)); + } + } else if (!strcmp(fmt, "localtime") || + !strcmp(fmt, "gmtime")) { + struct tm tm; + time_t ms =3D (time_t)pts; + const char *timefmt =3D argc >=3D 3 ? argv[2] : "%Y-%m-%d %H:%M:%S= "; + if (!strcmp(fmt, "localtime")) + localtime_r(&ms, &tm); + else + gmtime_r(&ms, &tm); + av_bprint_strftime(bp, timefmt, &tm); + } else { + av_log(ctx, AV_LOG_ERROR, "Invalid format '%s'\n", fmt); + return AVERROR(EINVAL); + } + return 0; +} + +static int func_frame_num(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + NetIntDrawTextContext *s =3D ctx->priv; + + av_bprintf(bp, "%d", (int)s->var_values[VAR_N]); + return 0; +} + +static int func_metadata(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + NetIntDrawTextContext *s =3D ctx->priv; + AVDictionaryEntry *e =3D av_dict_get(s->metadata, argv[0], NULL, 0); + + if (e && e->value) + av_bprintf(bp, "%s", e->value); + else if (argc >=3D 2) + av_bprintf(bp, "%s", argv[1]); + return 0; +} + +static int func_strftime(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + const char *fmt =3D argc ? argv[0] : "%Y-%m-%d %H:%M:%S"; + time_t now; + struct tm tm; + + time(&now); + if (tag =3D=3D 'L') + localtime_r(&now, &tm); + else + tm =3D *gmtime_r(&now, &tm); + av_bprint_strftime(bp, fmt, &tm); + return 0; +} + +static int func_eval_expr(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + NetIntDrawTextContext *s =3D ctx->priv; + double res; + int ret; + + ret =3D av_expr_parse_and_eval(&res, argv[0], var_names, s->var_values, + NULL, NULL, fun2_names, fun2, + &s->prng, 0, ctx); + if (ret < 0) + av_log(ctx, AV_LOG_ERROR, + "Expression '%s' for the expr text expansion function is no= t valid\n", + argv[0]); + else + av_bprintf(bp, "%f", res); + + return ret; +} + +static int func_eval_expr_int_format(AVFilterContext *ctx, AVBPrint *bp, + char *fct, unsigned argc, char **argv, int tag) +{ + NetIntDrawTextContext *s =3D ctx->priv; + double res; + int intval; + int ret; + unsigned int positions =3D 0; + char fmt_str[30] =3D "%"; + + /* + * argv[0] expression to be converted to `int` + * argv[1] format: 'x', 'X', 'd' or 'u' + * argv[2] positions printed (optional) + */ + + ret =3D av_expr_parse_and_eval(&res, argv[0], var_names, s->var_values, + NULL, NULL, fun2_names, fun2, + &s->prng, 0, ctx); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Expression '%s' for the expr text expansion function is no= t valid\n", + argv[0]); + return ret; + } + + if (!strchr("xXdu", argv[1][0])) { + av_log(ctx, AV_LOG_ERROR, "Invalid format '%c' specified," + " allowed values: 'x', 'X', 'd', 'u'\n", argv[1][0]); + return AVERROR(EINVAL); + } + + if (argc =3D=3D 3) { + ret =3D sscanf(argv[2], "%u", &positions); + if (ret !=3D 1) { + av_log(ctx, AV_LOG_ERROR, "expr_int_format(): Invalid number o= f positions" + " to print: '%s'\n", argv[2]); + return AVERROR(EINVAL); + } + } + + feclearexcept(FE_ALL_EXCEPT); + intval =3D res; +#if defined(FE_INVALID) && defined(FE_OVERFLOW) && defined(FE_UNDERFLOW) + if ((ret =3D fetestexcept(FE_INVALID|FE_OVERFLOW|FE_UNDERFLOW))) { + av_log(ctx, AV_LOG_ERROR, "Conversion of floating-point result to = int failed. Control register: 0x%08x. Conversion result: %d\n", ret, intval= ); + return AVERROR(EINVAL); + } +#endif + + if (argc =3D=3D 3) + av_strlcatf(fmt_str, sizeof(fmt_str), "0%u", positions); + av_strlcatf(fmt_str, sizeof(fmt_str), "%c", argv[1][0]); + + av_log(ctx, AV_LOG_DEBUG, "Formatting value %f (expr '%s') with spec '= %s'\n", + res, argv[0], fmt_str); + + av_bprintf(bp, fmt_str, intval); + + return 0; +} + +static const struct drawtext_function { + const char *name; + unsigned argc_min, argc_max; + int tag; /**< opaque argument to func */ + int (*func)(AVFilterContext *, AVBPrint *, char *, unsigned, char **, = int); +} functions[] =3D { + { "expr", 1, 1, 0, func_eval_expr }, + { "e", 1, 1, 0, func_eval_expr }, + { "expr_int_format", 2, 3, 0, func_eval_expr_int_format }, + { "eif", 2, 3, 0, func_eval_expr_int_format }, + { "pict_type", 0, 0, 0, func_pict_type }, + { "pts", 0, 3, 0, func_pts }, + { "gmtime", 0, 1, 'G', func_strftime }, + { "localtime", 0, 1, 'L', func_strftime }, + { "frame_num", 0, 0, 0, func_frame_num }, + { "n", 0, 0, 0, func_frame_num }, + { "metadata", 1, 2, 0, func_metadata }, +}; + +static int eval_function(AVFilterContext *ctx, AVBPrint *bp, char *fct, + unsigned argc, char **argv) +{ + unsigned i; + + for (i =3D 0; i < FF_ARRAY_ELEMS(functions); i++) { + if (strcmp(fct, functions[i].name)) + continue; + if (argc < functions[i].argc_min) { + av_log(ctx, AV_LOG_ERROR, "%%{%s} requires at least %d argumen= ts\n", + fct, functions[i].argc_min); + return AVERROR(EINVAL); + } + if (argc > functions[i].argc_max) { + av_log(ctx, AV_LOG_ERROR, "%%{%s} requires at most %d argument= s\n", + fct, functions[i].argc_max); + return AVERROR(EINVAL); + } + break; + } + if (i >=3D FF_ARRAY_ELEMS(functions)) { + av_log(ctx, AV_LOG_ERROR, "%%{%s} is not known\n", fct); + return AVERROR(EINVAL); + } + return functions[i].func(ctx, bp, fct, argc, argv, functions[i].tag); +} + +static int expand_function(AVFilterContext *ctx, AVBPrint *bp, char **rtex= t) +{ + const char *text =3D *rtext; + char *argv[16] =3D { NULL }; + unsigned argc =3D 0, i; + int ret; + + if (*text !=3D '{') { + av_log(ctx, AV_LOG_ERROR, "Stray %% near '%s'\n", text); + return AVERROR(EINVAL); + } + text++; + while (1) { + if (!(argv[argc++] =3D av_get_token(&text, ":}"))) { + ret =3D AVERROR(ENOMEM); + goto end; + } + if (!*text) { + av_log(ctx, AV_LOG_ERROR, "Unterminated %%{} near '%s'\n", *rt= ext); + ret =3D AVERROR(EINVAL); + goto end; + } + if (argc =3D=3D FF_ARRAY_ELEMS(argv)) + av_freep(&argv[--argc]); /* error will be caught later */ + if (*text =3D=3D '}') + break; + text++; + } + + if ((ret =3D eval_function(ctx, bp, argv[0], argc - 1, argv + 1)) < 0) + goto end; + ret =3D 0; + *rtext =3D (char *)text + 1; + +end: + for (i =3D 0; i < argc; i++) + av_freep(&argv[i]); + return ret; +} + +static int expand_text(AVFilterContext *ctx, char *text, AVBPrint *bp) +{ + int ret; + + av_bprint_clear(bp); + while (*text) { + if (*text =3D=3D '\\' && text[1]) { + av_bprint_chars(bp, text[1], 1); + text +=3D 2; + } else if (*text =3D=3D '%') { + text++; + if ((ret =3D expand_function(ctx, bp, &text)) < 0) + return ret; + } else { + av_bprint_chars(bp, *text, 1); + text++; + } + } + if (!av_bprint_is_complete(bp)) + return AVERROR(ENOMEM); + return 0; +} + +static int draw_glyphs(NetIntDrawTextContext *s, ni_frame_t *frame, + int width, int height, + FFDrawColor *color, + int x, int y, int borderw, int index) +{ + char *text =3D s->expanded_text.str; + uint32_t code =3D 0; + int i, x1, y1; + uint8_t *p; + Glyph *glyph =3D NULL; + int dst_linesize[NI_MAX_NUM_DATA_POINTERS] =3D {0}; + + dst_linesize[0] =3D frame->data_len[0] / height; + dst_linesize[1] =3D dst_linesize[2] =3D frame->data_len[1] / (height /= 2); + + for (i =3D 0, p =3D text; *p; i++) { + FT_Bitmap bitmap; + Glyph dummy =3D { 0 }; + GET_UTF8(code, *p ? *p++ : 0, code =3D 0xfffd; goto continue_on_in= valid;); +continue_on_invalid: + /* skip new line chars, just go to new line */ + if (code =3D=3D '\n' || code =3D=3D '\r' || code =3D=3D '\t') + continue; + + dummy.code =3D code; + dummy.fontsize =3D s->fontsize[index]; + glyph =3D av_tree_find(s->glyphs, &dummy, glyph_cmp, NULL); + + bitmap =3D borderw ? glyph->border_bitmap : glyph->bitmap; + + if (glyph->bitmap.pixel_mode !=3D FT_PIXEL_MODE_MONO && + glyph->bitmap.pixel_mode !=3D FT_PIXEL_MODE_GRAY) + return AVERROR(EINVAL); + + x1 =3D s->positions[i].x + s->x[index] + x - borderw; + y1 =3D s->positions[i].y + s->y[index] + y - borderw; + + ff_blend_mask(&s->dc, color, + frame->p_data, dst_linesize, width, height, + bitmap.buffer, bitmap.pitch, + bitmap.width, bitmap.rows, + bitmap.pixel_mode =3D=3D FT_PIXEL_MODE_MONO ? 0 : 3, + 0, x1, y1); + } + + return 0; +} + + +static void update_color_with_alpha(NetIntDrawTextContext *s, FFDrawColor = *color, const FFDrawColor incolor) +{ + *color =3D incolor; + color->rgba[3] =3D (color->rgba[3] * s->alpha) / 255; + ff_draw_color(&s->dc, color, color->rgba); +} + +static void update_alpha(NetIntDrawTextContext *s) +{ + double alpha =3D av_expr_eval(s->a_pexpr, s->var_values, &s->prng); + + if (isnan(alpha)) + return; + + if (alpha >=3D 1.0) + s->alpha =3D 255; + else if (alpha <=3D 0) + s->alpha =3D 0; + else + s->alpha =3D 256 * alpha; +} + +static void update_canvas_size(NetIntDrawTextContext *s, int x, int y, int= w, int h) +{ + if (s->x_start =3D=3D 0 && s->x_end =3D=3D -1 && + s->y_start =3D=3D 0 && s->y_end =3D=3D -1) { + s->x_start =3D x; + s->y_start =3D y; + s->x_end =3D x + w; + s->y_end =3D y + h; + return; + } + if (x < s->x_start) + s->x_start =3D x; + if (y < s->y_start) + s->y_start =3D y; + if (x + w > s->x_end) + s->x_end =3D x + w; + if (y + h > s->y_end) + s->y_end =3D y + h; +} + +static void update_watermark_internal(ni_scaler_watermark_params_t *multi_= watermark_params, int x, int y, int w, int h) +{ + if (w =3D=3D 0 || h =3D=3D 0) { + return; + } + if (multi_watermark_params->ui32Valid) { + uint32_t x_end =3D multi_watermark_params->ui32StartX + multi_wate= rmark_params->ui32Width; + uint32_t y_end =3D multi_watermark_params->ui32StartY + multi_wate= rmark_params->ui32Height; + multi_watermark_params->ui32StartX =3D FFMIN(multi_watermark_param= s->ui32StartX, x); + multi_watermark_params->ui32StartY =3D FFMIN(multi_watermark_param= s->ui32StartY, y); + x_end =3D FFMAX(x_end, x + w); + y_end =3D FFMAX(y_end, y + h); + multi_watermark_params->ui32Width =3D x_end - multi_watermark_para= ms->ui32StartX; + multi_watermark_params->ui32Height =3D y_end - multi_watermark_par= ams->ui32StartY; + } else { + multi_watermark_params->ui32Valid =3D 1; + multi_watermark_params->ui32StartX =3D x; + multi_watermark_params->ui32StartY =3D y; + multi_watermark_params->ui32Width =3D w; + multi_watermark_params->ui32Height =3D h; + } +} + +static void update_signal_watermark(int x0, int y0, int w0, int h0, + int x1, int y1, int w1, int h1, + NetIntDrawTextContext *s, int index) +{ + int inter_x_start =3D FFMAX(x0, x1); + int inter_y_start =3D FFMAX(y0, y1); + int inter_x_end =3D FFMIN(x0 + w0, x1 + w1); + int inter_y_end =3D FFMIN(y0 + h0, y1 + h1); + if (inter_x_start >=3D inter_x_end || inter_y_start >=3D inter_y_end) { + return; + } else { + av_log(s, AV_LOG_DEBUG, "index %d, x0 %d y0 %d w0 %d h0 %d\n", ind= ex, + x0, y0, w0, h0); + av_log(s, AV_LOG_DEBUG, "index %d, xstart %d ystart %d xend %d yen= d %d\n", index, + inter_x_start, inter_y_start, inter_x_end, inter_y_end); + update_watermark_internal(&(s->scaler_watermark_paras.multi_waterm= ark_params[index]), + inter_x_start, inter_y_start, + inter_x_end - inter_x_start, inter_y_end= - inter_y_start); + } +} + +static void update_watermark(NetIntDrawTextContext *s, int x, int y, int w= , int h) +{ + int frame_width =3D s->watermark_width0 + s->watermark_width1; + int frame_height =3D (s->watermark_height0 * 2) + s->watermark_height1; + if (x < 0) { + w =3D FFMAX(w + x, 0); + x =3D 0; + } + if (y < 0) { + h =3D FFMAX(h + y, 0); + y =3D 0; + } + if (x + w > frame_width) { + x =3D FFMIN(x, frame_width); + w =3D frame_width - x; + } + if (y + h > frame_height) { + y =3D FFMIN(y, frame_height); + h =3D frame_height - y; + } + + for (int watermark_idx =3D 0; watermark_idx < NI_MAX_SUPPORT_WATERMARK= _NUM; watermark_idx++) { + update_signal_watermark(x, y, w, h, + s->watermark_width0 * (watermark_idx % 2), + s->watermark_height0 * (watermark_idx / 2), + watermark_idx % 2 ? s->watermark_width1 : = s->watermark_width0, + watermark_idx > 3 ? s->watermark_height1 := s->watermark_height0, + s, watermark_idx); + } +} + +static void check_and_expand_canvas_size(NetIntDrawTextContext *s, int min= _filter_width, int min_filter_heigth) +{ + int x_distance =3D s->x_end - s->x_start; + int y_distance =3D s->y_end - s->y_start; + + if (x_distance < min_filter_width) { + if (s->x_start - 0 >=3D min_filter_width - x_distance) { + s->x_start -=3D min_filter_width - x_distance; + } + else { + s->x_end +=3D min_filter_width - x_distance; + } + } + + if (y_distance < min_filter_heigth) { + if (s->y_start - 0 >=3D min_filter_heigth - y_distance) { + s->y_start -=3D min_filter_heigth - y_distance; + } + else { + s->y_end +=3D min_filter_heigth - y_distance; + } + } +} + +static int draw_text(AVFilterContext *ctx, ni_frame_t *frame, + int width, int height, int64_t pts) +{ + NetIntDrawTextContext *s =3D ctx->priv; + AVFilterLink *inlink =3D ctx->inputs[0]; + + uint32_t code =3D 0, prev_code =3D 0; + int x =3D 0, y =3D 0, i =3D 0, j =3D 0, ret; + int max_text_line_w =3D 0, len; + int box_w, box_h; + char *text; + uint8_t *p; + int y_min =3D 32000, y_max =3D -32000; + int x_min =3D 32000, x_max =3D -32000; + FT_Vector delta; + Glyph *glyph =3D NULL, *prev_glyph =3D NULL; + Glyph dummy =3D { 0 }; + + time_t now =3D time(0); + struct tm ltime; + AVBPrint *bp =3D &s->expanded_text; + + FFDrawColor fontcolor; + FFDrawColor shadowcolor; + FFDrawColor bordercolor; + FFDrawColor boxcolor; + unsigned int dst_linesize[NI_MAX_NUM_DATA_POINTERS] =3D {0}; + dst_linesize[0] =3D frame->data_len[0] / height; + dst_linesize[1] =3D dst_linesize[2] =3D frame->data_len[1] / height / = 2; + + av_bprint_clear(bp); + + if (s->basetime !=3D AV_NOPTS_VALUE) + now =3D pts * av_q2d(ctx->inputs[0]->time_base) + s->basetime/1000= 000; + + s->upload_drawtext_frame =3D 0; + + for (i =3D 0; i < s->text_num; i++) { + switch (s->exp_mode) { + case EXP_NONE: + av_bprintf(bp, "%s", s->text[i]); + break; + case EXP_NORMAL: + if ((ret =3D expand_text(ctx, s->text[i], &s->expanded_text)) = < 0) + return ret; + break; + case EXP_STRFTIME: + localtime_r(&now, <ime); + av_bprint_strftime(bp, s->text[i], <ime); + break; + } + if (s->text_last_updated[i] =3D=3D NULL) { + s->upload_drawtext_frame =3D 1; + } else { + if (strcmp(s->text_last_updated[i], bp->str)) + s->upload_drawtext_frame =3D 1; + } + s->text_last_updated[i] =3D av_realloc(s->text_last_updated[i], bp= ->len+1); + strcpy(s->text_last_updated[i], bp->str); + + if (s->tc_opt_string) { + char tcbuf[AV_TIMECODE_STR_SIZE]; + FilterLink *li =3D ff_filter_link(inlink); + av_timecode_make_string(&s->tc, tcbuf, li->frame_count_out); + av_bprint_clear(bp); + av_bprintf(bp, "%s%s", s->text[i], tcbuf); + } + + if (!av_bprint_is_complete(bp)) + return AVERROR(ENOMEM); + text =3D s->expanded_text.str; + if ((len =3D s->expanded_text.len) > s->nb_positions) { + if (!(s->positions =3D + av_realloc(s->positions, len*sizeof(*s->positions)))) + return AVERROR(ENOMEM); + s->nb_positions =3D len; + } + + if (s->fontcolor_expr[i]) { + /* If expression is set, evaluate and replace the static value= */ + av_bprint_clear(&s->expanded_fontcolor); + if ((ret =3D expand_text(ctx, s->fontcolor_expr[i], &s->expand= ed_fontcolor)) < 0) + return ret; + if (!av_bprint_is_complete(&s->expanded_fontcolor)) + return AVERROR(ENOMEM); + av_log(s, AV_LOG_DEBUG, "Evaluated fontcolor is '%s'\n", s->ex= panded_fontcolor.str); + ret =3D av_parse_color(s->fontcolor[i].rgba, s->expanded_fontc= olor.str, -1, s); + if (ret) + return ret; + ff_draw_color(&s->dc, &s->fontcolor[i], s->fontcolor[i].rgba); + } + + x =3D 0; + y =3D 0; + max_text_line_w =3D 0; + + if ((ret =3D update_fontsize(ctx, i)) < 0) + return ret; + + /* load and cache glyphs */ + for (j =3D 0, p =3D text; *p; j++) { + GET_UTF8(code, *p ? *p++ : 0, code =3D 0xfffd; goto continue_o= n_invalid;); +continue_on_invalid: + /* get glyph */ + dummy.code =3D code; + dummy.fontsize =3D s->fontsize[i]; + glyph =3D av_tree_find(s->glyphs, &dummy, glyph_cmp, NULL); + if (!glyph) { + ret =3D load_glyph(ctx, &glyph, code, i); + if (ret < 0) + return ret; + } + + y_min =3D FFMIN(glyph->bbox.yMin, y_min); + y_max =3D FFMAX(glyph->bbox.yMax, y_max); + x_min =3D FFMIN(glyph->bbox.xMin, x_min); + x_max =3D FFMAX(glyph->bbox.xMax, x_max); + } + s->max_glyph_h =3D y_max - y_min; + s->max_glyph_w =3D x_max - x_min; + + /* compute and save position for each glyph */ + glyph =3D NULL; + for (j =3D 0, p =3D text; *p; j++) { + GET_UTF8(code, *p ? *p++ : 0, code =3D 0xfffd; goto continue_o= n_invalid2;); +continue_on_invalid2: + /* skip the \n in the sequence \r\n */ + if (prev_code =3D=3D '\r' && code =3D=3D '\n') + continue; + + prev_code =3D code; + if (is_newline(code)) { + + max_text_line_w =3D FFMAX(max_text_line_w, x); + y +=3D s->max_glyph_h + s->line_spacing; + x =3D 0; + continue; + } + + /* get glyph */ + prev_glyph =3D glyph; + dummy.code =3D code; + dummy.fontsize =3D s->fontsize[i]; + glyph =3D av_tree_find(s->glyphs, &dummy, glyph_cmp, NULL); + + /* kerning */ + if (s->use_kerning[i] && prev_glyph && glyph->code) { + FT_Get_Kerning(s->face[i], prev_glyph->code, glyph->code, + ft_kerning_default, &delta); + x +=3D delta.x >> 6; + } + + /* save position */ + s->positions[j].x =3D x + glyph->bitmap_left; + s->positions[j].y =3D y - glyph->bitmap_top + y_max; + if (code =3D=3D '\t') + x =3D (x / s->tabsize[i] + 1)*s->tabsize[i]; + else + x +=3D glyph->advance; + } + + max_text_line_w =3D FFMAX(x, max_text_line_w); + + s->var_values[VAR_TW] =3D s->var_values[VAR_TEXT_W] =3D max_text_l= ine_w; + s->var_values[VAR_TH] =3D s->var_values[VAR_TEXT_H] =3D y + s->max= _glyph_h; + + s->var_values[VAR_MAX_GLYPH_W] =3D s->max_glyph_w; + s->var_values[VAR_MAX_GLYPH_H] =3D s->max_glyph_h; + s->var_values[VAR_MAX_GLYPH_A] =3D s->var_values[VAR_ASCENT ] =3D = y_max; + s->var_values[VAR_MAX_GLYPH_D] =3D s->var_values[VAR_DESCENT] =3D = y_min; + + s->var_values[VAR_LINE_H] =3D s->var_values[VAR_LH] =3D s->max_gly= ph_h; + + s->x[i] =3D s->var_values[VAR_X] =3D av_expr_eval(s->x_pexpr[i], s= ->var_values, &s->prng); + s->y[i] =3D s->var_values[VAR_Y] =3D av_expr_eval(s->y_pexpr[i], s= ->var_values, &s->prng); + /* It is necessary if x is expressed from y */ + s->x[i] =3D s->var_values[VAR_X] =3D av_expr_eval(s->x_pexpr[i], s= ->var_values, &s->prng); + + update_alpha(s); + update_color_with_alpha(s, &fontcolor , s->fontcolor[i]); + update_color_with_alpha(s, &shadowcolor, s->shadowcolor); + update_color_with_alpha(s, &bordercolor, s->bordercolor); + update_color_with_alpha(s, &boxcolor , s->boxcolor[i]); + + box_w =3D max_text_line_w; + box_h =3D y + s->max_glyph_h; + + if (s->draw_box && s->boxborderw[i]) { + int bbsize[4]; + int count; + count =3D string_to_array(s->boxborderw[i], bbsize, 4); + if (count =3D=3D 1) { + s->bb_top[i] =3D s->bb_right[i] =3D s->bb_bottom[i] =3D s-= >bb_left[i] =3D bbsize[0]; + } else if (count =3D=3D 2) { + s->bb_top[i] =3D s->bb_bottom[i] =3D bbsize[0]; + s->bb_right[i] =3D s->bb_left[i] =3D bbsize[1]; + } else if (count =3D=3D 3) { + s->bb_top[i] =3D bbsize[0]; + s->bb_right[i] =3D s->bb_left[i] =3D bbsize[1]; + s->bb_bottom[i] =3D bbsize[2]; + } else if (count =3D=3D 4) { + s->bb_top[i] =3D bbsize[0]; + s->bb_right[i] =3D bbsize[1]; + s->bb_bottom[i] =3D bbsize[2]; + s->bb_left[i] =3D bbsize[3]; + } + } else { + s->bb_top[i] =3D s->bb_right[i] =3D s->bb_bottom[i] =3D s->bb_= left[i] =3D 0; + } + + if (s->fix_bounds) { + + /* calculate footprint of text effects */ + int borderoffset =3D s->borderw ? FFMAX(s->borderw, 0) : 0; + + int offsetleft =3D FFMAX3(FFMAX(s->bb_left[i], 0), borderoffse= t, + (s->shadowx < 0 ? FFABS(s->shadowx) : = 0)); + int offsettop =3D FFMAX3(FFMAX(s->bb_top[i], 0), borderoffset, + (s->shadowy < 0 ? FFABS(s->shadowy) : = 0)); + int offsetright =3D FFMAX3(FFMAX(s->bb_right[i], 0), borderoff= set, + (s->shadowx > 0 ? s->shadowx : 0)); + int offsetbottom =3D FFMAX3(FFMAX(s->bb_bottom[i], 0), bordero= ffset, + (s->shadowy > 0 ? s->shadowy : 0)); + + + if (s->x[i] - offsetleft < 0) s->x[i] =3D offsetleft; + if (s->y[i] - offsettop < 0) s->y[i] =3D offsettop; + + if (s->x[i] + box_w + offsetright > width) + s->x[i] =3D FFMAX(width - box_w - offsetright, 0); + if (s->y[i] + box_h + offsetbottom > height) + s->y[i] =3D FFMAX(height - box_h - offsetbottom, 0); + } + if (s->x[i] !=3D s->x_bak[i] || s->y[i] !=3D s->y_bak[i]) { + s->x_bak[i] =3D s->x[i]; + s->y_bak[i] =3D s->y[i]; + s->upload_drawtext_frame =3D 1; + } + /* draw box */ + if (s->draw_box) + ff_blend_rectangle(&s->dc, &boxcolor, + frame->p_data, dst_linesize, width, height, + s->x[i] - s->bb_left[i], s->y[i] - s->bb_to= p[i], + box_w + s->bb_left[i] + s->bb_right[i], box= _h + s->bb_top[i] + s->bb_bottom[i]); + + if (s->shadowx || s->shadowy) { + if ((ret =3D draw_glyphs(s, frame, width, height, + &shadowcolor, s->shadowx, s->shadowy, 0= , i)) < 0) + return ret; + } + + if (s->borderw) { + if ((ret =3D draw_glyphs(s, frame, width, height, + &bordercolor, 0, 0, s->borderw, i)) < 0) + return ret; + } + if ((ret =3D draw_glyphs(s, frame, width, height, + &fontcolor, 0, 0, 0, i)) < 0) + return ret; + + update_canvas_size(s, s->x[i] - s->bb_left[i], s->y[i] - s->bb_top= [i], + box_w + s->bb_left[i] + s->bb_right[i], box_h = + s->bb_top[i] + s->bb_bottom[i]); + update_watermark(s, s->x[i] - s->bb_left[i], s->y[i] - s->bb_top[i= ], + box_w + s->bb_left[i] + s->bb_right[i], box_h + s= ->bb_top[i] + s->bb_bottom[i]); + } + return 0; +} + +static int init_hwframe_uploader(AVFilterContext *ctx, NetIntDrawTextConte= xt *s, + AVFrame *frame, int txt_w, int txt_h) +{ + int ret; + AVHWFramesContext *hwframe_ctx; + AVHWFramesContext *out_frames_ctx; + AVHWFramesContext *main_frame_ctx; + AVNIDeviceContext *pAVNIDevCtx; + AVNIFramesContext *f_hwctx, *f_hwctx_output; + int cardno =3D ni_get_cardno(frame); + char buf[64] =3D {0}; + + main_frame_ctx =3D (AVHWFramesContext *)frame->hw_frames_ctx->data; + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + + av_log(ctx, AV_LOG_INFO, "%s out_frames_ctx->sw_format %d %s txt %dx%d= \n", + __func__, out_frames_ctx->sw_format, + av_get_pix_fmt_name(out_frames_ctx->sw_format), txt_w, txt_h); + + snprintf(buf, sizeof(buf), "%d", cardno); + + ret =3D av_hwdevice_ctx_create(&s->hwdevice, AV_HWDEVICE_TYPE_NI_QUADR= A, buf, + NULL, 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "failed to create AV HW device ctx\n"); + return ret; + } + + s->hwframe =3D av_hwframe_ctx_alloc(s->hwdevice); + if (!s->hwframe) + return AVERROR(ENOMEM); + + hwframe_ctx =3D (AVHWFramesContext *)s->hwframe->data; + hwframe_ctx->format =3D AV_PIX_FMT_NI_QUAD; + hwframe_ctx->sw_format =3D AV_PIX_FMT_RGBA; + hwframe_ctx->width =3D txt_w; + hwframe_ctx->height =3D txt_h; + + ret =3D av_hwframe_ctx_init(s->hwframe); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "failed to init AV HW device ctx\n"); + return ret; + } + + // Work around a hwdownload session start timestamp issue + f_hwctx =3D (AVNIFramesContext*) hwframe_ctx->hwctx; + f_hwctx_output =3D (AVNIFramesContext*) out_frames_ctx->hwctx; + f_hwctx_output->api_ctx.session_timestamp =3D + f_hwctx->api_ctx.session_timestamp; + + s->hw_frames_ctx =3D av_buffer_ref(s->hwframe); + if (!s->hw_frames_ctx) + return AVERROR(ENOMEM); + + // set up a scaler session for the in-place overlay + ret =3D ni_device_session_context_init(&s->api_ctx); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "ni overlay filter session context init failure\n"); + return ret; + } + + pAVNIDevCtx =3D (AVNIDeviceContext *)main_frame_ctx->device_ctx->hwctx; + s->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + s->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + + s->api_ctx.hw_id =3D cardno; + s->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->api_ctx.scaler_operation =3D s->use_watermark ? + NI_SCALER_OPCODE_WATERMARK : NI_SCALER= _OPCODE_IPOVLY; + s->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + av_log(ctx, AV_LOG_DEBUG, "%s open overlay session\n", __func__); + ret =3D ni_device_session_open(&s->api_ctx, NI_DEVICE_TYPE_SCALER); + if (ret !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't open scaler session on card %d\n", + cardno); + + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + return ret; + } + + s->session_opened =3D 1; + + ni_cpy_hwframe_ctx(main_frame_ctx, out_frames_ctx); + ni_device_session_copy(&s->api_ctx, &f_hwctx_output->api_ctx); + s->buffer_limit =3D 1; + if (s->use_watermark) { + // init the out pool for the overlay session when use watermark + ret =3D ff_ni_build_frame_pool(&s->api_ctx, frame->width, frame->h= eight, + main_frame_ctx->sw_format, 4, s->buff= er_limit); + + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", ret); + return ret; + } + } + + // if background frame has no alpha, set up an extra intermediate scal= er + // session for the crop operation + if (!s->main_has_alpha && !s->use_watermark) { + ret =3D ni_device_session_context_init(&s->crop_api_ctx); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "ni drawtext filter (crop) session context init failure= \n"); + return ret; + } + + s->crop_api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + s->crop_api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + + s->crop_api_ctx.hw_id =3D cardno; + s->crop_api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->crop_api_ctx.scaler_operation =3D NI_SCALER_OPCODE_CROP; + s->crop_api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + av_log(ctx, AV_LOG_DEBUG, "%s open crop session\n", __func__); + ret =3D ni_device_session_open(&s->crop_api_ctx, NI_DEVICE_TYPE_SC= ALER); + if (ret !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't open crop session on card %d\n", cardno); + + ni_device_session_close(&s->crop_api_ctx, 1, NI_DEVICE_TYPE_SC= ALER); + ni_device_session_context_clear(&s->crop_api_ctx); + return ret; + } + + s->crop_session_opened =3D 1; + + // init the out pool for the crop session, make it rgba + ret =3D ff_ni_build_frame_pool(&s->crop_api_ctx, txt_w, txt_h, + AV_PIX_FMT_RGBA, 1, 0); + + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", ret); + return ret; + } + } + + return 0; +} + +static int ni_drawtext_config_input(AVFilterContext *ctx, AVFrame *frame, + int txt_w, int txt_h) +{ + NetIntDrawTextContext *s =3D ctx->priv; + int ret; + + if (s->initialized) + return 0; + + ret =3D init_hwframe_uploader(ctx, s, frame, txt_w, txt_h); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "failed to initialize uploader session\n= "); + return ret; + } + + s->initialized =3D 1; + return 0; +} + +static int overlay_intersects_background( + const AVFilterContext *ctx, + int overlay_width, + int overlay_height, + const AVFrame *main) +{ + const NetIntDrawTextContext *s =3D (NetIntDrawTextContext *)ctx->priv; + + if (s->x_start >=3D main->width) + return 0; + + if (s->y_start >=3D main->height) + return 0; + + if (s->x_start + overlay_width <=3D 0) + return 0; + + if (s->y_start + overlay_height <=3D 0) + return 0; + + return 1; +} + +static void calculate_dst_rectangle( + int *px, + int *py, + int *pw, + int *ph, + int bgnd_x, + int bgnd_y, + int bgnd_w, + int bgnd_h, + int ovly_x, + int ovly_y, + int ovly_w, + int ovly_h) +{ + *px =3D FFMAX(0, ovly_x); + *py =3D FFMAX(0, ovly_y); + + if (ovly_x > 0) { + *pw =3D FFMIN(bgnd_w - ovly_x, ovly_w); + } else { + *pw =3D FFMIN(ovly_w + ovly_x, bgnd_w); + } + + if (ovly_y > 0) { + *ph =3D FFMIN(bgnd_h - ovly_y, ovly_h); + } else { + *ph =3D FFMIN(ovly_h + ovly_y, bgnd_h); + } +} + +static void init_watermark(NetIntDrawTextContext *s, int width, int height) +{ + s->watermark_width0 =3D width / 2; + s->watermark_width1 =3D width - s->watermark_width0; + s->watermark_height0 =3D height / 3; + s->watermark_height1 =3D height - (2 * s->watermark_height0); + for (int watermark_idx =3D 0; watermark_idx < NI_MAX_SUPPORT_WATERMARK= _NUM; watermark_idx++) { + s->scaler_watermark_paras.multi_watermark_params[watermark_idx].ui= 32StartX =3D 0; + s->scaler_watermark_paras.multi_watermark_params[watermark_idx].ui= 32StartY =3D 0; + s->scaler_watermark_paras.multi_watermark_params[watermark_idx].ui= 32Width =3D 0; + s->scaler_watermark_paras.multi_watermark_params[watermark_idx].ui= 32Height =3D 0; + s->scaler_watermark_paras.multi_watermark_params[watermark_idx].ui= 32Valid =3D 0; + } +} + +static void calculate_src_rectangle( + int *px, + int *py, + int *pw, + int *ph, + int bgnd_x, + int bgnd_y, + int bgnd_w, + int bgnd_h, + int ovly_x, + int ovly_y, + int ovly_w, + int ovly_h) + +{ + *px =3D (ovly_x > 0) ? 0 : -ovly_x; + *py =3D (ovly_y > 0) ? 0 : -ovly_y; + + if (ovly_x > 0) { + *pw =3D FFMIN(bgnd_w - ovly_x, ovly_w); + } else { + *pw =3D FFMIN(ovly_w + ovly_x, bgnd_w); + } + + if (ovly_y > 0) { + *ph =3D FFMIN(bgnd_h - ovly_y, ovly_h); + } else { + *ph =3D FFMIN(ovly_h + ovly_y, bgnd_h); + } +} + +static int do_intermediate_crop_and_overlay( + AVFilterContext *ctx, + AVFrame *overlay, + AVFrame *frame) { + NetIntDrawTextContext *s =3D (NetIntDrawTextContext *)ctx->priv; + AVHWFramesContext *main_frame_ctx; + niFrameSurface1_t *frame_surface; + ni_retcode_t retcode; + uint16_t ui16FrameIdx; + int main_scaler_format, ovly_scaler_format; + int flags; + int crop_x,crop_y,crop_w,crop_h; + int src_x,src_y,src_w,src_h; + + main_frame_ctx =3D (AVHWFramesContext *) frame->hw_frames_ctx->data; + main_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(main_frame_ctx->sw_format); + + ovly_scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA); + + // Allocate a ni_frame_t for the intermediate crop operation + retcode =3D ni_frame_buffer_alloc_hwenc(&s->crop_api_dst_frame.data.fr= ame, + overlay->width, + overlay->height, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't allocate interim crop frame\n"); + return AVERROR(ENOMEM); + } + + calculate_dst_rectangle(&crop_x, &crop_y, &crop_w, &crop_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x_start,2), FFALIGN(s->y_start,2), + overlay->width, overlay->height); + + frame_surface =3D (niFrameSurface1_t *)frame->data[3]; + + // Assign a device input frame. Send incoming frame index to crop sess= ion + retcode =3D ni_device_alloc_frame( + &s->crop_api_ctx, + FFALIGN(frame->width, 2), + FFALIGN(frame->height, 2), + main_scaler_format, + 0, + crop_w, + crop_h, + crop_x, + crop_y, + 0, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't assign input crop frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + // Allocate destination frame. This acquires a frame from the pool + retcode =3D ni_device_alloc_frame( + &s->crop_api_ctx, + FFALIGN(overlay->width, 2), + FFALIGN(overlay->height, 2), + ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA), + NI_SCALER_FLAG_IO, + crop_w, + crop_h, + 0, + 0, + 0, + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't allocate output crop frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + retcode =3D ni_device_session_read_hwdesc(&s->crop_api_ctx, + &s->crop_api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "No cropped output frame %d\n", retcode); + return AVERROR(ENOMEM); + } + + // Get the acquired frame + frame_surface =3D (niFrameSurface1_t *) + s->crop_api_dst_frame.data.frame.p_data[3]; + s->ui16CropFrameIdx =3D frame_surface->ui16FrameIdx; + + av_log(ctx, AV_LOG_DEBUG, "%s intrim crop frame idx [%u]\n", + __func__, s->ui16CropFrameIdx); + + // Overlay the icon over the intermediate cropped frame + + // Allocate a ni_frame_t for the intermediate overlay + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + overlay->width, + overlay->height, + 0); + + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, "Can't allocate interim ovly frame\n"); + return AVERROR(ENOMEM); + } + + frame_surface =3D (niFrameSurface1_t *)overlay->data[3]; + ui16FrameIdx =3D frame_surface->ui16FrameIdx; + + calculate_src_rectangle(&src_x, &src_y, &src_w, &src_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x_start,2), FFALIGN(s->y_start,2), + overlay->width, overlay->height); + + /* Assign input frame to intermediate overlay session */ + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(overlay->width, 2), + FFALIGN(overlay->height, 2), + ovly_scaler_format, + 0, + src_w, + src_h, + src_x, + src_y, + 0, + ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't assign input overlay frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + // In-place overlay frame. Send down frame index of background frame + /* Configure output, Premultiply alpha */ + flags =3D NI_SCALER_FLAG_IO | NI_SCALER_FLAG_PA; + + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(overlay->width, 2), + FFALIGN(overlay->height, 2), + ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA), + flags, + crop_w, + crop_h, + 0, + 0, + 0, + s->ui16CropFrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't overlay frame for output %d\n", + retcode); + return AVERROR(ENOMEM); + } + + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, + &s->api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't acquire intermediate frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + return NI_RETCODE_SUCCESS; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx =3D inlink->dst; + AVFilterLink *outlink =3D ctx->outputs[0]; + NetIntDrawTextContext *s =3D ctx->priv; + niFrameSurface1_t *logging_surface, *logging_surface_out; + uint8_t *p_dst, *p_src; + int y, txt_img_width, txt_img_height; + int ret; + + AVHWFramesContext *main_frame_ctx, *ovly_frame_ctx; + niFrameSurface1_t *frame_surface, *new_frame_surface; + AVFrame *overlay =3D NULL; + AVFrame *out =3D NULL; + uint16_t main_frame_idx =3D 0; + uint16_t ovly_frame_idx =3D 0; + int main_scaler_format, ovly_scaler_format; + int flags; + int src_x, src_y, src_w, src_h; + int dst_x, dst_y, dst_w, dst_h; + int start_row, stop_row, start_col, stop_col; + int dl_frame_linesize0, text_frame_linesize0; + int ovly_width, ovly_height; + + av_log(ctx, AV_LOG_DEBUG, "ni_drawtext %s %dx%d is_hw_frame %d\n", + __func__, frame->width, frame->height, + AV_PIX_FMT_NI_QUAD =3D=3D frame->format); + + if (s->reload) { + if ((ret =3D load_textfile(ctx)) < 0) { + av_frame_free(&frame); + return ret; + } +#if CONFIG_LIBFRIBIDI + if (s->text_shaping) + if ((ret =3D shape_text(ctx)) < 0) { + av_frame_free(&frame); + return ret; + } +#endif + } + + FilterLink *li =3D ff_filter_link(inlink); + s->var_values[VAR_N] =3D li->frame_count_out + s->start_number; + s->var_values[VAR_T] =3D frame->pts =3D=3D AV_NOPTS_VALUE ? + NAN : frame->pts * av_q2d(inlink->time_base); + + s->var_values[VAR_PICT_TYPE] =3D frame->pict_type; + s->var_values[VAR_PKT_DURATION] =3D frame->duration * av_q2d(inlink->t= ime_base); + s->metadata =3D frame->metadata; + s->x_start =3D 0; + s->x_end =3D -1; + s->y_start =3D 0; + s->y_end =3D -1; + init_watermark(s, frame->width, frame->height); + + main_frame_ctx =3D (AVHWFramesContext *)frame->hw_frames_ctx->data; + av_log(ctx, AV_LOG_DEBUG, "%s HW frame, sw_format %d %s, before drawte= xt " + "var_text_WxH %dx%d\n", + __func__, main_frame_ctx->sw_format, + av_get_pix_fmt_name(main_frame_ctx->sw_format), + (int)s->var_values[VAR_TEXT_W], (int)s->var_values[VAR_TEXT_H]); + + memset(s->dl_frame.data.frame.p_buffer, 0, + s->dl_frame.data.frame.buffer_size); + + draw_text(ctx, &(s->dl_frame.data.frame), frame->width, frame->height, + frame->pts); + check_and_expand_canvas_size(s, NI_MIN_RESOLUTION_WIDTH_SCALER, NI_MIN= _RESOLUTION_HEIGHT_SCALER); + + av_log(ctx, AV_LOG_DEBUG, "n:%d t:%f text_w:%d text_h:%d x:%d y:%d " + "shadowx:%d shadowy:%d\n", + (int)s->var_values[VAR_N], s->var_values[VAR_T], + (int)s->var_values[VAR_TEXT_W], (int)s->var_values[VAR_TEXT_H], + s->x_start, s->y_start, s->shadowx, s->shadowy); + + txt_img_width =3D FFALIGN(s->x_end - s->x_start, 2); + txt_img_height =3D FFALIGN(s->y_end - s->y_start, 2); + + if (s->use_watermark) { + ovly_width =3D frame->width; + ovly_height =3D frame->height; + } else { + ovly_width =3D txt_img_width; + ovly_height =3D txt_img_height; + } + // If overlay does not intersect the background, pass + // the frame through the drawtext filter. + if (!overlay_intersects_background(ctx, txt_img_width, txt_img_height, + frame)) { + return ff_filter_frame(outlink, frame); + } + + if (s->use_watermark) { + FilterLink *li =3D ff_filter_link(inlink); + int frame_count =3D li->frame_count_out; + for (int watermark_idx =3D 0; watermark_idx < NI_MAX_SUPPORT_WATER= MARK_NUM; watermark_idx++) { + if (s->scaler_watermark_paras.multi_watermark_params[watermark= _idx].ui32Valid) { + av_log(ctx, AV_LOG_DEBUG, "frame %d index %d, x %d, y %d, = w %d, h %d\n", + frame_count, watermark_idx, + s->scaler_watermark_paras.multi_watermark_params[water= mark_idx].ui32StartX, + s->scaler_watermark_paras.multi_watermark_params[water= mark_idx].ui32StartY, + s->scaler_watermark_paras.multi_watermark_params[water= mark_idx].ui32Width, + s->scaler_watermark_paras.multi_watermark_params[water= mark_idx].ui32Height); + } + } + } + + if (!s->initialized) { + if (s->initiated_upload_width =3D=3D 0) { + s->initiated_upload_width =3D frame->width > ovly_width ? ovly= _width + 4 : frame->width; + s->initiated_upload_height =3D frame->height > ovly_height ? o= vly_height + 4 : frame->height; + } + ret =3D ni_drawtext_config_input(ctx, frame, s->initiated_upload_w= idth, s->initiated_upload_height); + if (ret) { + av_log(ctx, AV_LOG_ERROR, "failed ni_drawtext config input\n"); + goto fail; + } + } + txt_img_width =3D ovly_width =3D s->initiated_upload_width; + txt_img_height =3D ovly_height =3D s->initiated_upload_height; + + // Clear the contents of up_frame to avoid accumulating old data + av_frame_free(&s->up_frame); + s->up_frame =3D av_frame_alloc(); + if (!s->up_frame) { + ret =3D AVERROR(ENOMEM); + goto fail; + } + + if (s->use_watermark) { + if (ni_scaler_set_watermark_params(&s->api_ctx, + &s->scaler_watermark_paras.mult= i_watermark_params[0])) { + av_log(ctx, AV_LOG_ERROR, "failed ni_drawtext set_watermark_pa= rams\n"); + goto fail; + } + // wrap the dl_frame ni_frame into AVFrame up_frame + // for RGBA format, only need to copy the first data + // in some situation, like linesize[0] =3D=3D align64(width*4) + // it will use zero copy, and it need to keep data[1] and data[2] = be null + // for watermark, it uploads the whole frame + s->up_frame->data[0] =3D s->dl_frame.data.frame.p_data[0]; + s->up_frame->linesize[0] =3D FFALIGN(ovly_width, 16) * 4; + } else { + av_log(ctx, AV_LOG_DEBUG, "%s alloc txt_frame %dx%d\n", __func__, + txt_img_width, txt_img_height); + if (ni_frame_buffer_alloc_dl(&(s->txt_frame.data.frame), + txt_img_width, txt_img_height, + NI_PIX_FMT_RGBA)) { + ret =3D AVERROR(ENOMEM); + goto fail; + } + + p_dst =3D s->txt_frame.data.frame.p_buffer; + memset(p_dst, 0, s->txt_frame.data.frame.buffer_size); + + start_row =3D s->y_start; + stop_row =3D start_row + txt_img_height; + dl_frame_linesize0 =3D FFALIGN(frame->width, 16); + text_frame_linesize0 =3D FFALIGN(txt_img_width, 16); + // if overlay intersects at the main top/bottom, only copy the ove= rlaying + // portion + if (start_row < 0) { + p_dst +=3D -1 * start_row * text_frame_linesize0 * 4; + start_row =3D 0; + } + if (stop_row > frame->height) { + stop_row =3D frame->height; + } + + // if overlay intersects at the main left/right, only copy the ove= rlaying + // portion + start_col =3D s->x_start; + stop_col =3D start_col + txt_img_width; + if (start_col < 0) { + p_dst +=3D (-4 * start_col); + start_col =3D 0; + } + if (stop_col > frame->width) { + stop_col =3D frame->width; + } + + for (y =3D start_row; y < stop_row; y++) { + p_src =3D s->dl_frame.data.frame.p_buffer + + (y * dl_frame_linesize0 + start_col) * 4; + + memcpy(p_dst, p_src, (stop_col - start_col) * 4); + p_dst +=3D text_frame_linesize0 * 4; + } + // wrap the txt ni_frame into AVFrame up_frame + // for RGBA format, only need to copy the first data + // in some situation, like linesize[0] =3D=3D align64(width*4) + // it will use zero copy, and it need to keep data[1] and data[2] = be null + // for inplace overlay, it updates the clip include text + s->up_frame->data[0] =3D s->txt_frame.data.frame.p_data[0]; + s->up_frame->linesize[0] =3D text_frame_linesize0 * 4; + } + + if (s->optimize_upload =3D=3D 0) //Force uploading drawtext frame by e= very frame + s->upload_drawtext_frame =3D 1; + + s->filtered_frame_count++; + if (s->filtered_frame_count =3D=3D s->framerate || s->keep_overlay = =3D=3D NULL) { + s->upload_drawtext_frame =3D 1; + s->filtered_frame_count =3D 0; + } + + if (s->upload_drawtext_frame) { + av_frame_free(&s->keep_overlay); + s->keep_overlay =3D NULL; + s->keep_overlay =3D overlay =3D av_frame_alloc(); + if (!overlay) { + ret =3D AVERROR(ENOMEM); + goto fail; + } + + av_frame_copy_props(overlay, frame); + overlay->width =3D ovly_width; + overlay->height =3D ovly_height; + overlay->format =3D AV_PIX_FMT_NI_QUAD; + overlay->color_range =3D AVCOL_RANGE_MPEG; + overlay->hw_frames_ctx =3D s->out_frames_ref; + + ret =3D av_hwframe_get_buffer(s->hw_frames_ctx, overlay, 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "failed to get buffer\n"); + av_frame_free(&overlay); + return ret; + } + + av_frame_copy_props(s->up_frame, frame); + s->up_frame->format =3D AV_PIX_FMT_RGBA; + s->up_frame->width =3D ovly_width; + s->up_frame->height =3D ovly_height; + ret =3D av_hwframe_transfer_data(overlay, // dst src flags + s->up_frame, 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "upload failed, ret =3D %d\n", ret); + return ret; + } + } + else { + overlay =3D s->keep_overlay; + } + // logging + logging_surface =3D (niFrameSurface1_t*)frame->data[3]; + logging_surface_out =3D (niFrameSurface1_t*)overlay->data[3]; + av_log(ctx, AV_LOG_DEBUG, + "vf_drawtext_ni:IN ui16FrameIdx =3D [%d] uploaded overlay =3D [= %d]\n", + logging_surface->ui16FrameIdx, logging_surface_out->ui16FrameId= x); + + // do the in place overlay + main_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(main_frame_ctx->sw_format); + + frame_surface =3D (niFrameSurface1_t *) frame->data[3]; + if (frame_surface =3D=3D NULL) { + av_frame_free(&overlay); + return AVERROR(EINVAL); + } + + main_frame_idx =3D frame_surface->ui16FrameIdx; + + frame_surface =3D (niFrameSurface1_t *) overlay->data[3]; + if (frame_surface =3D=3D NULL) { + av_frame_free(&overlay); + return AVERROR(EINVAL); + } + + ovly_frame_idx =3D frame_surface->ui16FrameIdx; + ovly_frame_ctx =3D (AVHWFramesContext *)overlay->hw_frames_ctx->data; + ovly_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(ovly_frame_ctx->sw_format); + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + // for rgba over yuv, do an intermediate crop and overlay + if (!s->main_has_alpha && !s->use_watermark) { + ret =3D do_intermediate_crop_and_overlay(ctx, overlay, frame); + if (ret < 0) { + av_frame_free(&overlay); + return ret; + } + + // Allocate a ni_frame for the overlay output + ret =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + outlink->w, + outlink->h, + 0); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, "Can't allocate inplace overlay fram= e\n"); + return AVERROR(ENOMEM); + } + + calculate_src_rectangle(&src_x, &src_y, &src_w, &src_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x_start,2),FFALIGN(s->y_start,2= ), + overlay->width, overlay->height); + + // Assign an input frame for overlay picture. Send the + // incoming hardware frame index to the scaler manager. + ret =3D ni_device_alloc_frame( + &s->api_ctx, + overlay->width, // ovly width + overlay->height, // ovly height + ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA), // ovly pix fmt + 0, // flags + src_w, // src rect width + src_h, // src rect height + 0, // src rect x + 0, // src rect y + 0, // n/a + s->ui16CropFrameIdx, // ovly frame idx + NI_DEVICE_TYPE_SCALER); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, "Can't assign input overlay frame %d= \n", + ret); + return AVERROR(ENOMEM); + } + + calculate_dst_rectangle(&dst_x, &dst_y, &dst_w, &dst_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x_start,2), FFALIGN(s->y_start,= 2), + overlay->width, overlay->height); + + // Allocate device output frame from the pool. We also send down t= he + // frame index of the background frame to the scaler manager. + + /* configure the output, premultiply alpha*/ + flags =3D NI_SCALER_FLAG_IO | NI_SCALER_FLAG_PA; + + ret =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(frame->width, 2), // main width + FFALIGN(frame->height, 2), // main height + main_scaler_format, // main pix fmt + flags, // flags + dst_w, // dst rect width + dst_h, // dst rect height + dst_x, // dst rect x + dst_y, // dst rect y + 0, // n/a + main_frame_idx, // main frame idx + NI_DEVICE_TYPE_SCALER); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, "Can't allocate overlay output %d\n", + ret); + return AVERROR(ENOMEM); + } + + // Set the new frame index + ret =3D ni_device_session_read_hwdesc(&s->api_ctx, + &s->api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, + "Can't acquire output overlay frame %d\n", ret); + return AVERROR(ENOMEM); + } + } else { + // we can perform an in-place overlay immediately for rgba over rg= ba, + // or use watermark, it overlay rgab over yuv/rgba + + av_log(ctx, AV_LOG_DEBUG, "%s overlay %s main %s\n", __func__, + av_get_pix_fmt_name(ovly_frame_ctx->sw_format), + av_get_pix_fmt_name(main_frame_ctx->sw_format)); + + /* Allocate ni_frame for the overlay output */ + ret =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + outlink->w, + outlink->h, + 0); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, "Cannot allocate in-place frame\n"); + return AVERROR(ENOMEM); + } + + if (!s->use_watermark) { + calculate_src_rectangle(&src_x, &src_y, &src_w, &src_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x_start,2), FFALIGN(s->y_start,2), + overlay->width, overlay->height); + } + + /* + * Assign input frame for overlay picture. Sends the + * incoming hardware frame index to the scaler manager. + */ + ret =3D ni_device_alloc_frame( + &s->api_ctx, + overlay->width, // overlay width + overlay->height, // overlay height + ovly_scaler_format, // overlay pix fmt + 0, // flags + s->use_watermark ? ovly_width : src_w, // src rect width + s->use_watermark ? ovly_height : src_h, // src rect height + s->use_watermark ? 0 : src_x, // src rect x + s->use_watermark ? 0 : src_y, // src rect y + 0, // n/a + ovly_frame_idx, // overlay frame idx + NI_DEVICE_TYPE_SCALER); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, + "Can't assign frame for overlay input %d\n", ret); + return AVERROR(ENOMEM); + } + + if (!s->use_watermark) { + /* Configure the output, Premultiply alpha */ + flags =3D NI_SCALER_FLAG_IO | NI_SCALER_FLAG_PA; + + calculate_dst_rectangle(&dst_x, &dst_y, &dst_w, &dst_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x_start,2), FFALIGN(s->y_st= art,2), + overlay->width, overlay->height); + } + ret =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(frame->width, 2), // main width + FFALIGN(frame->height, 2), // main height + main_scaler_format, // main pix fmt + s->use_watermark ? NI_SCALER_FLAG_IO : flags, // flags + s->use_watermark ? frame->width : dst_w, // dst rect = width + s->use_watermark ? frame->height : dst_h, // dst rect = height + s->use_watermark ? 0 : dst_x, // dst rect x + s->use_watermark ? 0 : dst_y, // dst rect y + 0, // n/a + main_frame_idx, // main frame idx + NI_DEVICE_TYPE_SCALER); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, + "Can't allocate frame for output ovly %d\n", ret); + return AVERROR(ENOMEM); + } + + ret =3D ni_device_session_read_hwdesc(&s->api_ctx, &s->api_dst_fra= me, + NI_DEVICE_TYPE_SCALER); + + if (ret !=3D NI_RETCODE_SUCCESS) { + av_frame_free(&overlay); + av_log(ctx, AV_LOG_ERROR, + "Can't acquire output frame of overlay %d\n", ret); + return AVERROR(ENOMEM); + } + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_drawtext"); +#endif + + if (s->use_watermark) { + out =3D av_frame_alloc(); + if (!out) { + return AVERROR(ENOMEM); + } + + av_frame_copy_props(out,frame); + + out->width =3D outlink->w; + out->height =3D outlink->h; + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + // av_hwframe_get_buffer(s->hw_frames_ctx, out, 0); + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + + if (!out->data[3]) { + av_frame_free(&out); + return AVERROR(ENOMEM); + } + + /* Copy the frame surface from the incoming frame */ + memcpy(out->data[3], frame->data[3], sizeof(niFrameSurface1_t)); + frame_surface =3D (niFrameSurface1_t *) out->data[3]; + new_frame_surface =3D (niFrameSurface1_t *) s->api_dst_frame.data.= frame.p_data[3]; + frame_surface->ui16FrameIdx =3D new_frame_surface->ui16FrameIdx; + frame_surface->ui16session_ID =3D new_frame_surface->ui16session_I= D; + frame_surface->device_handle =3D new_frame_surface->device_handle; + frame_surface->output_idx =3D new_frame_surface->output_idx; + frame_surface->src_cpu =3D new_frame_surface->src_cpu; + frame_surface->dma_buf_fd =3D 0; + + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + main_frame_ctx->sw_format); + + /* Remove ni-split specific assets */ + frame_surface->ui32nodeAddress =3D 0; + + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurfa= ce1_t), ff_ni_frame_free, NULL, 0); + + av_log(ctx, AV_LOG_DEBUG, + "%s:IN trace ui16FrameIdx =3D [%d] over [%d] --> out [%d]\n", + __func__, ovly_frame_idx, main_frame_idx, frame_surface->ui16F= rameIdx); + + av_frame_free(&frame); + return ff_filter_frame(outlink, out); + } else { + frame->color_range =3D AVCOL_RANGE_MPEG; + + if (!s->main_has_alpha) { + av_log(ctx, AV_LOG_DEBUG, + "%s:IN trace ui16FrameIdx =3D [%d] and [%d] and [%d] --> o= ut [%d]\n", + __func__, main_frame_idx, ovly_frame_idx, s->ui16CropFrame= Idx, + main_frame_idx); + } else { + av_log(ctx, AV_LOG_DEBUG, + "%s:IN trace ui16FrameIdx =3D [%d] and [%d] --> out [%d]\n", + __func__, main_frame_idx, ovly_frame_idx, main_frame_idx); + } + + if (!s->main_has_alpha) { + ni_hwframe_buffer_recycle((niFrameSurface1_t *) + s->crop_api_dst_frame.data.frame.p_dat= a[3], + (int32_t)s->crop_api_ctx.device_handle= ); + } + + return ff_filter_frame(outlink, frame); + } +fail: + return ret; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntDrawTextContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %lu available_frame %d outlink framequeue %lu frame_wanted %d -= return NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + .config_props =3D config_input, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_output, + }, +}; + +FFFilter ff_vf_drawtext_ni_quadra =3D { + .p.name =3D "ni_quadra_drawtext", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra draw text on top of video frames using libfreetype = library v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_drawtext_class, + .priv_size =3D sizeof(NetIntDrawTextContext), + .init =3D init, + .uninit =3D uninit, + .activate =3D activate, + FILTER_QUERY_FUNC(query_formats), + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + .process_command =3D command, + .flags =3D AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, +}; diff --git a/libavfilter/vf_flip_ni.c b/libavfilter/vf_flip_ni.c new file mode 100644 index 0000000000..905c290fb5 --- /dev/null +++ b/libavfilter/vf_flip_ni.c @@ -0,0 +1,469 @@ +/* + * Copyright (c) 2013 Stefano Sabatini + * Copyright (c) 2008 Vitor Sessak + * Copyright (c) 2022 NETINT Technologies Inc. + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * flip filter, based on the FFmpeg flip filter +*/ + +#include + +#include "libavutil/opt.h" + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "libavutil/avstring.h" + +typedef struct NetIntFlipContext { + const AVClass *class; + + AVBufferRef *out_frames_ref; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + + int flip_type; + bool initialized; + bool session_opened; + int64_t keep_alive_timeout; + int buffer_limit; +} NetIntFlipContext; + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D { AV_PIX_FMT_NI_QUAD, A= V_PIX_FMT_NONE }; + AVFilterFormats *fmts_list =3D NULL; + + fmts_list =3D ff_make_format_list(pix_fmts); + if (!fmts_list) { + return AVERROR(ENOMEM); + } + + return ff_set_common_formats(ctx, fmts_list); +} + +static av_cold int init(AVFilterContext *ctx) +{ + NetIntFlipContext *flip =3D ctx->priv; + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntFlipContext *flip =3D ctx->priv; + + if (flip->api_dst_frame.data.frame.p_buffer) { + ni_frame_buffer_free(&flip->api_dst_frame.data.frame); + } + + if (flip->session_opened) { + /* Close operation will free the device frames */ + ni_device_session_close(&flip->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&flip->api_ctx); + } + + av_buffer_unref(&flip->out_frames_ref); +} + +static int config_props(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + NetIntFlipContext *flip =3D ctx->priv; + AVFilterLink *inlink =3D ctx->inputs[0]; + AVHWFramesContext *in_frames_ctx, *out_frames_ctx; + + // Quadra 2D engine only supports even pixel widths and heights + outlink->w =3D FFALIGN(inlink->w, 2); + outlink->h =3D FFALIGN(inlink->h, 2); + + if (outlink->w > NI_MAX_RESOLUTION_WIDTH || + outlink->h > NI_MAX_RESOLUTION_HEIGHT) { + av_log(ctx, AV_LOG_ERROR, "Resolution %dx%d > %dx%d is not allowed= \n", + outlink->w, outlink->h, + NI_MAX_RESOLUTION_WIDTH, NI_MAX_RESOLUTION_HEIGHT); + return AVERROR(EINVAL); + } + + FilterLink *li =3D ff_filter_link(inlink); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *) li->hw_frames_ctx->data; + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X4 || + in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 not supported\n"); + return AVERROR(EINVAL); + } + + av_log(ctx, AV_LOG_VERBOSE, + "w:%d h:%d fmt:%s sar:%d/%d -> w:%d h:%d fmt:%s sar:%d/%d\n", + inlink->w, inlink->h, av_get_pix_fmt_name(inlink->format), + inlink->sample_aspect_ratio.num, inlink->sample_aspect_ratio.de= n, + outlink->w, outlink->h, av_get_pix_fmt_name(outlink->format), + outlink->sample_aspect_ratio.num, outlink->sample_aspect_ratio.= den); + + flip->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_re= f); + if (!flip->out_frames_ref) { + return AVERROR(ENOMEM); + } + + out_frames_ctx =3D (AVHWFramesContext *) flip->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D outlink->w; + out_frames_ctx->height =3D outlink->h; + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + out_frames_ctx->initial_pool_size =3D NI_FLIP_ID; // Repurposed as ide= ntity code + + av_hwframe_ctx_init(flip->out_frames_ref); + + FilterLink *lo =3D ff_filter_link(ctx->outputs[0]); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(flip->out_frames_ref); + + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + + return 0; +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntFlipContext *flip =3D ctx->priv; + AVHWFramesContext *out_frames_context; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_context =3D (AVHWFramesContext*)flip->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + flip->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&flip->api_ctx, + out_frames_context->width, + out_frames_context->height, + out_frames_context->sw_format, + pool_size, + flip->buffer_limit); +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx =3D inlink->dst; + AVFilterLink *outlink =3D inlink->dst->outputs[0]; + AVFrame *out =3D NULL; + NetIntFlipContext *flip =3D ctx->priv; + AVBufferRef *out_buffer_ref =3D flip->out_frames_ref; + AVHWFramesContext *in_frames_context =3D (AVHWFramesContext *) in->hw_= frames_ctx->data; + AVNIDeviceContext *av_ni_device_context =3D (AVNIDeviceContext *) in_f= rames_context->device_ctx->hwctx; + ni_retcode_t ni_retcode =3D NI_RETCODE_SUCCESS; + niFrameSurface1_t *frame_surface =3D (niFrameSurface1_t *) in->data[3]= , *frame_surface2 =3D NULL; + ni_frame_config_t input_frame_config =3D {0}; + uint32_t scaler_format; + int retcode =3D 0, card_number =3D ni_get_cardno(in); + + if (!frame_surface) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter frame_surface should not= be NULL\n"); + return AVERROR(EINVAL); + } + + if (!flip->initialized) { + ni_retcode =3D ni_device_session_context_init(&flip->api_ctx); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter session context init= failed with %d\n", ni_retcode); + retcode =3D AVERROR(EINVAL); + goto FAIL; + } + + flip->api_ctx.device_handle =3D flip->api_ctx.blk_io_handle =3D av= _ni_device_context->cards[card_number]; + + flip->api_ctx.hw_id =3D card_number; + flip->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + flip->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_FLIP; //Flip o= peration compatible with crop + flip->api_ctx.keep_alive_timeout =3D flip->keep_alive_timeout; + + ni_retcode =3D ni_device_session_open(&flip->api_ctx, NI_DEVICE_TY= PE_SCALER); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter device session open = failed with %d\n", ni_retcode); + retcode =3D ni_retcode; + /* Close operation will free the device frames */ + ni_device_session_close(&flip->api_ctx, 1, NI_DEVICE_TYPE_SCAL= ER); + ni_device_session_context_clear(&flip->api_ctx); + goto FAIL; + } + + flip->session_opened =3D true; + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + inlink->dst->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SI= ZE > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + ni_retcode =3D init_out_pool(inlink->dst); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter init out pool failed= with %d\n", ni_retcode); + goto FAIL; + } + + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)out_buf= fer_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_frames_= ctx->hwctx; + ni_cpy_hwframe_ctx(in_frames_context, out_frames_ctx); + ni_device_session_copy(&flip->api_ctx, &out_ni_ctx->api_ctx); + + AVHWFramesContext *pAVHFWCtx =3D (AVHWFramesContext *) in->hw_fram= es_ctx->data; + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(pAVHFWCtx->= sw_format); + + if ((in->color_range =3D=3D AVCOL_RANGE_JPEG) && !(desc->flags & A= V_PIX_FMT_FLAG_RGB)) { + av_log(ctx, AV_LOG_WARNING, "Full color range input, limited c= olor output\n"); + } + + flip->initialized =3D true; + } + + ni_retcode =3D ni_frame_buffer_alloc_hwenc(&flip->api_dst_frame.data.f= rame, + outlink->w, + outlink->h, + 0); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter frame buffer alloc hwenc= failed with %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + // Input. + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(in_frames_context->sw_= format); + input_frame_config.picture_format =3D scaler_format; + + input_frame_config.rgba_color =3D frame_surface->ui32nodeAddress; + input_frame_config.frame_index =3D frame_surface->ui16FrameIdx; + + input_frame_config.rectangle_x =3D 0; + input_frame_config.rectangle_y =3D 0; + input_frame_config.rectangle_width =3D input_frame_config.picture_widt= h =3D in->width; + input_frame_config.rectangle_height =3D input_frame_config.picture_hei= ght =3D in->height; + + if (flip->flip_type =3D=3D 0) { + //hflip + input_frame_config.orientation =3D 4; + } else if (flip->flip_type =3D=3D 1) { + //vflip + input_frame_config.orientation =3D 5; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + // use ni_device_config_frame() instead of ni_device_alloc_frame() + // such that input_frame_config's orientation can be configured + ni_retcode =3D ni_device_config_frame(&flip->api_ctx, &input_frame_con= fig); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter device config input fram= e failed with %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + ni_retcode =3D ni_device_alloc_frame(&flip->api_ctx, + outlink->w, + outlink->h, + scaler_format, + NI_SCALER_FLAG_IO, + 0, + 0, + 0, + 0, + 0, + -1, + NI_DEVICE_TYPE_SCALER); + + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter device alloc output fram= e failed with %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + out =3D av_frame_alloc(); + if (!out) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter av_frame_alloc returned = NULL\n"); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + av_frame_copy_props(out, in); + + out->width =3D outlink->w; + out->height =3D outlink->h; + out->format =3D AV_PIX_FMT_NI_QUAD; + out->color_range =3D AVCOL_RANGE_MPEG; + + out->hw_frames_ctx =3D av_buffer_ref(out_buffer_ref); + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + if (!out->data[3]) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter av_malloc returned NULL\= n"); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + memcpy(out->data[3], frame_surface, sizeof(niFrameSurface1_t)); + + ni_retcode =3D ni_device_session_read_hwdesc(&flip->api_ctx, + &flip->api_dst_frame, + NI_DEVICE_TYPE_SCALER); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter read hwdesc failed with = %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_flip"); +#endif + + frame_surface2 =3D (niFrameSurface1_t *) flip->api_dst_frame.data.fram= e.p_data[3]; + + frame_surface =3D (niFrameSurface1_t *) out->data[3]; + frame_surface->ui16FrameIdx =3D frame_surface2->ui16FrameIdx; + frame_surface->ui16session_ID =3D frame_surface2->ui16session_ID; + frame_surface->device_handle =3D frame_surface2->device_handle; + frame_surface->output_idx =3D frame_surface2->output_idx; + frame_surface->src_cpu =3D frame_surface2->src_cpu; + frame_surface->ui32nodeAddress =3D 0; + frame_surface->dma_buf_fd =3D 0; + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + in_frames_context->sw_format); + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + out->buf[0] =3D av_buffer_create(out->data[3], + sizeof(niFrameSurface1_t), + ff_ni_frame_free, + NULL, + 0); + if (!out->buf[0]) { + av_log(ctx, AV_LOG_ERROR, "ni flip filter av_buffer_create returne= d NULL\n"); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + av_frame_free(&in); + return ff_filter_frame(inlink->dst->outputs[0], out); + +FAIL: + av_frame_free(&in); + if (out) + av_frame_free(&out); + return retcode; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntFlipContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %u available_frame %d outlink framequeue %u frame_wanted %d - r= eturn NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntFlipContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption ni_flip_options[] =3D { + { "flip_type", "choose horizontal or vertical flip", OFFSET(flip_ty= pe), AV_OPT_TYPE_INT, {.i64 =3D 0}, 0, 1, FLAGS, "flip_type" }, + { "horizontal", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 0}, 0, 0, FL= AGS, "flip_type" }, + { "h", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 0}, 0, 0, FL= AGS, "flip_type" }, + { "veritcal", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 1}, 0, 0, FL= AGS, "flip_type" }, + { "v", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 1}, 0, 0, FL= AGS, "flip_type" }, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_flip); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_props, + }, +}; + +FFFilter ff_vf_flip_ni_quadra =3D { + .p.name =3D "ni_quadra_flip", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra flip the input video v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_flip_class, + .priv_size =3D sizeof(NetIntFlipContext), + .init =3D init, + .uninit =3D uninit, + .activate =3D activate, + FILTER_QUERY_FUNC(query_formats), + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, +}; diff --git a/libavfilter/vf_hvsplus_ni.c b/libavfilter/vf_hvsplus_ni.c new file mode 100644 index 0000000000..438098a896 --- /dev/null +++ b/libavfilter/vf_hvsplus_ni.c @@ -0,0 +1,1792 @@ +/* +* Copyright (c) 2024 NetInt +* +* This file is part of FFmpeg. +* +* FFmpeg is free software; you can redistribute it and/or +* modify it under the terms of the GNU Lesser General Public +* License as published by the Free Software Foundation; either +* version 2.1 of the License, or (at your option) any later version. +* +* FFmpeg is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* Lesser General Public License for more details. +* +* You should have received a copy of the GNU Lesser General Public +* License along with FFmpeg; if not, write to the Free Software +* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301= USA +*/ + +#include +#include +#include +#include +#include + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#if HAVE_IO_H +#include +#endif +#include "libavutil/avassert.h" +#include "libavutil/avstring.h" +#include "libavutil/common.h" +#include "libavutil/eval.h" +#include "libavutil/colorspace.h" +#include "libavutil/imgutils.h" +#include "libavutil/internal.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "libavutil/pixdesc.h" +#include "libavutil/time.h" +#include "libswscale/swscale.h" +#include "drawutils.h" +#include "ni_device_api.h" +#include "ni_util.h" +#include "video.h" + +#define NI_NUM_FRAMES_IN_QUEUE 8 + +// hvsplus related definition +typedef struct _ni_hvsplus_network_layer { + int32_t width; + int32_t height; + int32_t channel; + int32_t classes; + int32_t component; + int32_t output_number; + float *output; +} ni_hvsplus_network_layer_t; + +typedef struct _ni_hvsplus_nbsize { + int32_t width; + int32_t height; +} ni_hvsplus_nbsize_t; + +typedef struct _ni_hvsplus_network { + int32_t netw; + int32_t neth; + int32_t net_out_w; + int32_t net_out_h; + ni_network_data_t raw; + ni_hvsplus_network_layer_t *layers; +} ni_hvsplus_network_t; + +typedef struct HwPadContext { + uint8_t rgba_color[4]; ///< color for the padding area + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; +} HwPadContext; + +typedef struct HwCropContext { + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; +} HwCropContext; + +typedef struct AiContext { + ni_session_context_t api_ctx; + ni_session_data_io_t api_src_frame; + ni_session_data_io_t api_dst_frame; +} AiContext; + +typedef struct NetIntHvsplusContext { + const AVClass *class; + int level; + int initialized; + int devid; + int in_width, in_height; + int out_width, out_height; + int nb_width, nb_height; + int need_padding; + + AiContext *ai_ctx; + AVBufferRef *out_frames_ref; + HwPadContext *hwp_ctx; + HwCropContext *hwc_ctx; + + ni_hvsplus_network_t network; + + int keep_alive_timeout; /* keep alive timeout setting */ + int ai_timeout; + int channel_mode; + int buffer_limit; +} NetIntHvsplusContext; + +static const ni_hvsplus_nbsize_t nbSizes[] =3D { + {512, 288}, + {704, 396}, + {720, 1280}, + {960, 540}, + {1280, 720}, + {1920, 1080}, + {3840, 2160} +}; + +// Find the smallest NB size that is equal to or larger than the input size +// -1: not supported, 0: matched, >0: index of nbSize + 1 +static int findNBSize(int frameWidth, int frameHeight) +{ + + int numSizes =3D sizeof(nbSizes) / sizeof(nbSizes[0]); + int retval =3D -1; + + // Iterate through the existing NB sizes to find the smallest one that= fits + for (int i =3D 0; i < numSizes; i++) { + if (frameWidth =3D=3D nbSizes[i].width && frameHeight =3D=3D nbSiz= es[i].height) { + av_log(NULL, AV_LOG_INFO, "%s: matched w %d h %d\n", __func__,= nbSizes[i].width, nbSizes[i].height); + retval =3D 0; + break; + } else if (frameWidth <=3D nbSizes[i].width && frameHeight <=3D nb= Sizes[i].height) { + av_log(NULL, AV_LOG_INFO, "%s: w %d h %d\n", __func__, nbSizes= [i].width, nbSizes[i].height); + retval =3D i+1; + break; + } + } + return retval; +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats; + + static const enum AVPixelFormat pix_fmts[] =3D { + AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUV420P10LE, + AV_PIX_FMT_NI_QUAD, + AV_PIX_FMT_NONE, + }; + + formats =3D ff_make_format_list(pix_fmts); + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static void cleanup_ai_context(AVFilterContext *ctx, NetIntHvsplusContext = *s) +{ + ni_retcode_t retval; + AiContext *ai_ctx =3D s->ai_ctx; + + if (ai_ctx) { + ni_frame_buffer_free(&ai_ctx->api_src_frame.data.frame); + + retval =3D + ni_device_session_close(&ai_ctx->api_ctx, 1, NI_DEVICE_TYPE_AI= ); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Error: failed to close ai session. retval %d\n", retv= al); + } + if (ai_ctx->api_ctx.hw_action !=3D NI_CODEC_HW_ENABLE) { +#ifdef _WIN32 + if (ai_ctx->api_ctx.device_handle !=3D NI_INVALID_DEVICE_HANDL= E) { + ni_device_close(ai_ctx->api_ctx.device_handle); + } +#elif __linux__ + if (ai_ctx->api_ctx.device_handle !=3D NI_INVALID_DEVICE_HANDL= E) { + ni_device_close(ai_ctx->api_ctx.device_handle); + } + if (ai_ctx->api_ctx.blk_io_handle !=3D NI_INVALID_DEVICE_HANDL= E) { + ni_device_close(ai_ctx->api_ctx.blk_io_handle); + } +#endif + ni_packet_buffer_free(&ai_ctx->api_dst_frame.data.packet); + ai_ctx->api_ctx.device_handle =3D NI_INVALID_DEVICE_HANDLE; + ai_ctx->api_ctx.blk_io_handle =3D NI_INVALID_DEVICE_HANDLE; + } else { + ni_frame_buffer_free(&ai_ctx->api_dst_frame.data.frame); + } + ni_device_session_context_clear(&ai_ctx->api_ctx); + av_free(ai_ctx); + s->ai_ctx =3D NULL; + } +} + + +static int init_ai_context(AVFilterContext *ctx, NetIntHvsplusContext *s, + AVFrame *frame) +{ + ni_retcode_t retval; + AiContext *ai_ctx; + ni_hvsplus_network_t *network =3D &s->network; + int hwframe =3D frame->format =3D=3D AV_PIX_FMT_NI_QUAD ? 1 : 0; + int ret; + AVHWFramesContext *pAVHFWCtx; + AVNIDeviceContext *pAVNIDevCtx; + AVHWFramesContext *out_frames_ctx; + AVNIFramesContext *f_hwctx; + int cardno; + int format; + int options; + + av_log(ctx, AV_LOG_INFO, "%s: %d x %d format %s\n", __func__, + s->out_width, s->out_height, av_get_pix_fmt_name(frame->format)= ); + + ai_ctx =3D av_mallocz(sizeof(AiContext)); + if (!ai_ctx) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to allocate ai context\n"= ); + return AVERROR(ENOMEM); + } + s->ai_ctx =3D ai_ctx; + retval =3D ni_device_session_context_init(&ai_ctx->api_ctx); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: ai session context init failure\= n"); + return AVERROR(EIO); + } + + if (hwframe) { + pAVHFWCtx =3D (AVHWFramesContext*) frame->hw_frames_ctx->data; + pAVNIDevCtx =3D (AVNIDeviceContext*) pAVHFWCtx->device_ctx->hwctx; + cardno =3D ni_get_cardno(frame); + + ai_ctx->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + ai_ctx->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + ai_ctx->api_ctx.hw_action =3D NI_CODEC_HW_ENABLE; + ai_ctx->api_ctx.hw_id =3D cardno; + } else { + ai_ctx->api_ctx.hw_id =3D s->devid; + } + + ai_ctx->api_ctx.device_type =3D NI_DEVICE_TYPE_AI; + ai_ctx->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retval =3D ni_device_session_open(&ai_ctx->api_ctx, NI_DEVICE_TYPE_AI); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to open ai session. retva= l %d\n", + retval); + ret =3D AVERROR(EIO); + goto failed_out; + } + + // Configure NB file + av_log(ctx, AV_LOG_DEBUG, "%s: out w %d h %d NB w %d h %d sw_format %s= pixel_format %d\n", __func__, + s->out_width, s->out_height, s->nb_width, s->nb_height, av_get= _pix_fmt_name(hwframe?pAVHFWCtx->sw_format:frame->format), + ai_ctx->api_ctx.pixel_format); + + ai_ctx->api_ctx.active_video_width =3D s->nb_width; + ai_ctx->api_ctx.active_video_height =3D s->nb_height; + ai_ctx->api_ctx.hvsplus_level =3D s->level; + ai_ctx->api_ctx.pixel_format =3D ff_ni_ffmpeg_to_libxcoder_pix_fmt( + (hwframe ? pAVHFWCtx->sw_format : frame->format)); + + retval =3D ni_ai_config_hvsplus(&ai_ctx->api_ctx, &network->raw); + + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to configure ai session. = retval %d\n", + retval); + ret =3D AVERROR(EIO); + goto failed_out; + } + + if (!hwframe) { + return 0; + } + out_frames_ctx =3D (AVHWFramesContext*) s->out_frames_ref->data; + f_hwctx =3D (AVNIFramesContext*) out_frames_ctx->hwctx; + f_hwctx->api_ctx.session_timestamp =3D ai_ctx->api_ctx.session_timesta= mp; + + // Create frame pool + format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + options =3D NI_AI_FLAG_IO | NI_AI_FLAG_PC; + if (s->buffer_limit) + options |=3D NI_AI_FLAG_LM; + + /* Allocate a pool of frames by the AI */ + retval =3D ni_device_alloc_frame(&ai_ctx->api_ctx, FFALIGN(s->nb_width= , 2), + FFALIGN(s->nb_height, 2), format, options, 0, // rec width + 0, // rec height + 0, // rec X pos + 0, // rec Y pos + 8, // rgba color/pool size + 0, // frame index + NI_DEVICE_TYPE_AI); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to create buffer pool\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + retval =3D ni_frame_buffer_alloc_hwenc(&ai_ctx->api_dst_frame.data.fra= me, + FFALIGN(s->nb_width, 2), FFALIGN(s->nb_height, 2), 0); + + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to allocate ni dst frame\= n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + return 0; + + failed_out: cleanup_ai_context(ctx, s); + return ret; +} + +static void ni_destroy_network(AVFilterContext *ctx, + ni_hvsplus_network_t *network) { + if (network) { + int i; + + if (network->layers) { + for (i =3D 0; i < network->raw.output_num; i++) { + av_freep(&network->layers[i].output); + } + + av_freep(&network->layers); + } + } +} + +static int ni_create_network(AVFilterContext *ctx, ni_hvsplus_network_t *n= etwork) +{ + int ret; + int i; + ni_network_data_t *ni_network =3D &network->raw; + + av_log(ctx, AV_LOG_INFO, "network input number %d, output number %d\n", + ni_network->input_num, ni_network->output_num); + + if (ni_network->input_num =3D=3D 0 || ni_network->output_num =3D=3D 0)= { + av_log(ctx, AV_LOG_ERROR, "Error: invalid network layer\n"); + return AVERROR(EINVAL); + } + + network->layers =3D + av_malloc(sizeof(ni_hvsplus_network_layer_t) * ni_network->output_= num); + if (!network->layers) { + av_log(ctx, AV_LOG_ERROR, "Error: cannot allocate network layer me= mory\n"); + return AVERROR(ENOMEM); + } + memset(network->layers, 0, + sizeof(ni_hvsplus_network_layer_t) * ni_network->output_num); + + for (i =3D 0; i < ni_network->output_num; i++) { + network->layers[i].channel =3D ni_network->linfo.out_param[i].si= zes[0]; + network->layers[i].width =3D ni_network->linfo.out_param[i].si= zes[1]; + network->layers[i].height =3D ni_network->linfo.out_param[i].si= zes[2]; + network->layers[i].component =3D 3; + network->layers[i].classes =3D + (network->layers[i].channel / network->layers[i].component) - + (4 + 1); + network->layers[i].output_number =3D + ni_ai_network_layer_dims(&ni_network->linfo.out_param[i]); + av_assert0(network->layers[i].output_number =3D=3D + network->layers[i].width * network->layers[i].height * + network->layers[i].channel); + + network->layers[i].output =3D + av_malloc(network->layers[i].output_number * sizeof(float)); + if (!network->layers[i].output) { + av_log(ctx, AV_LOG_ERROR, + "Error: failed to allocate network layer %d output buff= er\n", i); + ret =3D AVERROR(ENOMEM); + goto out; + } + + av_log(ctx, AV_LOG_DEBUG, "%s: network layer %d: w %d, h %d, ch %d= , co %d, cl %d\n", __func__, i, + network->layers[i].width, network->layers[i].height, + network->layers[i].channel, network->layers[i].component, + network->layers[i].classes); + } + + network->netw =3D ni_network->linfo.in_param[0].sizes[1]; + network->neth =3D ni_network->linfo.in_param[0].sizes[2]; + network->net_out_w =3D ni_network->linfo.out_param[0].sizes[1]; + network->net_out_h =3D ni_network->linfo.out_param[0].sizes[2]; + + return 0; +out: + ni_destroy_network(ctx, network); + return ret; +} + +static av_cold int init_hwframe_pad(AVFilterContext *ctx, NetIntHvsplusCon= text *s, + enum AVPixelFormat format, + AVFrame *frame) +{ + ni_retcode_t retval; + HwPadContext *hwp_ctx; + int ret; + AVHWFramesContext *pAVHFWCtx; + AVNIDeviceContext *pAVNIDevCtx; + int cardno; + + av_log(ctx, AV_LOG_INFO, "%s: format %s\n", __func__, av_get_pix_fmt_n= ame(format)); + + hwp_ctx =3D av_mallocz(sizeof(HwPadContext)); + if (!hwp_ctx) { + av_log(ctx, AV_LOG_ERROR, "Error: could not allocate hwframe ctx\n= "); + return AVERROR(ENOMEM); + } + s->hwp_ctx =3D hwp_ctx; + ni_device_session_context_init(&hwp_ctx->api_ctx); + + pAVHFWCtx =3D (AVHWFramesContext *)frame->hw_frames_ctx->data; + pAVNIDevCtx =3D (AVNIDeviceContext *)pAVHFWCtx->device_ctx->hwctx; + cardno =3D ni_get_cardno(frame); + + hwp_ctx->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + hwp_ctx->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + hwp_ctx->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + hwp_ctx->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_PAD; + hwp_ctx->api_ctx.hw_id =3D cardno; + hwp_ctx->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + hwp_ctx->rgba_color[0] =3D 0; + hwp_ctx->rgba_color[1] =3D 0; + hwp_ctx->rgba_color[2] =3D 0; + hwp_ctx->rgba_color[3] =3D 255; + + + retval =3D ni_device_session_open(&hwp_ctx->api_ctx, NI_DEVICE_TYPE_SC= ALER); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: could not open scaler session\n"= ); + ret =3D AVERROR(EIO); + ni_device_session_close(&hwp_ctx->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&hwp_ctx->api_ctx); + goto out; + } + + s->buffer_limit =3D 1; + + /* Create scale frame pool on device */ + retval =3D ff_ni_build_frame_pool(&hwp_ctx->api_ctx, s->nb_width, + s->nb_height, format, + DEFAULT_NI_FILTER_POOL_SIZE, s->buffer= _limit); + + if (retval < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: could not build frame pool\n"); + ret =3D AVERROR(EIO); + ni_device_session_close(&hwp_ctx->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&hwp_ctx->api_ctx); + goto out; + } + + return 0; +out: + av_free(hwp_ctx); + return ret; +} + +static void cleanup_hwframe_pad(NetIntHvsplusContext *s) +{ + HwPadContext *hwp_ctx =3D s->hwp_ctx; + + if (hwp_ctx) { + ni_frame_buffer_free(&hwp_ctx->api_dst_frame.data.frame); + ni_device_session_close(&hwp_ctx->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&hwp_ctx->api_ctx); + av_free(hwp_ctx); + s->hwp_ctx =3D NULL; + } +} + +static av_cold int init_hwframe_crop(AVFilterContext *ctx, NetIntHvsplusCo= ntext *s, + enum AVPixelFormat format, + AVFrame *frame) +{ + ni_retcode_t retval; + HwCropContext *hwc_ctx; + int ret; + AVHWFramesContext *pAVHFWCtx; + AVNIDeviceContext *pAVNIDevCtx; + int cardno; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + av_log(ctx, AV_LOG_INFO, "%s: format %s frame pool for w %d h %d\n", + __func__, av_get_pix_fmt_name(format), s->in_width, s->in_heig= ht); + + hwc_ctx =3D av_mallocz(sizeof(HwCropContext)); + if (!hwc_ctx) { + av_log(ctx, AV_LOG_ERROR, "Error: could not allocate hwframe ctx\n= "); + return AVERROR(ENOMEM); + } + s->hwc_ctx =3D hwc_ctx; + ni_device_session_context_init(&hwc_ctx->api_ctx); + + pAVHFWCtx =3D (AVHWFramesContext *)frame->hw_frames_ctx->data; + pAVNIDevCtx =3D (AVNIDeviceContext *)pAVHFWCtx->device_ctx->hwctx; + cardno =3D ni_get_cardno(frame); + + hwc_ctx->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + hwc_ctx->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + hwc_ctx->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + hwc_ctx->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_CROP; + hwc_ctx->api_ctx.hw_id =3D cardno; + hwc_ctx->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retval =3D ni_device_session_open(&hwc_ctx->api_ctx, NI_DEVICE_TYPE_SC= ALER); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: could not open scaler session\n"= ); + ret =3D AVERROR(EIO); + ni_device_session_close(&hwc_ctx->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&hwc_ctx->api_ctx); + goto out; + } + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + s->buffer_limit =3D 1; + + /* Create scale frame pool on device */ + retval =3D ff_ni_build_frame_pool(&hwc_ctx->api_ctx, s->in_width, + s->in_height, format, + pool_size, s->buffer_limit); + + if (retval < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: could not build frame pool\n"); + ret =3D AVERROR(EIO); + ni_device_session_close(&hwc_ctx->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&hwc_ctx->api_ctx); + goto out; + } + + return 0; +out: + av_free(hwc_ctx); + return ret; +} + +static void cleanup_hwframe_crop(NetIntHvsplusContext *s) +{ + HwCropContext *hwc_ctx =3D s->hwc_ctx; + + if (hwc_ctx) { + ni_frame_buffer_free(&hwc_ctx->api_dst_frame.data.frame); + ni_device_session_close(&hwc_ctx->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&hwc_ctx->api_ctx); + av_free(hwc_ctx); + s->hwc_ctx =3D NULL; + } +} + +static av_cold int init(AVFilterContext *ctx) +{ + NetIntHvsplusContext *s =3D ctx->priv; + + s->initialized =3D 0; + s->nb_width =3D -1; + s->nb_height =3D -1; + s->need_padding =3D 0; + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntHvsplusContext *s =3D ctx->priv; + ni_hvsplus_network_t *network =3D &s->network; + + cleanup_ai_context(ctx, s); + + ni_destroy_network(ctx, network); + + av_buffer_unref(&s->out_frames_ref); + s->out_frames_ref =3D NULL; + + if (s->need_padding) { + cleanup_hwframe_pad(s); + cleanup_hwframe_crop(s); + } +} + +static int config_input(AVFilterContext *ctx, AVFrame *frame) +{ + NetIntHvsplusContext *s =3D ctx->priv; + int hwframe =3D frame->format =3D=3D AV_PIX_FMT_NI_QUAD ? 1 : 0; + int ret; + + if (s->initialized) + return 0; + + ret =3D init_ai_context(ctx, s, frame); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to initialize ai context\= n"); + return ret; + } + + ret =3D ni_create_network(ctx, &s->network); + if (ret !=3D 0) { + goto fail_out; + } + + if (hwframe && s->need_padding) { + AVHWFramesContext *pAVHFWCtx =3D (AVHWFramesContext*) frame->hw_fr= ames_ctx->data; + av_log(ctx, AV_LOG_INFO, "%s: hw frame sw format %s\n", __func__, = av_get_pix_fmt_name(pAVHFWCtx->sw_format)); + + ret =3D init_hwframe_pad(ctx, s, pAVHFWCtx->sw_format, frame); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Error: could not initialized hwframe pad context\n"); + goto fail_out; + } + + ret =3D init_hwframe_crop(ctx, s, pAVHFWCtx->sw_format, frame); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, + "Error: could not initialized hwframe crop context\n"); + goto fail_out; + } + } + + s->initialized =3D 1; + return 0; + +fail_out: + cleanup_ai_context(ctx, s); + + ni_destroy_network(ctx, &s->network); + + return ret; +} + +static int output_config_props_internal(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink =3D outlink->src->inputs[0]; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + NetIntHvsplusContext *s =3D ctx->priv; + int out_width, out_height; + + if (s->out_width =3D=3D -1 || s->out_height =3D=3D -1) { + out_width =3D inlink->w; + out_height =3D inlink->h; + s->out_width =3D out_width; + s->out_height =3D out_height; + } else { + out_width =3D s->out_width; + out_height =3D s->out_height; + } + + s->in_width =3D inlink->w; + s->in_height =3D inlink->h; + + av_log(ctx, AV_LOG_INFO, "%s: need_padding %d s->out_width %d s->out_h= eight %d\n", __func__, s->need_padding, s->out_width, s->out_height); + + outlink->w =3D out_width; + outlink->h =3D out_height; + + FilterLink *li =3D ff_filter_link(inlink); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_DEBUG, "sw frame\n"); + return 0; + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (in_frames_ctx->format !=3D AV_PIX_FMT_NI_QUAD) { + av_log(ctx, AV_LOG_ERROR, "Error: pixel format not supported, form= at=3D%d\n", in_frames_ctx->format); + return AVERROR(EINVAL); + } + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X4 || + in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 not supported\n"); + return AVERROR(EINVAL); + } + + s->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_ref); + if (!s->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D outlink->w; + out_frames_ctx->height =3D outlink->h; + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + out_frames_ctx->initial_pool_size =3D NI_HVSPLUS_ID; + + av_log(ctx, AV_LOG_INFO, "%s: w %d h %d\n", __func__, out_frames_ctx->= width, out_frames_ctx->height); + + FilterLink *lo =3D ff_filter_link(ctx->outputs[0]); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + return 0; +} + +static int output_config_props(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink =3D outlink->src->inputs[0]; + NetIntHvsplusContext *s =3D ctx->priv; + int out_width, out_height, retval, ret; + + av_log(ctx, AV_LOG_DEBUG, "%s: inlink src %s dst %s filter %p w %d h %= d\n", __func__, inlink->src->name, inlink->dst->name, s, inlink->w, inlink-= >h); + av_log(ctx, AV_LOG_DEBUG, "%s: outlink src %s dst %s filter %p w %d h = %d\n", __func__, outlink->src->name, outlink->dst->name, s, outlink->w, out= link->h); + + FilterLink *li =3D ff_filter_link(inlink); + if ((li->hw_frames_ctx =3D=3D NULL) && (inlink->format =3D=3D AV_PIX_F= MT_NI_QUAD)) { + av_log(ctx, AV_LOG_ERROR, "Error: No hw context provided on input\= n"); + return AVERROR(EINVAL); + } + + if (s->out_width =3D=3D -1 || s->out_height =3D=3D -1) { + out_width =3D inlink->w; + out_height =3D inlink->h; + } else { + out_width =3D s->out_width; + out_height =3D s->out_height; + } + + // Find the width and height to be used for the AI hvs filter. + // If they match the supporting sizes of network binary, proceed. + // If they don't match, padding and cropping are needed before and aft= er hvsplus filter respectively. + // If they are greater than 4k, it is not supported. Then, exit. + retval =3D findNBSize(inlink->w, inlink->h); + if (retval < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: hvsplus doesn't support resoluti= on greater than 4K (width %d height %d).\n", out_width, out_height); + return AVERROR(EINVAL); + } + + if (retval =3D=3D 0) { + s->nb_width =3D inlink->w; + s->nb_height =3D inlink->h; + } else { + s->nb_width =3D nbSizes[retval-1].width; + s->nb_height =3D nbSizes[retval-1].height; + s->need_padding =3D 1; + } + + av_log(ctx, AV_LOG_DEBUG, "%s: inlink w %d h %d NB w %d h %d need_padd= ing %d\n", + __func__, inlink->w, inlink->h, s->nb_width, s->nb_height, s->= need_padding); + + ret =3D output_config_props_internal(outlink); + + return ret; +} + +static int av_to_niframe_copy(NetIntHvsplusContext *s, ni_frame_t *dst, co= nst AVFrame *src, int nb_planes) +{ + int dst_stride[4],src_height[4], hpad[4], vpad[4], linesize[4]; + int i, j, h; + uint8_t *src_line, *dst_line, YUVsample, *sample, *dest; + uint16_t lastidx; + bool tenBit; + + av_log(NULL, AV_LOG_DEBUG, "%s: src width %d height %d nb w %d nb h %d= format %s linesize %d %d %d nb_planes %d\n", __func__, + src->width, src->height, s->nb_width, s->nb_height, av_get_pix_= fmt_name(src->format), + src->linesize[0], src->linesize[1], src->linesize[2], nb_planes= ); + + linesize[0] =3D src->linesize[0]; + linesize[1] =3D src->linesize[1]; + linesize[2] =3D src->linesize[2]; + linesize[3] =3D 0; + + switch (src->format) { + case AV_PIX_FMT_YUV420P: + case AV_PIX_FMT_YUVJ420P: + dst_stride[0] =3D FFALIGN(s->nb_width, 128); + dst_stride[1] =3D FFALIGN((s->nb_width / 2), 128); + dst_stride[2] =3D dst_stride[1]; + dst_stride[3] =3D 0; + + linesize[0] =3D FFALIGN(src->width, 2); + linesize[1] =3D FFALIGN(src->width, 2) / 2; + linesize[2] =3D linesize[1]; + + hpad[0] =3D FFMAX(dst_stride[0] - linesize[0], 0); + hpad[1] =3D FFMAX(dst_stride[1] - linesize[1], 0); + hpad[2] =3D FFMAX(dst_stride[2] - linesize[2], 0); + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D FFALIGN(src->height, 2) / 2; + src_height[2] =3D FFALIGN(src->height, 2) / 2; + src_height[3] =3D 0; + + vpad[0] =3D FFALIGN(s->nb_height, 2) - src->height; + vpad[1] =3D (FFALIGN(s->nb_height, 2) / 2) - (FFALIGN(src->height,= 2) / 2); + vpad[2] =3D (FFALIGN(s->nb_height, 2) / 2) - (FFALIGN(src->height,= 2) / 2); + vpad[3] =3D 0; + + tenBit =3D false; + break; + case AV_PIX_FMT_YUV420P10LE: + dst_stride[0] =3D FFALIGN(s->nb_width * 2, 128); + dst_stride[1] =3D FFALIGN(s->nb_width, 128); + dst_stride[2] =3D dst_stride[1]; + dst_stride[3] =3D 0; + + linesize[0] =3D src->width * 2; + linesize[1] =3D src->width; + linesize[2] =3D linesize[1]; + + hpad[0] =3D FFMAX(dst_stride[0] - linesize[0], 0); + hpad[1] =3D FFMAX(dst_stride[1] - linesize[1], 0); + hpad[2] =3D FFMAX(dst_stride[2] - linesize[2], 0); + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D FFALIGN(src->height, 2) / 2; + src_height[2] =3D FFALIGN(src->height, 2) / 2; + src_height[3] =3D 0; + + vpad[0] =3D FFALIGN(s->nb_height, 2) - src->height; + vpad[1] =3D (FFALIGN(s->nb_height, 2) / 2) - (FFALIGN(src->height,= 2) / 2); + vpad[2] =3D (FFALIGN(s->nb_height, 2) / 2) - (FFALIGN(src->height,= 2) / 2); + vpad[3] =3D 0; + + tenBit =3D true; + break; + case AV_PIX_FMT_NV12: + dst_stride[0] =3D FFALIGN(src->width, 128); + dst_stride[1] =3D dst_stride[0]; + dst_stride[2] =3D 0; + dst_stride[3] =3D 0; + hpad[0] =3D FFMAX(dst_stride[0] - linesize[0], 0); + hpad[1] =3D FFMAX(dst_stride[1] - linesize[1], 0); + hpad[2] =3D 0; + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D FFALIGN(src->height, 2) / 2; + src_height[2] =3D 0; + src_height[3] =3D 0; + + vpad[0] =3D FFALIGN(src_height[0], 2) - src_height[0]; + vpad[1] =3D FFALIGN(src_height[1], 2) - src_height[1]; + vpad[2] =3D 0; + vpad[3] =3D 0; + + tenBit =3D false; + break; + case AV_PIX_FMT_NV16: + dst_stride[0] =3D FFALIGN(src->width, 64); + dst_stride[1] =3D dst_stride[0]; + dst_stride[2] =3D 0; + dst_stride[3] =3D 0; + hpad[0] =3D 0; + hpad[1] =3D 0; + hpad[2] =3D 0; + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D src->height; + src_height[2] =3D 0; + src_height[3] =3D 0; + + vpad[0] =3D 0; + vpad[1] =3D 0; + vpad[2] =3D 0; + vpad[3] =3D 0; + + tenBit =3D false; + break; + case AV_PIX_FMT_P010LE: + dst_stride[0] =3D FFALIGN(src->width * 2, 128); + dst_stride[1] =3D dst_stride[0]; + dst_stride[2] =3D 0; + dst_stride[3] =3D 0; + hpad[0] =3D FFMAX(dst_stride[0] - linesize[0], 0); + hpad[1] =3D FFMAX(dst_stride[1] - linesize[1], 0); + hpad[2] =3D 0; + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D FFALIGN(src->height, 2) / 2; + src_height[2] =3D 0; + src_height[3] =3D 0; + + vpad[0] =3D FFALIGN(src_height[0], 2) - src_height[0]; + vpad[1] =3D FFALIGN(src_height[1], 2) - src_height[1]; + vpad[2] =3D 0; + vpad[3] =3D 0; + + tenBit =3D true; + break; + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_BGR0: + dst_stride[0] =3D FFALIGN(src->width, 16) * 4; + dst_stride[1] =3D 0; + dst_stride[2] =3D 0; + dst_stride[3] =3D 0; + hpad[0] =3D FFMAX(dst_stride[0] - linesize[0], 0); + hpad[1] =3D 0; + hpad[2] =3D 0; + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D 0; + src_height[2] =3D 0; + src_height[3] =3D 0; + + vpad[0] =3D 0; + vpad[1] =3D 0; + vpad[2] =3D 0; + vpad[3] =3D 0; + + tenBit =3D false; + break; + case AV_PIX_FMT_YUYV422: + case AV_PIX_FMT_UYVY422: + dst_stride[0] =3D FFALIGN(src->width, 16) * 2; + dst_stride[1] =3D 0; + dst_stride[2] =3D 0; + dst_stride[3] =3D 0; + hpad[0] =3D FFMAX(dst_stride[0] - linesize[0], 0); + hpad[1] =3D 0; + hpad[2] =3D 0; + hpad[3] =3D 0; + + src_height[0] =3D src->height; + src_height[1] =3D 0; + src_height[2] =3D 0; + src_height[3] =3D 0; + + vpad[0] =3D 0; + vpad[1] =3D 0; + vpad[2] =3D 0; + vpad[3] =3D 0; + + tenBit =3D false; + break; + default: + av_log(NULL, AV_LOG_ERROR, "Error: Pixel format %s not supported\n= ", + av_get_pix_fmt_name(src->format)); + return AVERROR(EINVAL); + } + av_log(NULL, AV_LOG_DEBUG, "%s: dst_stride %d %d %d linesize %d %d %d = hpad %d %d %d\n", __func__, + dst_stride[0], dst_stride[1], dst_stride[2], + src->linesize[0], src->linesize[1], src->linesize[2], + hpad[0], hpad[1], hpad[2]); + av_log(NULL, AV_LOG_DEBUG, "%s: src_height %d %d %d vpad %d %d %d tenB= it %d\n", __func__, + src_height[0], src_height[1], src_height[2], + vpad[0], vpad[1], vpad[2], tenBit); + + dst_line =3D dst->p_buffer; + for (i =3D 0; i < nb_planes; i++) { + src_line =3D src->data[i]; + for (h =3D 0; h < src_height[i]; h++) { + memcpy(dst_line, src_line, FFMIN(linesize[i], dst_stride[i])); + + if (h =3D=3D 0) + av_log(NULL, AV_LOG_DEBUG, "%s: i %d h %d to %d memcpy siz= e %d\n", __func__, i, h, src_height[i]-1, FFMIN(linesize[i], dst_stride[i])= ); + + if (hpad[i]) { + lastidx =3D linesize[i]; + + if (tenBit) { + sample =3D &src_line[lastidx - 2]; + dest =3D &dst_line[lastidx]; + + /* two bytes per sample */ + for (j =3D 0; j < hpad[i] / 2; j++) { + memcpy(dest, sample, 2); + dest +=3D 2; + } + if (h =3D=3D 0) + av_log(NULL, AV_LOG_DEBUG, "%s: i %d hpad %d to %d= memset size %d value %d %d tenBit\n", __func__, i, h, src_height[i]-1, hpa= d[i], sample[0], sample[1]); + + } else { + YUVsample =3D dst_line[lastidx - 1]; + memset(&dst_line[lastidx], YUVsample, hpad[i]); + + if (h =3D=3D 0) + av_log(NULL, AV_LOG_DEBUG, "%s: i %d hpad %d to %d= memset size %d value %d\n", __func__, i, h, src_height[i]-1, hpad[i], YUVs= ample); + } + } + + src_line +=3D src->linesize[i]; + dst_line +=3D dst_stride[i]; + } + + /* Extend the height by cloning the last line */ + src_line =3D dst_line - dst_stride[i]; + for (h =3D 0; h < vpad[i]; h++) { + memcpy(dst_line, src_line, dst_stride[i]); + + av_log(NULL, AV_LOG_DEBUG, "%s: h %d memcpy vpad size %d\n", _= _func__, h, dst_stride[i]); + + dst_line +=3D dst_stride[i]; + } + } + + return 0; +} + +static int ni_to_avframe_copy(NetIntHvsplusContext *s, AVFrame *dst, const= ni_packet_t *src, int nb_planes) +{ + int src_linesize[4], src_height[4], dst_height[4]; + int i, h; + uint8_t *src_line, *dst_line; + + av_log(NULL, AV_LOG_DEBUG, "%s: dst width %d height %d nb w %d nb h %d= format %s nb_planes %d\n", __func__, + dst->width, dst->height, s->nb_width, s->nb_height, av_get_pix_= fmt_name(dst->format), nb_planes); + + dst_height[0] =3D dst->height; + dst_height[1] =3D dst->height / 2; + dst_height[2] =3D dst_height[1]; + dst_height[3] =3D 0; + + switch (dst->format) { + case AV_PIX_FMT_YUV420P: + case AV_PIX_FMT_YUVJ420P: + src_linesize[0] =3D FFALIGN(s->nb_width, 128); + src_linesize[1] =3D FFALIGN(s->nb_width / 2, 128); + src_linesize[2] =3D src_linesize[1]; + src_linesize[3] =3D 0; + + src_height[0] =3D s->nb_height; + src_height[1] =3D FFALIGN(s->nb_height, 2) / 2; + src_height[2] =3D src_height[1]; + src_height[3] =3D 0; + break; + case AV_PIX_FMT_YUV420P10LE: + src_linesize[0] =3D FFALIGN(s->nb_width * 2, 128); + src_linesize[1] =3D FFALIGN(s->nb_width, 128); + src_linesize[2] =3D src_linesize[1]; + src_linesize[3] =3D 0; + + src_height[0] =3D s->nb_height; + src_height[1] =3D FFALIGN(s->nb_height, 2) / 2; + src_height[2] =3D src_height[1]; + src_height[3] =3D 0; + break; + case AV_PIX_FMT_NV12: + src_linesize[0] =3D FFALIGN(dst->width, 128); + src_linesize[1] =3D FFALIGN(dst->width, 128); + src_linesize[2] =3D 0; + src_linesize[3] =3D 0; + + src_height[0] =3D dst->height; + src_height[1] =3D FFALIGN(dst->height, 2) / 2; + src_height[2] =3D 0; + src_height[3] =3D 0; + break; + case AV_PIX_FMT_NV16: + src_linesize[0] =3D FFALIGN(dst->width, 64); + src_linesize[1] =3D FFALIGN(dst->width, 64); + src_linesize[2] =3D 0; + src_linesize[3] =3D 0; + + src_height[0] =3D dst->height; + src_height[1] =3D dst->height; + src_height[2] =3D 0; + src_height[3] =3D 0; + break; + case AV_PIX_FMT_YUYV422: + case AV_PIX_FMT_UYVY422: + src_linesize[0] =3D FFALIGN(dst->width, 16) * 2; + src_linesize[1] =3D 0; + src_linesize[2] =3D 0; + src_linesize[3] =3D 0; + + src_height[0] =3D dst->height; + src_height[1] =3D 0; + src_height[2] =3D 0; + src_height[3] =3D 0; + break; + case AV_PIX_FMT_P010LE: + src_linesize[0] =3D FFALIGN(dst->width * 2, 128); + src_linesize[1] =3D FFALIGN(dst->width * 2, 128); + src_linesize[2] =3D 0; + src_linesize[3] =3D 0; + + src_height[0] =3D dst->height; + src_height[1] =3D FFALIGN(dst->height, 2) / 2; + src_height[2] =3D 0; + src_height[3] =3D 0; + break; + case AV_PIX_FMT_RGBA: + case AV_PIX_FMT_BGRA: + case AV_PIX_FMT_ABGR: + case AV_PIX_FMT_ARGB: + case AV_PIX_FMT_BGR0: + src_linesize[0] =3D FFALIGN(dst->width, 16) * 4; + src_linesize[1] =3D 0; + src_linesize[2] =3D 0; + src_linesize[3] =3D 0; + + src_height[0] =3D dst->height; + src_height[1] =3D 0; + src_height[2] =3D 0; + src_height[3] =3D 0; + break; + default: + av_log(NULL, AV_LOG_ERROR, "Error: Unsupported pixel format %s\n", + av_get_pix_fmt_name(dst->format)); + return AVERROR(EINVAL); + } + av_log(NULL, AV_LOG_DEBUG, "%s: src_linesize %d %d %d src_height %d %d= %d dst linesize %d %d %d dst_height %d %d %d\n", __func__, + src_linesize[0], src_linesize[1], src_linesize[2], + src_height[0], src_height[1], src_height[2], + dst->linesize[0], dst->linesize[1], dst->linesize[2], + dst_height[0], dst_height[1], dst_height[2]); + + src_line =3D src->p_data; + for (i =3D 0; i < nb_planes; i++) { + dst_line =3D dst->data[i]; + + for (h =3D 0; h < src_height[i]; h++) { + if (h < dst_height[i]) { + memcpy(dst_line, src_line, + FFMIN(src_linesize[i], dst->linesize[i])); + if (h =3D=3D 0) + av_log(NULL, AV_LOG_DEBUG, "%s: i %d h %d to %d memcpy= size %d\n", __func__, i, h, src_height[i]-1, FFMIN(src_linesize[i], dst->= linesize[i])); + dst_line +=3D FFMIN(src_linesize[i], dst->linesize[i]); + } + src_line +=3D src_linesize[i]; + } + } + + return 0; +} + +static int ni_hwframe_pad(AVFilterContext *ctx, NetIntHvsplusContext *s, A= VFrame *in, + int w, int h, + niFrameSurface1_t **filt_frame_surface) +{ + HwPadContext *pad_ctx =3D s->hwp_ctx; + uint32_t ui32RgbaColor, scaler_format; + ni_retcode_t retcode; + niFrameSurface1_t *frame_surface, *new_frame_surface; + AVHWFramesContext *pAVHFWCtx; + + frame_surface =3D (niFrameSurface1_t *)in->data[3]; + + pAVHFWCtx =3D (AVHWFramesContext *)in->hw_frames_ctx->data; + + av_log(ctx, AV_LOG_DEBUG, "%s: in frame surface frameIdx %d sw_format = %s w %d h %d\n", __func__, + frame_surface->ui16FrameIdx, av_get_pix_fmt_name(pAVHFWCtx->sw_= format), w, h); + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + + retcode =3D ni_frame_buffer_alloc_hwenc(&pad_ctx->api_dst_frame.data.f= rame, + w, h, 0); + if (retcode !=3D NI_RETCODE_SUCCESS) + return AVERROR(ENOMEM); + + av_log(ctx, AV_LOG_DEBUG, + "%s: inlink->w =3D %d;inlink->h =3D %d;outlink->w =3D %d;outlin= k->h =3D %d\n", __func__, + in->width, in->height, s->nb_width, s->nb_height); + av_log(ctx, AV_LOG_DEBUG, + "%s: s->w=3D%d;s->h=3D%d;s->x=3D%d;s->y=3D%d;c=3D%02x:%02x:%02x= :%02x\n", __func__, w, + h, 0, 0, pad_ctx->rgba_color[0], pad_ctx->rgba_color[1], + pad_ctx->rgba_color[2], pad_ctx->rgba_color[3]); + + /* + * Allocate device input frame. This call won't actually allocate a fr= ame, + * but sends the incoming hardware frame index to the scaler manager + */ + retcode =3D ni_device_alloc_frame(&pad_ctx->api_ctx, + FFALIGN(in->width, 2), + FFALIGN(in->height, 2), + scaler_format, + 0, // input frame + in->width, // src rectangle width + in->height, // src rectangle height + 0, // src rectangle x =3D 0 + 0, // src rectangle y =3D 0 + frame_surface->ui32nodeAddress, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(NULL, AV_LOG_ERROR, "Error: Can't allocate device input fra= me %d\n", retcode); + return AVERROR(ENOMEM); + } + + /* Scaler uses BGRA color, or ARGB in little-endian */ + ui32RgbaColor =3D (pad_ctx->rgba_color[3] << 24) | (pad_ctx->rgba_colo= r[0] << 16) | + (pad_ctx->rgba_color[1] << 8) | pad_ctx->rgba_color[2]; + + /* Allocate device destination frame. This will acquire a frame from t= he pool */ + retcode =3D ni_device_alloc_frame(&pad_ctx->api_ctx, + FFALIGN(s->nb_width,2), + FFALIGN(s->nb_height,2), + scaler_format, + NI_SCALER_FLAG_IO, // output frame + in->width, // dst rectangle= width + in->height, // dst rectangle= height + 0, //s->x, // dst rect= angle x + 0, //s->y, // dst rect= angle y + ui32RgbaColor, // rgba color + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(NULL, AV_LOG_ERROR, "Error: Can't allocate device output fr= ame %d\n", retcode); + return AVERROR(ENOMEM); + } + + /* Set the new frame index */ + ni_device_session_read_hwdesc( + &pad_ctx->api_ctx, &pad_ctx->api_dst_frame, NI_DEVICE_TYPE_SCALER); + new_frame_surface =3D + (niFrameSurface1_t *)pad_ctx->api_dst_frame.data.frame.p_data[3]; + + new_frame_surface->ui16width =3D s->nb_width; + new_frame_surface->ui16height =3D s->nb_height; + + *filt_frame_surface =3D new_frame_surface; + + return 0; +} + +static int ni_hwframe_crop(AVFilterContext *ctx, NetIntHvsplusContext *s, = AVFrame *in, + int w, int h, + niFrameSurface1_t **filt_frame_surface) +{ + AiContext *ai_ctx =3D s->ai_ctx; + HwCropContext *crop_ctx =3D s->hwc_ctx; + uint32_t scaler_format; + ni_retcode_t retcode; + niFrameSurface1_t *frame_surface, *new_frame_surface; + AVHWFramesContext *pAVHFWCtx; + + frame_surface =3D (niFrameSurface1_t *) ai_ctx->api_dst_frame.data.fra= me.p_data[3]; //(niFrameSurface1_t *)in->data[3]; + if (frame_surface =3D=3D NULL) { + av_log(NULL, AV_LOG_ERROR, "Error: frame_surface is NULL\n"); + return AVERROR(EINVAL); + } + + pAVHFWCtx =3D (AVHWFramesContext *)in->hw_frames_ctx->data; + + av_log(ctx, AV_LOG_DEBUG, "%s: in frame surface frameIdx %d sw_format = %s w %d h %d\n", __func__, + frame_surface->ui16FrameIdx, av_get_pix_fmt_name(pAVHFWCtx->sw_= format), w, h); + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + + retcode =3D ni_frame_buffer_alloc_hwenc(&crop_ctx->api_dst_frame.data.= frame, s->nb_width, s->nb_height, // w, h, + 0); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(NULL, AV_LOG_ERROR, "Error: Cannot allocate memory\n"); + return AVERROR(ENOMEM); + } + + av_log(ctx, AV_LOG_DEBUG, + "%s: inlink->w =3D %d;inlink->h =3D %d;outlink->w =3D %d;outlin= k->h =3D %d\n", __func__, + s->nb_width, s->nb_height, w, h); + + av_log(ctx, AV_LOG_DEBUG, "%s: x:%d y:%d x+w:%d y+h:%d\n", __func__, + 0, 0, w, h); + + /* + * Allocate device input frame. This call won't actually allocate a fr= ame, + * but sends the incoming hardware frame index to the scaler manager + */ + retcode =3D ni_device_alloc_frame(&crop_ctx->api_ctx, + FFALIGN(s->nb_width, 2), + FFALIGN(s->nb_height, 2), + scaler_format, + 0, // input frame + w, // src rectangle width + h, // src rectangle height + 0, // src rectangle x + 0, // src rectangle y + frame_surface->ui32nodeAddress, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(NULL, AV_LOG_ERROR, "Error: Can't assign input frame %d\n",= retcode); + return AVERROR(ENOMEM); + } + + /* Allocate device destination frame This will acquire a frame from th= e pool */ + retcode =3D ni_device_alloc_frame(&crop_ctx->api_ctx, + FFALIGN(w,2), + FFALIGN(h,2), + scaler_format, + NI_SCALER_FLAG_IO, + 0, + 0, + 0, + 0, + 0, + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(NULL, AV_LOG_ERROR, "Error: Can't allocate device output fr= ame %d\n", retcode); + return AVERROR(ENOMEM); + } + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc( + &crop_ctx->api_ctx, &crop_ctx->api_dst_frame, NI_DEVICE_TYPE_SCALE= R); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "%s: Error: Can't allocate device output= frame %d\n", __func__, retcode); + return AVERROR(ENOMEM); + } + + new_frame_surface =3D + (niFrameSurface1_t *)crop_ctx->api_dst_frame.data.frame.p_data[3]; + + new_frame_surface->ui16width =3D w; + new_frame_surface->ui16height =3D h; + + *filt_frame_surface =3D new_frame_surface; + + return 0; +} + +static int filter_frame_internal(AVFilterLink *link, AVFrame *in) +{ + AVFilterContext *ctx =3D link->dst; + AVHWFramesContext *in_frames_context =3D NULL; //=3D (AVHWFramesContex= t *) in->hw_frames_ctx->data; + NetIntHvsplusContext *s =3D ctx->priv; + AVFrame *out =3D NULL; + ni_retcode_t retval; + int ret; + AiContext *ai_ctx; + ni_hvsplus_network_t *network =3D &s->network; + int nb_planes; + int64_t start_t; + int hwframe =3D in->format =3D=3D AV_PIX_FMT_NI_QUAD ? 1 : 0; + + av_log(ctx, AV_LOG_DEBUG, "%s: filter %p hwframe %d format %s\n", __fu= nc__, s, hwframe, av_get_pix_fmt_name(in->format)); + + if (!s->initialized) { + AVHWFramesContext *pAVHFWCtx; + if (hwframe) { + pAVHFWCtx =3D (AVHWFramesContext *) in->hw_frames_ctx->data; + } + + AVFilterLink *outlink =3D link->dst->outputs[0]; + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + ctx->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SIZE > 1) = ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + ret =3D config_input(ctx, in); + if (ret) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to config input\n"); + goto failed_out; + } + if (hwframe) { + av_hwframe_ctx_init(s->out_frames_ref); + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)s->= out_frames_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_fra= mes_ctx->hwctx; + ni_cpy_hwframe_ctx(pAVHFWCtx, out_frames_ctx); + ni_device_session_copy(&s->ai_ctx->api_ctx, &out_ni_ctx->api_c= tx); + } + } + + ai_ctx =3D s->ai_ctx; + out =3D av_frame_alloc(); + if (!out) { + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + av_frame_copy_props(out, in); + + av_log(ctx, AV_LOG_DEBUG, "%s: out_width %d out_height %d in width %d = height %d\n", + __func__, s->out_width, s->out_height, in->width, in->height); + + if (hwframe) { + niFrameSurface1_t *frame_surface; + niFrameSurface1_t *hvsplus_surface; + niFrameSurface1_t *out_surface; + niFrameSurface1_t *frame_surface2; + int ai_out_format; + niFrameSurface1_t dst_surface =3D {0}; + + in_frames_context =3D (AVHWFramesContext *) in->hw_frames_ctx->dat= a; + + out->width =3D (s->need_padding) ? in->width : s->nb_width; + out->height =3D (s->need_padding) ? in->height : s->nb_height; + + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + if (s->need_padding) { + ret =3D ni_hwframe_pad(ctx, s, in, s->nb_width, s->nb_height, = //network->netw, network->neth, + &frame_surface); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Error run hwframe pad\n"); + goto failed_out; + } + + av_log(ctx, AV_LOG_DEBUG, "filt frame surface frameIdx %d\n", + frame_surface->ui16FrameIdx); + + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + } else { + // To hvsplus + frame_surface =3D (niFrameSurface1_t *)in->data[3]; + } + + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + if (!out->data[3]) { + av_log(ctx, AV_LOG_ERROR, "Error: ni hvsplus filter av_malloc = returned NULL\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + memcpy(out->data[3], frame_surface, sizeof(niFrameSurface1_t)); + av_log(ctx, AV_LOG_DEBUG, "%s: input frame surface frameIdx %d ui1= 6width %d ui16height %d\n", + __func__, frame_surface->ui16FrameIdx, frame_surface->ui16w= idth, frame_surface->ui16height); + + start_t =3D av_gettime(); + + /* set output buffer */ + ai_out_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(in_frames_context-= >sw_format); + + av_log(ctx, AV_LOG_DEBUG, "%s: in sw_format %s ai_out_format %d\n"= , __func__, + av_get_pix_fmt_name(in_frames_context->sw_format), ai_out_f= ormat); + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + do { + if (s->channel_mode) { + retval =3D ni_device_alloc_dst_frame(&(ai_ctx->api_ctx), &= dst_surface, NI_DEVICE_TYPE_AI); + } else { + if (s->need_padding) { + av_log(ctx, AV_LOG_DEBUG, "%s: 1. Set output hw frame = in Ai w %d h %d\n", + __func__, s->nb_width, s->nb_height); + retval =3D ni_device_alloc_frame( + &ai_ctx->api_ctx, FFALIGN(s->nb_width, 2), FFALIGN= (s->nb_height,2), + ai_out_format, NI_AI_FLAG_IO, 0, 0, + 0, 0, 0, -1, NI_DEVICE_TYPE_AI); + } else { + av_log(ctx, AV_LOG_DEBUG, "%s: 1. Set output hw frame = in Ai w %d h %d\n", + __func__, s->out_width, s->out_height); + retval =3D ni_device_alloc_frame( + &ai_ctx->api_ctx, FFALIGN(s->out_width, 2), FFALIG= N(s->out_height,2), + ai_out_format, NI_AI_FLAG_IO, 0, 0, + 0, 0, 0, -1, NI_DEVICE_TYPE_AI); + } + } + + if (retval < NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to alloc hw outpu= t frame\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + if (av_gettime() - start_t > s->ai_timeout * 1000000) { + av_log(ctx, AV_LOG_ERROR, "Error: alloc hw output timeout\= n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + } while (retval !=3D NI_RETCODE_SUCCESS); + + if (s->channel_mode) { + // copy input hw frame to dst hw frame + ni_frameclone_desc_t frame_clone_desc =3D {0}; + frame_clone_desc.ui16DstIdx =3D dst_surface.ui16FrameIdx; + frame_clone_desc.ui16SrcIdx =3D frame_surface->ui16FrameIdx; + if (in_frames_context->sw_format =3D=3D AV_PIX_FMT_YUV420P) { + // only support yuv420p + if (s->need_padding) { + // offset Y size + frame_clone_desc.ui32Offset =3D NI_VPU_ALIGN128(s->nb_= width) * NI_VPU_CEIL(s->nb_height, 2); + // copy U+V size + frame_clone_desc.ui32Size =3D NI_VPU_ALIGN128(s->nb_wi= dth / 2) * NI_VPU_CEIL(s->nb_height, 2); + } else { + // offset Y size + frame_clone_desc.ui32Offset =3D NI_VPU_ALIGN128(s->out= _width) * NI_VPU_CEIL(s->out_height, 2); + // copy U+V size + frame_clone_desc.ui32Size =3D NI_VPU_ALIGN128(s->out_w= idth / 2) * NI_VPU_CEIL(s->out_height, 2); + } + retval =3D ni_device_clone_hwframe(&ai_ctx->api_ctx, &fram= e_clone_desc); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to clone hw i= nput frame\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + } else { + av_log(ctx, AV_LOG_ERROR, "Error: support yuv420p only, cu= rrent fmt %d\n", + in_frames_context->sw_format); + ret =3D AVERROR(EINVAL); + goto failed_out; + } + } + + av_log(ctx, AV_LOG_DEBUG, "%s: 2. Set input hw frame in Ai w %d h = %d\n", + __func__, frame_surface->ui16width, frame_= surface->ui16height); + + /* set input buffer */ + retval =3D ni_device_alloc_frame(&ai_ctx->api_ctx, 0, 0, 0, 0, 0, = 0, 0, 0, + frame_surface->ui32nodeAddress, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_AI); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to alloc hw input fra= me\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + /* Set the new frame index */ + start_t =3D av_gettime(); + do{ + av_log(ctx, AV_LOG_DEBUG, "%s: 3. Read hw frame from Ai w %d h= %d\n", + __func__, out->width, out->hei= ght); + retval =3D ni_device_session_read_hwdesc( + &ai_ctx->api_ctx, &s->ai_ctx->api_dst_frame, NI_DEVICE_TYP= E_AI); + + if (retval < NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to read hwdesc,re= tval=3D%d\n", retval); + ret =3D AVERROR(EINVAL); + goto failed_out; + } + if (av_gettime() - start_t > s->ai_timeout * 1000000) { + av_log(ctx, AV_LOG_ERROR, "Error: alloc hw output timeout\= n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + } while (retval !=3D NI_RETCODE_SUCCESS); + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_hvsplus"); +#endif + + if (s->need_padding) { + + hvsplus_surface =3D (niFrameSurface1_t *) ai_ctx->api_dst_fram= e.data.frame.p_data[3]; + + ni_hwframe_buffer_recycle(frame_surface, frame_surface->device= _handle); + + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + + memcpy(out->data[3], ai_ctx->api_dst_frame.data.frame.p_data[3= ], sizeof(niFrameSurface1_t)); + + ret =3D ni_hwframe_crop(ctx, s, in, in->width, in->height, &fr= ame_surface2); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Error run hwframe crop\n"); + goto failed_out; + } + + ni_hwframe_buffer_recycle(hvsplus_surface, hvsplus_surface->de= vice_handle); + + av_log(ctx, AV_LOG_DEBUG, "filt frame surface frameIdx %d\n", + frame_surface2->ui16FrameIdx); + } else { + frame_surface2 =3D (niFrameSurface1_t *) ai_ctx->api_dst_frame= .data.frame.p_data[3]; + } + + out_surface =3D (niFrameSurface1_t *) out->data[3]; + + av_log(ctx, AV_LOG_DEBUG,"ai pre process, idx=3D%d\n", frame_surfa= ce2->ui16FrameIdx); + + out_surface->ui16FrameIdx =3D frame_surface2->ui16FrameIdx; + out_surface->ui16session_ID =3D frame_surface2->ui16session_ID; + out_surface->device_handle =3D frame_surface2->device_handle; + out_surface->output_idx =3D frame_surface2->output_idx; + out_surface->src_cpu =3D frame_surface2->src_cpu; + out_surface->ui32nodeAddress =3D 0; + out_surface->dma_buf_fd =3D 0; + out_surface->ui16width =3D out->width; + out_surface->ui16height =3D out->height; + ff_ni_set_bit_depth_and_encoding_type(&out_surface->bit_depth, + &out_surface->encoding_type, + in_frames_context->sw_format); + + av_log(ctx, AV_LOG_DEBUG, "%s: need_padding %d 4. Read hw frame fr= om Ai w %d %d h %d %d\n", + __func__, s->need_= padding, out->width, s->out_width, out->height, s->out_height); + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurfa= ce1_t), ff_ni_frame_free, NULL, 0); + + if (!out->buf[0]) { + av_log(ctx, AV_LOG_ERROR, "Error: ni hvsplus filter av_buffer_= create returned NULL\n"); + ret =3D AVERROR(ENOMEM); + av_log(NULL, AV_LOG_DEBUG, "Recycle trace ui16FrameIdx =3D [%d= ] DevHandle %d\n", + out_surface->ui16FrameIdx, out_surface->device_handle); + retval =3D ni_hwframe_buffer_recycle(out_surface, out_surface-= >device_handle); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(NULL, AV_LOG_ERROR, "ERROR: Failed to recycle trace= ui16FrameIdx =3D [%d] DevHandle %d\n", + out_surface->ui16FrameIdx, out_surface->device_han= dle); + } + goto failed_out; + } + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + } else { + out->width =3D s->out_width; + out->height =3D s->out_height; + + out->format =3D in->format; + av_log(ctx, AV_LOG_DEBUG, "%s: format %s allocate frame %d x %d\n"= , __func__, av_get_pix_fmt_name(in->format), out->width, out->height); + if (av_frame_get_buffer(out, 32) < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: Could not allocate the AVFra= me buffers\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + start_t =3D av_gettime(); + // sw frame: step 1: allocate + retval =3D ni_ai_frame_buffer_alloc(&ai_ctx->api_src_frame.data.fr= ame, + &network->raw); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: cannot allocate ai frame\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + nb_planes =3D av_pix_fmt_count_planes(in->format); + if (s->channel_mode) { + if (in->format !=3D AV_PIX_FMT_YUV420P && in->format !=3D AV_P= IX_FMT_YUVJ420P) { + av_log(ctx, AV_LOG_ERROR, "Error: support yuv420p and yuvj= 420p only, current fmt %d\n", + in->format); + ret =3D AVERROR(EINVAL); + goto failed_out; + } + nb_planes =3D 1; // only copy Y data + } + // sw frame: step 2: pad and setup frame + retval =3D av_to_niframe_copy(s, &ai_ctx->api_src_frame.data.frame= , in, nb_planes); + if (retval < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: hvsplus cannot copy frame\n"= ); + ret =3D AVERROR(EIO); + goto failed_out; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + /* write frame */ + // sw frame: step 3: write a frame to AI + do { + retval =3D ni_device_session_write( + &ai_ctx->api_ctx, &ai_ctx->api_src_frame, NI_DEVICE_TYPE_A= I); + if (retval < 0) { + av_log(ctx, AV_LOG_ERROR, + "Error: failed to write ai session: retval %d\n", r= etval); + ret =3D AVERROR(EIO); + goto failed_out; + } + + if (av_gettime() - start_t > s->ai_timeout * 1000000) { + av_log(ctx, AV_LOG_ERROR, "Error: write sw frame to AI tim= eout\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + } while (retval =3D=3D 0); + // sw frame: step 4: alloc frame for read + retval =3D ni_ai_packet_buffer_alloc(&ai_ctx->api_dst_frame.data.p= acket, + &network->raw); + if (retval !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Error: failed to allocate ni packet= \n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + + start_t =3D av_gettime(); + // sw frame: step 5: read a frame from AI + do { + retval =3D ni_device_session_read(&ai_ctx->api_ctx, &ai_ctx->a= pi_dst_frame, NI_DEVICE_TYPE_AI); + if (retval < 0) { + av_log(NULL,AV_LOG_ERROR,"Error: read AI data retval %d\n"= , retval); + ret =3D AVERROR(EIO); + goto failed_out; + } else if (retval > 0) { + if (av_gettime() - start_t > s->ai_timeout * 1000000) { + av_log(ctx, AV_LOG_ERROR, "Error: read sw frame from A= I timeout\n"); + ret =3D AVERROR(ENOMEM); + goto failed_out; + } + } + } while (retval =3D=3D 0); +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_hvsplus"); +#endif + nb_planes =3D av_pix_fmt_count_planes(out->format); + if (s->channel_mode) { + if (out->format !=3D AV_PIX_FMT_YUV420P && out->format !=3D AV= _PIX_FMT_YUVJ420P) { + av_log(ctx, AV_LOG_ERROR, "Error: support yuv420p and yuvj= 420p only, current fmt %d\n", + out->format); + ret =3D AVERROR(EINVAL); + goto failed_out; + } + nb_planes =3D 1; // only copy Y data + // copy U/V data from the input sw frame + memcpy(out->data[1], in->data[1], in->height * in->linesize[1]= / 2); + memcpy(out->data[2], in->data[2], in->height * in->linesize[2]= / 2); + } + // sw frame: step 6: crop + retval =3D ni_to_avframe_copy(s, out, &ai_ctx->api_dst_frame.data.= packet, nb_planes); + if (retval < 0) { + av_log(ctx, AV_LOG_ERROR, "Error: hvsplus cannot copy ai frame= to avframe\n"); + ret =3D AVERROR(EIO); + goto failed_out; + } + } + + av_frame_free(&in); + return ff_filter_frame(link->dst->outputs[0], out); + +failed_out: + if (out) + av_frame_free(&out); + + av_frame_free(&in); + return ret; +} + +static int filter_frame(AVFilterLink *link, AVFrame *in) +{ + AVFilterContext *ctx =3D link->dst; + int ret; + + if (in =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "Error: in frame is null\n"); + return AVERROR(EINVAL); + } + + ret =3D filter_frame_internal(link, in); + + return ret; +} + +#define OFFSET(x) offsetof(NetIntHvsplusContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption ni_hvsplus_options[] =3D { + { "devid", "device to operate in swframe mode", OFFSET(devid), = AV_OPT_TYPE_INT, {.i64 =3D 0}, -1, INT_MAX, FLAGS }, + { "level", "level of modification", OFFSET(level), = AV_OPT_TYPE_INT, {.i64 =3D 2}, 1, 2, FLAGS }, + { "width", "Specify the output frame width.", OFFSET(out_width), = AV_OPT_TYPE_INT, {.i64 =3D -1}, -1, NI_MAX_RESOLUTION_WIDTH, FLAGS }, + { "height", "Specify the output frame height.", OFFSET(out_height), = AV_OPT_TYPE_INT, {.i64 =3D -1}, -1, NI_MAX_RESOLUTION_HEIGHT, FLAGS }, + { "mode", "filter mode", OFFSET(channel_mode)= , AV_OPT_TYPE_INT, {.i64 =3D 0}, 0, 1, FLAGS, "mod= e" }, + { "YUV", "process channels Y, U, and V", 0, AV_OPT_TYPE_CONST, = {.i64 =3D 0}, 0, 0, FLAGS, "mode" }, + { "Y_only", "process only channel Y", 0, AV_OPT_TYPE_CONST, = {.i64 =3D 1}, 0, 0, FLAGS, "mode" }, + { "timeout", "Timeout for AI operations", OFFSET(ai_timeout), = AV_OPT_TYPE_INT, {.i64 =3D NI_DEFAULT_KEEP_ALIVE_TIMEOUT}, NI_MIN_KEEP_A= LIVE_TIMEOUT, NI_MAX_KEEP_ALIVE_TIMEOUT, FLAGS, "AI_timeout" }, + NI_FILT_OPTION_KEEPALIVE10, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_hvsplus); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D output_config_props, + }, +}; + +FFFilter ff_vf_hvsplus_ni_quadra =3D { + .p.name =3D "ni_quadra_hvsplus", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra hvsplus v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_hvsplus_class, + .priv_size =3D sizeof(NetIntHvsplusContext), + .init =3D init, + .uninit =3D uninit, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +}; diff --git a/libavfilter/vf_hwupload_ni_quadra.c b/libavfilter/vf_hwupload_= ni_quadra.c new file mode 100644 index 0000000000..e479482b72 --- /dev/null +++ b/libavfilter/vf_hwupload_ni_quadra.c @@ -0,0 +1,297 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +#include "libavutil/buffer.h" +#include "libavutil/log.h" +#include "libavutil/opt.h" + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "video.h" + +typedef struct NetIntUploadContext { + const AVClass *class; + int device_idx; + const char *device_name; + AVBufferRef *hwdevice; + AVBufferRef *hwframe; + int keep_alive_timeout; /* keep alive timeout setting */ +} NetIntUploadContext; + +static int query_formats(AVFilterContext *ctx) +{ + NetIntUploadContext *nictx =3D ctx->priv; + AVHWFramesConstraints *constraints =3D NULL; + const enum AVPixelFormat *input_pix_fmts, *output_pix_fmts; + AVFilterFormats *input_formats =3D NULL; + int err, i; + + if (!nictx->hwdevice) + return AVERROR(ENOMEM); + + constraints =3D av_hwdevice_get_hwframe_constraints(nictx->hwdevice, N= ULL); + if (!constraints) { + err =3D AVERROR(EINVAL); + goto fail; + } + + input_pix_fmts =3D constraints->valid_sw_formats; + output_pix_fmts =3D constraints->valid_hw_formats; + + input_formats =3D ff_make_format_list(output_pix_fmts); + if (!input_formats) { + err =3D AVERROR(ENOMEM); + goto fail; + } + if (input_pix_fmts) { + for (i =3D 0; input_pix_fmts[i] !=3D AV_PIX_FMT_NONE; i++) { + err =3D ff_add_format(&input_formats, input_pix_fmts[i]); + if (err < 0) + goto fail; + } + } + + if ((err =3D ff_formats_ref(input_formats, &ctx->inputs[0]->outcfg.for= mats)) < 0 || + (err =3D ff_formats_ref(ff_make_format_list(output_pix_fmts), + &ctx->outputs[0]->incfg.formats)) < 0) + goto fail; + + av_hwframe_constraints_free(&constraints); + return 0; + +fail: + av_buffer_unref(&nictx->hwdevice); + av_hwframe_constraints_free(&constraints); + return err; +} + +static av_cold int init(AVFilterContext *ctx) +{ + NetIntUploadContext *s =3D ctx->priv; + char buf[64] =3D { 0 }; + + snprintf(buf, sizeof(buf), "%d", s->device_idx); + + if (s->device_name) { + int tmp_guid_id; + tmp_guid_id =3D ni_rsrc_get_device_by_block_name(s->device_name, N= I_DEVICE_TYPE_UPLOAD); + if (tmp_guid_id !=3D NI_RETCODE_FAILURE) { + av_log(ctx, AV_LOG_VERBOSE,"User set uploader device_name=3D%s= . This will replace uploader_device_id\n",s->device_name); + memset(buf, 0, sizeof(buf)); + snprintf(buf, sizeof(buf), "%d", tmp_guid_id); + } + else { + av_log(ctx, AV_LOG_VERBOSE, "Uploader device_name=3D%s not fou= nd. Use default value of uploader device_num=3D%d instead.\n",s->device_nam= e,s->device_idx); + } + } + + return av_hwdevice_ctx_create(&s->hwdevice, AV_HWDEVICE_TYPE_NI_QUADRA= , buf, NULL, 0); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntUploadContext *s =3D ctx->priv; + + av_buffer_unref(&s->hwframe); + av_buffer_unref(&s->hwdevice); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink =3D ctx->inputs[0]; + NetIntUploadContext *s =3D ctx->priv; + AVNIFramesContext *pub_ctx; + AVHWFramesContext *hwframe_ctx; + int ret; + + av_buffer_unref(&s->hwframe); + + if (inlink->format =3D=3D outlink->format) { + // The input is already a hardware format, so we just want to + // pass through the input frames in their own hardware context. + FilterLink *li =3D ff_filter_link(inlink); + if (!li->hw_frames_ctx) { + av_log(ctx, AV_LOG_ERROR, "No input hwframe context.\n"); + return AVERROR(EINVAL); + } + FilterLink *lo =3D ff_filter_link(outlink); + lo->hw_frames_ctx =3D av_buffer_ref(li->hw_frames_ctx); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + return 0; + } + + s->hwframe =3D av_hwframe_ctx_alloc(s->hwdevice); + if (!s->hwframe) + return AVERROR(ENOMEM); + + hwframe_ctx =3D (AVHWFramesContext*)s->hwframe->data; + hwframe_ctx->format =3D AV_PIX_FMT_NI_QUAD; + hwframe_ctx->sw_format =3D inlink->format; + hwframe_ctx->width =3D inlink->w; + hwframe_ctx->height =3D inlink->h; + pub_ctx =3D (AVNIFramesContext*)hwframe_ctx->hwctx; + pub_ctx->keep_alive_timeout =3D s->keep_alive_timeout; + FilterLink *li =3D ff_filter_link(inlink); + pub_ctx->framerate =3D li->frame_rate; + + ret =3D av_hwframe_ctx_init(s->hwframe); + if (ret < 0) + return ret; + + FilterLink *lo =3D ff_filter_link(outlink); + lo->hw_frames_ctx =3D av_buffer_ref(s->hwframe); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + return 0; +} + +static int filter_frame(AVFilterLink *link, AVFrame *in) +{ + AVFilterContext *ctx =3D link->dst; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *out =3D NULL; + int ret; + + if (in->format =3D=3D outlink->format) + return ff_filter_frame(outlink, in); + + out =3D ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) { + ret =3D AVERROR(ENOMEM); + goto fail; + } + + out->width =3D in->width; + out->height =3D in->height; + + ret =3D av_hwframe_transfer_data(out, in, 0); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Error transferring data to the Quadra\n= "); + goto fail; + } + + ret =3D av_frame_copy_props(out, in); + if (ret < 0) + goto fail; + + av_frame_free(&in); + + return ff_filter_frame(ctx->outputs[0], out); + +fail: + av_frame_free(&in); + av_frame_free(&out); + return ret; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + FilterLink *lo =3D ff_filter_link(outlink); + AVHWFramesContext *hwfc =3D (AVHWFramesContext *) lo->hw_frames_ctx->d= ata; + AVNIFramesContext *f_hwctx =3D (AVNIFramesContext*) hwfc->hwctx; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + av_log(ctx, AV_LOG_TRACE, "%s: ready %u inlink framequeue %u available= _frame %d outlink framequeue %u frame_wanted %d\n", + __func__, ctx->ready, ff_inlink_queued_frames(inlink), ff_inlink_c= heck_available_frame(inlink), ff_inlink_queued_frames(outlink), ff_outlink_= frame_wanted(outlink)); + + if (ff_inlink_check_available_frame(inlink)) { + if (inlink->format !=3D outlink->format) { + ret =3D ni_device_session_query_buffer_avail(&f_hwctx->api_ctx= , NI_DEVICE_TYPE_UPLOAD); + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW= \n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u in= link framequeue %u available_frame %d outlink framequeue %u frame_wanted %d= - return NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(= inlink), ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(o= utlink), ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntUploadContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) +// default device_idx -1 for uploader to auto balance +static const AVOption ni_upload_options[] =3D { + { "device", "Number of the device to use", OFFSET(device_idx), AV_OP= T_TYPE_INT, {.i64 =3D -1}, -1, INT_MAX, FLAGS}, + { "devname", "Name of the device to use", OFFSET(device_name), AV_OP= T_TYPE_STRING, {.str =3D NULL}, CHAR_MIN, CHAR_MAX, FLAGS}, + NI_FILT_OPTION_KEEPALIVE, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_upload); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_output, + }, +}; + +FFFilter ff_vf_hwupload_ni_quadra =3D { + .p.name =3D "ni_quadra_hwupload", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra upload a system memory frame to a device v" NI_XCOD= ER_REVISION), + .p.priv_class =3D &ni_upload_class, + .priv_size =3D sizeof(NetIntUploadContext), + .init =3D init, + .uninit =3D uninit, + .activate =3D activate, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +}; diff --git a/libavfilter/vf_overlay_ni.c b/libavfilter/vf_overlay_ni.c new file mode 100644 index 0000000000..7be8f45cec --- /dev/null +++ b/libavfilter/vf_overlay_ni.c @@ -0,0 +1,1397 @@ +/* + * Copyright (c) 2010 Stefano Sabatini + * Copyright (c) 2010 Baptiste Coudurier + * Copyright (c) 2007 Bobby Bingham + * Copyright (c) 2021 NetInt + * + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * overlay one video on top of another + */ + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "libavutil/common.h" +#include "libavutil/eval.h" +#include "libavutil/avstring.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" +#include "libavutil/timestamp.h" +#include "libavutil/hwcontext.h" +#include "drawutils.h" +#include "framesync.h" +#include "video.h" +#include + +static const char *const var_names[] =3D { + "main_w", "W", ///< width of the main video + "main_h", "H", ///< height of the main video + "overlay_w", "w", ///< width of the overlay video + "overlay_h", "h", ///< height of the overlay video + "hsub", + "vsub", + "x", + "y", + "t", + NULL +}; + +enum var_name { + VAR_MAIN_W, VAR_MW, + VAR_MAIN_H, VAR_MH, + VAR_OVERLAY_W, VAR_OW, + VAR_OVERLAY_H, VAR_OH, + VAR_HSUB, + VAR_VSUB, + VAR_X, + VAR_Y, + VAR_T, + VAR_VARS_NB +}; + +#define MAIN 0 +#define OVERLAY 1 + +typedef struct NetIntOverlayContext { + const AVClass *class; + int x, y; ///< position of overlaid picture + + uint8_t main_has_alpha; + uint8_t overlay_has_alpha; + int alpha_format; + + FFFrameSync fs; + + int hsub, vsub; ///< chroma subsampling values + + double var_values[VAR_VARS_NB]; + char *x_expr, *y_expr; + + AVExpr *x_pexpr, *y_pexpr; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + + AVBufferRef* out_frames_ref; + + int initialized; + int session_opened; + int crop_session_opened; + int keep_alive_timeout; /* keep alive timeout setting */ + int inplace; + int buffer_limit; + uint16_t ui16CropFrameIdx; + ni_session_context_t crop_api_ctx; + ni_session_data_io_t crop_api_dst_frame; +} NetIntOverlayContext; + +static int process_frame(FFFrameSync *fs); +static int process_frame_inplace(FFFrameSync *fs); + +static int set_expr(AVExpr **pexpr, const char *expr, const char *option, = void *log_ctx) +{ + int ret; + AVExpr *old =3D NULL; + + if (*pexpr) + old =3D *pexpr; + ret =3D av_expr_parse(pexpr, expr, var_names, + NULL, NULL, NULL, NULL, 0, log_ctx); + if (ret < 0) { + av_log(log_ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s' for %s\n", + expr, option); + *pexpr =3D old; + return ret; + } + + av_expr_free(old); + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + /* We only accept hardware frames */ + static const enum AVPixelFormat pix_fmts[] =3D + {AV_PIX_FMT_NI_QUAD, AV_PIX_FMT_NONE}; + AVFilterFormats *formats; + + formats =3D ff_make_format_list(pix_fmts); + + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static int init_framesync(AVFilterContext *ctx) +{ + NetIntOverlayContext *s =3D ctx->priv; + int ret, i; + + s->fs.on_event =3D s->inplace ? process_frame_inplace : process_frame; + s->fs.opaque =3D s; + ret =3D ff_framesync_init(&s->fs, ctx, ctx->nb_inputs); + if (ret < 0) + return ret; + + for (i =3D 0; i < ctx->nb_inputs; i++) { + FFFrameSyncIn *in =3D &s->fs.in[i]; + in->before =3D EXT_STOP; + in->after =3D EXT_INFINITY; + in->sync =3D i ? 1 : 2; + in->time_base =3D ctx->inputs[i]->time_base; + } + + return ff_framesync_configure(&s->fs); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntOverlayContext *s =3D ctx->priv; + + ff_framesync_uninit(&s->fs); + av_expr_free(s->x_pexpr); s->x_pexpr =3D NULL; + av_expr_free(s->y_pexpr); s->y_pexpr =3D NULL; + + if (s->api_dst_frame.data.frame.p_buffer) { + ni_frame_buffer_free(&s->api_dst_frame.data.frame); + } + + if (s->crop_api_dst_frame.data.frame.p_buffer) { + ni_frame_buffer_free(&s->crop_api_dst_frame.data.frame); + } + + if (s->session_opened) { + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + } + + if (s->crop_session_opened) { + ni_device_session_close(&s->crop_api_ctx, 1, NI_DEVICE_TYPE_SCALER= ); + ni_device_session_context_clear(&s->crop_api_ctx); + } + + av_buffer_unref(&s->out_frames_ref); +} + +static inline int normalize_xy(double d, int chroma_sub) +{ + if (isnan(d)) + return INT_MAX; + return (int)d & ~((1 << chroma_sub) - 1); +} + +static void eval_expr(AVFilterContext *ctx) +{ + NetIntOverlayContext *s =3D ctx->priv; + + s->var_values[VAR_X] =3D av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->var_values[VAR_Y] =3D av_expr_eval(s->y_pexpr, s->var_values, NULL); + s->var_values[VAR_X] =3D av_expr_eval(s->x_pexpr, s->var_values, NULL); + s->x =3D normalize_xy(s->var_values[VAR_X], s->hsub); + s->y =3D normalize_xy(s->var_values[VAR_Y], s->vsub); +} + +static int overlay_intersects_background( + const AVFilterContext *ctx, + const AVFrame *overlay, + const AVFrame *main) +{ + const NetIntOverlayContext *s =3D (NetIntOverlayContext *) ctx->priv; + + if (s->x >=3D main->width) + return 0; + + if (s->y >=3D main->height) + return 0; + + if (s->x + overlay->width <=3D 0) + return 0; + + if (s->y + overlay->height <=3D 0) + return 0; + + return 1; +} + +static void calculate_src_rectangle( + int *px, + int *py, + int *pw, + int *ph, + int bgnd_x, + int bgnd_y, + int bgnd_w, + int bgnd_h, + int ovly_x, + int ovly_y, + int ovly_w, + int ovly_h) + +{ + *px =3D (ovly_x > 0) ? 0 : -ovly_x; + *py =3D (ovly_y > 0) ? 0 : -ovly_y; + + if (ovly_x > 0) { + *pw =3D FFMIN(bgnd_w - ovly_x, ovly_w); + } else { + *pw =3D FFMIN(ovly_w + ovly_x, bgnd_w); + } + + if (ovly_y > 0) { + *ph =3D FFMIN(bgnd_h - ovly_y, ovly_h); + } else { + *ph =3D FFMIN(ovly_h + ovly_y, bgnd_h); + } +} + +static void calculate_dst_rectangle( + int *px, + int *py, + int *pw, + int *ph, + int bgnd_x, + int bgnd_y, + int bgnd_w, + int bgnd_h, + int ovly_x, + int ovly_y, + int ovly_w, + int ovly_h) +{ + *px =3D FFMAX(0, ovly_x); + *py =3D FFMAX(0, ovly_y); + + if (ovly_x > 0) { + *pw =3D FFMIN(bgnd_w - ovly_x, ovly_w); + } else { + *pw =3D FFMIN(ovly_w + ovly_x, bgnd_w); + } + + if (ovly_y > 0) { + *ph =3D FFMIN(bgnd_h - ovly_y, ovly_h); + } else { + *ph =3D FFMIN(ovly_h + ovly_y, bgnd_h); + } +} + +static const enum AVPixelFormat alpha_pix_fmts[] =3D { + AV_PIX_FMT_ARGB, AV_PIX_FMT_ABGR, AV_PIX_FMT_RGBA, + AV_PIX_FMT_BGRA, AV_PIX_FMT_NONE +}; + +static int config_input_overlay(AVFilterLink *inlink) +{ + AVFilterContext *ctx =3D inlink->dst; + NetIntOverlayContext *s =3D inlink->dst->priv; + AVHWFramesContext *in_frames_ctx; + const AVPixFmtDescriptor *pix_desc; + int ret; + + FilterLink *li =3D ff_filter_link(inlink); + if (li =3D=3D NULL) { + av_log(inlink->dst, AV_LOG_ERROR, "No hw context provided on input= \n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + if (!in_frames_ctx) { + return AVERROR(EINVAL); + } + + pix_desc =3D av_pix_fmt_desc_get(in_frames_ctx->sw_format); + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 10b not supported for overlay!\= n"); + return AVERROR(EINVAL); + } + + /* Finish the configuration by evaluating the expressions + now when both inputs are configured. */ + s->var_values[VAR_MAIN_W ] =3D s->var_values[VAR_MW] =3D ctx->inputs= [MAIN ]->w; + s->var_values[VAR_MAIN_H ] =3D s->var_values[VAR_MH] =3D ctx->inputs= [MAIN ]->h; + s->var_values[VAR_OVERLAY_W] =3D s->var_values[VAR_OW] =3D ctx->inputs= [OVERLAY]->w; + s->var_values[VAR_OVERLAY_H] =3D s->var_values[VAR_OH] =3D ctx->inputs= [OVERLAY]->h; + s->var_values[VAR_HSUB] =3D 1<log2_chroma_w; + s->var_values[VAR_VSUB] =3D 1<log2_chroma_h; + s->var_values[VAR_X] =3D NAN; + s->var_values[VAR_Y] =3D NAN; + s->var_values[VAR_T] =3D NAN; + + if ((ret =3D set_expr(&s->x_pexpr, s->x_expr, "x", ctx)= ) < 0 || + (ret =3D set_expr(&s->y_pexpr, s->y_expr, "y", ctx)= ) < 0) + return ret; + + s->overlay_has_alpha =3D ff_fmt_is_in(in_frames_ctx->sw_format, + alpha_pix_fmts); + + av_log(ctx, AV_LOG_VERBOSE, + "main w:%d h:%d fmt:%s overlay w:%d h:%d fmt:%s\n", + ctx->inputs[MAIN]->w, ctx->inputs[MAIN]->h, + av_get_pix_fmt_name(ctx->inputs[MAIN]->format), + ctx->inputs[OVERLAY]->w, ctx->inputs[OVERLAY]->h, + av_get_pix_fmt_name(ctx->inputs[OVERLAY]->format)); + return 0; +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntOverlayContext *s =3D ctx->priv; + AVHWFramesContext *out_frames_ctx; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + s->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&s->api_ctx, out_frames_ctx->width, + out_frames_ctx->height, out_frames_ctx->= sw_format, + pool_size, + s->buffer_limit); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + NetIntOverlayContext *s =3D ctx->priv; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + int ret =3D 0; + + outlink->w =3D ctx->inputs[MAIN]->w; + outlink->h =3D ctx->inputs[MAIN]->h; + FilterLink *li =3D ff_filter_link(ctx->inputs[MAIN]); + FilterLink *lo =3D ff_filter_link(outlink); + lo->frame_rate =3D li->frame_rate; + outlink->time_base =3D ctx->inputs[MAIN]->time_base; + + ret =3D init_framesync(ctx); + if (ret < 0) + return ret; + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (!s->inplace) { + FilterLink *lt; + AVHWFramesContext *tmp_frames_ctx; + + s->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_r= ef); + if (!s->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D outlink->w; + out_frames_ctx->height =3D outlink->h; + + av_hwframe_ctx_init(s->out_frames_ref); + + lt =3D ff_filter_link(ctx->inputs[OVERLAY]); + tmp_frames_ctx =3D (AVHWFramesContext *)lt->hw_frames_ctx->data; + + // HW does not support NV12 Compress + RGB -> NV12 Compress + if (((in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X= 4) || + (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X= 4)) && + ((tmp_frames_ctx->sw_format >=3D AV_PIX_FMT_ARGB) && + (tmp_frames_ctx->sw_format <=3D AV_PIX_FMT_BGRA))) { + out_frames_ctx->sw_format =3D AV_PIX_FMT_NV12; + av_log(ctx, AV_LOG_WARNING, "Overlay output is changed to nv12= \n"); + } else { + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + } + + out_frames_ctx->initial_pool_size =3D + NI_OVERLAY_ID; // Repurposed as identity code + } else { + s->out_frames_ref =3D av_buffer_ref(li->hw_frames_ctx); + } + + av_buffer_unref(&lo->hw_frames_ctx); + + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + return ret; +} + +static int do_intermediate_crop_and_overlay(AVFilterContext *ctx, + AVFrame *overlay, AVFrame *fra= me) +{ + NetIntOverlayContext *s =3D (NetIntOverlayContext *) ctx->priv; + AVHWFramesContext *main_frame_ctx,*ovly_frame_ctx; + niFrameSurface1_t *frame_surface; + ni_retcode_t retcode; + uint16_t ui16FrameIdx; + int main_scaler_format,ovly_scaler_format; + int flags; + int crop_x,crop_y,crop_w,crop_h; + int src_x,src_y,src_w,src_h; + + main_frame_ctx =3D (AVHWFramesContext *) frame->hw_frames_ctx->data; + main_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(main_frame_ctx->sw_format); + + ovly_frame_ctx =3D (AVHWFramesContext *) overlay->hw_frames_ctx->data; + ovly_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(ovly_frame_ctx->sw_format); + + /* Allocate a ni_frame_t for the intermediate crop operation */ + retcode =3D ni_frame_buffer_alloc_hwenc(&s->crop_api_dst_frame.data.fr= ame, + ctx->inputs[OVERLAY]->w, + ctx->inputs[OVERLAY]->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't allocate interim crop frame\n"); + return AVERROR(ENOMEM); + } + + calculate_dst_rectangle(&crop_x, &crop_y, &crop_w, &crop_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x,2), FFALIGN(s->y,2), + overlay->width, overlay->height); + + frame_surface =3D (niFrameSurface1_t *) frame->data[3]; + + /* Assign a device input frame. Send incoming frame index to crop sess= ion */ + retcode =3D ni_device_alloc_frame( + &s->crop_api_ctx, + FFALIGN(ctx->inputs[MAIN]->w, 2), + FFALIGN(ctx->inputs[MAIN]->h, 2), + main_scaler_format, + 0, + crop_w, + crop_h, + crop_x, + crop_y, + 0, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't assign input crop frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + /* Allocate destination frame. This acquires a frame from the pool */ + retcode =3D ni_device_alloc_frame( + &s->crop_api_ctx, + FFALIGN(ctx->inputs[OVERLAY]->w, 2), + FFALIGN(ctx->inputs[OVERLAY]->h, 2), + ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA), + NI_SCALER_FLAG_IO, + crop_w, + crop_h, + 0, + 0, + 0, + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't allocate output crop frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + retcode =3D ni_device_session_read_hwdesc(&s->crop_api_ctx, + &s->crop_api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "No cropped output frame %d\n", retcode); + return AVERROR(ENOMEM); + } + + /* Get the acquired frame */ + frame_surface =3D (niFrameSurface1_t *) + s->crop_api_dst_frame.data.frame.p_data[3]; + s->ui16CropFrameIdx =3D frame_surface->ui16FrameIdx; + + /* Overlay the icon over the intermediate cropped frame */ + + /* Allocate a ni_frame_t for the intermediate overlay */ + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + ctx->inputs[OVERLAY]->w, + ctx->inputs[OVERLAY]->h, + 0); + + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, "Can't allocate interim ovly frame\n"); + return AVERROR(ENOMEM); + } + + frame_surface =3D (niFrameSurface1_t *) overlay->data[3]; + ui16FrameIdx =3D frame_surface->ui16FrameIdx; + + calculate_src_rectangle(&src_x, &src_y, &src_w, &src_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x,2), FFALIGN(s->y,2), + overlay->width, overlay->height); + + /* Assign input frame to intermediate overlay session */ + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(ctx->inputs[OVERLAY]->w, 2), + FFALIGN(ctx->inputs[OVERLAY]->h, 2), + ovly_scaler_format, + 0, + src_w, + src_h, + src_x, + src_y, + 0, + ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't assign input overlay frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + /* In-place overlay frame. Send down frame index of background frame */ + flags =3D NI_SCALER_FLAG_IO; /* Configure outpu= t */ + flags |=3D s->alpha_format ? NI_SCALER_FLAG_PA : 0; /* Premultiply/str= aight */ + + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(ctx->inputs[OVERLAY]->w, 2), + FFALIGN(ctx->inputs[OVERLAY]->h, 2), + ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA), + flags, + crop_w, + crop_h, + 0, + 0, + 0, + s->ui16CropFrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't overlay frame for output %d\n", + retcode); + return AVERROR(ENOMEM); + } + + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, + &s->api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't acquire intermediate frame %d\n", + retcode); + return AVERROR(ENOMEM); + } + + return NI_RETCODE_SUCCESS; +} + +static int process_frame(FFFrameSync *fs) +{ + AVFilterContext *ctx =3D fs->parent; + NetIntOverlayContext *s =3D (NetIntOverlayContext *) ctx->priv; + AVHWFramesContext *main_frame_ctx,*ovly_frame_ctx; + AVNIDeviceContext *pAVNIDevCtx; + AVFilterLink *inlink_main =3D ctx->inputs[MAIN]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFilterLink *inlink_overlay =3D ctx->inputs[OVERLAY]; + AVFrame *frame =3D NULL; + AVFrame *overlay =3D NULL; + AVFrame *out =3D NULL; + niFrameSurface1_t *frame_surface,*new_frame_surface; + int flags, main_cardno, ovly_cardno; + int main_scaler_format, ovly_scaler_format; + ni_retcode_t retcode; + uint16_t tempFIDOverlay =3D 0; + uint16_t tempFIDFrame =3D 0; + + av_log(ctx, AV_LOG_TRACE, "%s: ready %u inlink framequeue %u available= _frame %d inlink_overlay framequeue %u available_frame %d outlink framequeu= e %u frame_wanted %d\n", + __func__, ctx->ready, + ff_inlink_queued_frames(inlink_main), ff_inlink_check_available_fr= ame(inlink_main), + ff_inlink_queued_frames(inlink_overlay), ff_inlink_check_available= _frame(inlink_overlay), + ff_inlink_queued_frames(outlink), ff_outlink_frame_wanted(outlink)= ); + + // Consume from inlink framequeue only when outlink framequeue is empt= y, to prevent filter from exhausting all pre-allocated device buffers + if (ff_inlink_check_available_frame(outlink)) + return FFERROR_NOT_READY; + + /* ff_framesync_get_frame() always returns 0 for hw frames */ + ff_framesync_get_frame(fs, OVERLAY, &overlay, 0); + + if (!overlay) { + ff_framesync_get_frame(fs, MAIN, &frame, 1); + return ff_filter_frame(ctx->outputs[0], frame); + } + + ff_framesync_get_frame(fs, MAIN, &frame, 0); + + frame->pts =3D + av_rescale_q(fs->pts, fs->time_base, ctx->outputs[0]->time_base); + + if (overlay) { + s->var_values[VAR_OVERLAY_W] =3D s->var_values[VAR_OW] =3D overlay= ->width; + s->var_values[VAR_OVERLAY_H] =3D s->var_values[VAR_OH] =3D overlay= ->height; + } + + s->var_values[VAR_MAIN_W ] =3D s->var_values[VAR_MW] =3D frame->widt= h; + s->var_values[VAR_MAIN_H ] =3D s->var_values[VAR_MH] =3D frame->heig= ht; + s->var_values[VAR_T] =3D frame->pts =3D=3D AV_NOPTS_VALUE ? + NAN : frame->pts * av_q2d(inlink_main->time_ba= se); + + // This can satisfy some customers or demos to modify the location whe= n using ni_overlay + set_expr(&s->x_pexpr, s->x_expr,"x", ctx); + set_expr(&s->y_pexpr, s->y_expr,"y", ctx); + + eval_expr(ctx); + av_log(ctx, AV_LOG_DEBUG, "x:%f xi:%d y:%f yi:%d t:%f\n", + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y, + s->var_values[VAR_T]); + + main_frame_ctx =3D (AVHWFramesContext *) frame->hw_frames_ctx->data; + main_scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(main_frame_ctx->s= w_format); + + main_cardno =3D ni_get_cardno(frame); + + if (overlay) { + ovly_frame_ctx =3D (AVHWFramesContext *) overlay->hw_frames_ctx->d= ata; + ovly_scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(ovly_frame_ct= x->sw_format); + ovly_cardno =3D ni_get_cardno(overlay); + + if (main_cardno !=3D ovly_cardno) { + av_log(ctx, AV_LOG_ERROR, + "Main/Overlay frames on different cards\n"); + return AVERROR(EINVAL); + } + } else { + ovly_scaler_format =3D 0; + } + + if (!s->initialized) { + retcode =3D ni_device_session_context_init(&s->api_ctx); + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "ni overlay filter session context init failure\n"); + return retcode; + } + + pAVNIDevCtx =3D (AVNIDeviceContext *)main_frame_ctx->device_ctx->h= wctx; + s->api_ctx.device_handle =3D pAVNIDevCtx->cards[main_cardno]; + s->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[main_cardno]; + + s->api_ctx.hw_id =3D main_cardno; + s->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_OVERLAY; + s->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retcode =3D ni_device_session_open(&s->api_ctx, NI_DEVICE_TYPE_SCA= LER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't open device session on card %= d\n", + main_cardno); + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + return retcode; + } + + s->session_opened =3D 1; + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + inlink_main->dst->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUE= UE_SIZE > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + + retcode =3D init_out_pool(inlink_main->dst); + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", retcod= e); + return retcode; + } + + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)s->out_= frames_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_frames_= ctx->hwctx; + ni_cpy_hwframe_ctx(main_frame_ctx, out_frames_ctx); + ni_device_session_copy(&s->api_ctx, &out_ni_ctx->api_ctx); + + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(main_frame_= ctx->sw_format); + + if (((frame && frame->color_range =3D=3D AVCOL_RANGE_JPEG) || + (overlay && overlay->color_range =3D=3D AVCOL_RANGE_JPEG)) && = !(desc->flags & AV_PIX_FMT_FLAG_RGB)) { + av_log(ctx, AV_LOG_WARNING, + "WARNING: Full color range input, limited color range o= utput\n"); + } + + if (av_buffer_get_ref_count(frame->buf[0]) > 1) { + av_log(ctx, AV_LOG_WARNING, + "WARNING: In-place overlay being used after split " + "filter may cause corruption\n"); + } + + s->initialized =3D 1; + } + + /* Allocate a ni_frame for the overlay output */ + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + return AVERROR(ENOMEM); + } + + if (overlay) { + frame_surface =3D (niFrameSurface1_t *)overlay->data[3]; + tempFIDOverlay =3D frame_surface->ui16FrameIdx; + } else { + frame_surface =3D NULL; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + /* + * Assign an input frame for overlay picture. Send the + * incoming hardware frame index to the scaler manager. + */ + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + overlay ? FFALIGN(overlay->width, 2) : 0, + overlay ? FFALIGN(overlay->height, 2) : 0, + ovly_scaler_format, + (frame_surface && frame_surface->encoding_type =3D=3D 2) ? NI_SCAL= ER_FLAG_CMP : 0, + overlay ? FFALIGN(overlay->width, 2) : 0, + overlay ? FFALIGN(overlay->height, 2) : 0, + s->x, + s->y, + frame_surface ? (int)frame_surface->ui32nodeAddress : 0, + frame_surface ? frame_surface->ui16FrameIdx : 0, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't assign frame for overlay input %d= \n", + retcode); + return AVERROR(ENOMEM); + } + + frame_surface =3D (niFrameSurface1_t *) frame->data[3]; + if (frame_surface =3D=3D NULL) { + return AVERROR(EINVAL); + } + + tempFIDFrame =3D frame_surface->ui16FrameIdx; + /* + * Allocate device output frame from the pool. We also send down the f= rame index + * of the background frame to the scaler manager. + */ + flags =3D (s->alpha_format ? NI_SCALER_FLAG_PA : 0) | NI_SCALER_FLAG_I= O; + flags |=3D (frame_surface && frame_surface->encoding_type =3D=3D 2) ? = NI_SCALER_FLAG_CMP : 0; + retcode =3D ni_device_alloc_frame(&s->api_ctx, + FFALIGN(frame->width, 2), + FFALIGN(frame->height, 2), + main_scaler_format, + flags, + FFALIGN(frame->width, 2), + FFALIGN(frame->height, 2), + 0, // x + 0, // y + frame_surface->ui32nodeAddress, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_DEBUG, "Can't allocate frame for output %d\n", + retcode); + return AVERROR(ENOMEM); + } + + out =3D av_frame_alloc(); + if (!out) { + return AVERROR(ENOMEM); + } + + av_frame_copy_props(out,frame); + + out->width =3D outlink->w; + out->height =3D outlink->h; + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + + if (!out->data[3]) { + av_frame_free(&out); + return AVERROR(ENOMEM); + } + + /* Copy the frame surface from the incoming frame */ + memcpy(out->data[3], frame->data[3], sizeof(niFrameSurface1_t)); + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, &s->api_dst_fra= me, + NI_DEVICE_TYPE_SCALER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't acquire output frame %d\n", retcode); + av_frame_free(&out); + return AVERROR(ENOMEM); + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_overlay"); +#endif + + frame_surface =3D (niFrameSurface1_t *) out->data[3]; + new_frame_surface =3D (niFrameSurface1_t *) s->api_dst_frame.data.fram= e.p_data[3]; + frame_surface->ui16FrameIdx =3D new_frame_surface->ui16FrameIdx; + frame_surface->ui16session_ID =3D new_frame_surface->ui16session_ID; + frame_surface->device_handle =3D new_frame_surface->device_handle; + frame_surface->output_idx =3D new_frame_surface->output_idx; + frame_surface->src_cpu =3D new_frame_surface->src_cpu; + frame_surface->dma_buf_fd =3D 0; + + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + main_frame_ctx->sw_format); + + /* Remove ni-split specific assets */ + frame_surface->ui32nodeAddress =3D 0; + + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + av_log(ctx, AV_LOG_DEBUG, + "%s:IN trace ui16FrameIdx =3D [%d] and [%d] --> out [%d] \n", _= _FILE__, + tempFIDFrame, tempFIDOverlay, frame_surface->ui16FrameIdx); + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurface1_= t), ff_ni_frame_free, NULL, 0); + + return ff_filter_frame(ctx->outputs[0], out); +} + +static int process_frame_inplace(FFFrameSync *fs) +{ + AVFilterContext *ctx =3D fs->parent; + NetIntOverlayContext *s =3D (NetIntOverlayContext *) ctx->priv; + AVHWFramesContext *main_frame_ctx,*ovly_frame_ctx; + AVNIDeviceContext *pAVNIDevCtx; + AVFilterLink *inlink_main =3D ctx->inputs[MAIN]; + AVFilterLink *inlink_overlay =3D ctx->inputs[OVERLAY]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + AVFrame *overlay =3D NULL; + AVFrame *out =3D NULL; + niFrameSurface1_t *frame_surface; + ni_retcode_t retcode; + uint16_t ovly_frame_idx =3D 0; + uint16_t main_frame_idx =3D 0; + int flags, main_cardno, ovly_cardno; + int main_scaler_format, ovly_scaler_format; + int src_x, src_y, src_w, src_h; + int dst_x, dst_y, dst_w, dst_h; + + av_log(ctx, AV_LOG_TRACE, "%s: ready %u inlink framequeue %u available= _frame %d " + "inlink_overlay framequeue %u available_frame %d outlink frameq= ueue %u " + "frame_wanted %d\n", __func__, ctx->ready, ff_inlink_queued_fra= mes(inlink_main), + ff_inlink_check_available_frame(inlink_main), + ff_inlink_queued_frames(inlink_overlay), + ff_inlink_check_available_frame(inlink_overlay), ff_inlink_queu= ed_frames(outlink), + ff_outlink_frame_wanted(outlink)); + + // Consume from inlink framequeue only when outlink framequeue is empt= y, to prevent filter from exhausting all pre-allocated device buffers + if (ff_inlink_check_available_frame(outlink)) + return FFERROR_NOT_READY; + + ff_framesync_get_frame(fs, OVERLAY, &overlay, 0); + + if (!overlay) { + ff_framesync_get_frame(fs, MAIN, &frame, 1); + return ff_filter_frame(ctx->outputs[0], frame); + } + + ff_framesync_get_frame(fs, MAIN, &frame, 0); + + frame->pts =3D + av_rescale_q(fs->pts, fs->time_base, ctx->outputs[0]->time_base); + + s->var_values[VAR_OVERLAY_W] =3D s->var_values[VAR_OW] =3D overlay->wi= dth; + s->var_values[VAR_OVERLAY_H] =3D s->var_values[VAR_OH] =3D overlay->he= ight; + s->var_values[VAR_MAIN_W ] =3D s->var_values[VAR_MW] =3D frame->widt= h; + s->var_values[VAR_MAIN_H ] =3D s->var_values[VAR_MH] =3D frame->heig= ht; + s->var_values[VAR_T] =3D frame->pts =3D=3D AV_NOPTS_VALUE ? + NAN : frame->pts * av_q2d(ctx->inputs[0]->time= _base); + + // Allow location modification + set_expr(&s->x_pexpr, s->x_expr, "x", ctx); + set_expr(&s->y_pexpr, s->y_expr, "y", ctx); + + eval_expr(ctx); + av_log(ctx, AV_LOG_DEBUG, "x:%f xi:%d y:%f yi:%d t:%f\n", + s->var_values[VAR_X], s->x, + s->var_values[VAR_Y], s->y, + s->var_values[VAR_T]); + + main_frame_ctx =3D (AVHWFramesContext *) frame->hw_frames_ctx->data; + main_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(main_frame_ctx->sw_format); + + frame_surface =3D (niFrameSurface1_t *) frame->data[3]; + + if (frame_surface =3D=3D NULL) + return AVERROR(EINVAL); + + main_frame_idx =3D frame_surface->ui16FrameIdx; + + frame_surface =3D (niFrameSurface1_t *) overlay->data[3]; + + if (frame_surface =3D=3D NULL) + return AVERROR(EINVAL); + + ovly_frame_idx =3D frame_surface->ui16FrameIdx; + + main_cardno =3D ni_get_cardno(frame); + + ovly_frame_ctx =3D (AVHWFramesContext *) overlay->hw_frames_ctx->data; + ovly_scaler_format =3D + ff_ni_ffmpeg_to_gc620_pix_fmt(ovly_frame_ctx->sw_format); + ovly_cardno =3D ni_get_cardno(overlay); + + if (main_cardno !=3D ovly_cardno) { + av_log(ctx, AV_LOG_ERROR, "Main/Overlay frames on different cards\= n"); + return AVERROR(EINVAL); + } + + // If overlay does not intersect the background, pass + // the frame through the overlay filter. + if (!overlay_intersects_background(ctx, overlay, frame)) { + out =3D av_frame_clone(frame); + + if (!out) { + av_log(ctx, AV_LOG_ERROR, "Can't clone frame\n"); + return AVERROR(ENOMEM); + } + return ff_filter_frame(ctx->outputs[0], out); + } + + if (!s->initialized) { + /* Set up a scaler session for the in-place overlay */ + retcode =3D ni_device_session_context_init(&s->api_ctx); + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "ni overlay filter session context init failure\n"); + return retcode; + } + + pAVNIDevCtx =3D (AVNIDeviceContext *)main_frame_ctx->device_ctx->h= wctx; + s->api_ctx.device_handle =3D pAVNIDevCtx->cards[main_cardno]; + s->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[main_cardno]; + + s->api_ctx.hw_id =3D main_cardno; + s->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_IPOVLY; + s->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retcode =3D ni_device_session_open(&s->api_ctx, NI_DEVICE_TYPE_SCA= LER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't open device session on card %= d\n", + main_cardno); + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + return retcode; + } + + s->session_opened =3D 1; + + // If the in-place overlay is rgba over yuv, we need to set up + // an extra intermediate crop session. + if (s->overlay_has_alpha && !s->main_has_alpha) { + /* Set up a scaler session for the crop operation */ + retcode =3D ni_device_session_context_init(&s->crop_api_ctx); + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "ni overlay filter (crop) session context init fail= ure\n"); + return retcode; + } + + s->crop_api_ctx.device_handle =3D pAVNIDevCtx->cards[main_card= no]; + s->crop_api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[main_card= no]; + + s->crop_api_ctx.hw_id =3D main_cardno; + s->crop_api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->crop_api_ctx.scaler_operation =3D NI_SCALER_OPCODE_CROP; + s->crop_api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retcode =3D ni_device_session_open(&s->crop_api_ctx, + NI_DEVICE_TYPE_SCALER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't open device session on card %d\n", main_card= no); + ni_device_session_close(&s->crop_api_ctx, 1, NI_DEVICE_TYP= E_SCALER); + ni_device_session_context_clear(&s->crop_api_ctx); + return retcode; + } + + s->crop_session_opened =3D 1; + + /* init the out pool for the crop session, make it rgba */ + retcode =3D ff_ni_build_frame_pool(&s->crop_api_ctx, overlay->= width, + overlay->height, + AV_PIX_FMT_RGBA, 1, 0); + + if (retcode < 0) { + av_log(ctx, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", re= tcode); + return retcode; + } + } + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + inlink_main->dst->extra_hw_frames =3D ((DEFAULT_FRAME_THREAD_QU= EUE_SIZE > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0) - DEFAULT_NI_FILTER_PO= OL_SIZE; + } + + retcode =3D init_out_pool(inlink_main->dst); + if (retcode < 0) { + av_log(inlink_main->dst, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", retcod= e); + return retcode; + } + + s->initialized =3D 1; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + /* For rgba over yuv, we do an intermediate crop and overlay */ + if (s->overlay_has_alpha && !s->main_has_alpha) { + retcode =3D do_intermediate_crop_and_overlay(ctx, overlay, frame); + + if (retcode < 0) + return retcode; + + /* Allocate a ni_frame for the overlay output */ + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.fra= me, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't allocate inplace overlay fram= e\n"); + return AVERROR(ENOMEM); + } + + calculate_src_rectangle(&src_x, &src_y, &src_w, &src_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x,2),FFALIGN(s->y,2), overlay->= width, overlay->height); + + /* + * Assign an input frame for overlay picture. Send the + * incoming hardware frame index to the scaler manager. + */ + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(overlay->width, 2), // ovly width + FFALIGN(overlay->height, 2), // ovly height + ff_ni_ffmpeg_to_gc620_pix_fmt(AV_PIX_FMT_RGBA), // ovly pix fmt + 0, // flags + src_w, // src rect width + src_h, // src rect height + 0, // src rect x + 0, // src rect y + 0, // n/a + s->ui16CropFrameIdx, // ovly frame idx + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't assign input overlay frame %d= \n", + retcode); + return AVERROR(ENOMEM); + } + + calculate_dst_rectangle(&dst_x, &dst_y, &dst_w, &dst_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x,2), FFALIGN(s->y, 2), + overlay->width, overlay->height); + + /* + * Allocate device output frame from the pool. We also send down t= he + * frame index of the background frame to the scaler manager. + */ + + /* configure the output */ + flags =3D NI_SCALER_FLAG_IO; + /* premultiply vs straight alpha */ + flags |=3D (s->alpha_format) ? NI_SCALER_FLAG_PA : 0; + + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(frame->width, 2), // main width + FFALIGN(frame->height, 2), // main height + main_scaler_format, // main pix fmt + flags, // flags + dst_w, // dst rect width + dst_h, // dst rect height + dst_x, // dst rect x + dst_y, // dst rect y + 0, // n/a + main_frame_idx, // main frame idx + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Can't allocate overlay output %d\n", + retcode); + return AVERROR(ENOMEM); + } + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, + &s->api_dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't acquire output overlay frame %d\n", retcode); + return AVERROR(ENOMEM); + } + } else { + /* Not rgba over yuv. For yuv over yuv, yuv over rgba, */ + /* rgba over rgba, we can perform an in-place overlay immediately.= */ + + /* Allocate ni_frame for the overlay output */ + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.fra= me, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Cannot allocate in-place frame\n"); + return AVERROR(ENOMEM); + } + + calculate_src_rectangle(&src_x, &src_y, &src_w, &src_h, + 0, 0, frame->width, frame->height, + FFALIGN(s->x,2), FFALIGN(s->y,2), + overlay->width, overlay->height); + + /* + * Assign input frame for overlay picture. Sends the + * incoming hardware frame index to the scaler manager. + */ + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(overlay->width, 2), // overlay width + FFALIGN(overlay->height, 2), // overlay height + ovly_scaler_format, // overlay pix fmt + 0, // flags + src_w, // src rect width + src_h, // src rect height + src_x, // src rect x + src_y, // src rect y + 0, // n/a + ovly_frame_idx, // overlay frame idx + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't assign frame for overlay input %d\n", retcode); + return AVERROR(ENOMEM); + } + + /* In-place overlay frame. Send down frame index of background fra= me */ + + /* Configure the output */ + flags =3D NI_SCALER_FLAG_IO; + /* Premultiply vs straight alpha */ + flags |=3D s->alpha_format ? NI_SCALER_FLAG_PA : 0; + + calculate_dst_rectangle(&dst_x,&dst_y,&dst_w,&dst_h, + 0,0,frame->width,frame->height, + FFALIGN(s->x,2),FFALIGN(s->y,2), + overlay->width,overlay->height); + + retcode =3D ni_device_alloc_frame( + &s->api_ctx, + FFALIGN(frame->width, 2), // main width + FFALIGN(frame->height, 2), // main height + main_scaler_format, // main pix fmt + flags, // flags + dst_w, // dst rect width + dst_h, // dst rect height + dst_x, // dst rect x + dst_y, // dst rect y + 0, // n/a + main_frame_idx, // main frame idx + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't allocate frame for output ovly %d\n", retcode); + return AVERROR(ENOMEM); + } + + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, &s->api_dst= _frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, + "Can't acquire output frame of overlay %d\n", retcode); + return AVERROR(ENOMEM); + } + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_overlay"); +#endif + + /* Do an in-place overlay onto the background frame */ + out =3D av_frame_clone(frame); + + if (!out) { + av_log(ctx, AV_LOG_ERROR, "Cannot clone frame\n"); + return AVERROR(ENOMEM); + } + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + if (s->overlay_has_alpha && !s->main_has_alpha) { + av_log(ctx, AV_LOG_DEBUG, + "%s:IN trace ui16FrameIdx =3D [%d] and [%d] and [%d] --> out [= %d]\n", + __func__, main_frame_idx, ovly_frame_idx, s->ui16CropFrameIdx, + main_frame_idx); + } else { + av_log(ctx, AV_LOG_DEBUG, + "%s:IN trace ui16FrameIdx =3D [%d] and [%d] --> out [%d]\n", + __func__, main_frame_idx, ovly_frame_idx, main_frame_idx); + } + + if (s->overlay_has_alpha && !s->main_has_alpha) { + ni_hwframe_buffer_recycle((niFrameSurface1_t *) + s->crop_api_dst_frame.data.frame.p_data[= 3], + (int32_t)s->crop_api_ctx.device_handle); + } + + return ff_filter_frame(ctx->outputs[0], out); +} + +/** + * Blend image in src to destination buffer dst at position (x, y). + */ +static int config_input_main(AVFilterLink *inlink) +{ + NetIntOverlayContext *s =3D inlink->dst->priv; + AVHWFramesContext *in_frames_ctx; + const AVPixFmtDescriptor *pix_desc; + = + FilterLink *li =3D ff_filter_link(inlink); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(inlink->dst, AV_LOG_ERROR, "No hw context provided on input= \n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + if (!in_frames_ctx) { + return AVERROR(EINVAL); + } + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(inlink->dst, AV_LOG_ERROR, "tile4x4 10b not supported for o= verlay!\n"); + return AVERROR(EINVAL); + } + + s->main_has_alpha =3D ff_fmt_is_in(in_frames_ctx->sw_format, alpha_pix= _fmts); + + pix_desc =3D av_pix_fmt_desc_get(in_frames_ctx->sw_format); + + s->hsub =3D pix_desc->log2_chroma_w; + s->vsub =3D pix_desc->log2_chroma_h; + + return 0; +} + +static int activate(AVFilterContext *ctx) +{ + NetIntOverlayContext *s =3D ctx->priv; + return ff_framesync_activate(&s->fs); +} + +#define OFFSET(x) offsetof(NetIntOverlayContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption ni_overlay_options[] =3D { + { "x", "set the x expression", OFFSET(x_expr), AV_OPT_TYPE= _STRING, {.str =3D "0"}, CHAR_MIN, CHAR_MAX, FLAGS }, + { "y", "set the y expression", OFFSET(y_expr), AV_OPT_TYPE= _STRING, {.str =3D "0"}, CHAR_MIN, CHAR_MAX, FLAGS }, + { "alpha", "alpha format", OFFSET(alpha_format), AV_OPT_TYPE= _INT, {.i64 =3D 0}, 0, 1, FLAGS, "alpha_format" }, + { "straight", "", 0, AV_OPT_TYPE_CONST, {.i64 =3D 0}, .flags = =3D FLAGS, .unit =3D "alpha_format" }, + { "premultiplied", "", 0, AV_OPT_TYPE_CONST, {.i64 =3D 1}, .flags = =3D FLAGS, .unit =3D "alpha_format" }, + { "inplace", "overlay in-place", OFFSET(inplace), AV_OPT_TYPE= _BOOL, {.i64 =3D 0}, 0, 1, FLAGS }, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +// NOLINTNEXTLINE(clang-diagnostic-deprecated-declarations) +FRAMESYNC_DEFINE_CLASS(ni_overlay, NetIntOverlayContext, fs); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "main", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_input_main, + }, + { + .name =3D "overlay", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_input_overlay, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_output, + }, +}; + +FFFilter ff_vf_overlay_ni_quadra =3D { + .p.name =3D "ni_quadra_overlay", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra overlay a video source on top of the input v" NI_XC= ODER_REVISION), + .p.priv_class =3D &ni_overlay_class, + .priv_size =3D sizeof(NetIntOverlayContext), + .uninit =3D uninit, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), + .preinit =3D ni_overlay_framesync_preinit, + .activate =3D activate, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, +}; diff --git a/libavfilter/vf_pad_ni.c b/libavfilter/vf_pad_ni.c new file mode 100644 index 0000000000..f935377f84 --- /dev/null +++ b/libavfilter/vf_pad_ni.c @@ -0,0 +1,703 @@ +/* + * Copyright (c) 2007 Bobby Bingham + * Copyright (c) 2020 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * video padding filter + */ + +#include /* DBL_MAX */ + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "video.h" +#include "libavutil/avstring.h" +#include "libavutil/common.h" +#include "libavutil/eval.h" +#include "libavutil/pixdesc.h" +#include "libavutil/colorspace.h" +#include "libavutil/imgutils.h" +#include "libavutil/parseutils.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" +#include "drawutils.h" + +static const char *const var_names[] =3D { + "in_w", "iw", + "in_h", "ih", + "out_w", "ow", + "out_h", "oh", + "x", + "y", + "a", + "sar", + "dar", + "hsub", + "vsub", + NULL +}; + +enum var_name { + VAR_IN_W, VAR_IW, + VAR_IN_H, VAR_IH, + VAR_OUT_W, VAR_OW, + VAR_OUT_H, VAR_OH, + VAR_X, + VAR_Y, + VAR_A, + VAR_SAR, + VAR_DAR, + VAR_HSUB, + VAR_VSUB, + VARS_NB +}; + +typedef struct NetIntPadContext { + const AVClass *class; + int w, h; ///< output dimensions, a value of 0 will resu= lt in the input size + int x, y; ///< offsets of the input area with respect to= the padded area + int in_w, in_h; ///< width and height for the padded input vid= eo, which has to be aligned to the chroma values in order to avoid chroma i= ssues + int inlink_w, inlink_h; + AVRational aspect; + + char *w_expr; ///< width expression string + char *h_expr; ///< height expression string + char *x_expr; ///< width expression string + char *y_expr; ///< height expression string + uint8_t rgba_color[4]; ///< color for the padding area + FFDrawContext draw; + FFDrawColor color; + + AVBufferRef *out_frames_ref; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + + int initialized; + int session_opened; + int keep_alive_timeout; /* keep alive timeout setting */ + + int auto_skip; + int skip_filter; + int buffer_limit; +} NetIntPadContext; + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D + {AV_PIX_FMT_NI_QUAD, AV_PIX_FMT_NONE}; + AVFilterFormats *formats; + + formats =3D ff_make_format_list(pix_fmts); + + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntPadContext *s =3D ctx->priv; + + if (s->api_dst_frame.data.frame.p_buffer) + ni_frame_buffer_free(&s->api_dst_frame.data.frame); + + if (s->session_opened) { + /* Close operation will free the device frames */ + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + } + + av_buffer_unref(&s->out_frames_ref); +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntPadContext *s =3D ctx->priv; + AVHWFramesContext *out_frames_ctx; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_ctx =3D (AVHWFramesContext*)s->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + s->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&s->api_ctx, + out_frames_ctx->width, out_frames_ctx->h= eight, + out_frames_ctx->sw_format, + pool_size, + s->buffer_limit); +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx =3D inlink->dst; + NetIntPadContext *s =3D ctx->priv; + AVRational adjusted_aspect =3D s->aspect; + int ret; + double var_values[VARS_NB], res; + char *expr; + AVHWFramesContext *avhwctx; + + if (inlink->format =3D=3D AV_PIX_FMT_NI_QUAD) { + FilterLink *li =3D ff_filter_link(inlink); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + avhwctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (ff_draw_init(&s->draw, avhwctx->sw_format, 0) < 0) + return AVERROR(EINVAL); + } else { + if (ff_draw_init(&s->draw, inlink->format, 0) < 0) + return AVERROR(EINVAL); + } + + ff_draw_color(&s->draw, &s->color, s->rgba_color); + + var_values[VAR_IN_W] =3D var_values[VAR_IW] =3D inlink->w; + var_values[VAR_IN_H] =3D var_values[VAR_IH] =3D inlink->h; + var_values[VAR_OUT_W] =3D var_values[VAR_OW] =3D NAN; + var_values[VAR_OUT_H] =3D var_values[VAR_OH] =3D NAN; + var_values[VAR_A] =3D (double) inlink->w / inlink->h; + var_values[VAR_SAR] =3D inlink->sample_aspect_ratio.num ? + (double) inlink->sample_aspect_ratio.num / inlink->sample_aspect_r= atio.den : 1; + var_values[VAR_DAR] =3D var_values[VAR_A] * var_values[VAR_SAR]; + var_values[VAR_HSUB] =3D 1 << s->draw.hsub_max; + var_values[VAR_VSUB] =3D 1 << s->draw.vsub_max; + + /* evaluate width and height */ + av_expr_parse_and_eval(&res, s->w_expr, var_names, var_values, NULL, N= ULL, + NULL, NULL, NULL, 0, ctx); + s->w =3D (int)res; + var_values[VAR_OUT_W] =3D var_values[VAR_OW] =3D res; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->h_expr), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto eval_fail; + s->h =3D (int)res; + var_values[VAR_OUT_H] =3D var_values[VAR_OH] =3D res; + if (!s->h) + var_values[VAR_OUT_H] =3D var_values[VAR_OH] =3D s->h =3D inlink->= h; + + /* evaluate the width again, as it may depend on the evaluated output = height */ + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->w_expr), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto eval_fail; + s->w =3D (int)res; + var_values[VAR_OUT_W] =3D var_values[VAR_OW] =3D res; + if (!s->w) + var_values[VAR_OUT_W] =3D var_values[VAR_OW] =3D s->w =3D inlink->= w; + + if (adjusted_aspect.num && adjusted_aspect.den) { + adjusted_aspect =3D av_div_q(adjusted_aspect, inlink->sample_aspec= t_ratio); + if (s->h < av_rescale(s->w, adjusted_aspect.den, adjusted_aspect.n= um)) { + s->h =3D av_rescale(s->w, adjusted_aspect.den, adjusted_aspect= .num); + var_values[VAR_OUT_H] =3D var_values[VAR_OH] =3D (double)s->h; + } else { + s->w =3D av_rescale(s->h, adjusted_aspect.num, adjusted_aspect= .den); + var_values[VAR_OUT_W] =3D var_values[VAR_OW] =3D (double)s->w; + } + } + + /* evaluate x and y */ + av_expr_parse_and_eval(&res, s->x_expr, var_names, var_values, NULL, N= ULL, + NULL, NULL, NULL, 0, ctx); + s->x =3D (int)res; + var_values[VAR_X] =3D res; + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->y_expr), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto eval_fail; + s->y =3D (int)res; + var_values[VAR_Y] =3D res; + /* evaluate x again, as it may depend on the evaluated y value */ + if ((ret =3D av_expr_parse_and_eval(&res, (expr =3D s->x_expr), + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx= )) < 0) + goto eval_fail; + s->x =3D (int)res; + var_values[VAR_X] =3D res; + + if (s->x < 0 || s->x + inlink->w > s->w) { + var_values[VAR_X] =3D (double)(s->w - inlink->w) / 2.0; + s->x =3D (int)var_values[VAR_X]; + } + if (s->y < 0 || s->y + inlink->h > s->h) { + var_values[VAR_Y] =3D (double)(s->h - inlink->h) / 2.0; + s->y =3D (int)var_values[VAR_Y]; + } + + /* sanity check params */ + if (s->w < 0 || s->h < 0) { + av_log(ctx, AV_LOG_ERROR, "Negative values are not acceptable.\n"); + return AVERROR(EINVAL); + } + + s->w =3D ff_draw_round_to_sub(&s->draw, 0, -1, s->w); + s->h =3D ff_draw_round_to_sub(&s->draw, 1, -1, s->h); + s->x =3D ff_draw_round_to_sub(&s->draw, 0, -1, s->x); + s->y =3D ff_draw_round_to_sub(&s->draw, 1, -1, s->y); + s->in_w =3D ff_draw_round_to_sub(&s->draw, 0, -1, inlink->w); + s->in_h =3D ff_draw_round_to_sub(&s->draw, 1, -1, inlink->h); + s->inlink_w =3D inlink->w; + s->inlink_h =3D inlink->h; + + av_log(ctx, AV_LOG_VERBOSE, "w:%d h:%d -> w:%d h:%d x:%d y:%d color:0x= %02X%02X%02X%02X\n", + inlink->w, inlink->h, s->w, s->h, s->x, s->y, + s->rgba_color[0], s->rgba_color[1], s->rgba_color[2], s->rgba_c= olor[3]); + + if (s->x < 0 || s->y < 0 || + s->w <=3D 0 || s->h <=3D 0 || + (unsigned)s->x + (unsigned)inlink->w > s->w || + (unsigned)s->y + (unsigned)inlink->h > s->h) { + av_log(ctx, AV_LOG_ERROR, + "Input area %d:%d:%d:%d not within the padded area 0:0:%d:%= d or zero-sized\n", + s->x, s->y, s->x + inlink->w, s->y + inlink->h, s->w, s->h); + return AVERROR(EINVAL); + } + + if (s->w > NI_MAX_RESOLUTION_WIDTH || s->h > NI_MAX_RESOLUTION_HEIGHT)= { + av_log(ctx, AV_LOG_ERROR, "Padded value (%dx%d) > 8192, not allowe= d\n", s->w, s->h); + return AVERROR(EINVAL); + } + + return 0; + +eval_fail: + av_log(ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s'\n", expr); + return ret; +} + +static int config_output(AVFilterLink *outlink) +{ + NetIntPadContext *s =3D outlink->src->priv; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + AVFilterContext *ctx; + + outlink->w =3D s->w; + outlink->h =3D s->h; + + ctx =3D (AVFilterContext *)outlink->src; + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_YUYV422 || + in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_UYVY422) { + av_log(ctx, AV_LOG_ERROR, "yuyv/uyvy not supported\n"); + return AVERROR(EINVAL); + } + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X4 || + in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 not supported\n"); + return AVERROR(EINVAL); + } + + //skip the color range check + if (s->auto_skip && + s->w =3D=3D in_frames_ctx->width && + s->h =3D=3D in_frames_ctx->height && + s->x =3D=3D 0 && + s->y =3D=3D 0 + ) { + //skip hardware pad + s->skip_filter =3D 1; + + FilterLink *lt =3D ff_filter_link(outlink->src->inputs[0]); + s->out_frames_ref =3D av_buffer_ref(lt->hw_frames_ctx); + if (!s->out_frames_ref) { + return AVERROR(ENOMEM); + } + FilterLink *lo =3D ff_filter_link(outlink); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + + return 0; + } + + s->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_ref); + if (!s->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)s->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D s->w; + out_frames_ctx->height =3D s->h; + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + out_frames_ctx->initial_pool_size =3D + NI_PAD_ID; // Repurposed as identity code + + av_hwframe_ctx_init(s->out_frames_ref); + + FilterLink *lo =3D ff_filter_link(outlink); + av_buffer_unref(&lo->hw_frames_ctx); + + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + NetIntPadContext *s =3D inlink->dst->priv; + AVFilterLink *outlink =3D inlink->dst->outputs[0]; + AVFrame *out =3D NULL; + niFrameSurface1_t* frame_surface,*new_frame_surface; + AVHWFramesContext *pAVHFWCtx; + AVNIDeviceContext *pAVNIDevCtx; + ni_retcode_t retcode; + uint32_t ui32RgbaColor, scaler_format; + uint16_t tempFID; + int cardno; + + frame_surface =3D (niFrameSurface1_t *) in->data[3]; + if (frame_surface =3D=3D NULL) { + return AVERROR(EINVAL); + } + + pAVHFWCtx =3D (AVHWFramesContext *)in->hw_frames_ctx->data; + pAVNIDevCtx =3D (AVNIDeviceContext *)pAVHFWCtx->device_ctx->hwct= x; + cardno =3D ni_get_cardno(in); + + if (s->skip_filter) { + //skip hardware pad + return ff_filter_frame(inlink->dst->outputs[0], in); + } + + if (!s->initialized) { + retcode =3D ni_device_session_context_init(&s->api_ctx); + if (retcode < 0) { + av_log(inlink->dst, AV_LOG_ERROR, + "ni pad filter session context init failure\n"); + goto fail; + } + + s->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + s->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + + s->api_ctx.hw_id =3D cardno; + s->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + s->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_PAD; + s->api_ctx.keep_alive_timeout =3D s->keep_alive_timeout; + + retcode =3D ni_device_session_open(&s->api_ctx, NI_DEVICE_TYPE_SCA= LER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(inlink->dst, AV_LOG_ERROR, + "Can't open device session on card %d\n", cardno); + + /* Close operation will free the device frames */ + ni_device_session_close(&s->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&s->api_ctx); + goto fail; + } + + s->session_opened =3D 1; + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + inlink->dst->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SI= ZE > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + + retcode =3D init_out_pool(inlink->dst); + if (retcode < 0) { + av_log(inlink->dst, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", retcod= e); + goto fail; + } + + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)s->out_= frames_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_frames_= ctx->hwctx; + ni_cpy_hwframe_ctx(pAVHFWCtx, out_frames_ctx); + ni_device_session_copy(&s->api_ctx, &out_ni_ctx->api_ctx); + + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(pAVHFWCtx->= sw_format); + + if ((in->color_range =3D=3D AVCOL_RANGE_JPEG) && !(desc->flags & A= V_PIX_FMT_FLAG_RGB)) { + av_log(inlink->dst, AV_LOG_WARNING, + "WARNING: Full color range input, limited color range o= utput\n"); + } + + s->initialized =3D 1; + } + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + + retcode =3D ni_frame_buffer_alloc_hwenc(&s->api_dst_frame.data.frame, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + av_log(inlink->dst, AV_LOG_DEBUG, + "inlink->w =3D %d;inlink->h =3D %d;outlink->w =3D %d;outlink->h= =3D %d\n", + inlink->w, inlink->h, outlink->w, outlink->h); + av_log(inlink->dst, AV_LOG_DEBUG, + "s->w=3D%d;s->h=3D%d;s->x=3D%d;s->y=3D%d;c=3D%02x:%02x:%02x:%02= x\n", s->w, + s->h, s->x, s->y, s->rgba_color[0], s->rgba_color[1], + s->rgba_color[2], s->rgba_color[3]); + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + /* + * Allocate device input frame. This call won't actually allocate a fr= ame, + * but sends the incoming hardware frame index to the scaler manager + */ + retcode =3D ni_device_alloc_frame(&s->api_ctx, + FFALIGN(in->width, 2), + FFALIGN(in->height, 2), + scaler_format, + 0, // input frame + in->width, // src rectangle width + in->height, // src rectangle height + 0, // src rectangle x =3D 0 + 0, // src rectangle y =3D 0 + frame_surface->ui32nodeAddress, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(inlink->dst, AV_LOG_DEBUG, "Can't allocate device input fra= me %d\n", retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Scaler uses BGRA color, or ARGB in little-endian */ + ui32RgbaColor =3D (s->rgba_color[3] << 24) | (s->rgba_color[0] << 16) | + (s->rgba_color[1] << 8) | s->rgba_color[2]; + + /* Allocate device destination frame. This will acquire a frame from t= he pool */ + retcode =3D ni_device_alloc_frame(&s->api_ctx, + FFALIGN(outlink->w,2), + FFALIGN(outlink->h,2), + scaler_format, + NI_SCALER_FLAG_IO, // output frame + in->width, // dst rectangle width + in->height, // dst rectangle height + s->x, // dst rectangle x + s->y, // dst rectangle y + ui32RgbaColor, // rgba color + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(inlink->dst, AV_LOG_DEBUG, + "Can't allocate device output frame %d\n", retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + out =3D av_frame_alloc(); + if (!out) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + av_frame_copy_props(out,in); + + out->width =3D s->w; + out->height =3D s->h; + + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref); + + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + + if (!out->data[3]) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Copy the frame surface from the incoming frame */ + memcpy(out->data[3], in->data[3], sizeof(niFrameSurface1_t)); + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc(&s->api_ctx, &s->api_dst_fra= me, + NI_DEVICE_TYPE_SCALER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(inlink->dst, AV_LOG_ERROR, + "Can't acquire output frame %d\n",retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_pad"); +#endif + + tempFID =3D frame_surface->ui16FrameIdx; + frame_surface =3D (niFrameSurface1_t *) out->data[3]; + new_frame_surface =3D (niFrameSurface1_t *) s->api_dst_frame.data.fram= e.p_data[3]; + frame_surface->ui16FrameIdx =3D new_frame_surface->ui16FrameIdx; + frame_surface->ui16session_ID =3D new_frame_surface->ui16session_ID; + frame_surface->device_handle =3D new_frame_surface->device_handle; + frame_surface->output_idx =3D new_frame_surface->output_idx; + frame_surface->src_cpu =3D new_frame_surface->src_cpu; + frame_surface->dma_buf_fd =3D 0; + + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + pAVHFWCtx->sw_format); + + /* Remove ni-split specific assets */ + frame_surface->ui32nodeAddress =3D 0; + + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + av_log(inlink->dst, AV_LOG_DEBUG, + "vf_pad_ni.c:IN trace ui16FrameIdx =3D [%d] --> out [%d] \n", t= empFID, + frame_surface->ui16FrameIdx); + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurface1_= t), ff_ni_frame_free, NULL, 0); + + av_frame_free(&in); + + return ff_filter_frame(inlink->dst->outputs[0], out); + +fail: + av_frame_free(&in); + if (out) + av_frame_free(&out); + return retcode; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntPadContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %u available_frame %d outlink framequeue %u frame_wanted %d - r= eturn NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntPadContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption ni_pad_options[] =3D { + { "width", "set the pad area width expression", = OFFSET(w_expr), AV_OPT_TYPE_STRING, {.str =3D "iw"}, CHAR_MIN, CHAR= _MAX, FLAGS }, + { "w", "set the pad area width expression", = OFFSET(w_expr), AV_OPT_TYPE_STRING, {.str =3D "iw"}, CHAR_MIN, CHAR= _MAX, FLAGS }, + { "height", "set the pad area height expression", = OFFSET(h_expr), AV_OPT_TYPE_STRING, {.str =3D "ih"}, CHAR_MIN, CHAR= _MAX, FLAGS }, + { "h", "set the pad area height expression", = OFFSET(h_expr), AV_OPT_TYPE_STRING, {.str =3D "ih"}, CHAR_MIN, CHAR= _MAX, FLAGS }, + { "x", "set the x offset expression for the input image position"= , OFFSET(x_expr), AV_OPT_TYPE_STRING, {.str =3D "0"}, CHAR_MIN, CHAR= _MAX, FLAGS }, + { "y", "set the y offset expression for the input image position"= , OFFSET(y_expr), AV_OPT_TYPE_STRING, {.str =3D "0"}, CHAR_MIN, CHAR= _MAX, FLAGS }, + { "color", "set the color of the padded area border", = OFFSET(rgba_color), AV_OPT_TYPE_COLOR, {.str =3D "black"}, .flags =3D = FLAGS }, + { "aspect", "pad to fit an aspect instead of a resolution", = OFFSET(aspect), AV_OPT_TYPE_RATIONAL, {.dbl =3D 0}, 0, DBL_MAX, FLAGS= }, + NI_FILT_OPTION_AUTO_SKIP, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_pad); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + .config_props =3D config_input, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_output, + }, +}; + +FFFilter ff_vf_pad_ni_quadra =3D { + .p.name =3D "ni_quadra_pad", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra pad the input video v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_pad_class, + .priv_size =3D sizeof(NetIntPadContext), + .uninit =3D uninit, + .activate =3D activate, + .flags_internal=3D FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +}; diff --git a/libavfilter/vf_rotate_ni.c b/libavfilter/vf_rotate_ni.c new file mode 100644 index 0000000000..3ce394a131 --- /dev/null +++ b/libavfilter/vf_rotate_ni.c @@ -0,0 +1,764 @@ +/* + * Copyright (c) 2013 Stefano Sabatini + * Copyright (c) 2008 Vitor Sessak + * Copyright (c) 2022 NETINT Technologies Inc. + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * rotation filter, based on the FFmpeg rotate filter +*/ + +#include + +#include "libavutil/eval.h" +#include "libavutil/parseutils.h" +#include "libavutil/opt.h" + +#include "nifilter.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "filters.h" +#include "libavutil/avstring.h" + +#define BUFFER_WIDTH_PIXEL_ALIGNMENT 16 + +static const char * const var_names[] =3D { + "in_w" , "iw", ///< width of the input video + "in_h" , "ih", ///< height of the input video + "out_w", "ow", ///< width of the input video + "out_h", "oh", ///< height of the input video + "hsub", "vsub", + NULL +}; + +enum var_name { + VAR_IN_W , VAR_IW, + VAR_IN_H , VAR_IH, + VAR_OUT_W, VAR_OW, + VAR_OUT_H, VAR_OH, + VAR_HSUB, VAR_VSUB, + VAR_VARS_NB +}; + +typedef struct NetIntRotContext { + const AVClass *class; + + char *angle_expr_str; + AVExpr *angle_expr; + + char *outw_expr_str, *outh_expr_str; + int outw, outh; + + char *fillcolor_str; + uint8_t fillcolor[4]; + bool fillcolor_enable; + + int hsub, vsub; + + double var_values[VAR_VARS_NB]; + + AVBufferRef *out_frames_ref; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + + ni_frame_config_t output_frame_config; + + bool initialized; + bool session_opened; + int64_t keep_alive_timeout; + + int auto_skip; + int skip_filter; + int buffer_limit; +} NetIntRotContext; + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D { AV_PIX_FMT_NI_QUAD, A= V_PIX_FMT_NONE }; + AVFilterFormats *fmts_list =3D NULL; + + fmts_list =3D ff_make_format_list(pix_fmts); + if (!fmts_list) { + return AVERROR(ENOMEM); + } + + return ff_set_common_formats(ctx, fmts_list); +} + +static av_cold int init(AVFilterContext *ctx) +{ + NetIntRotContext *rot =3D ctx->priv; + + if (!strcmp(rot->fillcolor_str, "none")) { + rot->fillcolor_enable =3D false; + } else if (av_parse_color(rot->fillcolor, rot->fillcolor_str, -1, ctx)= >=3D 0) { + rot->fillcolor_enable =3D true; + } else { + return AVERROR(EINVAL); + } + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntRotContext *rot =3D ctx->priv; + + av_expr_free(rot->angle_expr); + rot->angle_expr =3D NULL; + + if (rot->api_dst_frame.data.frame.p_buffer) { + ni_frame_buffer_free(&rot->api_dst_frame.data.frame); + } + + if (rot->session_opened) { + /* Close operation will free the device frames */ + ni_device_session_close(&rot->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&rot->api_ctx); + } + + av_buffer_unref(&rot->out_frames_ref); + +} + +static double get_rotated_w(void *opaque, double angle) +{ + NetIntRotContext *rot =3D opaque; + double inw =3D rot->var_values[VAR_IN_W]; + double inh =3D rot->var_values[VAR_IN_H]; + float sinx =3D (float)sin(angle); + float cosx =3D (float)cos(angle); + + return FFMAX(0, inh * sinx) + FFMAX(0, -inw * cosx) + + FFMAX(0, inw * cosx) + FFMAX(0, -inh * sinx); +} + +static double get_rotated_h(void *opaque, double angle) +{ + NetIntRotContext *rot =3D opaque; + double inw =3D rot->var_values[VAR_IN_W]; + double inh =3D rot->var_values[VAR_IN_H]; + float sinx =3D (float)sin(angle); + float cosx =3D (float)cos(angle); + + return FFMAX(0, -inh * cosx) + FFMAX(0, -inw * sinx) + + FFMAX(0, inh * cosx) + FFMAX(0, inw * sinx); +} + +static double (* const func1[])(void *, double) =3D { + get_rotated_w, + get_rotated_h, + NULL +}; + +static const char * const func1_names[] =3D { + "rotw", + "roth", + NULL +}; + +static int config_props(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + NetIntRotContext *rot =3D ctx->priv; + AVFilterLink *inlink =3D ctx->inputs[0]; + AVHWFramesContext *in_frames_ctx, *out_frames_ctx; + const AVPixFmtDescriptor *pixdesc =3D av_pix_fmt_desc_get(inlink->form= at); + int ret; + double res; + char *expr; + + rot->hsub =3D pixdesc->log2_chroma_w; + rot->vsub =3D pixdesc->log2_chroma_h; + + rot->var_values[VAR_IN_W] =3D rot->var_values[VAR_IW] =3D inlink->w; + rot->var_values[VAR_IN_H] =3D rot->var_values[VAR_IH] =3D inlink->h; + rot->var_values[VAR_HSUB] =3D 1<hsub; + rot->var_values[VAR_VSUB] =3D 1<vsub; + rot->var_values[VAR_OUT_W] =3D rot->var_values[VAR_OW] =3D NAN; + rot->var_values[VAR_OUT_H] =3D rot->var_values[VAR_OH] =3D NAN; + + av_expr_free(rot->angle_expr); + rot->angle_expr =3D NULL; + ret =3D av_expr_parse(&rot->angle_expr, + // NOLINTNEXTLINE(clang-analyzer-deadcode.DeadStor= es) + expr =3D rot->angle_expr_str, + var_names, + func1_names, + func1, + NULL, + NULL, + 0, + ctx); + if (ret < 0) { + av_log(ctx, + AV_LOG_ERROR, + "Error occurred parsing angle expression '%s'\n", + rot->angle_expr_str); + return ret; + } + +#define SET_SIZE_EXPR(name, opt_name) do { = \ + ret =3D av_expr_parse_and_eval(&res, expr =3D rot->name##_expr_str, = \ + var_names, rot->var_values, = \ + func1_names, func1, NULL, NULL, rot, 0, c= tx); \ + if (ret < 0 || isnan(res) || isinf(res) || res <=3D 0) { = \ + av_log(ctx, AV_LOG_ERROR, = \ + "Error parsing or evaluating expression for option %s: " = \ + "invalid expression '%s' or non-positive or indefinite valu= e %f\n", \ + opt_name, expr, res); = \ + return ret; = \ + } = \ +} while (0) + + /* evaluate width and height */ + av_expr_parse_and_eval(&res, + // NOLINTNEXTLINE(clang-analyzer-deadcode.DeadS= tores) + expr =3D rot->outw_expr_str, + var_names, + rot->var_values, + func1_names, + func1, + NULL, + NULL, + rot, + 0, + ctx); + rot->var_values[VAR_OUT_W] =3D rot->var_values[VAR_OW] =3D res; + // NOLINTNEXTLINE(bugprone-incorrect-roundings, bugprone-narrowing-con= versions) + rot->outw =3D ceil(res); + + SET_SIZE_EXPR(outh, "out_h"); + rot->var_values[VAR_OUT_H] =3D rot->var_values[VAR_OH] =3D ceil(res); + + // NOLINTNEXTLINE(bugprone-incorrect-roundings, bugprone-narrowing-con= versions) + rot->outh =3D ceil(res); + + /* evaluate the width again, as it may depend on the evaluated output = height */ + SET_SIZE_EXPR(outw, "out_w"); + rot->var_values[VAR_OUT_W] =3D rot->var_values[VAR_OW] =3D ceil(res); + // NOLINTNEXTLINE(bugprone-incorrect-roundings, bugprone-narrowing-con= versions) + rot->outw =3D ceil(res); + + // Quadra 2D engine only supports even pixel widths and heights + rot->outw =3D FFALIGN(rot->outw, 2); + rot->outh =3D FFALIGN(rot->outh, 2); + + outlink->w =3D rot->outw; + outlink->h =3D rot->outh; + + if (outlink->w > NI_MAX_RESOLUTION_WIDTH || + outlink->h > NI_MAX_RESOLUTION_HEIGHT) { + av_log(ctx, AV_LOG_ERROR, "Resolution %dx%d > %dx%d is not allowed= \n", + outlink->w, outlink->h, + NI_MAX_RESOLUTION_WIDTH, NI_MAX_RESOLUTION_HEIGHT); + return AVERROR(EINVAL); + } + + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *) li->hw_frames_ctx->data; + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X4 || + in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 not supported\n"); + return AVERROR(EINVAL); + } + + av_log(ctx, AV_LOG_VERBOSE, + "w:%d h:%d fmt:%s sar:%d/%d -> w:%d h:%d fmt:%s sar:%d/%d\n", + inlink->w, inlink->h, av_get_pix_fmt_name(inlink->format), + inlink->sample_aspect_ratio.num, inlink->sample_aspect_ratio.de= n, + outlink->w, outlink->h, av_get_pix_fmt_name(outlink->format), + outlink->sample_aspect_ratio.num, outlink->sample_aspect_ratio.= den); + + //skip the color range check + if (rot->auto_skip && + av_expr_eval(rot->angle_expr, rot->var_values, rot) =3D=3D 0 && + in_frames_ctx->width =3D=3D outlink->w && + in_frames_ctx->height =3D=3D outlink->h + ) { + //skip hardware rotate + rot->skip_filter =3D 1; + + FilterLink *lt =3D ff_filter_link(outlink->src->inputs[0]); + rot->out_frames_ref =3D av_buffer_ref(lt->hw_frames_ctx); + if (!rot->out_frames_ref) { + return AVERROR(ENOMEM); + } + FilterLink *lo =3D ff_filter_link(outlink); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(rot->out_frames_ref); + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + return 0; + } + + rot->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_ref= ); + if (!rot->out_frames_ref) { + return AVERROR(ENOMEM); + } + + out_frames_ctx =3D (AVHWFramesContext *) rot->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D rot->outw; + out_frames_ctx->height =3D rot->outh; + out_frames_ctx->sw_format =3D in_frames_ctx->sw_format; + out_frames_ctx->initial_pool_size =3D NI_ROTATE_ID; // Repurposed as i= dentity code + + av_hwframe_ctx_init(rot->out_frames_ref); + + FilterLink *lo =3D ff_filter_link(ctx->outputs[0]); + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(rot->out_frames_ref); + + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + + return 0; +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntRotContext *rot =3D ctx->priv; + AVHWFramesContext *out_frames_context; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_context =3D (AVHWFramesContext*)rot->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + rot->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&rot->api_ctx, + out_frames_context->width, + out_frames_context->height, + out_frames_context->sw_format, + pool_size, + rot->buffer_limit); +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx =3D inlink->dst; + AVFilterLink *outlink =3D inlink->dst->outputs[0]; + AVFrame *out =3D NULL; + NetIntRotContext *rot =3D ctx->priv; + AVBufferRef *out_buffer_ref =3D rot->out_frames_ref; + AVHWFramesContext *in_frames_context =3D (AVHWFramesContext *) in->hw_= frames_ctx->data; + AVNIDeviceContext *av_ni_device_context =3D (AVNIDeviceContext *) in_f= rames_context->device_ctx->hwctx; + ni_retcode_t ni_retcode =3D NI_RETCODE_SUCCESS; + niFrameSurface1_t *frame_surface =3D (niFrameSurface1_t *) in->data[3]= , *frame_surface2 =3D NULL; + ni_frame_config_t input_frame_config =3D {0}; + uint32_t scaler_format; + int retcode =3D 0, rgba_color =3D 255 /* black opaque */, card_number = =3D ni_get_cardno(in); + int aligned_picture_width, rotated_picture_width, rotated_picture_heig= ht; + double angle; + + if (!frame_surface) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter frame_surface should n= ot be NULL\n"); + return AVERROR(EINVAL); + } + + //skip hardware rotate + if (rot->skip_filter) { + return ff_filter_frame(outlink, in); + } + + if (!rot->initialized) { + ni_retcode =3D ni_device_session_context_init(&rot->api_ctx); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter session context in= it failed with %d\n", ni_retcode); + retcode =3D AVERROR(EINVAL); + goto FAIL; + } + + rot->api_ctx.device_handle =3D rot->api_ctx.blk_io_handle =3D av_n= i_device_context->cards[card_number]; + + rot->api_ctx.hw_id =3D card_number; + rot->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + rot->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_ROTATE; + rot->api_ctx.keep_alive_timeout =3D rot->keep_alive_timeout; + + ni_retcode =3D ni_device_session_open(&rot->api_ctx, NI_DEVICE_TYP= E_SCALER); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter device session ope= n failed with %d\n", ni_retcode); + retcode =3D ni_retcode; + + /* Close operation will free the device frames */ + ni_device_session_close(&rot->api_ctx, 1, NI_DEVICE_TYPE_SCALE= R); + ni_device_session_context_clear(&rot->api_ctx); + goto FAIL; + } + + rot->session_opened =3D true; + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + inlink->dst->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SI= ZE > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + + ni_retcode =3D init_out_pool(inlink->dst); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter init out pool fail= ed with %d\n", ni_retcode); + goto FAIL; + } + + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)out_buf= fer_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_frames_= ctx->hwctx; + ni_cpy_hwframe_ctx(in_frames_context, out_frames_ctx); + ni_device_session_copy(&rot->api_ctx, &out_ni_ctx->api_ctx); + + AVHWFramesContext *pAVHFWCtx =3D (AVHWFramesContext *) in->hw_fram= es_ctx->data; + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(pAVHFWCtx->= sw_format); + + if ((in->color_range =3D=3D AVCOL_RANGE_JPEG) && !(desc->flags & A= V_PIX_FMT_FLAG_RGB)) { + av_log(ctx, AV_LOG_WARNING, "Full color range input, limited c= olor output\n"); + } + + rot->initialized =3D true; + } + + ni_retcode =3D ni_frame_buffer_alloc_hwenc(&rot->api_dst_frame.data.fr= ame, + outlink->w, + outlink->h, + 0); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter frame buffer alloc hwe= nc failed with %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + // Input. + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(in_frames_context->sw_= format); + input_frame_config.picture_format =3D scaler_format; + + input_frame_config.rgba_color =3D frame_surface->ui32nodeAddress; + input_frame_config.frame_index =3D frame_surface->ui16FrameIdx; + + aligned_picture_width =3D FFALIGN(in->width, BUFFER_WIDTH_PIXEL_ALIGNM= ENT); + + angle =3D av_expr_eval(rot->angle_expr, rot->var_values, rot); + if (angle =3D=3D 0.0) { + // input_frame_config.orientation =3D 0; // initialized to zero, u= nnecessary assignment + input_frame_config.picture_width =3D in->width; + input_frame_config.picture_height =3D in->height; + + input_frame_config.rectangle_width =3D FFMIN(outlink->w, in->width= ); + input_frame_config.rectangle_height =3D FFMIN(outlink->h, in->heig= ht); + + rotated_picture_width =3D in->width; + rotated_picture_height =3D in->height; + } else if ((angle =3D=3D -M_PI_2 * 3.0) || (angle =3D=3D M_PI_2)) { + // -270.0=C2=B0 || 90.0=C2=B0 + input_frame_config.orientation =3D 1; + input_frame_config.picture_width =3D aligned_picture_width; + input_frame_config.picture_height =3D in->height; + + input_frame_config.rectangle_width =3D FFMIN(outlink->w, in->heigh= t); + input_frame_config.rectangle_height =3D FFMIN(outlink->h, in->widt= h); + + rotated_picture_width =3D in->height; + rotated_picture_height =3D aligned_picture_width; + } else if ((angle =3D=3D -M_PI) || (angle =3D=3D M_PI)) { + // -180.0=C2=B0 || 180.0=C2=B0 + input_frame_config.orientation =3D 2; + input_frame_config.picture_width =3D aligned_picture_width; + input_frame_config.picture_height =3D in->height; + + input_frame_config.rectangle_width =3D FFMIN(outlink->w, in->width= ); + input_frame_config.rectangle_height =3D FFMIN(outlink->h, in->heig= ht); + + rotated_picture_width =3D aligned_picture_width; + rotated_picture_height =3D in->height; + } else if ((angle =3D=3D -M_PI_2) || (angle =3D=3D M_PI_2 * 3.0)) { + // -90.0=C2=B0 || 270.0=C2=B0 + input_frame_config.orientation =3D 3; + input_frame_config.picture_width =3D aligned_picture_width; + input_frame_config.picture_height =3D in->height; + + input_frame_config.rectangle_width =3D FFMIN(outlink->w, in->heigh= t); + input_frame_config.rectangle_height =3D FFMIN(outlink->h, in->widt= h); + + rotated_picture_width =3D in->height; + rotated_picture_height =3D aligned_picture_width; + } else { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter does not support rotat= ion of %.1f radians\n", angle); + retcode =3D AVERROR(EINVAL); + goto FAIL; + } + + input_frame_config.rectangle_x =3D + (rotated_picture_width > input_frame_config.rectangle_width) ? + (rotated_picture_width / 2) - (input_frame_config.rectangle_width = / 2) : 0; + input_frame_config.rectangle_y =3D + (rotated_picture_height > input_frame_config.rectangle_height) ? + (rotated_picture_height / 2) - (input_frame_config.rectangle_heigh= t / 2) : 0; + if (aligned_picture_width - in->width) { + switch (input_frame_config.orientation) { + case 1: // 90=C2=B0 + input_frame_config.rectangle_y =3D + (in->width > input_frame_config.rectangle_height) ? + (in->width / 2) - (input_frame_config.rectangle_height / 2= ) : 0; + break; + case 2: // 180=C2=B0 + input_frame_config.rectangle_x =3D + aligned_picture_width - in->width + + ((in->width > input_frame_config.rectangle_width) ? + (in->width / 2) - (input_frame_config.rectangle_width / 2= ) : 0); + break; + case 3: // 270=C2=B0 + input_frame_config.rectangle_y =3D + aligned_picture_width - in->width + + ((in->width > input_frame_config.rectangle_height) ? + (in->width / 2) - (input_frame_config.rectangle_height / = 2) : 0); + break; + default: + break; + } + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + // use ni_device_config_frame() instead of ni_device_alloc_frame() + // such that input_frame_config's orientation can be configured + ni_retcode =3D ni_device_config_frame(&rot->api_ctx, &input_frame_conf= ig); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter device config input fr= ame failed with %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + // Output. + + if (rot->fillcolor_enable) { + rgba_color =3D (rot->fillcolor[3] << 24) | + (rot->fillcolor[0] << 16) | + (rot->fillcolor[1] << 8) | + rot->fillcolor[2]; + } + + rot->output_frame_config.picture_width =3D outlink->w; + rot->output_frame_config.picture_height =3D outlink->h; + rot->output_frame_config.rectangle_width =3D input_frame_config.rectan= gle_width; + rot->output_frame_config.rectangle_height =3D input_frame_config.recta= ngle_height; + rot->output_frame_config.rectangle_x =3D + (rot->output_frame_config.picture_width > rot->output_frame_config= .rectangle_width) ? + (rot->output_frame_config.picture_width / 2) - (rot->output_frame_= config.rectangle_width / 2) : 0; + rot->output_frame_config.rectangle_y =3D + (rot->output_frame_config.picture_height > rot->output_frame_confi= g.rectangle_height) ? + (rot->output_frame_config.picture_height / 2) - (rot->output_frame= _config.rectangle_height / 2) : 0; + rot->output_frame_config.rgba_color =3D rgba_color; + + ni_retcode =3D ni_device_alloc_frame(&rot->api_ctx, + rot->output_frame_config.picture_wi= dth, + rot->output_frame_config.picture_he= ight, + scaler_format, + NI_SCALER_FLAG_IO, + rot->output_frame_config.rectangle_= width, + rot->output_frame_config.rectangle_= height, + rot->output_frame_config.rectangle_= x, + rot->output_frame_config.rectangle_= y, + rot->output_frame_config.rgba_color, + -1, + NI_DEVICE_TYPE_SCALER); + + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter device alloc output fr= ame failed with %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + out =3D av_frame_alloc(); + if (!out) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter av_frame_alloc returne= d NULL\n"); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + av_frame_copy_props(out, in); + + out->width =3D rot->outw; + out->height =3D rot->outh; + out->format =3D AV_PIX_FMT_NI_QUAD; + out->color_range =3D AVCOL_RANGE_MPEG; + + out->hw_frames_ctx =3D av_buffer_ref(out_buffer_ref); + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + if (!out->data[3]) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter av_malloc returned NUL= L\n"); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + memcpy(out->data[3], frame_surface, sizeof(niFrameSurface1_t)); + + ni_retcode =3D ni_device_session_read_hwdesc(&rot->api_ctx, + &rot->api_dst_frame, + NI_DEVICE_TYPE_SCALER); + if (ni_retcode !=3D NI_RETCODE_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter read hwdesc failed wit= h %d\n", ni_retcode); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_rotate"); +#endif + + frame_surface2 =3D (niFrameSurface1_t *) rot->api_dst_frame.data.frame= .p_data[3]; + + frame_surface =3D (niFrameSurface1_t *) out->data[3]; + frame_surface->ui16FrameIdx =3D frame_surface2->ui16FrameIdx; + frame_surface->ui16session_ID =3D frame_surface2->ui16session_ID; + frame_surface->device_handle =3D frame_surface2->device_handle; + frame_surface->output_idx =3D frame_surface2->output_idx; + frame_surface->src_cpu =3D frame_surface2->src_cpu; + frame_surface->ui32nodeAddress =3D 0; + frame_surface->dma_buf_fd =3D 0; + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + in_frames_context->sw_format); + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + out->buf[0] =3D av_buffer_create(out->data[3], + sizeof(niFrameSurface1_t), + ff_ni_frame_free, + NULL, + 0); + if (!out->buf[0]) { + av_log(ctx, AV_LOG_ERROR, "ni rotate filter av_buffer_create retur= ned NULL\n"); + retcode =3D AVERROR(ENOMEM); + goto FAIL; + } + + av_frame_free(&in); + return ff_filter_frame(inlink->dst->outputs[0], out); + +FAIL: + av_frame_free(&in); + if (out) + av_frame_free(&out); + return retcode; +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntRotContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %u available_frame %d outlink framequeue %u frame_wanted %d - r= eturn NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntRotContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM) + +static const AVOption ni_rotate_options[] =3D { + { "angle", "set angle (in radians)", OFFSET(angle_expr_str),= AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "a", "set angle (in radians)", OFFSET(angle_expr_str),= AV_OPT_TYPE_STRING, {.str=3D"0"}, 0, 0, FLAGS }, + { "out_w", "set output width expression", OFFSET(outw_expr_str), = AV_OPT_TYPE_STRING, {.str=3D"iw"}, 0, 0, FLAGS }, + { "ow", "set output width expression", OFFSET(outw_expr_str), = AV_OPT_TYPE_STRING, {.str=3D"iw"}, 0, 0, FLAGS }, + { "out_h", "set output height expression", OFFSET(outh_expr_str), = AV_OPT_TYPE_STRING, {.str=3D"ih"}, 0, 0, FLAGS }, + { "oh", "set output height expression", OFFSET(outh_expr_str), = AV_OPT_TYPE_STRING, {.str=3D"ih"}, 0, 0, FLAGS }, + { "fillcolor", "set background fill color", OFFSET(fillcolor_str), = AV_OPT_TYPE_STRING, {.str=3D"black"}, 0, 0, FLAGS }, + { "c", "set background fill color", OFFSET(fillcolor_str), = AV_OPT_TYPE_STRING, {.str=3D"black"}, 0, 0, FLAGS }, + NI_FILT_OPTION_AUTO_SKIP, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_rotate); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_props, + }, +}; + +FFFilter ff_vf_rotate_ni_quadra =3D { + .p.name =3D "ni_quadra_rotate", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra rotate the input video v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_rotate_class, + .priv_size =3D sizeof(NetIntRotContext), + .init =3D init, + .uninit =3D uninit, + .activate =3D activate, + FILTER_QUERY_FUNC(query_formats), + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, +}; diff --git a/libavfilter/vf_scale_ni.c b/libavfilter/vf_scale_ni.c new file mode 100644 index 0000000000..1388abf590 --- /dev/null +++ b/libavfilter/vf_scale_ni.c @@ -0,0 +1,958 @@ +/* + * Copyright (c) 2007 Bobby Bingham + * Copyright (c) 2020 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * scale video filter + */ + +#include +#include + +#include "nifilter.h" +#include "filters.h" +#include "formats.h" +#include "libavutil/mem.h" +#include "fftools/ffmpeg_sched.h" +#include "scale_eval.h" +#include "video.h" +#include "libavutil/avstring.h" +#include "libavutil/internal.h" +#include "libavutil/mathematics.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "libavutil/pixdesc.h" +#include "libavutil/imgutils.h" +#include "libavutil/avassert.h" +#include "libavutil/eval.h" +#include "libswscale/swscale.h" + +static const char *const var_names[] =3D { + "in_w", "iw", + "in_h", "ih", + "out_w", "ow", + "out_h", "oh", + "a", + "sar", + "dar", + "hsub", + "vsub", + "ohsub", + "ovsub", + "main_w", + "main_h", + "main_a", + "main_sar", + "main_dar", "mdar", + "main_hsub", + "main_vsub", + NULL +}; + +enum var_name { + VAR_IN_W, VAR_IW, + VAR_IN_H, VAR_IH, + VAR_OUT_W, VAR_OW, + VAR_OUT_H, VAR_OH, + VAR_A, + VAR_SAR, + VAR_DAR, + VAR_HSUB, + VAR_VSUB, + VAR_OHSUB, + VAR_OVSUB, + VARS_NB +}; + +enum OutputFormat { + OUTPUT_FORMAT_YUV420P, + OUTPUT_FORMAT_YUYV422, + OUTPUT_FORMAT_UYVY422, + OUTPUT_FORMAT_NV12, + OUTPUT_FORMAT_ARGB, + OUTPUT_FORMAT_RGBA, + OUTPUT_FORMAT_ABGR, + OUTPUT_FORMAT_BGRA, + OUTPUT_FORMAT_YUV420P10LE, + OUTPUT_FORMAT_NV16, + OUTPUT_FORMAT_BGR0, + OUTPUT_FORMAT_P010LE, + OUTPUT_FORMAT_AUTO, + OUTPUT_FORMAT_NB +}; + +enum AVPixelFormat ff_output_fmt[] =3D { + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUYV422, AV_PIX_FMT_UYVY422, + AV_PIX_FMT_NV12, AV_PIX_FMT_ARGB, AV_PIX_FMT_RGBA, + AV_PIX_FMT_ABGR, AV_PIX_FMT_BGRA, AV_PIX_FMT_YUV420P10LE, + AV_PIX_FMT_NV16, AV_PIX_FMT_BGR0, AV_PIX_FMT_P010LE}; + +typedef struct NetIntScaleContext { + const AVClass *class; + AVDictionary *opts; + + /** + * New dimensions. Special values are: + * 0 =3D original width/height + * -1 =3D keep original aspect + * -N =3D try to keep aspect but make sure it is divisible by N + */ + int w, h; + char *size_str; + + char *w_expr; ///< width expression string + char *h_expr; ///< height expression string + + char *flags_str; + + char *in_color_matrix; + char *out_color_matrix; + + int force_original_aspect_ratio; + int force_divisible_by; + int format; + + enum AVPixelFormat out_format; + AVBufferRef *out_frames_ref; + AVBufferRef *out_frames_ref_1; + + ni_session_context_t api_ctx; + ni_session_data_io_t api_dst_frame; + ni_scaler_params_t params; + + int initialized; + int session_opened; + int keep_alive_timeout; /* keep alive timeout setting */ + int output_compressed; + + int auto_skip; + int skip_filter; + int autoselect; + int buffer_limit; + AVExpr *w_pexpr; + AVExpr *h_pexpr; + double var_values[VARS_NB]; +} NetIntScaleContext; + +FFFilter ff_vf_scale_ni; +static int config_props(AVFilterLink *outlink); + +static int check_exprs(AVFilterContext *ctx) +{ + NetIntScaleContext *scale =3D ctx->priv; + unsigned vars_w[VARS_NB] =3D { 0 }, vars_h[VARS_NB] =3D { 0 }; + + if (!scale->w_pexpr && !scale->h_pexpr) + return AVERROR(EINVAL); + + if (scale->w_pexpr) + av_expr_count_vars(scale->w_pexpr, vars_w, VARS_NB); + if (scale->h_expr) + av_expr_count_vars(scale->h_pexpr, vars_h, VARS_NB); + + if (vars_w[VAR_OUT_W] || vars_w[VAR_OW]) { + av_log(ctx, AV_LOG_ERROR, "Width expression cannot be self-referen= cing: '%s'.\n", scale->w_expr); + return AVERROR(EINVAL); + } + + if (vars_h[VAR_OUT_H] || vars_h[VAR_OH]) { + av_log(ctx, AV_LOG_ERROR, "Height expression cannot be self-refere= ncing: '%s'.\n", scale->h_expr); + return AVERROR(EINVAL); + } + + if ((vars_w[VAR_OUT_H] || vars_w[VAR_OH]) && + (vars_h[VAR_OUT_W] || vars_h[VAR_OW])) { + av_log(ctx, AV_LOG_WARNING, "Circular references detected for widt= h '%s' and height '%s' - possibly invalid.\n", scale->w_expr, scale->h_expr= ); + } + + return 0; +} + +static int scale_parse_expr(AVFilterContext *ctx, char *str_expr, AVExpr *= *pexpr_ptr, const char *var, const char *args) +{ + NetIntScaleContext *scale =3D ctx->priv; + int ret, is_inited =3D 0; + char *old_str_expr =3D NULL; + AVExpr *old_pexpr =3D NULL; + + if (str_expr) { + old_str_expr =3D av_strdup(str_expr); + if (!old_str_expr) + return AVERROR(ENOMEM); + av_opt_set(scale, var, args, 0); + } + + if (*pexpr_ptr) { + old_pexpr =3D *pexpr_ptr; + *pexpr_ptr =3D NULL; + is_inited =3D 1; + } + + ret =3D av_expr_parse(pexpr_ptr, args, var_names, + NULL, NULL, NULL, NULL, 0, ctx); + if (ret < 0) { + av_log(ctx, AV_LOG_ERROR, "Cannot parse expression for %s: '%s'\n"= , var, args); + goto revert; + } + + ret =3D check_exprs(ctx); + if (ret < 0) + goto revert; + + if (is_inited && (ret =3D config_props(ctx->outputs[0])) < 0) + goto revert; + + av_expr_free(old_pexpr); + old_pexpr =3D NULL; + av_freep(&old_str_expr); + + return 0; + +revert: + av_expr_free(*pexpr_ptr); + *pexpr_ptr =3D NULL; + if (old_str_expr) { + av_opt_set(scale, var, old_str_expr, 0); + av_free(old_str_expr); + } + if (old_pexpr) + *pexpr_ptr =3D old_pexpr; + + return ret; +} + +static int scale_eval_dimensions(AVFilterContext *ctx) +{ + NetIntScaleContext *scale =3D ctx->priv; + const AVFilterLink *inlink =3D ctx->inputs[0]; + const AVFilterLink *outlink =3D ctx->outputs[0]; + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(inlink->format); + const AVPixFmtDescriptor *out_desc =3D av_pix_fmt_desc_get(outlink->fo= rmat); + char *expr; + int ret; + double res; + const AVPixFmtDescriptor *main_desc; + const AVFilterLink *main_link; + + scale->var_values[VAR_IN_W] =3D scale->var_values[VAR_IW] =3D inlink-= >w; + scale->var_values[VAR_IN_H] =3D scale->var_values[VAR_IH] =3D inlink-= >h; + scale->var_values[VAR_OUT_W] =3D scale->var_values[VAR_OW] =3D NAN; + scale->var_values[VAR_OUT_H] =3D scale->var_values[VAR_OH] =3D NAN; + scale->var_values[VAR_A] =3D (double) inlink->w / inlink->h; + scale->var_values[VAR_SAR] =3D inlink->sample_aspect_ratio.num ? + (double) inlink->sample_aspect_ratio.num / inlink->sample_aspect_r= atio.den : 1; + scale->var_values[VAR_DAR] =3D scale->var_values[VAR_A] * scale->var= _values[VAR_SAR]; + scale->var_values[VAR_HSUB] =3D 1 << desc->log2_chroma_w; + scale->var_values[VAR_VSUB] =3D 1 << desc->log2_chroma_h; + scale->var_values[VAR_OHSUB] =3D 1 << out_desc->log2_chroma_w; + scale->var_values[VAR_OVSUB] =3D 1 << out_desc->log2_chroma_h; + + res =3D av_expr_eval(scale->w_pexpr, scale->var_values, NULL); + scale->var_values[VAR_OUT_W] =3D scale->var_values[VAR_OW] =3D (int) r= es =3D=3D 0 ? inlink->w : (int) res; + + res =3D av_expr_eval(scale->h_pexpr, scale->var_values, NULL); + if (isnan(res)) { + expr =3D scale->h_expr; + ret =3D AVERROR(EINVAL); + goto fail; + } + scale->var_values[VAR_OUT_H] =3D scale->var_values[VAR_OH] =3D (int) r= es =3D=3D 0 ? inlink->h : (int) res; + + res =3D av_expr_eval(scale->w_pexpr, scale->var_values, NULL); + if (isnan(res)) { + expr =3D scale->w_expr; + ret =3D AVERROR(EINVAL); + goto fail; + } + scale->var_values[VAR_OUT_W] =3D scale->var_values[VAR_OW] =3D (int) r= es =3D=3D 0 ? inlink->w : (int) res; + + scale->w =3D (int)scale->var_values[VAR_OUT_W]; + scale->h =3D (int)scale->var_values[VAR_OUT_H]; + + return 0; + +fail: + av_log(ctx, AV_LOG_ERROR, + "Error when evaluating the expression '%s'.\n", expr); + return ret; +} + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] =3D + {AV_PIX_FMT_NI_QUAD, AV_PIX_FMT_NONE}; + AVFilterFormats *formats; + + formats =3D ff_make_format_list(pix_fmts); + + if (!formats) + return AVERROR(ENOMEM); + + return ff_set_common_formats(ctx, formats); +} + +static av_cold int init(AVFilterContext *ctx) +{ + NetIntScaleContext *scale =3D ctx->priv; + int ret; + + if (scale->size_str && (scale->w_expr || scale->h_expr)) { + av_log(ctx, AV_LOG_ERROR, + "Size and width/height expressions cannot be set at the sam= e time.\n"); + return AVERROR(EINVAL); + } + + if (scale->w_expr && !scale->h_expr) + FFSWAP(char *, scale->w_expr, scale->size_str); + + if (scale->size_str) { + char buf[32]; + + if ((ret =3D av_parse_video_size(&scale->w, &scale->h, scale->size= _str)) < 0) { + av_log(ctx, AV_LOG_ERROR, + "Invalid size '%s'\n", scale->size_str); + return ret; + } + snprintf(buf, sizeof(buf)-1, "%d", scale->w); + av_opt_set(scale, "w", buf, 0); + snprintf(buf, sizeof(buf)-1, "%d", scale->h); + av_opt_set(scale, "h", buf, 0); + } + if (!scale->w_expr) + av_opt_set(scale, "w", "iw", 0); + if (!scale->h_expr) + av_opt_set(scale, "h", "ih", 0); + + ret =3D scale_parse_expr(ctx, NULL, &scale->w_pexpr, "width", scale->w= _expr); + if (ret < 0) + return ret; + + ret =3D scale_parse_expr(ctx, NULL, &scale->h_pexpr, "height", scale->= h_expr); + if (ret < 0) + return ret; + + av_log(ctx, AV_LOG_VERBOSE, "w:%s h:%s\n", scale->w_expr, scale->h_exp= r); + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + NetIntScaleContext *scale =3D ctx->priv; + + av_expr_free(scale->w_pexpr); + av_expr_free(scale->h_pexpr); + scale->w_pexpr =3D scale->h_pexpr =3D NULL; + av_dict_free(&scale->opts); + + if (scale->api_dst_frame.data.frame.p_buffer) + ni_frame_buffer_free(&scale->api_dst_frame.data.frame); + + if (scale->session_opened) { + /* Close operation will free the device frames */ + ni_device_session_close(&scale->api_ctx, 1, NI_DEVICE_TYPE_SCALER); + ni_device_session_context_clear(&scale->api_ctx); + } + + av_buffer_unref(&scale->out_frames_ref); + av_buffer_unref(&scale->out_frames_ref_1); +} + +static int init_out_pool(AVFilterContext *ctx) +{ + NetIntScaleContext *s =3D ctx->priv; + AVHWFramesContext *out_frames_ctx; + int pool_size =3D DEFAULT_NI_FILTER_POOL_SIZE; + + out_frames_ctx =3D (AVHWFramesContext*)s->out_frames_ref->data; + pool_size +=3D ctx->extra_hw_frames > 0 ? ctx->extra_hw_frames : 0; + s->buffer_limit =3D 1; + + /* Create frame pool on device */ + return ff_ni_build_frame_pool(&s->api_ctx, out_frames_ctx->width, + out_frames_ctx->height, + s->out_format, pool_size, + s->buffer_limit); +} + +static int config_props(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink0 =3D outlink->src->inputs[0]; + AVFilterLink *inlink =3D outlink->src->inputs[0]; + // AVFilterLink *inlink =3D outlink->src->inputs[0]; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx; + NetIntScaleContext *scale =3D ctx->priv; + int w, h, ret, h_shift, v_shift; + + if ((ret =3D scale_eval_dimensions(ctx)) < 0) + return ret; + + w =3D scale->w; + h =3D scale->h; + + ff_scale_adjust_dimensions(inlink, &w, &h, + scale->force_original_aspect_ratio, + scale->force_divisible_by, 1.f); + + if (w > NI_MAX_RESOLUTION_WIDTH || h > NI_MAX_RESOLUTION_HEIGHT) { + av_log(ctx, AV_LOG_ERROR, "Scaled value (%dx%d) > 8192 not allowed= \n", w, h); + return AVERROR(EINVAL); + } + + if ((w <=3D 0) || (h <=3D 0)) { + av_log(ctx, AV_LOG_ERROR, "Scaled value (%dx%d) not allowed\n", w,= h); + return AVERROR(EINVAL); + } + + FilterLink *li =3D ff_filter_link(inlink); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (in_frames_ctx->sw_format =3D=3D AV_PIX_FMT_NI_QUAD_10_TILE_4X4) { + av_log(ctx, AV_LOG_ERROR, "tile4x4 10b not supported for scale!\n"= ); + return AVERROR(EINVAL); + } + + /* Set the output format */ + if (scale->format =3D=3D OUTPUT_FORMAT_AUTO) { + scale->out_format =3D in_frames_ctx->sw_format; + } else { + scale->out_format =3D ff_output_fmt[scale->format]; + } + if (scale->out_format =3D=3D AV_PIX_FMT_NI_QUAD_8_TILE_4X4) + scale->output_compressed =3D 1; + else + scale->output_compressed =3D 0; + + av_pix_fmt_get_chroma_sub_sample(scale->out_format, &h_shift, &v_shift= ); + + outlink->w =3D FFALIGN(w, (1 << h_shift)); + outlink->h =3D FFALIGN(h, (1 << v_shift)); + + if (inlink0->sample_aspect_ratio.num) { + outlink->sample_aspect_ratio =3D av_mul_q((AVRational){outlink->h = * inlink0->w, outlink->w * inlink0->h}, inlink0->sample_aspect_ratio); + } else { + outlink->sample_aspect_ratio =3D inlink0->sample_aspect_ratio; + } + + av_log(ctx, AV_LOG_VERBOSE, + "w:%d h:%d fmt:%s sar:%d/%d -> w:%d h:%d fmt:%s sar:%d/%d\n", + inlink->w, inlink->h, av_get_pix_fmt_name(inlink->format), + inlink->sample_aspect_ratio.num, inlink->sample_aspect_ratio.de= n, + outlink->w, outlink->h, av_get_pix_fmt_name(outlink->format), + outlink->sample_aspect_ratio.num, outlink->sample_aspect_ratio.= den); + + //skip the color range check + if (scale->auto_skip && + inlink->w =3D=3D outlink->w && + inlink->h =3D=3D outlink->h && + in_frames_ctx->sw_format =3D=3D scale->out_format && + ( + (!scale->in_color_matrix && (!scale->out_color_matrix || strcmp(s= cale->out_color_matrix, "bt709") =3D=3D 0)) || + (!scale->out_color_matrix && (!scale->in_color_matrix || strcmp(s= cale->in_color_matrix, "bt709") =3D=3D 0)) || + (scale->in_color_matrix && scale->out_color_matrix && strcmp(scal= e->in_color_matrix, scale->out_color_matrix) =3D=3D 0) + ) + ) { + //skip hardware scale + scale->skip_filter =3D 1; + + FilterLink *lo =3D ff_filter_link(outlink); + scale->out_frames_ref =3D av_buffer_ref(li->hw_frames_ctx); + if (!scale->out_frames_ref) { + return AVERROR(ENOMEM); + } + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(scale->out_frames_ref); + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + return 0; + } + + scale->out_frames_ref =3D av_hwframe_ctx_alloc(in_frames_ctx->device_r= ef); + if (!scale->out_frames_ref) + return AVERROR(ENOMEM); + + out_frames_ctx =3D (AVHWFramesContext *)scale->out_frames_ref->data; + + out_frames_ctx->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx->width =3D outlink->w; + out_frames_ctx->height =3D outlink->h; + out_frames_ctx->sw_format =3D scale->out_format; + out_frames_ctx->initial_pool_size =3D + NI_SCALE_ID; // Repurposed as identity code + + av_hwframe_ctx_init(scale->out_frames_ref);//call this? + + FilterLink *lt =3D ff_filter_link(ctx->outputs[0]); + av_buffer_unref(<->hw_frames_ctx); + lt->hw_frames_ctx =3D av_buffer_ref(scale->out_frames_ref); + if (!lt->hw_frames_ctx) + return AVERROR(ENOMEM); + + return 0; +} + +static int config_props_ref(AVFilterLink *outlink) +{ + AVFilterContext *ctx =3D outlink->src; + AVFilterLink *inlink =3D outlink->src->inputs[1]; + + // AVFilterLink *inlink =3D outlink->src->inputs[0]; + NetIntScaleContext *scale =3D ctx->priv; + outlink->w =3D inlink->w; + outlink->h =3D inlink->h; + outlink->sample_aspect_ratio =3D inlink->sample_aspect_ratio; + outlink->time_base =3D inlink->time_base; + outlink->format =3D inlink->format; + + FilterLink *li =3D ff_filter_link(inlink); + FilterLink *lo =3D ff_filter_link(outlink); + lo->frame_rate =3D li->frame_rate; + scale->out_frames_ref_1 =3D av_buffer_ref(li->hw_frames_ctx); + if (!scale->out_frames_ref_1) { + return AVERROR(ENOMEM); + } + av_buffer_unref(&lo->hw_frames_ctx); + lo->hw_frames_ctx =3D av_buffer_ref(scale->out_frames_ref_1); + if (!lo->hw_frames_ctx) { + return AVERROR(ENOMEM); + } + + av_log(ctx, AV_LOG_VERBOSE, + "w:%d h:%d fmt:%s sar:%d/%d -> w:%d h:%d fmt:%s sar:%d/%d\n", + inlink->w, inlink->h, av_get_pix_fmt_name(inlink->format), + inlink->sample_aspect_ratio.num, inlink->sample_aspect_ratio.den, + outlink->w, outlink->h, av_get_pix_fmt_name(outlink->format), + outlink->sample_aspect_ratio.num, outlink->sample_aspect_ratio.den= ); + + return 0; +} + +static int request_frame(AVFilterLink *outlink) +{ + return ff_request_frame(outlink->src->inputs[0]); +} + +static int request_frame_ref(AVFilterLink *outlink) +{ + return ff_request_frame(outlink->src->inputs[1]); +} + +/* Process a received frame */ +static int filter_frame(AVFilterLink *link, AVFrame *in) +{ + NetIntScaleContext *scale =3D link->dst->priv; + AVFilterLink *outlink =3D link->dst->outputs[0]; + AVFrame *out =3D NULL; + niFrameSurface1_t* frame_surface,*new_frame_surface; + AVHWFramesContext *pAVHFWCtx; + AVNIDeviceContext *pAVNIDevCtx; + ni_retcode_t retcode; + int scaler_format, cardno; + uint16_t tempFID; + uint16_t options; + + frame_surface =3D (niFrameSurface1_t *) in->data[3]; + if (frame_surface =3D=3D NULL) { + return AVERROR(EINVAL); + } + + pAVHFWCtx =3D (AVHWFramesContext *) in->hw_frames_ctx->data; + pAVNIDevCtx =3D (AVNIDeviceContext *)pAVHFWCtx->device_ctx->hwct= x; + cardno =3D ni_get_cardno(in); + + if (scale->skip_filter) { + //skip hardware scale + return ff_filter_frame(link->dst->outputs[0], in); + } + + if (!scale->initialized) { + retcode =3D ni_device_session_context_init(&scale->api_ctx); + if (retcode < 0) { + av_log(link->dst, AV_LOG_ERROR, + "ni scale filter session context init failure\n"); + goto fail; + } + + scale->api_ctx.device_handle =3D pAVNIDevCtx->cards[cardno]; + scale->api_ctx.blk_io_handle =3D pAVNIDevCtx->cards[cardno]; + + scale->api_ctx.hw_id =3D cardno; + scale->api_ctx.device_type =3D NI_DEVICE_TYPE_SCALER; + scale->api_ctx.scaler_operation =3D NI_SCALER_OPCODE_SCALE; + scale->api_ctx.keep_alive_timeout =3D scale->keep_alive_timeout; + + av_log(link->dst, AV_LOG_INFO, + "Open scaler session to card %d, hdl %d, blk_hdl %d\n", car= dno, + scale->api_ctx.device_handle, scale->api_ctx.blk_io_handle); + + retcode =3D + ni_device_session_open(&scale->api_ctx, NI_DEVICE_TYPE_SCALER); + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_ERROR, + "Can't open device session on card %d\n", cardno); + + /* Close operation will free the device frames */ + ni_device_session_close(&scale->api_ctx, 1, NI_DEVICE_TYPE_SCA= LER); + ni_device_session_context_clear(&scale->api_ctx); + goto fail; + } + + scale->session_opened =3D 1; + + if (scale->autoselect) { + if (outlink->w <=3D 540 || outlink->h <=3D 540) + scale->params.filterblit =3D 1; + else + scale->params.filterblit =3D 2; + } + +#if ((LIBXCODER_API_VERSION_MAJOR > 2) || = \ + (LIBXCODER_API_VERSION_MAJOR =3D=3D 2 && LIBXCODER_API_VERSION_MINOR= >=3D 76)) + if (scale->params.scaler_param_b !=3D 0 || scale->params.scaler_pa= ram_c !=3D 0.75) { + scale->params.enable_scaler_params =3D true; + } else { + scale->params.enable_scaler_params =3D false; + } +#endif + if (scale->params.filterblit) { + retcode =3D ni_scaler_set_params(&scale->api_ctx, &(scale->par= ams)); + if (retcode < 0) + goto fail; + } + + if (!((av_strstart(outlink->dst->filter->name, "ni_quadra", NULL))= || (av_strstart(outlink->dst->filter->name, "hwdownload", NULL)))) { + link->dst->extra_hw_frames =3D (DEFAULT_FRAME_THREAD_QUEUE_SIZE= > 1) ? DEFAULT_FRAME_THREAD_QUEUE_SIZE : 0; + } + + retcode =3D init_out_pool(link->dst); + if (retcode < 0) { + av_log(link->dst, AV_LOG_ERROR, + "Internal output allocation failed rc =3D %d\n", retcod= e); + goto fail; + } + + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)scale->= out_frames_ref->data; + AVNIFramesContext *out_ni_ctx =3D (AVNIFramesContext *)out_frames_= ctx->hwctx; + ni_cpy_hwframe_ctx(pAVHFWCtx, out_frames_ctx); + ni_device_session_copy(&scale->api_ctx, &out_ni_ctx->api_ctx); + + const AVPixFmtDescriptor *desc =3D av_pix_fmt_desc_get(pAVHFWCtx->= sw_format); + + if ((in->color_range =3D=3D AVCOL_RANGE_JPEG) && !(desc->flags & A= V_PIX_FMT_FLAG_RGB)) { + av_log(link->dst, AV_LOG_WARNING, + "WARNING: Full color range input, limited color range o= utput\n"); + } + + scale->initialized =3D 1; + } + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(pAVHFWCtx->sw_format); + + retcode =3D ni_frame_buffer_alloc_hwenc(&scale->api_dst_frame.data.fra= me, + outlink->w, + outlink->h, + 0); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark(NULL); +#endif + + options =3D 0; + if (scale->in_color_matrix && strcmp(scale->in_color_matrix,"bt2020") = =3D=3D 0) + options |=3D NI_SCALER_FLAG_CS; + options |=3D (frame_surface->encoding_type =3D=3D 2) ? NI_SCALER_FLAG_= CMP : 0; + + /* + * Allocate device input frame. This call won't actually allocate a fr= ame, + * but sends the incoming hardware frame index to the scaler manager + */ + retcode =3D ni_device_alloc_frame(&scale->api_ctx, + FFALIGN(in->width, 2), + FFALIGN(in->height, 2), + scaler_format, + options, + 0, + 0, + 0, + 0, + 0, + frame_surface->ui16FrameIdx, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_DEBUG, + "Can't assign input frame %d\n", retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + scaler_format =3D ff_ni_ffmpeg_to_gc620_pix_fmt(scale->out_format); + + options =3D NI_SCALER_FLAG_IO; + if (scale->out_color_matrix && strcmp(scale->out_color_matrix, "bt2020= ") =3D=3D 0) + options |=3D NI_SCALER_FLAG_CS; + options |=3D (scale->output_compressed) ? NI_SCALER_FLAG_CMP : 0; + + /* Allocate hardware device destination frame. This acquires a frame f= rom the pool */ + retcode =3D ni_device_alloc_frame(&scale->api_ctx, + FFALIGN(outlink->w,2), + FFALIGN(outlink->h,2), + scaler_format, + options, + 0, + 0, + 0, + 0, + 0, + -1, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_DEBUG, + "Can't allocate device output frame %d\n", retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + out =3D av_frame_alloc(); + if (!out) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + av_frame_copy_props(out,in); + + out->width =3D outlink->w; + out->height =3D outlink->h; + + out->format =3D AV_PIX_FMT_NI_QUAD; + + /* Quadra 2D engine always outputs limited color range */ + out->color_range =3D AVCOL_RANGE_MPEG; + + /* Reference the new hw frames context */ + out->hw_frames_ctx =3D av_buffer_ref(scale->out_frames_ref); + + out->data[3] =3D av_malloc(sizeof(niFrameSurface1_t)); + + if (!out->data[3]) { + retcode =3D AVERROR(ENOMEM); + goto fail; + } + + /* Copy the frame surface from the incoming frame */ + memcpy(out->data[3], in->data[3], sizeof(niFrameSurface1_t)); + + /* Set the new frame index */ + retcode =3D ni_device_session_read_hwdesc(&scale->api_ctx, &scale->api= _dst_frame, + NI_DEVICE_TYPE_SCALER); + + if (retcode !=3D NI_RETCODE_SUCCESS) { + av_log(link->dst, AV_LOG_ERROR, + "Can't acquire output frame %d\n",retcode); + retcode =3D AVERROR(ENOMEM); + goto fail; + } + +#ifdef NI_MEASURE_LATENCY + ff_ni_update_benchmark("ni_quadra_scale"); +#endif + + tempFID =3D frame_surface->ui16FrameIdx; + frame_surface =3D (niFrameSurface1_t *)out->data[3]; + new_frame_surface =3D (niFrameSurface1_t *)scale->api_dst_frame.data.f= rame.p_data[3]; + frame_surface->ui16FrameIdx =3D new_frame_surface->ui16FrameIdx; + frame_surface->ui16session_ID =3D new_frame_surface->ui16session_ID; + frame_surface->device_handle =3D new_frame_surface->device_handle; + frame_surface->output_idx =3D new_frame_surface->output_idx; + frame_surface->src_cpu =3D new_frame_surface->src_cpu; + frame_surface->dma_buf_fd =3D 0; + + ff_ni_set_bit_depth_and_encoding_type(&frame_surface->bit_depth, + &frame_surface->encoding_type, + scale->out_format); + + /* Remove ni-split specific assets */ + frame_surface->ui32nodeAddress =3D 0; + + frame_surface->ui16width =3D out->width; + frame_surface->ui16height =3D out->height; + + av_log(link->dst, AV_LOG_DEBUG, + "vf_scale_ni.c:IN trace ui16FrameIdx =3D [%d] --> out [%d]\n", + tempFID, frame_surface->ui16FrameIdx); + + out->buf[0] =3D av_buffer_create(out->data[3], sizeof(niFrameSurface1_= t), + ff_ni_frame_free, NULL, 0); + + av_frame_free(&in); + + return ff_filter_frame(link->dst->outputs[0], out); + +fail: + av_frame_free(&in); + if (out) + av_frame_free(&out); + return retcode; +} + +static int filter_frame_ref(AVFilterLink *link, AVFrame *in) +{ + AVFilterLink *outlink =3D link->dst->outputs[1]; + return ff_filter_frame(outlink, in); +} + +static int activate(AVFilterContext *ctx) +{ + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFilterLink *outlink =3D ctx->outputs[0]; + AVFrame *frame =3D NULL; + int ret =3D 0; + NetIntScaleContext *s =3D inlink->dst->priv; + + // Forward the status on output link to input link, if the status is s= et, discard all queued frames + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + + av_log(ctx, AV_LOG_TRACE, "%s: ready %u inlink framequeue %u available= _frame %d outlink framequeue %u frame_wanted %d\n", + __func__, ctx->ready, ff_inlink_queued_frames(inlink), ff_inlink_c= heck_available_frame(inlink), ff_inlink_queued_frames(outlink), ff_outlink_= frame_wanted(outlink)); + + if (ff_inlink_check_available_frame(inlink)) { + if (s->initialized) { + ret =3D ni_device_session_query_buffer_avail(&s->api_ctx, NI_D= EVICE_TYPE_SCALER); + } + + if (ret =3D=3D NI_RETCODE_ERROR_UNSUPPORTED_FW_VERSION) { + av_log(ctx, AV_LOG_WARNING, "No backpressure support in FW\n"); + } else if (ret < 0) { + av_log(ctx, AV_LOG_WARNING, "%s: query ret %d, ready %u inlink= framequeue %u available_frame %d outlink framequeue %u frame_wanted %d - r= eturn NOT READY\n", + __func__, ret, ctx->ready, ff_inlink_queued_frames(inlink)= , ff_inlink_check_available_frame(inlink), ff_inlink_queued_frames(outlink)= , ff_outlink_frame_wanted(outlink)); + return FFERROR_NOT_READY; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) + return ret; + + ret =3D filter_frame(inlink, frame); + if (ret >=3D 0) { + ff_filter_set_ready(ctx, 300); + } + return ret; + } + + // We did not get a frame from input link, check its status + FF_FILTER_FORWARD_STATUS(inlink, outlink); + + // We have no frames yet from input link and no EOF, so request some. + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntScaleContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption ni_scale_options[] =3D { + { "w", "Output video width", OFFSET(w_expr), AV_OPT_TYPE_STRIN= G, .flags =3D FLAGS }, + { "width", "Output video width", OFFSET(w_expr), AV_OPT_TYPE_STRIN= G, .flags =3D FLAGS }, + { "h", "Output video height", OFFSET(h_expr), AV_OPT_TYPE_STRIN= G, .flags =3D FLAGS }, + { "height", "Output video height", OFFSET(h_expr), AV_OPT_TYPE_STRIN= G, .flags =3D FLAGS }, + { "size", "set video size", OFFSET(size_str), AV_OPT_TYPE_STRIN= G, {.str =3D NULL}, 0, FLAGS }, + { "s", "set video size", OFFSET(size_str), AV_OPT_TYPE_STRIN= G, {.str =3D NULL}, 0, FLAGS }, + { "in_color_matrix", "set input YCbCr type", OFFSET(in_color_matrix= ), AV_OPT_TYPE_STRING, {.str =3D NULL}, .flags =3D FLAGS, "color" }, + { "out_color_matrix", "set output YCbCr type", OFFSET(out_color_matri= x), AV_OPT_TYPE_STRING, {.str =3D NULL}, .flags =3D FLAGS, "color"}, + { "bt709", NULL, 0, AV_OPT_TYPE_CONST, {.str =3D "bt709"}, = 0, 0, FLAGS, "color" }, + { "bt2020", NULL, 0, AV_OPT_TYPE_CONST, {.str =3D "bt2020"}, = 0, 0, FLAGS, "color" }, + { "force_original_aspect_ratio", "decrease or increase w/h if necessar= y to keep the original AR", OFFSET(force_original_aspect_ratio), AV_OPT_TYP= E_INT, {.i64 =3D 0}, 0, 2, FLAGS, "force_oar" }, + { "format", "set_output_format", OFFSET(format), AV_OPT_TYPE_INT, {.i6= 4=3DOUTPUT_FORMAT_AUTO}, 0, OUTPUT_FORMAT_NB-1, FLAGS, "format" }, + { "yuv420p", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_Y= UV420P}, .flags =3D FLAGS, .unit =3D "format" }, + { "yuyv422", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_Y= UYV422}, .flags =3D FLAGS, .unit =3D "format" }, + { "uyvy422", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_U= YVY422}, .flags =3D FLAGS, .unit =3D "format" }, + { "nv12", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_N= V12}, .flags =3D FLAGS, .unit =3D "format" }, + { "argb", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_A= RGB}, .flags =3D FLAGS, .unit =3D "format" }, + { "rgba", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_R= GBA}, .flags =3D FLAGS, .unit =3D "format" }, + { "abgr", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_A= BGR}, .flags =3D FLAGS, .unit =3D "format" }, + { "bgra", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_B= GRA}, .flags =3D FLAGS, .unit =3D "format" }, + { "yuv420p10le", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_Y= UV420P10LE}, .flags =3D FLAGS, .unit =3D "format" }, + { "nv16", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_N= V16}, .flags =3D FLAGS, .unit =3D "format" }, + { "bgr0", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_B= GR0}, .flags =3D FLAGS, .unit =3D "format" }, + { "p010le", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_P= 010LE}, .flags =3D FLAGS, .unit =3D "format" }, + { "auto", "", 0, AV_OPT_TYPE_CONST, {.i64=3DOUTPUT_FORMAT_A= UTO}, .flags =3D FLAGS, .unit =3D "format"}, + { "disable", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 0}, 0, 0, FLAGS, "= force_oar" }, + { "decrease", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 1}, 0, 0, FLAGS, "= force_oar" }, + { "increase", NULL, 0, AV_OPT_TYPE_CONST, {.i64 =3D 2}, 0, 0, FLAGS, "= force_oar" }, + { "force_divisible_by", "enforce that the output resolution is divisib= le by a defined integer when force_original_aspect_ratio is used", OFFSET(f= orce_divisible_by), AV_OPT_TYPE_INT, {.i64 =3D 1}, 1, 256, FLAGS }, + { "filterblit", "filterblit enable", OFFSET(params.filterblit), AV_OPT= _TYPE_INT, {.i64=3D0}, 0, 4, FLAGS }, +#if ((LIBXCODER_API_VERSION_MAJOR > 2) || (LIBXCODER_API_VERSION_MAJOR =3D= =3D 2 && LIBXCODER_API_VERSION_MINOR>=3D 76)) + { "param_b", "Parameter B for bicubic", OFFSET(params.scaler_param_b),= AV_OPT_TYPE_DOUBLE, {.dbl=3D0.0}, 0, 1, FLAGS }, + { "param_c", "Parameter C for bicubic", OFFSET(params.scaler_param_c),= AV_OPT_TYPE_DOUBLE, {.dbl=3D0.75}, 0, 1, FLAGS }, +#endif + { "autoselect", "auto select filterblit mode according to resolution",= OFFSET(autoselect), AV_OPT_TYPE_BOOL, {.i64=3D0}, 0, 1, FLAGS }, + NI_FILT_OPTION_AUTO_SKIP, + NI_FILT_OPTION_KEEPALIVE, + NI_FILT_OPTION_BUFFER_LIMIT, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_scale); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .filter_frame =3D filter_frame, + }, +}; + +static const AVFilterPad outputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_props, + }, +}; + +FFFilter ff_vf_scale_ni_quadra =3D { + .p.name =3D "ni_quadra_scale", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra video scaler v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_scale_class, + .priv_size =3D sizeof(NetIntScaleContext), + .init =3D init, + .uninit =3D uninit, + .activate =3D activate, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, + FILTER_INPUTS(inputs), + FILTER_OUTPUTS(outputs), + FILTER_QUERY_FUNC(query_formats), +}; diff --git a/libavfilter/vf_split_ni.c b/libavfilter/vf_split_ni.c new file mode 100644 index 0000000000..7537127132 --- /dev/null +++ b/libavfilter/vf_split_ni.c @@ -0,0 +1,529 @@ +/* + * Copyright (c) 2007 Bobby Bingham + * Copyright (c) 2021 NetInt + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-130= 1 USA + */ + +/** + * @file + * audio and video splitter + */ + +#include + +#include "libavutil/attributes.h" +#include "libavutil/internal.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "nifilter.h" +#include "version.h" +#include +#include "avfilter_internal.h" +#include "framequeue.h" +#include "avfilter.h" +#include "audio.h" +#include "formats.h" +#include "filters.h" +#include "video.h" + +typedef struct NetIntSplitContext { + const AVClass *class; + bool initialized; + int nb_output0; + int nb_output1; + int nb_output2; + int total_outputs; + int frame_contexts_applied; + ni_split_context_t src_ctx; + AVBufferRef *out_frames_ref[3]; +} NetIntSplitContext; + +static int config_output(AVFilterLink *link); + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat input_pix_fmts[] =3D { + AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUV420P10LE, + AV_PIX_FMT_NV12, + AV_PIX_FMT_P010LE, + AV_PIX_FMT_NI_QUAD, + AV_PIX_FMT_NONE, + }; + static const enum AVPixelFormat output_pix_fmts[] =3D { + AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUV420P10LE, + AV_PIX_FMT_NV12, + AV_PIX_FMT_P010LE, + AV_PIX_FMT_NI_QUAD, + AV_PIX_FMT_NONE, + }; + AVFilterFormats *in_fmts =3D ff_make_format_list(input_pix_fmts); + AVFilterFormats *out_fmts =3D ff_make_format_list(output_pix_fmts); + + // NOLINTNEXTLINE(clang-diagnostic-unused-result) + ff_formats_ref(in_fmts, &ctx->inputs[0]->outcfg.formats); + // NOLINTNEXTLINE(clang-diagnostic-unused-result) + ff_formats_ref(out_fmts, &ctx->outputs[0]->incfg.formats); + + return 0; +} + +static av_cold int split_init(AVFilterContext *ctx) +{ + NetIntSplitContext *s =3D ctx->priv; + int i, ret; + + av_log(ctx, AV_LOG_DEBUG, "ni_quadra_split INIT out0,1,2 =3D %d %d %d = ctx->nb_outputs =3D %d\n", + s->nb_output0, s->nb_output1, s->nb_output2, + ctx->nb_outputs); + if (s->nb_output2 && s->nb_output1 =3D=3D 0) { + //swap them for reorder to use out1 first + s->nb_output1 =3D s->nb_output2; + s->nb_output2 =3D 0; + av_log(ctx, AV_LOG_DEBUG, "ni_quadra_split INIT out2 moved to out1= \n"); + } + + s->total_outputs =3D s->nb_output0 + s->nb_output1 + s->nb_output2; + + for (i =3D 0; i < s->total_outputs; i++) { + char name[32]; + AVFilterPad pad =3D { 0 }; + + snprintf(name, sizeof(name), "output%d", i); + pad.type =3D ctx->filter->inputs[0].type; + pad.name =3D av_strdup(name); + if (!pad.name) { + return AVERROR(ENOMEM); + } + pad.config_props =3D config_output; + + if (!pad.name) + return AVERROR(ENOMEM); + + if ((ret =3D ff_append_outpad_free_name(ctx, &pad)) < 0) { + av_freep(&pad.name); + return ret; + } + } + + return 0; +} + +static av_cold void split_uninit(AVFilterContext *ctx) +{ + int i; + NetIntSplitContext *s =3D ctx->priv; + for (i =3D 0; i < ctx->nb_outputs; i++) + av_freep(&ctx->output_pads[i].name); + + for (i =3D 0; i < 3; i++) { + if (s->out_frames_ref[i]) + av_buffer_unref(&s->out_frames_ref[i]); + } +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *avctx =3D inlink->dst; + NetIntSplitContext *s =3D avctx->priv; + AVHWFramesContext *ctx; + ni_split_context_t *p_split_ctx_dst =3D &s->src_ctx; + AVNIFramesContext *src_ctx; + ni_split_context_t *p_split_ctx_src; + int i; + s->frame_contexts_applied =3D -1; + FilterLink *li =3D ff_filter_link(inlink); + + if (li->hw_frames_ctx =3D=3D NULL) { + for (i =3D 0; i < 3; i++) { + s->src_ctx.w[i] =3D inlink->w; + s->src_ctx.h[i] =3D inlink->h; + s->src_ctx.f[i] =3D -1; + s->src_ctx.f8b[i] =3D -1; + } + } else { + ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + src_ctx =3D (AVNIFramesContext*) ctx->hwctx; + p_split_ctx_src =3D &src_ctx->split_ctx; + memcpy(p_split_ctx_dst, p_split_ctx_src, sizeof(ni_split_context_t= )); + for (i =3D 0; i < 3; i++) { + s->frame_contexts_applied =3D 0; + av_log(avctx, AV_LOG_DEBUG, "[%d] %d x %d f8b %d\n", i, + p_split_ctx_dst->w[i], p_split_ctx_dst->h[i], + p_split_ctx_dst->f8b[i]); + } + if (p_split_ctx_dst->enabled =3D=3D 0) { + for (i =3D 0; i < 3; i++) { + s->src_ctx.w[i] =3D inlink->w; + s->src_ctx.h[i] =3D inlink->h; + s->src_ctx.f[i] =3D -1; + s->src_ctx.f8b[i] =3D -1; + } + } + } + + return 0; +} + +static int init_out_hwctxs(AVFilterContext *ctx) +{ + NetIntSplitContext *s =3D ctx->priv; + AVHWFramesContext *in_frames_ctx; + AVHWFramesContext *out_frames_ctx[3]; + enum AVPixelFormat out_format; + int i, j; + + FilterLink *li =3D ff_filter_link(ctx->inputs[0]); + if (li->hw_frames_ctx =3D=3D NULL) { + av_log(ctx, AV_LOG_ERROR, "No hw context provided on input\n"); + return AVERROR(EINVAL); + } + + in_frames_ctx =3D (AVHWFramesContext *)li->hw_frames_ctx->data; + + if (s->src_ctx.enabled =3D=3D 1) { + for (i =3D 0; i < 3; i++) { + s->out_frames_ref[i] =3D + av_hwframe_ctx_alloc(in_frames_ctx->device_ref); + if (!s->out_frames_ref[i]) + return AVERROR(ENOMEM); + out_frames_ctx[i] =3D (AVHWFramesContext *)s->out_frames_ref[i= ]->data; + + out_frames_ctx[i]->format =3D AV_PIX_FMT_NI_QUAD; + out_frames_ctx[i]->width =3D s->src_ctx.w[i]; + out_frames_ctx[i]->height =3D s->src_ctx.h[i]; + + if (s->src_ctx.f[i] =3D=3D -1) { + return AVERROR(EINVAL); + } + + switch (s->src_ctx.f[i]) { + case NI_PIXEL_PLANAR_FORMAT_PLANAR: // yuv420p or p10 + out_format =3D (s->src_ctx.f8b[i] =3D=3D 1) ? AV_PIX_F= MT_YUV420P + : AV_PIX_FMT_YUV= 420P10LE; + break; + case NI_PIXEL_PLANAR_FORMAT_TILED4X4: // tiled + out_format =3D (s->src_ctx.f8b[i] =3D=3D 1) + ? AV_PIX_FMT_NI_QUAD_8_TILE_4X4 + : AV_PIX_FMT_NI_QUAD_10_TILE_4X4; + break; + case NI_PIXEL_PLANAR_FORMAT_SEMIPLANAR: // NV12 + out_format =3D (s->src_ctx.f8b[i] =3D=3D 1) ? AV_PIX_F= MT_NV12 + : AV_PIX_FMT_P01= 0LE; + break; + default: + av_log(ctx, AV_LOG_ERROR, "PPU%d invalid pixel format = %d in hwframe ctx\n", i, s->src_ctx.f[i]); + return AVERROR(EINVAL); + } + out_frames_ctx[i]->sw_format =3D out_format; + out_frames_ctx[i]->initial_pool_size =3D -1; // already has it= s own pool + + /* Don't check return code, this will intentionally fail */ + av_hwframe_ctx_init(s->out_frames_ref[i]); + + ni_cpy_hwframe_ctx(in_frames_ctx, out_frames_ctx[i]); + ((AVNIFramesContext *) out_frames_ctx[i]->hwctx)->split_ctx.en= abled =3D 0; + } + + for (i =3D 0; i < ctx->nb_outputs; i++) { + FilterLink *lo =3D ff_filter_link(ctx->outputs[i]); + av_buffer_unref(&lo->hw_frames_ctx); + if (i < s->nb_output0) { + j =3D 0; + } else if (i < s->nb_output0 + s->nb_output1) { + j =3D 1; + } else { + j =3D 2; + } + lo->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref[j]); + + av_log(ctx, AV_LOG_DEBUG, "NI:%s:out\n", + (s->src_ctx.f[j] =3D=3D 0) + ? "semiplanar" + : (s->src_ctx.f[j] =3D=3D 2) ? "tiled" : "planar"); + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + av_log(ctx, AV_LOG_DEBUG, + "ni_split superframe config_output_hwctx[%d] %p\n", i, + lo->hw_frames_ctx); + } + } else { // no possibility of using extra outputs + for (i =3D 0; i < ctx->nb_outputs; i++) { + FilterLink *lo =3D ff_filter_link(ctx->outputs[i]); + av_buffer_unref(&lo->hw_frames_ctx); + if (i < s->nb_output0) { + lo->hw_frames_ctx =3D av_buffer_ref(li->hw_frames_ctx); + } + if (!lo->hw_frames_ctx) + return AVERROR(ENOMEM); + + av_log(ctx, AV_LOG_DEBUG, "ni_split config_output_hwctx[%d] %p= \n", + i, lo->hw_frames_ctx); + } + av_log(ctx, AV_LOG_DEBUG, + "ni_split config_output_hwctx set direct to output\n"); + } + return 0; +} + +static int config_output(AVFilterLink *link) +{ + // config output sets all outputs at a time since there's no + // easy way to track the target output based on inlink. + // fairly trivial assignments here so no performance worries + AVFilterContext *ctx =3D link->src; + NetIntSplitContext *s =3D ctx->priv; + int i, ret; + + for (i =3D 0; i < ctx->nb_outputs; i++) { + if (i < s->nb_output0) { + ctx->outputs[i]->w =3D s->src_ctx.w[0]; + ctx->outputs[i]->h =3D s->src_ctx.h[0]; + } else if (i < s->nb_output0 + s->nb_output1) { + ctx->outputs[i]->w =3D s->src_ctx.w[1]; + ctx->outputs[i]->h =3D s->src_ctx.h[1]; + } else { + ctx->outputs[i]->w =3D s->src_ctx.w[2]; + ctx->outputs[i]->h =3D s->src_ctx.h[2]; + } + av_log(ctx, AV_LOG_DEBUG, + "ni_split config_output[%d] w x h =3D %d x %d\n", i, + ctx->outputs[i]->w, ctx->outputs[i]->h); + } + if (s->frame_contexts_applied =3D=3D 0) { + s->frame_contexts_applied =3D 1; // run once per set ni_split, not= per output + ret =3D init_out_hwctxs(ctx); + if (ret < 0) + return ret; + } + return 0; +} + +static int filter_ni_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx =3D inlink->dst; + NetIntSplitContext *s =3D ctx->priv; + int i, ret =3D AVERROR_EOF; + int output_index; + niFrameSurface1_t *p_data3; + + if (!s->initialized) { + for (i =3D 0; i < 3; i++) { + AVHWFramesContext *in_frames_ctx =3D (AVHWFramesContext *)fram= e->hw_frames_ctx->data; + AVHWFramesContext *out_frames_ctx =3D (AVHWFramesContext *)s->= out_frames_ref[i]->data; + ni_cpy_hwframe_ctx(in_frames_ctx, out_frames_ctx); + AVNIFramesContext *ni_frames_ctx =3D (AVNIFramesContext *)out_= frames_ctx->hwctx; + ni_frames_ctx->split_ctx.enabled =3D 0; + } + s->initialized =3D 1; + } + + for (i =3D 0; i < ctx->nb_outputs; i++) { + AVFrame *buf_out; + FilterLinkInternal* const li =3D ff_link_internal(ctx->outputs[i]); + if (li->status_in) + continue; + + buf_out =3D av_frame_alloc(); + if (!buf_out) { + ret =3D AVERROR(ENOMEM); + break; + } + av_frame_copy_props(buf_out, frame); + + buf_out->format =3D frame->format; + + if (i < s->nb_output0) { + output_index =3D 0; + } else if (i < s->nb_output0 + s->nb_output1) { + if (!frame->buf[1]) { + ret =3D AVERROR(ENOMEM); + av_frame_free(&buf_out); + break; + } + output_index =3D 1; + } else { + if (!frame->buf[2]) { + ret =3D AVERROR(ENOMEM); + av_frame_free(&buf_out); + break; + } + output_index =3D 2; + } + buf_out->buf[0] =3D av_buffer_ref(frame->buf[output_index]); + buf_out->hw_frames_ctx =3D av_buffer_ref(s->out_frames_ref[output_= index]); + buf_out->data[3] =3D buf_out->buf[0]->data; + p_data3 =3D (niFrameSurface1_t*)((uint8_t*)buf_out->data[3]); + + buf_out->width =3D ctx->outputs[i]->w =3D p_data3->ui16width; + buf_out->height =3D ctx->outputs[i]->h =3D p_data3->ui16height; + + av_log(ctx, AV_LOG_DEBUG, "output %d supplied WxH =3D %d x %d FID = %d offset %d\n", + i, buf_out->width, buf_out->height, + p_data3->ui16FrameIdx, p_data3->ui32nodeAddress); + + ret =3D ff_filter_frame(ctx->outputs[i], buf_out); + if (ret < 0) + break; + } + return ret; +} + +static int filter_std_frame(AVFilterLink *inlink, AVFrame *frame) +{//basically clone of native split + AVFilterContext *ctx =3D inlink->dst; + NetIntSplitContext *s =3D ctx->priv; + int i, ret =3D AVERROR_EOF; + if (s->nb_output0 < 2) { + av_log(ctx, AV_LOG_ERROR, "ni_split must have at least 2 outputs f= or Standard split!\n"); + ret =3D AVERROR(EINVAL); + return ret; + } + if (s->nb_output1) { + av_log(ctx, AV_LOG_ERROR, "ni_split output1 or output2 param must = not be used for Standard splitting!\n"); + ret =3D AVERROR(E2BIG); + return ret; + } + + for (i =3D 0; i < ctx->nb_outputs; i++) { + AVFrame *buf_out; + FilterLinkInternal* const li =3D ff_link_internal(ctx->outputs[i]); + if (li->status_in) + continue; + buf_out =3D av_frame_clone(frame); + if (!buf_out) { + ret =3D AVERROR(ENOMEM); + break; + } + + ret =3D ff_filter_frame(ctx->outputs[i], buf_out); + if (ret < 0) + break; + } + return ret; +} + +static int activate(AVFilterContext *ctx) +{ + NetIntSplitContext *s =3D ctx->priv; + AVFilterLink *inlink =3D ctx->inputs[0]; + AVFrame *frame; + int status, ret, nb_eofs =3D 0; + int64_t pts; + + for (int i =3D 0; i < ctx->nb_outputs; i++) { + nb_eofs +=3D ff_outlink_get_status(ctx->outputs[i]) =3D=3D AVERROR= _EOF; + } + + if (nb_eofs =3D=3D ctx->nb_outputs) { + ff_inlink_set_status(inlink, AVERROR_EOF); + return 0; + } + + ret =3D ff_inlink_consume_frame(inlink, &frame); + if (ret < 0) { + return ret; + } + if (ret > 0) { + av_log(ctx, AV_LOG_TRACE, "out0,1,2 =3D %d %d %d total =3D %d\n", + s->nb_output0, s->nb_output1, s->nb_output2, + ctx->nb_outputs); + + av_log(ctx, AV_LOG_DEBUG, "ni_split: filter_frame, in format=3D%d,= Sctx %d\n", + frame->format, + s->src_ctx.enabled); + + if (frame->format =3D=3D AV_PIX_FMT_NI_QUAD && s->src_ctx.enabled = =3D=3D 1) + { + ret =3D filter_ni_frame(inlink, frame); + } + else + { + ret =3D filter_std_frame(inlink, frame); + } + + av_frame_free(&frame); + if (ret < 0) { + return ret; + } else { + ff_filter_set_ready(ctx, 300); + } + } + + if (ff_inlink_acknowledge_status(inlink, &status, &pts)) { + for (int i =3D 0; i < ctx->nb_outputs; i++) { + if (ff_outlink_get_status(ctx->outputs[i])) { + continue; + } + ff_outlink_set_status(ctx->outputs[i], status, pts); + } + return 0; + } + + for (int i =3D 0; i < ctx->nb_outputs; i++) { + if (ff_outlink_get_status(ctx->outputs[i])) { + continue; + } + + if (ff_outlink_frame_wanted(ctx->outputs[i])) { + ff_inlink_request_frame(inlink); + return 0; + } + } + + return FFERROR_NOT_READY; +} + +#define OFFSET(x) offsetof(NetIntSplitContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption ni_split_options[] =3D { + { "output0", "Copies of output0", OFFSET(nb_output0), AV_OPT_TYPE_INT,= {.i64 =3D 2}, 0, INT_MAX, FLAGS }, + { "output1", "Copies of output1", OFFSET(nb_output1), AV_OPT_TYPE_INT,= {.i64 =3D 0}, 0, INT_MAX, FLAGS }, + { "output2", "Copies of output2", OFFSET(nb_output2), AV_OPT_TYPE_INT,= {.i64 =3D 0}, 0, INT_MAX, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(ni_split); + +static const AVFilterPad inputs[] =3D { + { + .name =3D "default", + .type =3D AVMEDIA_TYPE_VIDEO, + .config_props =3D config_input, + }, +}; + +FFFilter ff_vf_split_ni_quadra =3D { + .p.name =3D "ni_quadra_split", + .p.description =3D NULL_IF_CONFIG_SMALL( + "NETINT Quadra demux input from decoder post-processor unit (PPU) = to N video outputs v" NI_XCODER_REVISION), + .p.priv_class =3D &ni_split_class, + .priv_size =3D sizeof(NetIntSplitContext), + .init =3D split_init, + .uninit =3D split_uninit, + .p.flags =3D AVFILTER_FLAG_DYNAMIC_OUTPUTS, + .flags_internal =3D FF_FILTER_FLAG_HWFRAME_AWARE, + .activate =3D activate, + FILTER_INPUTS(inputs), + FILTER_QUERY_FUNC(query_formats), +}; -- = 2.25.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".