[FFmpeg-devel] [RFC] fate rsync switch to git

[FFmpeg-devel] [RFC] fate rsync switch to git

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 817 bytes --]

Hi all

I did hear (at fosdem?)
about the idea to switch from rsync to git for managing the fate samples
i thought the idea sounds interresting but isnt rsync more efficient ?

If someone wants to try (it would be interresting to see how it
compares to rsync in terms of bandwidth, speed, latency, load, diskspace)

FFmpeg and fateserver git are here:

https://git.ffmpeg.org/ffmpeg.git
https://git.ffmpeg.org/fateserver.git

I do remember we had some embeded systems and VMs that where short on
disk space.

thx
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you fake or manipulate statistics in a paper in physics you will never
get a job again.
If you fake or manipulate statistics in a paper in medicin you will get
a job for life at the pharma industry.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[Jan Ekström]

On Tue, Feb 20, 2024 at 4:31 PM Michael Niedermayer
<michael@niedermayer.cc> wrote:
>
> Hi all
>
> I did hear (at fosdem?)
> about the idea to switch from rsync to git for managing the fate samples
> i thought the idea sounds interresting but isnt rsync more efficient ?
>

Do note that the idea was that this would only be for management of
the main archive, so it would not affect clients/runners rsync'ing
from the main archive.

Of course clients which want to sync directly from git could do that,
but the idea would be to keep the sync requirements same for FATE
clients/runners: if you are only running tests, rsync is enough.

As after all, the primary reasons for having the samples in git would
be versioning, more concrete known states in a public archive (I would
probably not call this a "backup", but it would mean we would have the
history in multiple places at least), as well as - if we utilize
something like git{lab,hub} - easier workflow to adding new samples by
means of f.ex. merge/pull requests.

This idea originated from looking at how the dav1d project handled
their reference sample suite, which seems to have served them well
enough: https://code.videolan.org/videolan/dav1d-test-data

Regards,
Jan
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[Niklas Haas]

On Tue, 20 Feb 2024 20:43:30 +0200 Jan Ekström <jeebjp@gmail.com> wrote:
> Do note that the idea was that this would only be for management of
> the main archive, so it would not affect clients/runners rsync'ing
> from the main archive.
> 
> Of course clients which want to sync directly from git could do that,
> but the idea would be to keep the sync requirements same for FATE
> clients/runners: if you are only running tests, rsync is enough.
> 
> As after all, the primary reasons for having the samples in git would
> be versioning, more concrete known states in a public archive (I would
> probably not call this a "backup", but it would mean we would have the
> history in multiple places at least), as well as - if we utilize
> something like git{lab,hub} - easier workflow to adding new samples by
> means of f.ex. merge/pull requests.
> 
> This idea originated from looking at how the dav1d project handled
> their reference sample suite, which seems to have served them well
> enough: https://code.videolan.org/videolan/dav1d-test-data
> 
> Regards,
> Jan

Is there any reason (besides efficiency hit) not to make the FATE repo
a `git submodule` of the FFmpeg git repo? That way, commits which depend
on certain additions to fate-samples can explicitly depend on the
commits adding those files, developers can more easily see (e.g. via
`git status`) if the fate samples are out-of-date (or use `git pull
--recurse-submodules` to automate the process).

It will also make the samples repo historically consistent, e.g. if
somebody changes a detail about a file in a later commit, older commits
referencing the unmodified version will continue passing FATE tests. I'm
not sure if this has ever been a concern in the past, but it may well
be one in the future.

Worrying about the performance impact of rsync vs git-lfs (or equivalent
solutions) seems like premature optimization to me; and the ease of
maintenance, historical consistency, transparency in process, and
end-user convenience of a git repository seems to far outweigh the
drawbacks.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[epirat07]

On 21 Feb 2024, at 15:38, Niklas Haas wrote:

> On Tue, 20 Feb 2024 20:43:30 +0200 Jan Ekström <jeebjp@gmail.com> wrote:
>> Do note that the idea was that this would only be for management of
>> the main archive, so it would not affect clients/runners rsync'ing
>> from the main archive.
>>
>> Of course clients which want to sync directly from git could do that,
>> but the idea would be to keep the sync requirements same for FATE
>> clients/runners: if you are only running tests, rsync is enough.
>>
>> As after all, the primary reasons for having the samples in git would
>> be versioning, more concrete known states in a public archive (I would
>> probably not call this a "backup", but it would mean we would have the
>> history in multiple places at least), as well as - if we utilize
>> something like git{lab,hub} - easier workflow to adding new samples by
>> means of f.ex. merge/pull requests.
>>
>> This idea originated from looking at how the dav1d project handled
>> their reference sample suite, which seems to have served them well
>> enough: https://code.videolan.org/videolan/dav1d-test-data
>>
>> Regards,
>> Jan
>
> Is there any reason (besides efficiency hit) not to make the FATE repo
> a `git submodule` of the FFmpeg git repo? That way, commits which depend
> on certain additions to fate-samples can explicitly depend on the
> commits adding those files, developers can more easily see (e.g. via
> `git status`) if the fate samples are out-of-date (or use `git pull
> --recurse-submodules` to automate the process).

I am all for having it in git but do not like the idea of a git submodule
at all as they are a nightmare to work with, sometimes create absolutely
unworkable conflicts when rebasing and other oddities…

(We use submodules for the Icecast project, it was my idea back then and
I regret it…)

>
> It will also make the samples repo historically consistent, e.g. if
> somebody changes a detail about a file in a later commit, older commits
> referencing the unmodified version will continue passing FATE tests. I'm
> not sure if this has ever been a concern in the past, but it may well
> be one in the future.
>
> Worrying about the performance impact of rsync vs git-lfs (or equivalent
> solutions) seems like premature optimization to me; and the ease of
> maintenance, historical consistency, transparency in process, and
> end-user convenience of a git repository seems to far outweigh the
> drawbacks.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[Vittorio Giovara]

On Wed, Feb 21, 2024 at 3:50 PM <epirat07@gmail.com> wrote:

>
>
> On 21 Feb 2024, at 15:38, Niklas Haas wrote:
>
> > On Tue, 20 Feb 2024 20:43:30 +0200 Jan Ekström <jeebjp@gmail.com> wrote:
> >> Do note that the idea was that this would only be for management of
> >> the main archive, so it would not affect clients/runners rsync'ing
> >> from the main archive.
> >>
> >> Of course clients which want to sync directly from git could do that,
> >> but the idea would be to keep the sync requirements same for FATE
> >> clients/runners: if you are only running tests, rsync is enough.
> >>
> >> As after all, the primary reasons for having the samples in git would
> >> be versioning, more concrete known states in a public archive (I would
> >> probably not call this a "backup", but it would mean we would have the
> >> history in multiple places at least), as well as - if we utilize
> >> something like git{lab,hub} - easier workflow to adding new samples by
> >> means of f.ex. merge/pull requests.
> >>
> >> This idea originated from looking at how the dav1d project handled
> >> their reference sample suite, which seems to have served them well
> >> enough: https://code.videolan.org/videolan/dav1d-test-data
> >>
> >> Regards,
> >> Jan
> >
> > Is there any reason (besides efficiency hit) not to make the FATE repo
> > a `git submodule` of the FFmpeg git repo? That way, commits which depend
> > on certain additions to fate-samples can explicitly depend on the
> > commits adding those files, developers can more easily see (e.g. via
> > `git status`) if the fate samples are out-of-date (or use `git pull
> > --recurse-submodules` to automate the process).
>
> I am all for having it in git but do not like the idea of a git submodule
> at all as they are a nightmare to work with, sometimes create absolutely
> unworkable conflicts when rebasing and other oddities…
>
> (We use submodules for the Icecast project, it was my idea back then and
> I regret it…)
>
> >
> > It will also make the samples repo historically consistent, e.g. if
> > somebody changes a detail about a file in a later commit, older commits
> > referencing the unmodified version will continue passing FATE tests. I'm
> > not sure if this has ever been a concern in the past, but it may well
> > be one in the future.
> >
> > Worrying about the performance impact of rsync vs git-lfs (or equivalent
> > solutions) seems like premature optimization to me; and the ease of
> > maintenance, historical consistency, transparency in process, and
> > end-user convenience of a git repository seems to far outweigh the
> > drawbacks.


I don't mind submodules but I understand the complications. Maybe we could
design a system in which samples have attached a hash/branch/ref so that
when checking out older/outdated samples the correct sample itself can
still be used.

But overall I agree it would be much preferable to have the management
system in git over rsync.
-- 
Vittorio
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[Jean-Baptiste Kempf]

Yo,

On Tue, 20 Feb 2024, at 15:31, Michael Niedermayer wrote:
> I did hear (at fosdem?)
> about the idea to switch from rsync to git for managing the fate samples
> i thought the idea sounds interresting but isnt rsync more efficient ?

Yes, that's my idea.

The git part is not for others clients to sync, but just to manage the samples.

It allows to have versioning of the samples, with the extra ability to have branches and tags matching the ffmpeg branches.
It avoids the weird dance "waiting for FATE admin to upload thing", because their would be multiple approvers/committers to that repo.
It simplifies the workflow on the admin side, since with a git hook, the server does "git update" in the directory that will be rsync'd by the mirrors. And it does not change their workflow, still using rsync.

just my 5c.

-- 
Jean-Baptiste Kempf -  President
+33 672 704 734
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[Zhao Zhili]

> On Feb 21, 2024, at 09:39, Jean-Baptiste Kempf <jb@videolan.org> wrote:
> 
> Yo,
> 
> On Tue, 20 Feb 2024, at 15:31, Michael Niedermayer wrote:
>> I did hear (at fosdem?)
>> about the idea to switch from rsync to git for managing the fate samples
>> i thought the idea sounds interresting but isnt rsync more efficient ?
> 
> Yes, that's my idea.
> 
> The git part is not for others clients to sync, but just to manage the samples.

I use git and git-lfs to manage/sync ffmpeg-fate-sample. It’s easy to setup a mirror
repo to avoid a lot of network issues at some countries.

> 
> It allows to have versioning of the samples, with the extra ability to have branches and tags matching the ffmpeg branches.
> It avoids the weird dance "waiting for FATE admin to upload thing", because their would be multiple approvers/committers to that repo.
> It simplifies the workflow on the admin side, since with a git hook, the server does "git update" in the directory that will be rsync'd by the mirrors. And it does not change their workflow, still using rsync.
> 
> just my 5c.
> 
> -- 
> Jean-Baptiste Kempf -  President
> +33 672 704 734
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] fate rsync switch to git

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2753 bytes --]

On Wed, Feb 21, 2024 at 02:39:42AM +0100, Jean-Baptiste Kempf wrote:
> Yo,
> 
> On Tue, 20 Feb 2024, at 15:31, Michael Niedermayer wrote:
> > I did hear (at fosdem?)
> > about the idea to switch from rsync to git for managing the fate samples
> > i thought the idea sounds interresting but isnt rsync more efficient ?
> 
> Yes, that's my idea.
> 
> The git part is not for others clients to sync, but just to manage the samples.
> 
> It allows to have versioning of the samples, with the extra ability to have branches and tags matching the ffmpeg branches.

versioning is fine

branches, we did not need, and adding branches sounds like a solution
searching for a problem


> It avoids the weird dance "waiting for FATE admin to upload thing", because their would be multiple approvers/committers to that repo.

we have 15 people in the samples group that have the power to upload samples,
the bottleneck is that only 2 of them actual do upload and noone else is
volunteering


> It simplifies the workflow on the admin side, since with a git hook, the server does "git update" in the directory that will be rsync'd by the mirrors. And it does not change their workflow, still using rsync.

It still requires people to push the new sample to git and that git is
a different architecture than how the samples normally where downloaded if you
keep rsync

I mean everyone has a rsync based checkout like now for "make fate"
updating with "make fate-rsync"

they add a sample to this and then they have no git to push that to
cuz, no .git directory is there.
This can be handled in a few ways but it mainly seems messy

what i suggest is that someone adds a
make fate-rsync-upload
that runs the rsync command
that way the other people who actually have fate samlpe upload rights
actually do the upload after testing a patch instead of waiting for 2 people
to do it.

Such "fate-rsync-upload" is easy and needs no change to architecture it also
doesnt make any future change to architecture any more difficult
so i suggets that to be done together with people volunteering to do fate sample
uploads themselfs if they freuqently review/approve patches that need such uploads

If people prefer switching to git then we need an volunteer to write and test the
related hook(s). I can create a git repo for it
Iam mainly hesitant now because it seems an additional layer not replacing a layer

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Rewriting code that is poorly written but fully understood is good.
Rewriting code that one doesnt understand is a sign that one is less smart
than the original author, trying to rewrite it will not make it better.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".