The issue is that avci->skip_samples will be overridden by any side-data.

When operating on raw files (adts, for example), the decoder is free
to decide the amount of samples to skip. Usually, this is the algorithmic
delay of the decoder.

When operating on more complete containers, like ISOBMFF, the amount of
samples to be skipped is recorded and signalled by the encoder.

However, it turns out many encoders have an arbitrary choice of padding
to insert at the start. Normally, they would signal the amount into
the container. But with ISOBMFF, there isn't just a single option -
the format has been extended multiple times, and has multiple ways
to signal padding. In the case of fdkaac-encoded samples, the STTS
is used, rather than the CTTS, which ends up with us leaving the padding
in.

But it's not just containers, as it turns out, most AAC encoders use
an arbitrary amount of padding at the start that may, or may not be
trimmed (usually, it won't be).

Furthermore, AAC has specific amount of algorithmic delay for SBR
operation. This delay is not accounter for anywhere. While it's an
option to skip the samples in the decoder, doing this in decode.c,
along with the rest of the skip adjustments, is a neater way, and
can be extended to other codecs.

Patch attached.