RDFTs are full of conventions that vary between implementations.
What I've gone for here is what's most common between
both fftw, avcodec's rdft and what we use, the equivalent of
which is DFT_R2C for forward and IDFT_C2R for inverse. The
other 2 conventions (IDFT_R2C and DFT_C2R) were not used at
all in our code, and their names are also not appropriate.

Like fftw, the input is written over with junk data.
Like avcodec's rft, the output of a C2R transform is
non-normalized (half the amplitude of the input).

Patch attached.