Solved with the hints from @zevele and @00lex! 
@Maurizion you probably were fooled, like me, by the fact that we might have a 64bit kernel, but the userland is only 32bit, so all compiles should be done for the 32bit version.
Doing a recap for the newcomers, to fix building of ha-av in HA 2023.6.x on a Core installation on Raspbian OS, you should do only the following part from the aforementioned guide, compiling only ffmpeg with the basic set of libraries (@00lex please correct me if I’m wrong). You should choose the most appropriate Git branch for your scenario altering the --branch release/6.0 option from the git clone command:
sudo apt -y install autoconf automake build-essential cmake doxygen git graphviz imagemagick libasound2-dev libass-dev libavcodec-dev libavdevice-dev libavfilter-dev libavformat-dev libavutil-dev libfreetype6-dev libgmp-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libopus-dev librtmp-dev libsdl2-dev libsdl2-image-dev libsdl2-mixer-dev libsdl2-net-dev libsdl2-ttf-dev libsnappy-dev libsoxr-dev libssh-dev libssl-dev libtool libv4l-dev libva-dev libvdpau-dev libvo-amrwbenc-dev libvorbis-dev libwebp-dev libx264-dev libx265-dev libxcb-shape0-dev libxcb-shm0-dev libxcb-xfixes0-dev libxcb1-dev libxml2-dev lzma-dev nasm pkg-config python3-dev python3-pip texinfo wget yasm zlib1g-dev libdrm-dev
git clone --branch release/6.0 --depth 1 https://github.com/FFmpeg/FFmpeg.git ~/FFmpeg
cd ~/FFmpeg
./configure \
--extra-cflags="-I/usr/local/include" \
--extra-ldflags="-L/usr/local/lib" \
--extra-libs="-lpthread -lm -latomic" \
--arch=armel \
--enable-gmp \
--enable-gpl \
--enable-libass \
--enable-libdrm \
--enable-libfreetype \
--enable-libmp3lame \
--enable-libopencore-amrnb \
--enable-libopencore-amrwb \
--enable-libopus \
--enable-librtmp \
--enable-libsnappy \
--enable-libsoxr \
--enable-libssh \
--enable-libvorbis \
--enable-libwebp \
--enable-libx264 \
--enable-libx265 \
--enable-libxml2 \
--enable-mmal \
--enable-nonfree \
--enable-version3 \
--target-os=linux \
--enable-pthreads \
--enable-openssl \
--enable-hardcoded-tables \
--enable-pic \
--disable-static \
--enable-shared
make -j$(nproc)
sudo make install
sudo ldconfig