avfilter/vf_libvmaf: fix filtering of >8 bit data