I don't understand how the details of the build process matter if the resulting files can be checked to be bit by bit identical? I can only think of something like Signal and Google conspiring to backdoor the binaries during the build process via this external binary blob. But if Google is part of this, they could also do it within Android which is not fully open source.

If you don't like this, you use the non-Play Store build instead (which supposedly doesn't include any binary blobs, but I haven't checked).