Even if you compile your own version of Signal, will your friends do it too? Will your grandma/grandpa do it as well? It only takes one person in the chain to be compromised by using the "real" app and then all your efforts would be defeated because now your messages have been exposed by this other person unknowingly.
Do phones have trusted execution environments? I suppose you could require the recipient provide attestation that it's running the expected binary. Of course, this is pointless if the hardware manufacturer shares their root keys with the government.
The backdoored app will hopefully not be called Signal, since Signal themselves would never do this. I hope they own a trademark on it and could enforce it against anyone who would try to upload a backdoored version under their name.