Secure element updates don't only need to have a valid signature and greater version. They also require the Owner user to authenticate successfully after booting in order for it to be accepted. This is what they refer to as insider attack resistance, since it protects against them being coerced by a government into removing the brute force protection for a locked device via an update.
GrapheneOS System Updater doesn't identify the device or user to the server. A massive portion of GrapheneOS users are using a VPN and some are using Tor so many of the IP addressed are VPN/Tor exit IPs shared between people. How would an update be targeted to a specific phone?