Indeed a golden factory firmware version that will be booted automatically if all else fails and that provides minimum connectivity is crucial.
> will be booted automatically if all else fails
I prefer to keep the factory firmware reset to a manual process that requires user intervention.
For example, holding down the reset button for 10 seconds after plugging the device in.
In my experience, it's not a good idea to have a device automatically roll back firmware and erase user data after failed boots. These mechanisms get triggered too easily during certain power outages (power comes on then goes off just long enough to cause multiple failed boots) or when users are doing simple things like rearranging their power cables.
Reverting to factory state is the last resort. You don't have users do it unless there is no other good state to return to on the device.
> Just because that version worked a decade ago, it doesn’t mean it’s compatible with the world today.
That's why I said you have to include this in your test procedures.
When you're planning for the long term you can accommodate for these things on your servers.