Please, the entirety of Hong Kong is like 40x50km.
There sure are some tall steep hills making some areas just not worth messing with.
Some of the outlining villages that are heavily populated are regarded as “remote” because they have limited public transport infrastructure. The density of the city combined with the mountainous topology makes it impossible to directly go to your destination.
Then, these abandoned villages are on outlying islands from these heavily populated villages. You need to hire a private charter boat or a “tour service” to get to.
So it can take 1-2 hours from the city center to get to one of the villages (if you’re going to Sai Kung Village, you’re on a single road prone to congestion during peak traffic), then you’re taking another 1-2 hour ride on a boat to the islands you want to go to.
Certainly possible as day trips, but also certainly an inconvenience.
People commute from Pakenham to Melbourne (~1:20), from Alpharetta to Atlanta (~1:30) etc.
For Hong Kong, the density and “business” driven expectations of the city make it so that a 1-2 hour journey sitting on a bus is considered a long route. People do not want to waste time traveling within the city.
It is common for some people to roll their eyes at the thought of traveling to Kowloon side if they live on Hong Kong side, even if it’s a 5minute train / 15 minute ferry trip.