Is my understanding wrong that LLMs are trained to emulate observed human behavior in their training data?
From that follows that LLMs fit to produce all kinds of human biases. Like preferring the first choice out of many, and the last our of many (primacy biases). Funnily the LLM might replicate the biases slightly wrong and by doing so produce new derived biases.
This understanding is incomplete in my opinion. LLMs are more than emulating observed behavior. In the pre-training phase tasks like masked language model indeed train the model to mimic what they read (which of course contains lots of bias); but in the RLHF phase, the model tries to generate the best response judged by human evaluations (who tries to eliminate as much bias as possible in the process). In other words, they are trained to meet human expectations in this later phase.
But human expectations are also not bias-free (e.g. from the preferring-the-first-choice phenomenon)
How can the RLHF phase eliminate bias if it uses a process(human input) that has the same biases as the pre-training(human input)?
During RLHF, the human evaluators are aware of such biases and are instructed to down-vote the model responses that incorporate such biases.