Bag of words, have mercy on us
I think this approach of thinking of LLMs as "bags of words" or, I might say boxes of text, is appropriate. AIs aren't honest or dishonest, but the box of text may be biased in one way or another. To me, the Internet is biased toward a lot of BS and inaccuracy. That's what I expect from LLMs.
Since source texts of varying quality are all mixed up in hard to understand ways, it's hard for the user to evaluate the quality of the LLM output. People who don't understand that rely on an appeal to authority, even though an LLM is not necessarily a good authority on anything.
Anthropomorphising AI is only natural, but it's interesting how AI seeming to be smart makes people think of it as an authority or even godlike.