That's botshit!
How to avoid your next bad impression caused by a chatbot.
The Intelligent Friend - The newsletter about the social, psychological and relational aspects of AI, based only on scientific papers.
Intro
Imagine that you have been working for the last three nights on a very important presentation. You are exhausted, but at the same time pumped up and ready to challenge everyone and show how much you are worth. Check the file again, everything is ok. The animations work. The information is all there. References are visible and inserted. The moment of presentation arrives. You do great, you're in the flow, you illustrate the concepts clearly and convincingly. Then comes the Q&A. They ask you a question about a reference they found strange. Go check it out. You check, but you can't find the reference. Then you remember: you had found that reference thanks to ChatGPT, and due to your haste you weren't able to verify it. The beautiful presentation remains, but you feel devastated by the bad impression.
The paper in a nutshell
Title: Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. Authors: Annigan et al. Year: 2024. Journal: / (working paper, coming soon on Business Horizons).
Main result: the authors build a framework that helps us detect and better manage the risks deriving from the wrong results generated by chatbots.
The right or the wrong joke?
Is the situation described familiar to you? I immagine anyone can say that in at least one ChatGPT interaction they have not misquoted a source or conveyed the wrong concept. This phenomenon, widely known as hallucination, is one of the most felt among people and widely reported. But as much as we may laugh about it, this can have very serious impacts on our work. In a striking decision, the Federal District Court in New York penalized two attorneys for filing a legal brief that was filled with fabricated cases and references, all of which were produced by ChatGPT1. Today's paper, which I found fascinating and interesting, provides a very important framework on how to best manage the risks deriving from hallucinations, coining the concept of botshit. Let's go find out.
Now, before we begin, I would like to make a specific. In this issue I will use the terms 'botshit' and 'bullshit' in a scientific way. I know it sounds like ironic, but it's the best way to actually illustrate what we're talking about.
Having said that, let's start: what is 'botshit'? Let's go back to the previous example. The moment you asked ChatGPT for information and he gave it incorrectly, it 'hallucinated'. As reported by the authors, by hallucination we mean "when an LLM generates seemingly realistic responses that are untrue, nonsensical, or unfaithful to the provided source input".2
Now, this is where the role of the chatbot stops. It's up to us to decide how to use the information, which we don't actually know is wrong. If you think about it, this thing has no impact or effect until we use it. ChatGPT can tell us that eating a lot of sugar is good for us, but if we don't eat it this only gives us temporary information. Or, as reported in the literature, it is a 'provisional knowledge'3.
However, when we incorporate that concept or suggestion into an action, into something that has an actual impact, the hallucination turns into 'botshit'. A botshit is nothing more than a hallucination that is used for a task non-critically by a human.
As mentioned, botshit can potentially have a very serious impact, and is instead distinguished from bullshit, which we define scientifically as "human-generated content that has no regard for the truth, which a human then uses in communication and decision-making tasks"45. That is: we say things without actually knowing or giving importance to the fact that they are untrue.
A real risk
Now, in their work the authors focus on a specific type of risk, namely the 'epistemic' one. By epistemic we mean very simply the risk linked to the production of knowledge. We who go to the presentation without verifying the source of what is suggested to us and then communicate it by incorporating it into the presentation itself is precisely the synthesis of an epistemic risk. We are running the risk of generating incorrect knowledge due to a chatbot.
But before going to discover the fascinating framework of scholars, I would like to focus on a question that I know will attract your attention: why do people say bullshit? And what are their characteristics? Funny as they may seem, the questions have important social implications and have seen a wide following among scholars. Among the most interesting reasons reported are these:
People tend to resort to fabricating information when they face strong social or professional pressure to express an opinion, especially if they believe their statements will be readily accepted as true without scrutiny6;
Workplace bullshitting is a social practice often necessary to navigate the professional environment. It can serve as a strategic language game to assimilate, accomplish tasks, and elevate one's status. Consequently, this behavior encourages others to engage in similar tactics to achieve comparable benefits;
Both bullshit and bot-generated content are more convincing and influential when they meet three criteria: (1) they are useful, beneficial, or invigorating for the audience; (2) they resonate with and flatter the audience's interests, beliefs, experiences, or attitudes; and (3) they appear credible due to their articulate presentation, the use of impressive jargon, and the perceived authority of the individual or chatbot producing them;
We are more inclined to believe a bullshit statement if we perceive it to be made by someone with prestigious scientific standing7. This is known as the 'Einstein Effect' (very relevant in the time of COVID-19, right?).
I wanted to make this specification because knowing the phenomenon well also helps to identify the risks, as specified by the authors. Therefore, starting from the risk management literature, the authors have built a framework to help individuals and organizations deal with the possibilities of botshit. This framework is made up of two dimensions:
"How important is chatbot response veracity for the task?";
"How easy is it to verify the veracity of the chatbot response?".
By crossing these two dimensions, four very different ways of operating with chatbots arise. Let's think about it. It is normal that if the risk associated with botshit is an unsound discussion with the long-time friend, we will not worry much and the impact will be limited. the case of our presentation that we mentioned previously is different. Similarly, if it is a date I can easily verify the information, but the case is different if I have asked for 50 references and I have 10 minutes to go and verify them.
Four modes of operating
From the intersection of the two dimensions four different situations arise, which you will probably see again in many situations of use of chatbots. It is very important to remember that this perspective is risk-oriented: when the risk severity of using chatbot content for work is catastrophic, ensuring the veracity of chatbot responses becomes essential in high-stakes investment decisions, mission-critical operations, and situations with little to no tolerance for failure. At the same time, When the risk severity of using chatbot content for work is low, the accuracy of chatbot responses becomes less important.
Furthermore, in addition to severity, as we specify, we consider the ease of verification. According to what the authors report, in risk management research, this concern is referred to as risk detection. Relatively difficult veracity verification occurs when it is costly and time-consuming to gather and confirm the truth claims surrounding the response content.
Let's find out in detail the different situations that can occur and how to deal with them:
Authenticated chatbot work: in this case the user sees both the veracity of the response as very crucial and as difficult to verify. Usually we find ourselves in a very delicate situation, in which there are tasks related to the legal or budgeting field, and in these specific cases the user must pay maximum attention when using information that can turn into botshit;
Automated chatbot work: in this case we have high relevance but at the same time the information is easy to verify. It is the classic case of routine tasks that we efficiently entrust to virtual assistants;
Augmented chatbot work: while in this case the fact that what we receive is correct is not important, at the same time we also have no simple ways of verifying its correctness. These are classic cases of brainstorming or organizing generated ideas. In this case it is very important to be aware of this dimension of difficulty and act above all to stimulate ideas and open perspectives, without immediately moving on to including information in the products (or tasks) we are working on;
Finally, autonomous chatbot work: it is easy to verify what the chatbot actually said, and at the same time the correctness is not actually relevant. Here we rely almost completely on the chatbot, which becomes our assistant in every way.
In addition to illustrating in detail these four 'ways' of operating with chatbots, the authors also identify possible strategies to reduce risks, categorized into three fundamental categories:
Technology-oriented guardrails: they help ensure that the mechanics and scope of a Large Language Model (LLM) are suitable for the specific type of chatbot work it is being utilized for;
Organization-oriented guardrails: these are the guidelines and policies that organizations develop to mitigate the risks associated with chatbot usage, and help ensure that chatbots are used appropriately and acceptably, focusing on the veracity, integrity, and responsible use of chatbot-generated content;
User-oriented guardrails: these pertain to the skills and practices that human users should employ to mitigate the risks of erroneous chatbot output in the workplace. Indeed, varying levels of critical thinking and fact-checking are expected for each of the four modes of chatbot work in our classification.
Take-aways
Beware for botshit! The use of chatbots can generate incorrect information (hallucinations) which when used by humans become 'botshit';
An impactful framework. Based on two dimensions, namely the ease of verifying the information provided by the chatbot and the importance of its correctness, the authors build a framework to orient yourself in four different 'ways' of working with chatbots;
Useful strategies. Finally, the authors identify three types of strategies useful for mitigating the risks associated with botshit: technology-oriented guardrails, organization-oriented guardrails, user-oriented guardrails.
Thank you for reading this issue of The Intelligent Friend and/or for subscribing. The relationships between humans and AI are a crucial topic and I am glad to be able to talk about it having you as a reader.
Has a friend of yours sent you this newsletter or are you not subscribed yet? You can subscribe here.
Surprise someone who deserves a gift or who you think would be interested in this newsletter. Share this post to your friend or colleague.
P.S. If you haven't already done so, in this questionnaire you can tell me a little about yourself and the wonderful things you do!
Weiser, B. (2023, June 22). ChatGPT lawyers are ordered to consider seeking forgiveness. The New York Times. Available at https://www.nytimes.com/2023/06/22/nyregion/ lawyers-chatgpt-schwartz-loduca.html
Alkaissi, H., & McFarlane, S. I. (2023, February 19). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus. Available at https://doi.org/10.7759/cureus.35179
Hannigan, T. R., Seidel, V. P., & Yakis-Douglas, B. (2018). Product innovation rumors as forms of open innovation. Research Policy, 47(5), 953-964.
Frankfurt, H. G. (2005). On bullshit. Princeton University Press.
McCarthy, I. P., Hannah, D., Pitt, L. F., & McCarthy, J. M. (2020). Confronting indifference toward truth: Dealing with workplace bullshit. Business Horizons, 63(3), 253-263.
Petrocelli, J. V. (2018). Antecedents of bullshitting. Journal of Experimental Social Psychology, 76, 249-258.
Hoogeveen, S., Haaf, J. M., Bulbulia, J. A., Ross, R. M., McKay, R., Altay, S., ... & van Elk, M. (2022). The Einstein effect provides global evidence for scientific source credibility effects and the influence of religiosity. Nature Human Behaviour, 6(4), 523-535.