Bias in AI-Enabled Chatbots: Implications, Causes, and Solutions

Abstract

Chatbots have been around for decades. However, the advancement in Natural Language Processing, Artificial Intelligence, and an explosion in the availability of data, which came with the advent of social media, have enabled a shift from command-driven chatbots to more ‘intelligent’ ones that much better at determining a user’s goal as well as context. 

While finding application in varied fields, AI-enabled chatbots have shown the tendency to internalize and reproduce societal biases, bigoted points of view, and profanities. Such reproductions have far-reaching implications in a world where human interactions are getting increasingly mediated by algorithms.

This paper undertakes a literature review and uses real-world examples to underline that the behavior of chatbots reflects our societal biases and it cannot be dealt with in isolation. It proposes the need to work on both design defects in the algorithm itself and also on the way data is chosen for training the algorithms.

Introduction

A revolution of sorts is unfolding in the digital space. Our online interactions with service providers, sellers, government agencies, and organizations is being increasingly mediated by algorithm-driven chatbots (Solomon, 2017). Chatbots can be defined as programmed software or conversational agents that can read and write messages “autonomously”, similar to how humans can perform a task (Mladenic, 1999). Simply put, a chatbot is a computer program that users can interact with in an informal, conversational way.

Industry surveys have found that chatbots, as conversational agents, are seen to enhance and improve customer interactions. According to a 2018 report by Gartner, it has been predicted that by the end of 2020, chatbots will power 85% of all customer service interactions (Hinds, 2018; Mor, 2017). Although this might seem improbable in the current scenario, it can be a possibility in the near future. From the historic 1960s Eliza to 2000s SmarterChild to development frameworks like IBM’s Watson and Microsoft’s Bot Framework, chatbots are a growing part of everyday computing (Hard, 2016). They can not only “mimic conversations” but can also “offer instant, digital connections” (Goasduff, n.d.).

The advance in the development of smart chatbots is tied intrinsically to the recent developments in the availability of Natural Language Processing (NLP) capabilities. At its core, NLP is the ability of taking a body of human-generated text and rendering it into machine readable language. Although computer scientists have worked on Natural Language Processing (NLP) since the 1950s, it was only with recent advances in machine learning and the availability of vast amounts of digital text and conversational data that has enabled engineers to make progress in NLP (Abdul-Kader & Woods, 2015; Olsson, 2009).

This paper explores different types of bots and discusses the implications and causes of biases in chatbots. Furthermore, it makes suggestions and recommends probable solutions that can help create intelligent chatbots with minimal biases.

Types of Chatbots

Chatbots can be categorized both on the level of sophistication of their underlying algorithm and on the kind of solutions they offer (Deloitte, 2018). In terms of sophistication, a chatbot may be as basic as a simple scripted chatbot, which looks for keywords and phrases in the input and provides predefined responses. Scripted bots, thus, have limited ability to carry on intelligent conversations as they are based on fixed, predefined scripts and templates that are programmed at the time of coding them. Every input from the user prompts the chatbot to take a particular action. One example of such a scripted bot is Kik, a bot maintained by leading makeup brand Sephora. It asks users questions about their makeup preferences and, based on the response, comes up with predefined recommendations.

At the other end of this spectrum are the intelligent chatbots like Alexa or Siri, which, according to a whitepaper by Deloitte (2018), employ “sophisticated weaving” of AI techniques “with complex conversational state tracking and integration into existing business services.” They are not only smart at handling queries but also adept in learning and saving new things with every conversation and using them “appropriately for future instances” (Marupaka, 2018). For example, Mitsuku, an award-winning human-like intelligent chatbot, understands a user’s mood and can chat with them about anything.

When viewed in terms of the solution a chatbot offers, they can be seen as coming in various degrees of services they offer. Such chatbots can vary from simple FAQ chatbots that can understand simple questions, search for, and respond with the most relevant answers, to virtual agents which can handle the entire functional role of an employee. For the purposes of this paper, I will be focusing on intelligent chatbots that forms the latter category of both the categorizations above.

In general, intelligent bots, are powered by AI and Machine Learning algorithms which leverage Natural Language Processing and are good at unsupervised learning. These chatbots are an improvement over the conventional scripted chatbots, which try to match user prompts to scripted responses. These scripts can range from simple to complex, depending on the choice of the programmer, but the chatbot cannot provide answers to questions for which answer scripts have not been coded.

Intelligent chatbots, on the other hand, have the ability to learn as they communicate. This means that every interaction they have makes them progressively more intelligent. Therefore, they are capable of coming up with handling new situations more effectively and give nuanced replies and suggestions (Hill, Ford, & Farreras, 2015).

 An intelligent bot can improve the user experience, as it is capable of a more “human” way of communication, by mimicking quirks of a human conversation such as employing humor and engaging in small talk (Toscano, 2016). These bots use new data to continuously train themselves and they are therefore faster and more efficient in incorporating new information.

While an improvement over scripted bots, intelligent bots have seen their own vulnerabilities exposed by fiascos such as the one that happened with Tay. Tay was Microsoft’s experimental AI-enabled chatbot, touted as being “smart” and “adaptive” (Mathur, Stavrakas, & Singh, 2016). It was released to the public on Twitter on 23 March 2016 for the purpose of entertainment and engagement. It started nicely and was able to engage its audience with coherent, intelligent, and nuanced replies. But then, within 16 hours of its launch, it quickly internalized the information fed to it by group of trolls and began a diatribe of racist, anti-Semitic, and sexist tweets (Neff & Nagy, 2016). In fact, Microsoft had to shut down the service in less than twenty-four hours of its launch. This incident was brought into limelight by AI experts to highlight a major flaw in intelligent bots: that they are prone to internalizing biases as they attempt to deliver relevant and engaging content.

Bias in Bots: Sources and Reasons

The question arises: what introduces biases in the so-called “intelligent” chatbots? To fully understand all the sources of biases, one must understand the constituents that make up a chatbot. According to a report by Deloitte (2017), a typical chatbot consists of three key components: (1) an interface, which is the front-end component that a user sees. It can be standalone or can be connected to social media, websites of service providers, email accounts, or messaging applications such as Facebook Messenger or Slack; (2) the Natural Language Processing module. This part is responsible for taking in a phrase input by the user into a structured object in a machine-readable code; and (3) a component that manages the content that the chatbot would use to respond to a human user. This repository of replies, in an intelligent chatbot, increases as it learns more ways of answering and keeping track of context (Deloitte, 2017).

The present generation of intelligent chatbot systems therefore have a machine learning algorithm driving their core. These algorithms need to be taught how to spot patterns in data, and this is normally done by feeding them large bodies of data, commonly known as training datasets. These systems are designed to spot patterns, and if the data is unrepresentative, or if the patterns reflect historical patterns of prejudice, then the decisions which they make may be unrepresentative or discriminatory as well. This can present problems when these systems are relied upon to make real world decisions.

 Within the AI community, this is commonly known as ‘bias’. Research (Caliskan, Bryson, & Narayanan, 2017) has also supported the hypothesis that applying machine learning to everyday human language reproduces existing societal bias. An empirical proof that could be cited to support the above conclusions is the case of Tay. For Tay, a major contributing factor to the corrupt, abusive, hate-speech was the actual text the bot was trained on. Tay learned from its users, including 4chan users. 4chan is a forum where users can anonymously post anything online, without the risk of accountability. 4chan users, as found in some studies, are notorious for using hateful, bigoted phrases and words to attack people of color and women. In another experiment researchers at MIT proved the criticality of data by training a psychopathic AI named Norman (Church, 2018). They used content from a Reddit thread about gruesome deaths to teach Norman and then gave him a Rorschach test. This psychological test, developed in 1921, helps measure thought disorder. Researchers found that Norman came up with gory images, as compared to the non-violent things other AIs would describe in the test.

But all the blame cannot lie on the door of the training data. Many opinions (Woolley & Howard, 2016; Neff & Nagy, 2016) point to the underlying algorithm as an equal culprit. For example, the fact that Microsoft had apparently failed to deploy a blacklist to moderate hate speech in Tay, which could censure words, was pointed out as a possible cause for Tay going rogue. In addition, according to a written evidence from Research Councils in UK (2018), some of the “bias and discrimination may emerge only when an algorithm processes particular data”. For example, in recent times, a controversy erupted when it emerged that Google’s visual identification algorithm was repeatedly failing to differentiate between gorillas and Black people (Snow, 2019). In the end, Google had to disable this feature of its algorithm.

 However, to understand the true genesis of the bias in a bot we must develop an understanding of how the digital identity of algorithms interact with the physical and the digital worlds. Learning from this work helps us understand that bias, identity, justice, and power are systemically entangled within our technology. Scholars like McPherson (2013) and Coleman (2009) underline the fact that bias in machines “cannot be treated in general, abstract ways or simply erased. We need to understand the specificities of the worlds we live in to respond to bias. We need to stay with the trouble” (Schlesinger, O'Hara, & Taylor, 2018).

Dangers of Biases in Bots

Typically, chatbots are “set up following a one-size-fits-all approach in which all users, regardless of needs, preferences, and degrees of digital literacy receive responses in the same language by way of the same underlying set of data and services” (Følstad & Brandtzæg, 2017). Thus, the threat of biases in AI-enabled chatbots raises several social, ethical, and even legal questions. To begin with, as AI makes inroads into our day-to-day lives, it is pertinent to worry if chatbots enabled by these smart algorithms, perpetuate and exacerbate social biases. Without a tempering human interference, chatbots that have internalized biases can keep using it unhindered to modify their interaction with users.

As the power of such platforms increases, they can become a source of reinforcement and mainstreaming of these fringe thoughts and philosophies such as gender bias, racism, and ethnic hatred. This is a real social problem and can lead to strife and tensions in the world we live in (Schlesinger et al., 2018).

As bots begin to take control of much of decision making and increasingly mediate social interactions, the issue of bias in chatbots can quickly lead to questions of societal ethics. (Dingler, Choudhury & Kostakos, 2018) Questions about fairness, transparency, and integrity are the first ones to get affected when a bot starts internalizing data that might train it to become biased and dishonest. This might undercut the faith users might have put into the bot. The chatbot algorithms should make provisions for incorporating enough checks to preclude bots from producing unethical behaviors (Alaieri & Vellino, 2016).   This is important not just for the reasons of good corporate behavior but also because it can have real commercial implications in the form of better market uptake.

Questions of ethics vis-a-vis a rogue chatbot brings into focus issues related to the legal implications of speeches and opinions as propagated by bots. Legal frameworks, as they exist in many jurisdictions, are inadequate or unprepared to deal with questions that might emerge from the actions of a biased or socially unacceptable behavior of a bot. For example, the kind of anti-Semitism and racist talk Tay was involved in might attract penalties and other legal punishments in the US. If spoken by a person, it can also be construed as a call to hate crime. However, when a bot, guided by algorithm, says the same thing, the legal system is in a fix on how to thrust responsibility for loss or damage caused by its speech.

Solutions and Conclusion

To comprehensively tackle the problem of bias in bots we must address the issue at the core of it. Most of chatbots that are branded as “intelligent” today are circumscribed by the constraints set by the machine learning algorithm and by the limitations in diversity of the world-view of the development team and the training data sets.

 However, even with the present constraints, the problem can be mitigated by adopting a tripartite strategy: (1) careful selection of data for training purposes, (2) due diligence in algorithm selection and design, and (3) incentivizing and rewarding responsible behavior by businesses. 

The “garbage in, garbage out” law points out that if we train machine learning or artificial intelligence algorithms on limited or biased data, it “will lead to biased models and discriminatory outcomes” (Kochi, 2018). Often the training data may tend to leave out economically or socially marginalized sections of societies as they are not active participants that are not part of processes that generate the data used for training the algorithm.  For example, people living in tribal areas and in many low-income countries may not find much representation. One of the most effective ways to build a relatively unbiased chatbot that can handle diversity is to train it on a database that reflects balanced and a wide variety of opinions. It is equally important to focus on training and sensitizing people developing databases for training a bot. A database team without concern for the racial, gender, or ethnic representation may produce a learning set that focuses only on what society deems normal: “white, cisgender, heterosexual men” (Ahmed, 2004).

The second stage of due diligence is the selection of algorithm. Algorithms are not universally successful or unfit. It is contextual. “Algorithms that work well in one context may discriminate in another” (Kochi, 2018). It is therefore important to involve a collaborative and comprehensive process to test and ascertain the fitness of the algorithm. 

Another issue that needs to be recognized is that one reason why avoiding objectionable talks is difficult for chatbots is because their understanding of the different contexts that their human counterparts use is often difficult to capture, even by the most sophisticated database. If we want to mitigate some of this difficulty, we must modify our orientation towards human-machine conversation. While it is difficult for one chatbot or database to achieve, a way forward is to use bots with specialized areas of expertise. Such specialized bots can then be trained in understanding context (Schlesinger et. al., 2018)

In conclusion, it is imperative to remember that chatbots talk and interact with real people. This fosters a condition were chatbots can become carriers of inherent social biases and divides. The presence of bias in intelligent chatbots is not just undesirable but also dangerous as it can have real world implications. However, removing it not something technology alone can solve.

It needs the coming together of sociologists, communities, computer scientists, and mathematicians. Sociologists, working with modern communities, and with their understanding of how social divides and biases get perpetuated can help mathematicians and computer scientists build algorithms that can counter bias more effectively. Recognizing this cross-cutting nature of the problem is the first step, but it is certainly an important one. An aware society can surely lead to more aware designing and training of not just intelligent but also aware chatbots. 

References

Abdul-Kader, S. A., & Woods, J. C. (2015). Survey on chatbot design techniques in speech conversation systems. International Journal of Advanced Computer Science and Applications, 6(7).
Ahmed, S. (2004). Declarations of whiteness: The non-performativity of anti-racism. borderlands, 3(2), 1-15.
Alaieri, F., & Vellino, A. (2016). Ethical decision making in robots: Autonomy, trust and responsibility. In International Conference on Social Robotics (pp. 159-168). Springer, Cham.
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186.
Church, N. (2018, June 06). MIT Creates Psychopath Artificial Intelligence. Retrieved from https://www.breitbart.com/tech/2018/06/05/mit-creates-psychopath-artificial-intelligence/
Coleman, B. (2009). Race as technology. Camera Obscura: Feminism, Culture, and Media Studies, 24(1 (70)), 177-207.
Conversational chatbots–Let's chat. (2017). Deloitte India. Retrieved from https://www2.deloitte.com/in/en/pages/strategy/articles/conversational-chatbots-lets-chat.html
Dingler, T., Choudhury, A., & Kostakos, V. (2018, October). Biased Bots: Conversational Agents to Overcome Polarization. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (pp. 1664-1668). ACM.
Hard, M. M. (2016). From Eliza to Alexa: Chatbots and the Future of Customer Experience.
Hill, J., Ford, W. R., & Farreras, I. G. (2015). Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in Human Behavior, 49, 245-250.
https://techcrunch.com/2015/09/29/forget-apps-now-the-bots-take-over/
Hinds, R. (2018, April 02). By 2020, You’re More Likely to Have a Conversation with This Than with Your Spouse. Retrieved from https://www.inc.com/rebecca-hinds/by-2020-youre-more-likely-to-have-a-conversation-with-this-than-with-your-spouse.html
Cognitive Engagement | Deloitte Netherlands. (2018, March). Retrieved from https://www2.deloitte.com/nl/nl/pages/technology/solutions/cognitive-engagement.html
Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the new world of HCI. interactions, 24(4), 38-42.
Goasduff, L. (n.d.). Chatbots Will Appeal to Modern Workers. Retrieved from https://www.gartner.com/smarterwithgartner/chatbots-will-appeal-to-modern-workers/
Kochi, E. (2018, March 22). How to Prevent Discriminatory Outcomes in Machine Learning. Retrieved from https://medium.com/@ericakochi/how-to-prevent-discriminatory-outcomes-in-machine-learning-3380ffb4f8b3
Marupaka, P. (2018, August 02). 5 Things That Are Triggering a New Era of Intelligent Chatbots. Retrieved from https://blog.kore.ai/5-things-that-are-triggering-a-new-era-of-intelligent-chatbotsMathur, V., Stavrakas, Y., & Singh, S. (2016). Intelligence analysis of Tay Twitter bot. In 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I) (pp. 231-236). IEEE.
McPherson, T. (2013). US operating systems at mid-century: The intertwining of race and UNIX. In Race after the Internet (pp. 27-43). Routledge.
Mladenic, D. (1999). Text-learning and related intelligent agents: A survey. IEEE Intelligent Systems,14(4), 44-54.
Mor, M. (2017, June 05). 9 Reasons Why Chatbots Will Rule Business. Retrieved from https://chatbotsmagazine.com/9-reasons-why-chatbots-will-rule-business-ec842dde45a9
Neff, G., & Nagy, P. (2016). Automation, algorithms, and politics| talking to Bots: Symbiotic agency and the case of Tay. International Journal of Communication, 10, 17.
Olsson, F. (2009). A literature survey of active machine learning in the context of natural language processing.
Schlesinger, A., O'Hara, K. P., & Taylor, A. S. (2018, April). Let's Talk About Race: Identity, Chatbots, and AI. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 315). ACM.
Snow, J. (2019, April 23). Google Photos still has a problem with gorillas. Retrieved from https://www.technologyreview.com/f/609959/google-photos-still-has-a-problem-with-gorillas/
Solomon, M. (2017, March 24). If Chatbots Win, Customers Lose, Says Zappos Customer Service Expert. Retrieved from https://www.forbes.com/sites/micahsolomon/2017/03/23/customers-lose-if-chatbots-win-says-zappos-customer-service-expert/
Toscano, J. (2016, October 26). The Ultimate Guide to Chatbots: Why they're disrupting UX and best practices for building. Retrieved from https://medium.muz.li/the-ultimate-guide-to-chatbots-why-theyre-disrupting-ux-and-best-practices-for-building-345e2150b682
UK Can Lead the Way on Ethical AI, Says Lords Committee - News from Parliament. (n.d.). Retrieved from https://www.parliament.uk/business/committees/committees-a-z/lords-select/ai-committee/news-parliament-2017/ai-report-published/
Woolley, S. C., & Howard, P. N. (2016). Automation, algorithms, and politics| Political communication, computational propaganda, and autonomous agents—Introduction. International Journal of Communication, 10, 9.