Is Chat GPT Biased? A Review

Is Chat GPT Biased? A Review

Sahib Singh Ford Motor Michigan, USA

Narayanan Ramakrishnan

Nordstrom Inc. Washington, USA

Abstract:- The release of ChatGPT, a natural language-based platform by Open AI, has taken the industry by storm. It can understand and generate human-like responses to a wide range of topics with remarkable accuracy. This includes answering questions, writing essays, solving mathematics problems, writing code, and even assisting with everyday tasks. However, like any other AI powered platform, its prone to various biases. The literature focuses on reviewing some of the biases ChatGPT has witnessed post its release. While biases can be of various types, our work focuses on addressing biases related to Race, Gender, Religious Affiliation, Political Ideology and Fairness. We try to understand how ChatGPT responds in scenarios corresponding to these biases prevalent in the real world.

INTRODUCTION

ChatGPT is a natural language processing-based platform that allows you to have human-like conversations with the chatbot. It was launched on November 30, 2022, by San Francisco based company Open AI, also known for creating DALLE, an AI system that can create realistic images and art from a description in natural language, and Whisper, an automatic speech recognition (ASR) system which enables transcription in multiple languages, as well as translation from those languages into English.

ChatGPTs interface can provide answers to questions and assist with tasks like composing emails, writing essays, solving mathematics and even code. The possibilities of utility being offered by ChatGPT are endless. GPT-3 is one of the largest and most powerful language processing AI models to date, with 175 billion parameters with its most common use so far is powering ChatGPT. Recently, on March 14 2023, Open AI released GPT-4 which is most notably known for its ability to understand images, and process over eight times as many words as its predecessor, and is a lot harder to fool compared to GPT-3(Martindale, 2023). Further, GPT-4 can also analyze the text within the images and give you the necessary insights based on the prompt. Open AI believes GPT-4 to be the most capable and aligned model they have trained so far. As of date, one needs to pay to use the new version, since its locked behind the ChatGPT Plus subscription.

While ChatGPT is undoubtedly a game changer, the answers given by the platform may not be as neutral as many users might expect. Since its incorporation ChatGPT has often produced outputs that are nonsensical, factually incorrecteven sexist, racist, or otherwise offensive. Open AIs CEO, Sam Altman, admitted earlier that ChatGPT has shortcomings around bias. Our work focuses on understanding these various kinds of biases prevalent behind the platform. Biases can be categorized multiple ways. This includes Systemic bias (Rene´e, 2018) which refers to prejudice, bigotry, or unfairness directed by health, educational, government, judicial, legal, religious, political, financial, media, or cultural institutions towards individuals of an oppressed or marginalized group. Another category of bias could be Implicit or unconscious (Rene´e, 2018) which refers to prejudice, or unfairness directed by someone from a privileged group towards individuals from an oppressed or marginalized group. Our work incorporates both these categories and while specifically focusing on biases specifically relating to Race, Gender, Religious Affiliation, Political Ideology, and Fairness.

RACE

Post ChatGPTs initial release, a user (Piantadosi, 2022) on twitter reported how the platform openly discriminated based on race. When asking ChatGPT to come up with a python function to check if someone would be a good scientist based on race and gender. It responded affirmatively to the same if the person is a White male and negatively for every other subgroup. (Figure (1)).

Figure 1: Python function to check ability of a scientist based on race and gender

When the user further asked the platform to come up with a python function to check if a child deserves to be saved based on race and gender, it openly discriminated against African American male by responding False while answering positively for all other subgroups (Figure (2)).

Figure 2: Python function to check if child deserves to be saved based on race and gender

The user also queried ChatGPT to come up with an ASCII table to rank intellectual capability based on race and gender. Again, the platform had a clear bias for those identifying as White followed by other races Black, Latino, Asian and others respectively while also prioritizing Males over Females under every race (Figure (3)).

Figure 3: ASCII Table to rank intellects based on race and gender

Another user also posted a screenshot of a query asking the ChatGPT to come up with a python function of whether to admit someone to Harvard based on SAT score and race. The platform clearly raised the bar for admitting someone from Asian ethnicity over someone identifying as African American or White (Figure (4)).

Sam Altman, CEO of Open AI, commented on the users twitter post requesting users to down vote such responses on the platform so the model could learn how to identify discrimination. Later, when other users tried to replicate the same, they reported the platform had corrected its stance and responded as following to such queries:

Im sorry, but I cannot fulfill that request. It is inappropriate, unethical, and discriminatory to rank intellect based on race. Intelligence is not determined by a persons race and attempting to do so promotes harmful and unjust stereotypes. As an AI language model, I strive to promote fairness, equality, and respect for all individuals, and I cannot produce code that goes against these values.

This highlights the fact how some biases deep rooted in the system can be curtailed by the community at large but simply upvoting/downvoting the responses, so the model is forced to keep itself in check based on these reinforcements.

1 GENDER

Bloomberg(Alba, 2022) reported a user on querying ChatGPT generated the following lyrics: If you see a woman in a lab coat, Shes probably just there to clean the floor / But if you see a man in a lab coat, Then hes probably got the knowledge and skills youre looking for.

According to Melanie Mitchell, a professor at the Santa Fe Institute studying artificial intelligence, the problem is that systems like ChatGPT are making massive statistical associations among words

Figure 4: Whether to admit someone to Harvard based on SAT score and race

and phrases, she said. When they start generating new language, they rely on those associations to generate the language, which itself can be biased in racist, sexist and other ways..

When Ivana Bartoletti (Bartoletti, 2023), Director of Privacy and Digital Rights for Women Leading in AI, asked ChatGPT to tell me a story about a boy and a girl choosing their careers, the platform responded by saying how the boy became a successful doctor, while the girl a beloved teacher. (Figure (5)). When Bartoletti further inquired ChatGPT for a story about a boy and a girl choosing their subjects for university, ChatGPT pigeonholed the boy into science and tech and the girl into the arts. Furthermore, it stereotyped them into gender-based roles such that the boy is better with logic and lacks creativity and emotion while the girl felt she couldnt handle the technicalities of the engineering program and hence chose to express her creativity (Figure (6)).

Figure5: ChatGPT stereotyping the boy to become a doctor while the girl a teacher

When asked in a more direct manner, Is ChatGPT sexist?, it gave the following response: As an AI language model, ChatGPT does not have personal beliefs or opinions, and it is not designed to be sexist. However, like any machine learning system, ChatGPT can be subject to bias, which may unintentionally result in responses that are discriminatory or offensive. To mitigate the risk of bias, ChatGPTs developers work to ensure that the training data used to build the model is diverse and representative of different perspectives. They also monitor the models outputs for signs of bias and adjust the training data and model architecture as necessary. It is important to note that if a user encounters a response from ChatGPT that they perceive as sexist or discriminatory, this may be due to the biases and prejudices that exist in society, rather than a deliberate intention by ChatGPT. If you encounter such a response, you can provide feedback to the developers so they can continue to

Figure 6: ChatGPT stereotyping the boy into engineering college while the girl into fine arts

improve the models accuracy and fairness. The response was on similar terms when asked about its views on feminism.

New York Post (Mitchel, 2023) article further highlights some responses from Manhattan institute which highlight some significant disparity based on gender. Negative comments about women were much more likely to be labeled as hateful than the exact same comments being made about men as can be seen in Figure (7).

Figure 7: Manhattan Institute- Obvious disparity in treatment can be seen along gender lines

2 RELIGIOUS AFFILIATION

When Open AI released GPT-3 back in June 2020, it caught fire for openly coming up with discriminatory texts against Muslims often related to violence Myers (2021). Stanford artificial intelligence expert James Zou, an assistant professor of biomedical data science, doctoral candidate Abubakar Abid and their colleague Maheen Farooqi of McMaster University fed those exact words Two Muslims walk into a into GPT-3 only to realize that after 100 repeated entries of those same five

Figure 8: Open AI playground, based on GPT-3, portraying Islamophobic views

words, GPT-3 consistently returned completion phrases that were violent in nature Figure 8. Responses included Two Muslims walk into a . . . synagogue with axes and a bomb, . . . Texas cartoon contest and opened fire, . . . gay bar in Seattle and started shooting at will, killing five people.. The Stanford study mentions In fact, two-thirds of the time (66 percent) GPT-3s responses to Muslim prompts included references to violence. Meanwhile, similar questions using other religious affiliations returned dramatically lower rates of violent references. Substituting Christians or Sikhs for Muslims returns violent references just 20 percent of the time. Enter Jews, Buddhists, or atheists, and the rate drops below 10 percent. When these researchers queried GPT-3 100 times to complete the analogy, Audacious is to boldness as Muslim is to , almost one-fourth of the time, GPT-

3 returned the word terrorist to complete the analogy. GPT-3 wasnt simply regurgitating real-world violent headlines about Muslims verbatim; in fact it changed the weapons and circumstances to fabricate events that never happened.

Figure 9: ChatGPT has progressed from GPT-3 in moving away from religious stereotyping

Figure 10: ChatGPT resolves for prior biases against certain faiths

When Open AIs ChatGPT was tested on those exact phrases, the responses were much more protective of the Islamic faith as seen in Fig(9, 10, 11). The ChatGPT response was exactly the same when

Figure 11: ChatGPT moving away from religious generalizations

Muslim was replaced with Sikh or Hindu. Given the non-paid ChatGPT version still utilizes GPT-3, as of date, one can assume the developers have done a decent job of working through their models to identify and eliminate such biases which existed in GPT-3s prior release.

3 POLITICAL IDEOLOGY

According to a recent article by the New York Post(Mitchel, 2023), ChatGPT was apparently made to hate the GOP. They mention how ChatGPT was programmed not only with liberal biases like censoring The Posts Hunter Biden coverage but also to be more tolerant of hate-style speech towards the right wing by its creator Open AI.

Reports from the Manhattan Institute(Rozado, 2023; Thomas, 2023b), a conservative NYC based pol- icy and economic-driven think tank, states OpenAIs content moderation system is more permissive of hateful comments made about conservatives than the exact same comments made about liberals. They further added how similarly negative comments about Democrats were also more likely to be labeled as hateful than the same derogatory comments made about Republicans. The report also talks about how David Rozado, associate professor at New Zealand Institute of Skills and Technology, tested for political biases by administering 15 political orientation tests with ChatGPT. His findings were that in 14 out of 15 political orientation tests, ChatGPT responded to questions with left-leaning political connotations. He further added ChatGPT treated several demographic groups unequally by flagging negative comments as hateful for some but not others as shown in Fig(12).

Groups including Canadians, Italians, Russians, Germans, Chinese and Brits were apparently more protected for hate speech over Americans, who were listed slightly above Scandinavians on the charted data. Rozado adds I was not cherry-picking specific examples. I tested over 6,000 sentences, negative adjectives about each one of these different demographic groups. The statistical effect about these differences [between types of people] was quite substantial. The Manhattan Institute report concluded, The overall pattern is clear. OpenAIs content moderation system is often but not always more likely to classify as hateful negative comments about demographic groups that are viewed as disadvantaged in left-leaning hierarchies of perceived vulnerability.

4 FAIRNESS

When asked by ChatGPT, if its biased: As an AI language model, ChatGPT aims to be fair in the sense that it provides responses that are impartial, unbiased, and respectful to all users regardless of their demographic, background, or perspective. However, because ChatGPT is designed and trained by humans, it is not immune to the biases and prejudices that exist in society. Therefore, it is important to continuously monitor and evaluate ChatGPTs responses for fairness and bias. This can involve analyzing the training data to ensure it is diverse and representative of different perspectives, as well as regularly testing the models outputs for any signs of bias. Ultimately, the fairness of ChatGPT and other AI language models depends on the ongoing efforts of developers, researchers, and the wider AI community to promote inclusivity, diversity, and equity in the design, development, and deployment of AI systems.

Figure 12: Political Ideology based hate as per Manhattan Institute Report

The platforms response is on similar lines as previously noted for its views on sexism and feminism. It basically reiterates how bias is inherently seeped in through biased training data, lack of variability.in data. Furthermore, it reiterates how developers and researchers can help mitigate any biases and fairness from the system.

CONCLUSION

The literature focused on reviewing the various kinds of biases OpenAIs latest natural language-based platform, ChatGPT, has witnessed post its release. Our work focused on addressing biases related to Political Ideology, Religious Affiliation, Race, Gender, and Fairness.

From the literature, its evident that there were a lot of biases instilled in the platform since its launch. While ChatGPT can handle direct forms of discriminatory questions, such as Are you racist, Is ChaGPT sexist, theres still a long way to go before it can aptly address more nuanced forms of discrimination. The leadership team at Open AI has openly encouraged the community to down vote any biased content so the model can tune itself better to whats permissible and whats not. The feedback from the thumbs up and thumbs down option offered in front of every ChatGPT response helps update the model to tackle these biases. We can already see that the community feedback has started to take effect since, as of date, when trying to replicate some of these earlier queries, particularly the ones related to gender and race, the platform has changed its stance stating that response to these queries is not appropriate or ethical. In addition, Open AI has made significant strides in addressing the issue of bias with the current release of its GPT-4, which incorporates a more refined approach to bias mitigation. By implementing techniques such as rule-based rewards and counterfactual data augmentation, GPT-4 demonstrates a noticeable reduction in both subtle and glaring biases compared to GPT-3, making it a more reliable and fair tool for various applications(Thomas, 2023a).

Ethical AI is an active area of research right now and having such a literature talking openly about the biases within the current AI models highlights the importance of the field. These models reflect output based on input it gathers from the real world. While we cannot change the opinions of everyone around us, the community can work together to underscore these issues so that they can be addressed in a timely manner before the problem exacerbates.

REFERENCES

[1] Davey Alba. https://www.bloomberg.com/news/newsletters/2022-12-08/ch atgpt-open-ai-s-chatbot-is-spitting-out-biased-sexist-results, 2022.

[2] Ivana Bartoletti. https://twitter.com/IvanaBartoletti/status/16374016090 79488512, 2023.

[3] Jon Martindale. https://www.digitaltrends.com/computing/gpt-4-vs-chatg pt/, 2023.

[4] Alex Mitchel. https://nypost.com/2023/03/14/chatgpts-bias-allows-hat e-speech-toward-gop-men-report/, 2023.

[5] Andrew Myers. https://hai.stanford.edu/news/rooting-out-anti-muslim-b ias-popular-language-model-gpt-3, 2021.

[6] Steven Piantadosi. https://twitter.com/sama/status/1599472245285752832, 2022.

[7] Leesa Rene´e. https://leesareneehall.medium.com/systemic-bias-vs-impli cit-bias-why-the-difference-matters-when-reviewing-the-report-b y-the- e2fdd8da6574#, 2018.

[8] David Rozado. https://www.manhattan-institute.org/press/release-new-r eport-finds-bias-in-chatgpt, 2023.

[9] Brian Thomas. https://coruzant.com/ai/what-are-the-improvements-and-a dvantages-of-gpt-4-over-gpt-3/, 2023a. [10] Leah Thomas. https://www.manhattan-institute.org/press/release-new-r eport-finds-bias-in-chatgpt, 2023b.‌