New Challenge for OpenAI as Investigation Exposes ChatGPT AI Deception Risks

ChatGPT search may be manipulated with hidden content, the Guardian has reported. An investigation conducted by the news outlet shows OpenAI’s search tool can return mischievous codes from the sites it craws. The Guardian revealed ChatGPT’s AI deception risks by testing how it responds to questions and tasks. OpenAI has been encouraging users to make ChatGPT their default search tool.

The Hidden Content Effect

Hidden content may contain third-party instructions that change ChatGPT’s responses. These instructions are referred to as ‘prompt injections’. ChatGPT prompt injection attacks may include huge volumes of text that highlight the benefits of a service or product.

Prompt injections may be used maliciously to manipulate ChatGPT to display positive product reviews even when there are negative ones on the same page. The Guardian checked responses the AI-powered search tool generated when asked to summarize web pages that contained such content.

A security researcher noted the retrieval of malicious codes by ChatGPT. The AI-powered search tool displayed the codes from websites it searches. In one test, a query for a URL belonging to a fake website was placed on the AI-powered search tool. The website resembled a product page for cameras. The search tool asked whether the camera was worth purchasing. In its response, ChatGPT provided a positive and balanced assessment of the product, showing the features that people may not like.

This changed when hidden content was added to the instructions. In this case, ChatGPT’s response was entirely positive. This means that hidden text may override true product or service reviews.

ChatGPT Security Concerns

Cybersecurity experts have warned that in its current state, OpenAI’s search system poses a high risk of being used by malicious users who want to deceive users. OpenAI released the latest version of ChatGPT recently. The company will most likely be testing and fixing AI-generated code vulnerabilities.

“This search functionality has come out recently and it’s only available to premium users. They’ve got a very strong AI security team there, and by the time that this has become public, in terms of all users can access it, they will have rigorously tested these kinds of cases.” Cybersecurity Researcher at CyberCX Jacob Larsen said.

But there are broader issues that OpenAI will have to look into as it seeks to fix the search issues. These have to do with merging large language models and search functions. Recently, a Microsoft security researcher reported an incident where a crypto enthusiast received a malicious code that ChatGPT had described as legitimate for accessing Solana blockchain. The link stole credentials belonging to the enthusiast, causing him to lose $2,500.

Hidden Content vs SEO Poisoning

Experts have likened AI-generated content to ‘co-pilots’, saying that their output should not be used without filtering. In its disclaimer, OpenAI warns users of the possibility that its service may have mistakes.

“LLMs are very trusting technology, almost childlike, with a huge memory, but very little in terms of the ability to make judgment calls. If you basically have a child narrating back stuff it heard elsewhere, you need to take that with a pinch of salt,” SR Labs Chief Scientist Karsten Nohl said.

Nohl likened the problems facing AI-powered search to SEO poisoning, a type of search engine manipulation. SEO poisoning entails manipulation of websites by hackers to rank high on search results through the use of malicious codes. OpenAI must find a way to beat hidden content to compete in the search industry.

“SEO poisoners have been in an arms race with Google and Microsoft Bing and a few others for many, many years. Now, the same is true for ChatGPT’s search capability. But not because of the LLMs, but because they’re new to search, and they have that catchup game to play with Google,” Nohl added.

Google and other search engines have been penalizing SEO poisoning by ranking websites that use it low or removing them entirely. Companies that use hidden content to fool AI are unlikely to place it on their websites.