Mediaboss Marketing

GPT-4 made big waves upon its release in March 2023, but finally, the cracks in the surface are beginning to show. Not only did ChatGPTs traffic drop by 9.7% in June,but a study published by Stanford University in July found that GPT-3.5 and GPT-4s performance on numerous tasks has gotten substantially worse over time.

In one notable example, when asked whether 17,077 was a prime number in March 2023, GPT-4 correctly answered with 97.6% accuracy, but this figure dropped to 2.4% in June. This was just one area of many where the capabilities of GPT-3.5 and GPT-4 declined over time.

James Zou, assistant professor at Stanford University, told Techopedia:

Our research shows that LLM drift is a major challenge in stable integration and deployment of LLMs in practice. Drift, or changes in LLMs behaviors, such as changes in its formatting or changes in its reasoning, can break downstream pipelines.

This highlights the importance of continuous monitoring of ChatGPTs behavior, which we are working on, Zou added.

Stanfords study, How is ChatGPTs behavior changing over time, looked to examine the performance of GPT-3.5 and GPT-4 across four key areas in March 2023 and June 2023.

A summary of each of these areas is listed below:

Although many have argued that GPT-4 has got lazier and dumber, with respect to ChatGPT, Zou believes its hard to say that ChatGPT is uniformly getting worse, but its certainly not always improving in all areas.

The reasons behind this lack of improvement, or decline in performance in some key areas, is hard to explain because its black box development approach means there is no transparency into how the organization is updating or fine-tuning its models behind the scenes.

However, Peter Welinder, OpenAIs VP of Product, has argued against critics whove suggested that GPT-4 is on the decline but suggests that users are just becoming more aware of its limitations.

No, we havent made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. Current hypothesis: When you use it more heavily, you start noticing issues you didnt see before, Welinder said in a Twitter post.

While increasing user awareness doesnt completely explain the decline in GPT-4s ability to solve math problems and generate code, Welinders comments do highlight that as user adoption increases, users and organizations will gradually develop greater awareness of the limitations posed by the technology.

Although there are many potential LLM use cases that can provide real value to organizations, the limitations of this technology are becoming more clear in a number of key areas.

For instance, another research paper, developed by Tencent AI lab researchers Wenxiang Jiao and Wenxuan Wang, found that the tool might not be as good at translating languages as is often suggested.

The report noted that while ChatGPT was competitive with commercial translation products like Google Translate in translating European languages, it lags behind significantly when translating low-resource or distant languages.

At the same time, many security researchers are critical of the capabilities of LLMs within cybersecurity workflows, with 64.2% of whitehat researchers reporting that ChatGPT displayed limited accuracy in identifying security vulnerabilities.

Likewise, open-source governance provider Endor Labs has released research indicating that LLMs can only accurately classify malware risk in just 5% of all cases.

Of course, its also impossible to overlook the tendency that LLMs have to hallucinate, invent facts, and state them to users as if they were correct.

Many of these issues stem from the fact that LLMs dont think but process user queries, leverage training data to infer context, and then predict a text output. This means it can predict both right and wrong answers (not to mention that bias or inaccuracies in the dataset can carry over into responses).

As such, they are a long way away from being able to live up to the hype of acting as a precursor to artificial general intelligence (AGI).

The public reception around ChatGPT is extremely mixed, with consumers sharing optimistic and pessimistic attitudes about the technologys capabilities.

On one hand, Capgemini Research Institute polled 10,000 respondents across Australia, Canada, France, Germany, Italy, Japan, the Netherlands, Norway, Singapore, Spain, Sweden, the UK, and the U.S. and found that 73% of consumers trust content written by generative AI.

Many of these users trusted generative AI solutions to the extent that they were willing to seek financial, medical, and relationship advice from a virtual assistant.

On the other side, there are many who are more anxious about the technology, with a survey conducted by Malwarebytes finding that not only did 63% of respondents not trust the information that LLMs produce, but 81% were concerned about possible security and safety risks.

It remains to be seen how this will change in the future, but its clear that hype around the technology isnt dead just yet, even if more and more performance issues are becoming apparent.

While generative AI solutions like ChatGPT still offer valuable use cases to enterprises, organizations need to be much more proactive about monitoring the performance of applications of this technology to avoid downstream challenges.

In an environment where the performance of LLMs like GPT-4 and GPT-3.5 is inconsistent at best or on the decline at worse, organizations cant afford to enable employees to blindly trust the output of these solutions and must continuously assess the output of these solutions to avoid being misinformed or spreading misinformation.

Zou said:

We recommend following our approach to periodically assess the LLMs responses on a set of questions that captures relevant application scenarios. In parallel, its also important to engineer the downstream pipeline to be robust to small changes in the LLMs.

For users that got caught up in the hype surrounding GPT, the reality of its performance limitations means its a flop. However, it can still be a valuable tool for organizations and users that remain mindful of its limitations and attempt to work around them.

Taking actions, such as double-checking the output of LLMs to make sure facts and other logical information are correct, can help ensure that users benefit from the technology without being misled.

Original post:

Why GPT-4 Is a Major Flop - Techopedia