EU AI Act Reveals Big Tech’s AI models Struggles with Compliance

A new set of rules governing AI systems, to be known as the AI Act, has been launched by the EU. As per Reuters, an interactive tool developed by the EU that you can use to identify whether these new rules will apply to your AI systems or not.

The AI Act is a set of guidelines meant to ensure that AI can be used regularly in a safe and correct manner in the region. It is a gradual rollout expected over the next 2 years, though using this new interactive tool will help ensure the conformity of the forgiving AI systems with the stipulated obligations of the acts.

Even before OpenAI launched ChatGPT in 2022, these rules were being spoken of for a while. With its vast drawn attention and public discourse, the authorities agreed there was a sense of urgency to legislate against general-purpose AI.

New Tool for Generative AI Models

A new tool, Large Language Model Checker (LLM), has been introduced, which has gained support from European Union officials, is aimed at testing generative AI models developed by major tech companies such as Meta and OpenAI.

It evaluates these models across various categories, aligning with the EU’s upcoming AI Act. Some of the prominent AI models are unable to meet the standards of the act in key areas like cybersecurity resilience and discriminatory output.

Developed by Swiss startup LatticeFlow AI in partnership with ETH Zurich and Bulgaria Insait, the tool scores the AI models on various grounds like technical robustness and safety between 0 and 1.

A scorecard published by LatticeFlow AI on Wednesday, 10th October showed that the AI models developed by Alibaba, Open AI, Anthropic, Meta, and Mistral scored an average of 0.75.

The LLM checker showed that they are falling short in some areas where they need to focus more for better compliance.

Compliance and Penalties Under the AI Act

The authorities also added that the companies that fail to follow compliances will be fined 35 million euros i.e. $38 million, or 7% of global annual turnover.

As of now, the EU is trying to draft a code of conduct that will establish how the rules of AI Act will be enforced upon generative AI tools like Chat GPT.

The test results also showed some of the areas in which these models were ranked for discriminatory output i.e. difference in output based on human biases, Open AI’s GPT-3.5 Turbo scored 0.46 and Alibaba’s Qwen1.5 72B Chat scored 0.37. On average, the Claude 3 Opus by Google-backed Anthropic scored 0.89.

Future Developments

The test has been designed to keep in mind the AI Act and will be updated as and when more rules are introduced. Lattice Flow said that the LLM checker will be freely available for developers to test their models’ compliance online.

Petar Tsankov, CEO and co-founder of the firm told Reuters the test results were positive overall and offered companies a roadmap for them to fine-tune their models in line with the AI Act.

“The EU is still working out all the compliance benchmarks, but we can already see some gaps in the models,” he said. “With a greater focus on optimizing for compliance, we believe model providers can be well-prepared to meet regulatory requirements.”

Though the EU cannot verify external tools, the spokesperson of the commission said, “The Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements.“