ACF: Introducing a New Framework for Classifying AI Capabilities

👋🏽 Hello friends! (Especially all the new ones who subscribed after yesterday’s insane A.I. news and newsletter.)

I don’t normally send back-to-back emails, but TechCrunch just published our AI Framework announcement. So without further ado, let me introduce you to…

ACF: The AI Classification Framework

The Turing test is no longer a useful way to assess AI. We need a new framework for assessing AIs and their capabilities.

Today, I am proud to co-announce the AI Classification Framework (ACF), a new system for classifying the capabilities of artificial intelligences. 

The AI Classification Framework is a comprehensive system for evaluating AI capabilities, based in part on the Theory of Multiple Intelligences by Harvard’s Howard Gardner. The Framework provides an objective way to measure the capabilities of an AI and identify areas of strength and weakness. Our hope is that it will help measure progress towards more sophisticated AI.

The ACF was developed by Chris Saad (the primary creator of the ACF, former head of platform at Uber), with contributions from Jeremiah Owyang (famed technology analyst) and myself (Ben Parr, co-founder of Octane AI and former Co-Editor of Mashable).

🚨You can get a copy of the ACF here, including detailed notes on what constitutes an AI’s capability at each level.🚨

🚨You can read more about the AI Classification Framework in Chris Saad’s announcement article on TechCrunch.🚨

How the AI Classification Framework Works

Similar to the Theory of Multiple intelligences, the AI Classification Framework has eight modalities:

• Logical-Mathematical

• Linguistic-verbal

• Visual-spatial

• Musical-rhythmic

• Bodily-kinesthetic

• Interpersonal

• Intrapersonal

• Existential

The ACF scores these capabilities on a scale of 0-5, where 0 = No Capability, 1 = Simple Aid, 2 = Advanced Aid, 3 = Complex Prompts, 4 = Context Aware, and 5 = Self Agency.

An AI with capabilities at level 0 would be considered an infant, while an AI at level 5 would be considered a super intelligence — beyond human intelligence. For example, an AI with a score of 3 in Linguistic-verbal intelligence should be able to analyze and generate expertly-written content based on natural language (ChatGPT scores a 3 on Linguistic-verbal), but a score of a 4 would need to be able to do this at a master level, taking in realtime context (like a human) — and be able to do it unprompted.

The ACF recognizes different AIs are specifically suited to different tasks — an AI for a self-driving car must have bodily-kinesthetic intelligence, but has no need for musical-rhythmic intelligence. AI image generators like DALL-E, Midjourney and Stable Diffusion must excel in visual-spatial intelligence. (They each currently score a “3” in this category.) The vast majority of AI systems today are, by necessity, narrow AIs.

The “holy grail” of AI, a humanlike AI or artificial general intelligence (AGI) that can perform any intellectual tasks a human can, would need to score in almost every category — something no AI has yet achieved.

A table with a complete description of each intelligence at each level is included in the AI Classification Framework.

Moving on from the Turing Test

It's been over 70 years since Alan Turing first proposed the Turing Test in 1950. The Turing Test was a revolutionary concept at the time — a way to measure the intelligence of machines by evaluating their ability to fool a person into thinking they were talking to another person.

The Turing Test has been used as a measure of artificial intelligence for decades, and it has been cited as a way to evaluate the success of AI projects. Yet while the Turing test has been a useful benchmark for determining an AI’s capabilities, we’ve passed the point where the test tells us anything about today’s generative AIs. The main goal of AI is no longer to fool humans, but to augment their work — or replace the work of a human entirely.

AI technology has evolved significantly since 1950, and the Turing Test is no longer a valid measure of AI capabilities.

As we enter a new era of AI development, it is clear that a new framework is needed to assess the capabilities of artificial intelligence. We need a new framework that can account for the rapid evolution of technology and assess the intelligence of today's AI.

Final Thoughts

We believe the AI Classification Framework gives developers, investors and business leaders a powerful tool to understand the capabilities of AI and make more informed decisions when building and deploying AI systems. Our hope is that this framework will help everyone benchmark modern AIs and their impact on society. Hopefully, this framework will help accelerate the development of artificial intelligence and the realization of its potential.

We’re excited to be launching the AI Classification Framework, and we hope it will change the way we think about AI.

If you have any questions about the ACF, please shoot Chris an email at [email protected], Jeremiah at [email protected] and/or me at [email protected].

Cheers,

~ Ben

P.S. I have (at least) one more announcement coming soon. Stay tuned.

Thanks for reading The Social Analyst by Ben Parr! Subscribe for free to receive new posts and support my work.