The latest AI model comparison takes an in-depth look at Anthropic’s Claude 3 Opus when pitted against industry heavyweights GPT-4 and Gemini 1.5 Pro. Having claimed that its Claude 3 Opus has surpassed GPT-4 in various popular benchmarks, Anthropic challenged us to test this assertion.
Claude 3 Opus vs GPT-4 vs Gemini 1.5 Pro
- The Apple Test: Claude 3 Opus, Gemini 1.5 Pro, and GPT-4 identify that three apples are presented to them with additional information. However, bereft of this information, Claude 3 Opus fails while the other models continue to get it right.
- Calculate the Time: Claude 3 Opus and Gemini 1.5 Pro failed to solve the first question on the time calculation presented to them. Although GPT-4 falters in the first question in this test its later outputs appear to vary.
- Evaluate the Weight: Claude 3 Opus incorrectly states that a kilo of feathers and a pound of steel weigh the same, while Gemini 1.5 Pro and GPT-4 provide correct responses.
- Maths Problem: Claude 3 Opus cannot solve a Math problem that needs the full calculation to solve before giving an answer. Gemini 1.5 Pro and GPT-4 provide the solution consistently and correctly.
- Follow User Instructions: Claude 3 Opus of the products, generates logical responses following the request notes. GPT-4 does fewer useful responses, than Claude 3 Opus. Gemini 1.5 Pro scores the least response in this note.
- Needle In a Haystack test: Claude 3 Opus fails to find the needle with 8K tokens as GPT-4 and Gemini 1.5 Pro provide the solution.
- Guess the movie (Vision Test): Claude 3 Opus can identify the movie by just glancing as GPT-4 is also able. Gemini takes the least points in this test.
Conclusion
Claude 3 Opus shows promise but falls short in tasks requiring common-sense reasoning and mathematical prowess compared to GPT-4 and Gemini 1.5 Pro. While it excels in following user instructions, its overall performance lags behind.
