Close Menu
    National News Brief
    Friday, November 20
    • Home
    • Business
    • Lifestyle
    • Science
    • Technology
    • International
    • Arts & Entertainment
    • Sports
    National News Brief
    Home » Large Language Model Performance Raises Stakes

    Large Language Model Performance Raises Stakes

    Team_NationalNewsBriefBy Team_NationalNewsBriefJuly 5, 2025 Technology No Comments3 Mins Read
    Share
    Facebook Twitter LinkedIn Pinterest Email


    Benchmarking large language models presents some unusual challenges. For one, the main purpose of many LLMs is to provide compelling text that’s indistinguishable from human writing. And success in that task may not correlate with metrics traditionally used to judge processor performance, such as instruction execution rate.

    RELATED: LLM Benchmarking Shows Capabilities Doubling Every 7 Months

    But there are solid reasons to persevere in attempting to gauge the performance of LLMs. Otherwise, it’s impossible to know quantitatively how much better LLMs are becoming over time—and to estimate when they might be capable of completing substantial and useful projects by themselves.

      Large Language Models are more challenged by tasks that have a high “messiness” score.Model Evaluation & Threat Research

    That was a key motivation behind work at Model Evaluation & Threat Research (METR). The organization, based in Berkeley, Calif., “researches, develops, and runs evaluations of frontier AI systems’ ability to complete complex tasks without human input.” In March, the group released a paper called Measuring AI Ability to Complete Long Tasks, which reached a startling conclusion: According to a metric it devised, the capabilities of key LLMs are doubling every seven months. This realization leads to a second conclusion, equally stunning: By 2030, the most advanced LLMs should be able to complete, with 50 percent reliability, a software-based task that takes humans a full month of 40-hour workweeks. And the LLMs would likely be able to do many of these tasks much more quickly than humans, taking only days, or even just hours.

    An LLM Might Write a Decent Novel by 2030

    Such tasks might include starting up a company, writing a novel, or greatly improving an existing LLM. The availability of LLMs with that kind of capability “would come with enormous stakes, both in terms of potential benefits and potential risks,” AI researcher Zach Stein-Perlman wrote in a blog post.

    At the heart of the METR work is a metric the researchers devised called “task-completion time horizon.” It’s the amount of time human programmers would take, on average, to do a task that an LLM can complete with some specified degree of reliability, such as 50 percent. A plot of this metric for some general-purpose LLMs going back several years [main illustration at top] shows clear exponential growth, with a doubling period of about seven months. The researchers also considered the “messiness” factor of the tasks, with “messy” tasks being those that more resembled ones in the “real world,” according to METR researcher Megan Kinniment. Messier tasks were more challenging for LLMs [smaller chart, above].

    If the idea of LLMs improving themselves strikes you as having a certain singularity–robocalypse quality to it, Kinniment wouldn’t disagree with you. But she does add a caveat: “You could get acceleration that is quite intense and does make things meaningfully more difficult to control without it necessarily resulting in this massively explosive growth,” she says. It’s quite possible, she adds, that various factors could slow things down in practice. “Even if it were the case that we had very, very clever AIs, this pace of progress could still end up bottlenecked on things like hardware and robotics.”

    From Your Site Articles

    Related Articles Around the Web



    Source link

    Team_NationalNewsBrief
    • Website

    Keep Reading

    Anthropic shares fall a day after blockbuster IPO

    White House removes cyber-security chief after series of tech outages

    GTA 6 breaks records as sales top $2.3bn – industry estimate

    Anthropic becomes one of world’s most valuable public companies in first-of-its kind IPO

    Don’t buy smart glasses, Dutch data watchdog advises

    UK joins in GTA 6 frenzy as game becomes available worldwide

    Add A Comment

    Comments are closed.

    Editors Picks

    Why Waiting for Monthly Financial Reports Is Creating Blind Spots and Slowing Your Growth

    July 11, 2025

    Targeting DOGE, labor group puts up billboards warning of heat deaths at national parks

    June 1, 2025

    Amy Robach & T.J. Holmes’ Ex-Spouses To Skip Couple’s Wedding

    May 21, 2026

    Israel kills dozens in Gaza, sends tanks into southern areas, medics say

    October 2, 2024

    Taylor Swift Feels ‘Violated’ After Private Texts Were Leaked

    January 27, 2026
    Categories
    • Arts & Entertainment
    • Business
    • International
    • Latest News
    • Lifestyle
    • Opinions
    • Politics
    • Science
    • Sports
    • Technology
    • Top Stories
    • Trending News
    • World Economy
    About us

    Welcome to National News Brief, your one-stop destination for staying informed on the latest developments from around the globe. Our mission is to provide readers with up-to-the-minute coverage across a wide range of topics, ensuring you never miss out on the stories that matter most.

    At National News Brief, we cover World News, delivering accurate and insightful reports on global events and issues shaping the future. Our Tech News section keeps you informed about cutting-edge technologies, trends in AI, and innovations transforming industries. Stay ahead of the curve with updates on the World Economy, including financial markets, economic policies, and international trade.

    Editors Picks

    Anthropic shares fall a day after blockbuster IPO

    November 20, 2026

    A Well-Deserved Place On American Coinage

    November 20, 2026

    Suns, Jazz Look to Steady Footing Before NBA Cup Clash

    November 20, 2026

    Angie Katsanevas ‘Can’t Imagine’ Dating After Split

    November 20, 2026
    Categories
    • Arts & Entertainment
    • Business
    • International
    • Latest News
    • Lifestyle
    • Opinions
    • Politics
    • Science
    • Sports
    • Technology
    • Top Stories
    • Trending News
    • World Economy
    • Privacy Policy
    • Disclaimer
    • Terms and Conditions
    • About us
    • Contact us
    Copyright © 2024 Nationalnewsbrief.com All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.