BYTE •
ARTIFICIAL INTELLIGENCEThe Problem With Measuring AI Productivity
Every time I read about some study measuring the productivity gain of AI-assisted development, I raise an eyebrow. I find them uninteresting and useful only for the online fights between “there is no AI productivity for software developers” and “Claude Code increased my productivity by a billion percent.” The problem is that the landscape is so varied that, honestly, you can stress the data to prove both assertions.
My personal opinion is that many papers on the subject are just messy, and often they do not test at all how people using AI efficiently for work are actually using it. There are many issues I identified: they give too-easy problems, they impose arbitrary time constraints, they give problems to people not familiar with the framework they have to use (a big no no), and, surprisingly, I still see a lot of studies where the test subjects use “copy-pasting code from the web chat interface back and forth.” That is a way to work with AI that I think every developer stopped using in 2024.
But setting this aside, my main problem is that they never measure a specific class of task: the tasks that I would not have even started without AI assistants. We can discuss “this AI agent increases productivity by 2%” or “5%” or “20%” for as long as we want, but the reality is that many software and features I made in the last year would not exist at all without AI agents. I have folders with a sea of tools, automations, and single-use scripts that I would have never started.
So how could I measure that? For me, that is an increase of infinite percent, because they didn’t help me complete a project faster; they are the reason some projects exist. In some sense, it is a kind of Pascal’s wager. How should I redact the claim “AI agents improve developer output by only 2%” when I have concrete evidence on my hard drive of things that would have remained annotated in my notebook for all eternity?
Maybe it is my ADHD talking. Probably other people function differently and can do everything by just deciding to. But for me, lowering the activation moat that blocks me from starting to work on something is 100% worth it, even if I end up doing 80% of the work by hand.
It is a personal thing. I know that. I also understand if you work differently. And that’s the point. I saw developers using these tools in wildly different ways, for very different purposes, and with very different outcomes, and I don’t think it’s possible or fair to reduce that to a universal percentage number in some random study.
MovingAI-Rust 2.1.0: Now panic-free!
A small update to my pathfinding benchmark parser
I released version 2.1.0 of my MovingAI benchmark parser for Rust. The main change is that I removed all panics from the library because idiomatic Rust code should not panic if it can return a Result object.
One surprising thing about LLMs nobody is talking about
Small reflections on the unexpected of syntactic networks.
Large Language Models (LLMs) maybe be controversial, somehow, but I am still amazed by their mathematical properties. In this article, I will reflect on my favorite one.
This website will always be open. For bots, too.
Because offering free and accessible information is the most important thing.
In period where everybody seems to care about the use of our content, I decided to take a stance: I will always make my content available to everybody, human and non-human alike. Because copyright is a already a problem as it is.
The Freakout-Free Guide to Generative AI
I saw a bit of a heated debate around generative AI and AI art. So this is my “guide” for the general public, where I try to answer some common questions. What is generative AI? How does it work? What are its problems? What are their advantages? Do we all die because of it? (no).
Machine Consciousness is Inevitable
Can a robot become self-conscious? It looks like an interesting question, but it is not: machine consciousness is just inevitable. The true interesting questions are hidden in the details of how such consciousness will emerge and how it will look like.
The Trolley Cart Problem is not an AI problem
Every time there is a discussion on the future of AI-powered Autonomous Vehicles, somebody put the Trolley Cart Problem (TCP) on the table. And every time this happens, I am annoyed. However, recently, I saw some mutual followers studying AI and Computer Science talking about how TCP is a fundamental problem for the future of AI and autonomous vehicles. So I think it is time to speak it loud: the Trolley Cart Problem is not an AI problem! Let’s see why.
Overview of Three Techniques for Procedural Storytelling
Inspired by a recent paper I read this week, I decided to explain the three major “classic solutions” to the generative storytelling problem: Simulation, Planning, and Context-Free Grammars. Let’s what they are and what to choose.
Marginalia: Rebooting AI by Gary Marcus and Ernest Davis
With this new year, let’s try a new format. Marginalia will be a series in which I’ll share notes and comments on interesting books I read. The name is directly inspired by the old word indicating the small notes on the margins of books.
It will be a chance to discuss my readings without the need to write a full-fledged article. I hope it will be interesting as a review of the book or as a discussion starter. So, let’s start.
The Winter of Virtual Assistants
Seven years and eight months have passed since the release of the first really popular commercial virtual assistant (VA). Yet, seven years later, virtual assistants can do only marginally better.
Sure, they understand better, they speak better, they have learned some new trick; but in the end, they are still a funny but useless experience. After the first fun moments of experimentation when you start talking to them – that is, where you keep asking them silly jokes or dumb questions – they quickly came back to be pretty dumb object. I am pretty sure that the vast majority of user use a VA just for timers, weather and – occasionally – asking for the event on our calendar.