James Shore: Measuring AI’s Impact on Delivery Speed

A framework is presented for rigorously measuring whether AI actually speeds up software delivery, arguing that naive metrics like lines of code, PR counts, or missed estimates are misleading because task sizes vary, estimates are inherently optimistic, and process changes accompanying AI adoption confound results. The proposed solution is a randomized within-subject repeated-measures trial: collect estimates before assignment, randomly assign tasks to 'with AI' or 'without AI' conditions, interleave the two approaches, and compare each person to their own baseline. This is part 2 of a four-part series (with an epilogue), following a piece on why naive AI ROI measurement fails and preceding upcoming installments on unintended consequences and business outcomes. References a METR study showing engineers overestimated AI speed benefits by nearly 50%.

Source: https://www.jamesshore.com/v2/blog/2026/measuring-ais-impact-on-delivery-speed. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.