We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of "added code" and "copy/pasted code" is increasing in proportion to “updated,” “deleted,” and “moved” code. In this regard, code generated during 2023 more resembles an itinerant contributor [...]
https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality
Code churn is a weird metric, because it is strongly affected by how commit-happy developers are (if you rewrite everything you write once, code churn will count it only if you commit it), which depends on e.g. code review practices. So, the effect on code churn (as described in the abstract, I didn't want to jump through the hoops to download the whole whitepaper) can be caused by e.g. organizations where code review standards are laxer increasing their code writing rate.
@isomer For the average ratio to change, you don't need to change any of the averaged ratios but just their weights. They seem to say that the average churn changed and imply that those who use copilot have larger churn than they would otherwise have. There is at least one obvious alternative explanation to that (that those who use copilot had larger churn already, and now are just writing more code with the same churn) which is not obviously wrong.
Obv. they could be more precise somewhere in the whitepaper, but I find a result where the central point is implied without being stated outright very suspicious.
@robryk their point is that the churn changed. Coz yeah, there's always gonna be churn. It's difficult to fit an entire paper in one post, I even had to truncate the summary.