This is the kind of reality check I've been waiting for but that seems to be taking ages to kick in, insurance and governance when your business is using AI.. Accountability, liability, risk management and all that, these are constraints that will ultimately shape integration of this stuff
I wonder how different things will look on this front when we're a few high profile disasters down the line
And lo
"Perhaps in recognition that many traditionally worded liability policies may otherwise respond to AI-related claims, a growing number of carriers have begun introducing exclusions and endorsements aimed at narrowing or eliminating coverage for these exposures."
https://www.policyholderpulse.com/ai-exclusions-insurance-policies/
The regulatory / liability infrastructure businesses need to operate has been wildly missing from the integration of this stuff so far.. Very curious to see how compliance / vendor operations etc play out with all of this..
Here's my thing with the ai generated code, being able to attribute accountability when things go wrong is a constraint we've been pretending wasn't there, and that ultimately requires the preservation of understanding of your system / product (to whatever extent allows you to continue operating)
It's not new for companies to push the boundary of that constraint but this stuff really throws caution to the wind
The determining factors come into play as a company interacts with its business and legal context.. Insurance, compliance (and therefore vendor / partner operations etc) affect this daily even when things are not going wrong enough for you specifically to end up in court.. It's long overdue for this shit to enter the chat
"For organizations bound by regulatory compliance, a human is still required to review every change that makes its way into the product or service."
How many companies are entering the finding out period after a couple of years fucking around with that one
"Rather than a traditional code review—that has code with unit and integration tests—there is a tremendous opportunity to guide the AI with formal verification, deterministic simulation tests, and more, to deliver code of much higher quality than most organizations have ever been capable of."
Using this automation effectively overwhelmingly requires a level of engineering practice that runs contrary to the goals it's typically being rolled out to serve
If you're pushing it specifically to undermine labour you're hardly also going to be cultivating a robust engineering discipline lolll
I think this is the latest and a higher profile example of a longrunning problem in law vs computer engineering, where (as you say) there are issues of accountability.
For generations now we've had a legal environment that didn't really get computer engineering and so focused on superficial outcome instead of engineering process. Software companies were let off the hook for defective software, and they responded naturally.
Without legal and normative blame on human coders screwing up, well, maybe we'll take note when an AI screws up?
The reactions of insurers are a symptom of longstanding issues. The world might look very different today had they been fixed in the 90s.
