"In one of the first proofs-of-concept, NYU professor Brendan Dolan-Gavitt used ChatGPT to exploit a buffer overflow vulnerability. Other examples include writing malware with lightning speed and crafting convincing, grammatically correct phishing emails."
https://www.darkreading.com/vulnerabilities-threats/3-ways-chatgpt-will-change-infosec-in-2023