In my view, AI is going to be more valuable in helping programmers understanding existing code rather than generating new code. Here @judell explores using AI to help write documentation
https://thenewstack.io/code-in-context-how-ai-can-help-improve-our-documentation/
@mfowler I literally spent last night feeding the C# class I wrote into a local AI and it documented it all with XMLDoc style comments. Got all that Visual Studio hover-for-information-goodness for free. And this was just a tiny 7B Mistral model. I'm blown away, truly.
Could I have written that myself? Sure, but that requires extra brain power for prose and explanations that I'd rather put toward the code.
@wagesj45 @mfowler Are you able to share any more on how this is done?
Personally, I think it's vital that this is a local process, but I've found the information out there about getting local models to ingest reference material is non-existent.
I'm just about to take ownership of python code based that is a few 10s of thousands of lines without documentation. Could be a good application of the tech.
@weebull I've been using that model since it is reasonably fast on CPU, and the context length is 32K, which is enough to fit lots of code.
You'll definitely have quirks with a model that size, and you could experiment with running bigger models like CodeBooga-34B-v0.1. And be sure to double check it, because like all AIs (especially smaller ones) it will goof.
@wagesj45 do you find that going class by class gives enough context for goods docs. If I did that as a human I wouldn't expect much more insight than just a "translation" from code to English.
@weebull for what I'm doing so far, it has been adequate. The more context and the more parameters in the model, the better. Mistral is overpowered for its size/speed, but it's also not something I'd put into production. If you are serious about using AI in this local context and also need it for enterprise, you're going to need a LOT of horsepower to run a bigger model and maybe even do some fine tuning.
I don't think you'll find a "drop in" solution, but it'll be an adventure!
@weebull The only "complex" thing is that I have a Linux server in my basement that I use to run oobabooga text-generation-webui. The model I'm running is Mistral-7B-Instruct-v0.2 (the 5-bit quantization from TheBloke on huggingface).
Then I basically just paste my code into the chat function. Something like this...
I want you to look at this C# class and analyze it. Then I want you to write the XMLDoc comments for the code. [extra instructions here].
```csharp
[code]
```