AI software coding
Perplexity surprised me with its programming skill. TLDR; it’s on a par with a super-fast junior programmer.
As a test I gave it the prompt:
write a go program to count files in a directory and subdirectories using gtk
To my surprise I got a complete program that worked. It got most of the idioms right, it got the required multi-threading (in golang terms go routines) right, etc. The associated text described dependencies and how it worked very clearly. There were some architectural choices that I would have made differently, a few bugs, and some security flaws. It also accepted the instructions to add a regex matching to the file name search and provided a revised program with explanation of the changes.
The issues were all ones that I would expect from a new junior programmer (but blindingly fast in comparison). The big ones were:
It chose the very simple but limited method of dealing with the fact that GTK is not thread safe. This not necessarily wrong, but it limits potential to expand functionality. GTK really interferes with the naturally multi-threaded nature of golang programs. I’m impressed it got this much right.
It did not properly handle ongoing user interaction. The text box to specify the directory worked. But when I changed the directory and told it to count again, it counted all the prior directories plus the new directory entry. I should have specified the expected user interaction. Again it’s about a junior programmer level considering the rather vague specification I gave it. It’s debatable whether to call this a bug or just a poor interpretation of what I wanted.
It did not sanitize the user inputs. I could include relative referencing in the directory input. Long term this is most worrisome. As AI generated code becomes more and more used, failure to follow good security programming practices will spread. I haven’t tried testing it by adding some instructions to sanitize the user inputs.
Update:
I told it to sanitize the user input. It didn’t. Instead it gave me seven sets of instructions, beginning with:
To sanitize user input in Go, you should focus on proper validation and encoding rather than trying to "clean" or modify the input directly. Here are some best practices for handling user input securely:
They were reasonable instructions and references.