An Experiment With AI-Assisted Programming
Anyone who has worked in programming recently has heard about “vibe coding” or programming with the assistance of large-language models. Personally, I’m always skeptical of trends that seem to have a lot of “hype” behind them, and I generally prefer programming without too much “smart assistance” (like auto-complete or tab-to-complete features in my text editor), so I’ve mostly stayed away from it, outside of a brief experiment with an AI-based code-completion a couple years ago.
However, in recent months, several programmers who’s experience I trust have noted that AI programming tools are rapidly improving, so I figured it was time to give them another try.
Tooling
At the moment, I program mostly on Windows and MacOS. On Windows, I’ve arrived at a workflow that I’m extremely happy with, using the Focus text editor along with the RAD Debugger (aka raddbg).
However, while Focus is cross-platform, the RAD Debugger is currently unavailable on MacOS, so around six months ago I started using the Zed editor on MacOS. Zed is a fairly performant text editor with an integrated debugger, and extensive support for AI tooling.
Setting up AI Assistance
In order to start doing some AI-assisted programming, I figured the easiest path forward would be to turn on Zed’s AI assistance features, which I had previously disabled. Next, I tried to use Zed’s “agentic editing” window, but discovered that I first needed to set up a “model” to use, so I signed up for a free trial of “Zed Pro”, which includes access to a handful of models from Anthropic and OpenAI. I chose to start with Anthropic’s “Claude Sonnet” model, since I had heard good things about it from other programmers. Now I guess I’m ready to start vibe coding!
The Project
Now I needed a project to try out the AI-assistance on, ideally something not too serious, but still more complicated than a basic “toy” project.
A few months ago, my wife asked me how many lines of code I had written that week. I told her that “lines of code” is often not a very useful way to measure developer productivity, and joked that she sounded a little bit like Elon Musk asking government employees “What did you do this week?”. But I realized that I didn’t know the answer, and that bothered me, since it seemed like the sort of thing that should be knowable.
So I hacked together a little Python script that would ping all of my GitHub repositories, find commits within some time range and count the lines of code added or removed. I wanted to turn the script into a web dashboard that I could use to track my work, but never got around to it…
Writing The App
… until now! I told Claude a little bit about the existing code, and mentioned that I had a template Dash app that I wanted to turn into a working dashboard, with data provided by the original script.
I took a little while for me to figure out a productive workflow, but eventually I settled into a cycle where I would ask the AI for a change, wait for the changes to complete, test them, and then use Zed’s “review” UI to either “keep” or “reject” the AI’s changes.
There is definitely a little bit of a learning curve for working with the AI… I started out asking very general requests, but gradually began to narrow my requests, and specifically ask the AI to work with existing constructs in the code. I also found it helpful to ask the AI for ideas about how to make the program better… even if a lot of those ideas weren’t good, it sometimes inspired me to think of additional ideas for myself.
After a few hours of AI-assisted work, I had a functioning dashboard with most of the features that I wanted!
Zed’s Workflow
While the workflow of using the AI assistant in Zed was pretty good overall, I did have a couple of complaints. The first issue is that I couldn’t figure out how to change the font size in Zed’s agent window… so it was a little bit annoying having to squint to read everything.
More importantly, I found that when I “rejected” the AI’s changes, the AI didn’t seem to know about it! In fact, it was confused that code it had previously written had vanished “for some reason”. Similarly, when I modified some of the AI-generated code manually, the AI didn’t seem to notice the changes when making future edits. It seems a bit backwards that I should have to tell the AI about the changes that I’m making to the code that it’s working on… can’t it just “see” the changes for itself?
Thoughts
Now that I’ve had some experience with AI-assisted programming, I have some more general thoughts that I’d like to share.
Legal/Moral
One question that I couldn’t avoid thinking about when programming with Claude Sonnet was “what code is this model trained on?” followed by “how is that code licensed?” I know there has been some controversy about AI models being trained on proprietary code and code that has been published under a more restrictive open-source license (and similarly controversy about AI being trained on copyrighted material in general).
While I don’t know much about the legality of training an AI model on material that is owned by a third-party, there is also a distinction between what is legal and what is moral. Personally, I think it’s immoral for AI companies to be making money from models that have potentially been trained on code written by other developers who are not being properly compensated for their work. Using these AI systems for a personal project like the one above doesn’t bother me too much, but if I were to use AI assistance for a more serious project, this is something that I would have to think about in more depth.
However, I think it is useful to evaluate technology on the basis of what it could be and not just what it currently is. I think it should be possible for an AI code assistant to be trained on properly licensed code, so I hope that in the future that can become possible.
Money
At the moment I have not yet had to pay any money for my AI usage. The Zed Pro free trial comes with $20 of included “credits”, and during my few hours of working on this project I went through roughly $11 worth of credits. After checking the Zed model pricing chart I learned that users are charged some amount of money per millions of tokens used.
If I extrapolate the cost of my AI-assisted programming session over the time I spend programming during a regular working day, I imagine it would cost around $25/day. If I work 20 days in a month, that would correspond to $500/month and $6,000/year. I guess the cost means different things to different people, but for me that seems like a pretty steep price for what I was able to get out of it.
Speed
Another concern that I had was the speed at which the AI assistant was able to work. A typical request during my programming session probably took 20–30 seconds to complete. While that isn’t a horribly long time in the grand scheme of things (perhaps faster than me writing the code myself?) it was just long enough for me to get distracted and start looking at other things on my computer.
Quality
While I was pleasantly surprised with the quality of the AI model’s output, it was far from perfect. Often times the AI model would attempt to fix a bug or add a feature and it just… wouldn’t work. I was always able to get something working eventually, but it sometimes took 3–4 attempts, often with me trying to guess what went wrong or give hints as to how I might have written the code that I was asking the AI to write.
Similarly, even when the AI did generate “correct” code, it would often have “holes” or missing parts. For example, originally the dashboard was showing all times in UTC, and I asked the AI to instead show everything in local time. The AI correctly did that… but only for one of the two plots (and even then it didn’t update the “hover” data for that plot). I had to keep asking the AI to make the change in more and more parts of the dashboard until it finally got them all.
The AI assistant also failed inhelping me to improve the performance of my “data collection” script. At the moment, collecting the data from the dashboard takes ~20 seconds. I asked the AI model that I wanted the collection time to be under 5 seconds, and while it tried a handful of optimizations they didn’t make any significant improvement.
Personal Growth
One of the main things I noticed from my AI-assisted programming session is that I don’t think I learned very much! Sure, I learned a little bit about better ways to prompt the AI model, but I consider that more “trivial knowledge” than “deep knowledge”.
I supposed that I could have taken the time to review the AI-generated code more closely. But, to be honest, the AI would generate a lot of code and it was faster for me to just test the app’s functionality rather than read the code thoroughly. If I had to fix a bug in the code without the AI’s help, I think it would be pretty tough… I would have to spend time learning the codebase properly before being comfortable to make changes. Some might argue that it doesn’t matter, since I could just ask the AI to fix the bug for me, but I don’t like the idea of being dependent on a tool to that extent.
I suppose I think of the AI assistance similar to a calculator. Calculator’s are great tools, but it’s still important to know how to do basic arithmetic even without using one. I’ve been thinking that I don’t mind using AI assistance for programming at two ends of the spectrum:
- Working on a program that I don’t need to maintain, especially if it relies on languages/libraries that I’m not interested in learning properly.
- Working on a program that I already know really well, using languages/libraries that I already know well.
Jobs?
Now for the hundred dollar question: when will AI take our jobs as programmers?
After this experience, I don’t think it will be for a while. While the AI tools are pretty capable (more capable than I thought they would be), they are definitely limited, and require some human guidance. I understand that this may not always be the case as this technology continues to develop, but I think it would require the models to develop further in a fundamental way… not just more training with more data.
Maybe more importantly for me… I didn’t enjoy the AI-assisted programming as much as I enjoy programming on my own. There’s probably a middle ground that maximizes both productivity and enjoyment, but for now I think I’ll stick with my old-school programming for the most part.
Questions
Ultimately, this experience left me with a number of questions:
- What is the most ethical large language model to use for programming? Ideally it would only be trained on permissively licensed open-source code and/or code that is explicitly licensed from the original developer(s) for training purposes.
- What is the most cost-effective model and pricing scheme to use for AI-assisted programming? Of the models provided with Zed Pro, it seems that Grok is the cheapest? Are there any models that offer discounted pricing for students or open-source developers?
- The project I was working on was using a number of technologies that I’m not super familiar with (Dash, GitHub API, etc). I’m curious how my experience would have been different working with systems that I’m more familiar with?
- Are there known methods for limiting the amount of code generated by the AI for a single request? That might make the generated code easier to review, while also limiting the models’ tendency to be overly verbose.
- What is the best way to use AI assistance to help with a performance problem? It definitely bothered me that the AI assistant seemed to struggle with that problem in particular.
If you have some answers to these questions, or suggestions for other things that I should try, please leave a comment below!
