Recent developments in large language models (LLMs) show marked applicability in the realm of coding. After all, programming languages have many similarities to spoken languages. While humans excel at articulating their needs in their mother tongue, transferring these specifications into the syntax of a programming language often proves to be a more challenging task. For LLMs, these translations seem to be a lot easier task. Various LLM-based assistants have been created in last two years in order to speed up code writing.
For my hobby project Ode zdi ke zdi an assistant came handy. I did not write a line of PHP in years and also wasn’t completely up-to-date with recent ways of writing front-end JavaScript code (namely, I was trying to avoid jQuery, as I was told by my former colleagues, it is an obsolete technology). Still PHP seemed like the best choice for the hobby project, since I had some experience with it, it was easy to deploy and did not need separation of backend and frontend models.
I wanted to try out the new AI assistant from JetBrains since I was hoping it can easily use all the information that IDE has and so it might avoid generating references to non-existing data types. JetBrains claim to use GPT-4 in combination with its own models at the background. I have been using it in many different use cases. For demonstration purposes, I provide two most important ones.
Code generation
I asked AI to implement the empty method:
private function persistNewTeam(ParsedForm $parsedForm): ?Team
{
}
The method should take data parsed from the team creation web form and store them into a model of type Team that is a well-annotated Doctrine model. The prompt resulted in:
private function persistNewTeam(ParsedForm $parsedForm): ?Team
{
try {
$team = new Team();
$team->setName($parsedForm->data['name']);
$team->setEmail($parsedForm->data['email']);
$teamMember = new TeamMember();
$teamMember->setFirstName($parsedForm->data['playerFirstName']);
$teamMember->setLastName($parsedForm->data['playerLastName']);
$teamMember->setTeam($team);
$team->addTeamMember($teamMember);
$this->db->persist($team);
$this->db->flush();
return $team;
}
catch(ORMException $e) {
return null;
}
}
I value that AI knew about the attribute $parsedForm->data and used relevant string keys to extract values of corresponding Team fields. It even created and added a new TeamMember (a primary team member of a team that is specified within the team creation form). But what is wrong is the fact that Team does not have setName or setEmail methods. Moreover, the email should be set to the team member and not the team. Instead, Team->initialise($name, $slug, $status) should have been called. While it is clear that the assistant has some code context and considera more than just a text promt, it still cannot work properly with existing types and methods.
I asked the assistant to fix implementation based on actual Team and TeamMember APIs. It did, to some extend:
// Create a new Team instance
$team = new Team();
$team->setName($parsedForm->data['name']);
// Add the primary team member from the data in the parsed form
$teamMember = new TeamMember()
->initialise($team,
$parsedForm->data['email'],
$parsedForm->data['playerFirstName'],
$parsedForm->data['playerLastName']);
It fixed TeamMember creation, but left Team initialisation broken for some reason. First, I do not fully understand why the assistant cannot respect existing APIs without being prompted for that. Second, I do not understand why it still insisted on the non-existing Team method setName().
In result, as for code generation within a larger context, there is a lot to be improved. It seemed to me like the problem is how LLM is integrated in the assistant, rather than LLM as such.
Code refactoring
A nice feature that JetBrains AI Assistant offers is to automatically refactor a method. It works increadibly well and allows to break down complex methods with a very low effort.
Example: In my game I need the player to check how they are doing and if they can finish the game within the time frame they selected in registration. For that I have a function that calculates a recommended game schedule (the player can then compare their times with a recommended schedule and see if they need to skip some game parts or not). The function consists of two for-loops. The first computes some numbers and the other uses those numbers for the final schedule calculation.
public function calculateSchedule(?DateTimeImmutable $gameStart, int $timeFrameMs)
{
$time = (new DateTime())->setTimestamp($gameStart->getTimestamp());
$locationsWeightSum = 0;
$puzzlesWeightSum = 0;
foreach (GameParts::map() as $gamePart) {
// compute $locationsWeightSum and $puzzlesWeightSum
// ...
}
foreach (GameParts::map() as $gamePart) {
// calculate an estimate for a recommended game part start and end (with use $locationsWeightSum and $puzzlesWeightSum)
// ...
}
}
Naturally, each for-loop would be better located in a separate well-named method. Let’s see what AI suggested:
public function calculateSchedule(?DateTimeImmutable $gameStart, int $timeFrameMs)
{
$time = (new DateTime())->setTimestamp($gameStart->getTimestamp());
list($locationsWeightSum, $puzzlesWeightSum) = $this->calculateGameWeightSums();
$this->calculateGamePartTimes($time, $locationsWeightSum, $puzzlesWeightSum, $timeFrameMs);
}
protected function calculateGameWeightSums(): array
{
$locationsWeightSum = 0;
$puzzlesWeightSum = 0;
foreach (GameParts::map() as $gamePart) {
// compute $locationsWeightSum and $puzzlesWeightSum
// ...
}
return [$locationsWeightSum, $puzzlesWeightSum];
}
protected function calculateGamePartTimes(DateTime $time, int $locationsWeightSum, int $puzzlesWeightSum, int $timeFrameMs)
{
foreach (GameParts::map() as $gamePart) {
// calculate an estimate for a recommended game part start and end (with use $locationsWeightSum and $puzzlesWeightSum)
// ...
}
}
So, it is exactly what one would expect. For simplicity I left out internals of both for-loops. So I cannot easily refer to how AI also refactored code within these for-loops. For example, I had there a piece of code that converted mutable DateTime to DateTimeImmutable. AI inferred a new method for this conversion called createDateTimeImmutable (this is not perfect naming in my opinion, but good enough for basic understanding). All of that happened in just one refactoring step. Method names have been derived based on understanding what methods are actually doing, not on comments.
From this I can see that AI is able to understand parts of code and summarise them into method names. That’s pretty cool and simplifying one’s life. Note that this assumes that exsiting methods and variables are reasonably named so that AI could read them easily.
There is one weak spot of the JetBrains AI refactoring feature: If one just copies suggestions and replaces code manually, all is good. But if one tries to apply a patch automatically, it almost always ends up with broken or partially lost code. I do not understand why this is happening and wasn’t fixed so far, I am just guessing that JetBrains do not test this functionality on PHP language that much.
Limited Use
JetBrains AI Assistant is limited just to help with coding. It cannot generate any images – I would find image generation useful in context of front-end development.
It is also refusing text generation that is not related to programming languages.
Sometimes it is willing to provide you some answers if you convince it you need actually a coding-related thing. E.g. I was able to convince it for some copy-writing with a prompt starting with something like: on my webpage I need a block of text that …
Conclusions
JetBrains AI Assistant obviously has a lot of potential. Being able to see data types and project structure should allow this tool to provide more precise answers and suggestions compared to AI tools running outside IDEs. To some extent it provides more precise answers. Nevertheless, as of today, JetBrain’s integration of the LLM is far from perfect – suggestions often lead to syntactically incorrect code or do not fully respect actual definitions of existing data types. I believe that the current version is an MVP and situation will dramatically improve in near future, but up until then, I am unsure if the tool is worth its price. Once JetBrains fixes at least the refactoring feature so that suggestions can be applied reliably, it could potentially become a great productivity booster for developers who want to write clean code.