The first international competition for Automated Program Repair (APR), witnessed 15 repair tools with 10 participating tools and
5 baseline tools. The competition carried out 3 tracks focusing on 3 different repair tasks, each track had two sub-tracks focused on two
prominent programming languages in the domain. The competition altogether evaluated 475 unique repair tasks.
This track is focused on fixing incorrect student submissions for CS assignments evaluated with Introductory Programming courses.
Repair tools will be fixing student assignment to provide tailor-made feedback for the incorrect submission.
Python Sub-Track
| Tool |
Patches Generated |
Incorrect Patches |
Partially Correct Patches |
Correct Patches |
Score |
| LLMR |
434 |
26 |
41 |
367 |
339.6 |
| Brafar |
98 |
12 |
12 |
74 |
308 |
| Refactory |
32 |
6 |
3 |
23 |
92 |
C Sub-Track
| Tool |
Patches Generated |
Incorrect Patches |
Partially Correct Patches |
Correct Patches |
Score |
| LLMR |
311 |
160 |
7 |
144 |
153.3 |
| VeriFix |
7 |
0 |
0 |
7 |
28 |
| F1X |
7 |
0 |
5 |
2 |
6 |
This track is focused on fixing incorrectly generated code by CodeX for general algorithmic problems curated in LeetCode .
Repair tools will be fixing LLM generated code which fails on provided test cases.
Java Sub-Track
| Tool |
Patches Generated |
Incorrect Patches |
Partially Correct Patches |
Correct Patches |
Score |
| ARJA-E |
25 |
0 |
0 |
25 |
20 |
| ET |
34 |
19 |
2 |
13 |
10 |
| APRER |
23 |
0 |
18 |
5 |
8 |
| LLMR |
25 |
25 |
0 |
0 |
-5.6 |
| TBar |
6 |
6 |
0 |
0 |
-24 |
| RepairLLAMA |
381 |
379 |
1 |
1 |
-105.7 |
| ARJA |
0 |
0 |
0 |
0 |
0 |
Python Sub-Track
| Tool |
Patches Generated |
Incorrect Patches |
Partially Correct Patches |
Correct Patches |
Score |
| RepairCATPython |
56 |
0 |
56 |
0 |
16 |
| Darjeeling |
0 |
0 |
0 |
0 |
0 |
| LLMR |
477 |
401 |
12 |
64 |
-26.3 |
This track is focused on fixing functional correctness errors in real-world software programs.
Repair tools will be fixing bugs with a failing test-case, accompanied by a set of passing test cases from a real-world application.
Java Sub-Track
| Tool |
Patches Generated |
Incorrect Patches |
Partially Correct Patches |
Correct Patches |
Score |
| ET |
6 |
0 |
1 |
5 |
6 |
| LLMR |
5 |
5 |
0 |
0 |
-2 |
| RepairLLAMA |
46 |
44 |
1 |
1 |
-31.67 |
| TBar |
0 |
0 |
0 |
0 |
0 |
| ARJA |
0 |
0 |
0 |
0 |
0 |
| APRER |
0 |
0 |
0 |
0 |
0 |
| ARJA-E |
0 |
0 |
0 |
0 |
0 |
| RepairCatJava |
0 |
0 |
0 |
0 |
0 |
| GRT5 |
0 |
0 |
0 |
0 |
0 |
C Sub-Track
| Tool |
Patches Generated |
Incorrect Patches |
Partially Correct Patches |
Correct Patches |
Score |
| F1X |
10 |
0 |
0 |
10 |
8 |
| Darjeeling |
0 |
0 |
0 |
0 |
0 |
| LLMR |
5 |
5 |
0 |
0 |
-1 |