Patent Application
20230376401
The embodiments relate generally to machine learning and auto-code generation, and more specifically to systems and methods for automatic program repair (APR) using retrieval-augmented patch generation (RAP-Gen).
Software developers often spend significant amount of time and energy for debugging and repairing their source code, rendering software development costly and time-consuming. Some existing automatic program repair tools may ease the difficulty and cost of program repair with use cases including search of patches at development time, build time or run time. For example, some search-based (also referred to as generate-and-validate) approach may search for repairs based on the fix patterns mined via manual heuristic rules or redundancy-based techniques. The redundancy-based techniques generally make a redundancy assumption that the fixed patch can often be found (or reconstructed) from elsewhere in the codebase (a donor code snippet). Thus, these conventional search-based techniques have limited accuracy and efficiency in repairing programs.
Therefore, there is a need for a more efficient way for automatic program repair.
In the figures, elements having the same designations have the same or similar functions.
As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and/or any training or learning models implemented thereon or therewith.
As used herein, the term “module” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.
Existing automatic program repair systems may reduce manual debugging efforts and improve software reliability. Conventional search-based techniques typically rely on heuristic rules or a redundancy assumption to mine fix patterns. Some deep learning-based approaches may automate the program repair process by training learning models to generate code repair patches. However, performance of such learning models is often limited by a fixed set of parameters to model the highly complex search space of program repair.
In view of the need for efficient and accurate code repair systems, embodiments described herein provide a retrieval-augmented patch generation framework to retrieve code patches, using a patch retriever based on relevant fix patterns. Specifically, a hybrid patch retriever may be configured for fix pattern mining that accounts for both lexical and semantic matching through sparse and dense retrieval based on the raw source code. The retriever is also a language-agnostic retriever as it does not require any language-specific features such as abstract syntax trees. One improvement from previous fix pattern mining models is that the retriever utilizes the top one relevant bug-fix pair as a guiding fix pattern for each buggy patch instead of clustering various fix templates. This strategy aligns with debugging behaviors of human developers, who often search for relevant bug-fix examples to distill some repair clues for bug fixing.
In one embodiment, a pretrained Transformer-based encoder-decoder model (e.g., a CodeT5 model) may be adopted as the foundation patch generator. CodeT5 is a generic programming language model pretrained on large source code corpora using code-aware language modeling objective. A two-stage training strategy may be used to train the pretrained encoder-decoder model to connect the patch retriever and CodeT5 patch generator. The patch retriever first searches for relevant bug fix patterns and then pass them to patch generator for synthesizing a fixed patch based on both the source buggy code and the external (retrieved) bug fix knowledge. The retrieved fix pattern may then be directly appended into the source buggy patch. In this way, the retriever may be integrated with any sequence-to-sequence learning based model for retrieval in fix-pattern mining for program repair.
Memory 120 may be used to store software executed by computing device 100 and/or one or more data structures used during operation of computing device 100. Memory 120 may include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.
Processor 110 and/or memory 120 may be arranged in any suitable physical arrangement. In some embodiments, processor 110 and/or memory 120 may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor 110 and/or memory 120 may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor 110 and/or memory 120 may be located in one or more data centers and/or cloud computing facilities.
In some examples, memory 120 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 110) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory 120 includes instructions for the automatic program repair module 130 that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. An automatic program repair module 130 may receive input 140 that includes an input such as a program bug, and/or the like via the data interface 115. The automatic program repair module 130 may generate an output 150 such as a code patch.
In some embodiments, the automatic program repair module 130 includes the retriever encoder submodule 131, the patch retriever submodule 132, and the patch generator submodule 133. In one embodiment, the automatic program repair module 130 and its submodules 131-133 may be implemented by hardware, software and/or a combination thereof.
Some examples of computing devices, such as computing device 200 may include non-transitory, tangible, machine readable media that include executable code that when run by one or more processors (e.g., processor 110) may cause the one or more processors to perform the processes of method. Some common forms of machine-readable media that may include the processes of method are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.
The user device 210, data vendor servers 245, 270 and 280, and the server 230 may communicate with each other over a network 260. User device 210 may be utilized by a user 240 (e.g., a driver, a system admin, etc.) to access the various features available for user device 210, which may include processes and/or applications associated with the server 230 to receive an output data anomaly report.
User device 210, data vendor server 245, and the server 230 may each include one or more processors, memories, and other appropriate components for executing instructions such as program code and/or data stored on one or more computer readable mediums to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer readable media such as memories or data storage devices internal and/or external to various components of system 200, and/or accessible over network 260.
User device 210 may be implemented as a communication device that may utilize appropriate hardware and software configured for wired and/or wireless communication with data vendor server 245 and/or the server 230. For example, in one embodiment, user device 210 may be implemented as an autonomous driving vehicle, a personal computer (PC), a smart phone, laptop/tablet computer, wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other type of wearable computing device, implantable communication devices, and/or other types of computing devices capable of transmitting and/or receiving data, such as an IPAD® from APPLE®. Although only one communication device is shown, a plurality of communication devices may function similarly.
User device 210 of
In various embodiments, user device 210 includes other applications 216 as may be desired in particular embodiments to provide features to user device 210. For example, other applications 216 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) over network 260, or other types of applications. Other applications 216 may also include communication applications, such as email, texting, voice, social networking, and IM applications that allow a user to send and receive emails, calls, texts, and other notifications through network 260. For example, the other application 216 may be an email or instant messaging application that receives a prediction result message from the server 230. Other applications 216 may include device interfaces and other display modules that may receive input and/or output information. For example, other applications 216 may contain software programs for asset management, executable by a processor, including a graphical user interface (GUI) configured to provide an interface to the user 240 to view the buggy code and/or fixed code.
User device 210 may further include database 218 stored in a transitory and/or non-transitory memory of user device 210, which may store various applications and data and be utilized during execution of various modules of user device 210. Database 218 may store user profile relating to the user 240, predictions previously viewed or saved by the user 240, historical data received from the server 230, and/or the like. In some embodiments, database 218 may be local to user device 210. However, in other embodiments, database 218 may be external to user device 210 and accessible by user device 210, including cloud storage systems and/or databases that are accessible over network 260.
User device 210 includes at least one network interface component 219 adapted to communicate with data vendor server 245 and/or the server 230. In various embodiments, network interface component 219 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and/or various other types of wired and/or wireless network communication devices including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.
Data vendor server 245 may correspond to a server that hosts one or more of the databases 203a-n (or collectively referred to as 203) to provide training datasets including pairs of buggy code and fixed code to the server 230. The database 203 may be implemented by one or more relational database, distributed databases, cloud databases, and/or the like.
The data vendor server 245 includes at least one network interface component 226 adapted to communicate with user device 210 and/or the server 230. In various embodiments, network interface component 226 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and/or various other types of wired and/or wireless network communication devices including microwave, radio frequency, infrared, Bluetooth, and near field communication devices. For example, in one implementation, the data vendor server 245 may send asset information from the database 203, via the network interface 226, to the server 230.
The server 230 may be housed with the automatic program repair module 130 and its submodules described in
The database 232 may be stored in a transitory and/or non-transitory memory of the server 230. In one implementation, the database 232 may store data obtained from the data vendor server 245. In one implementation, the database 232 may store parameters of the automatic program repair module 130. In one implementation, the database 232 may store previously generated fixed patch of code and the corresponding input feature vectors.
In some embodiments, database 232 may be local to the server 230. However, in other embodiments, database 232 may be external to the server 230 and accessible by the server 230, including cloud storage systems and/or databases that are accessible over network 260.
The server 230 includes at least one network interface component 233 adapted to communicate with user device 210 and/or data vendor servers 245, 270 or 280 over network 260. In various embodiments, network interface component 233 may comprise a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and/or various other types of wired and/or wireless network communication devices including microwave, radio frequency (RF), and infrared (IR) communication devices.
Network 260 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 260 may include the Internet or one or more intranets, landline networks, wireless networks, and/or other appropriate types of networks. Thus, network 260 may correspond to small scale communication networks, such as a private or local area network, or a larger scale network, such as a wide area network or the Internet, accessible by the various components of system 200.
The task formulation of the retrieval-augmented patch generation for automatic program repair is described as follows.
be a program repair dataset consisting of |D| bug-fix pairs (XiYi), where Xi and Yi are i-th buggy and fixed program patch, respectively. A codebase C (e.g., codebase 302) containing a large collection of previous bug-fix pairs
where (Bj, Fj) denotes the j-th bug-fix pair. Given a buggy program patch Xi 308 in D, a patch retriever 304 retrieves one or more most relevant bug-fix pair(s) (Bj, Fj) in the codebase C based on a relevance scoring function ƒϕ(Xj, Bj) parameterized by ϕ.
In some embodiments, the original input sequence Xi 308 is augmented with the retrieved bug-fix pair to form a new input sequence 312, e.g., {circumflex over (X)}i=Xi⊕Bj⊕Fj, where ⊕ denotes the concatenation operation. A patch generator 306 (e.g., using a sequence-to-sequence (seq2seq) generator, and also referred to as a sequence generator 306) may then generate Yi 316 from {circumflex over (X)}i 312 in an autoregressive manner. The framework 300 may learn the probability Pθ(Yi|{circumflex over (X)}i)=Πk=1nPθ(Yi,k|{circumflex over (X)}iYi,1:Yi,k-1) with the patch generator 306 parameterized by θ, where Yi,1:Yi,k-1 is the previous sequence before the k-th token and n denotes the number of tokens in the target sequence Yi. In some embodiments, the external codebase C 302 may be regarded as a non-parametric memory, and the retrieved bug-fix pair 310 may be regarded as a guiding fix pattern for the patch generation model 306. In probabilistic terms, the retrieval Zj(Bj, Fj) may be formulated as a latent variable, which may be approximated by top-1 in some cases. Formally,
where Zj* is the top-1 retrieved output from the retriever Pϕ(Zj|Xi). The top-1 approximation may be adopted for improved efficiency, as marginalization over k>1 makes the training and inference complicated and inefficient. In some embodiments, top-k (e.g., k=2,3,5) with the Fréchet inception distance (FiD) method may be used.
As shown in the example of
Lexical-based Retriever. In some embodiments, the lexical-based retriever (e.g., BM25) may be implemented using a term-based retriever, and may use sparse vector representation for lexical matching. The lexical-based retriever may convert each code patch as bag-of-words representation, and compute a lexical similarity between the query patch Xi and a candidate patch Bj. The computed similarity score is represented as ƒϕ(Xi, Bj)=BM25(Xi, Bj). In an example, a sparse term-based retriever may be sensitive to the choice of identifier naming in source code that does not impact the code semantics.
Semantic-based Retriever. In some embodiments, the semantic based retriever may be implemented using Dense Passage Retriever (DPR), and may retrieve relevant patches via measuring their semantic similarity. In some embodiments, to encode the code patch, an encoder (e.g., a Transformer-based encoder) may be used to map each patch to a fixed-size dense vector. A DPR may be initialized from an encoder of a pretrained transformer-based neural network model (e.g., Code Bidirectional Encoder Representations from Transformers (CodeBERT), etc.). The encoder may be pretrained using a large code repository in one or more programming languages (e.g., GitHub code repositories in six programming languages). In an example, the final layer hidden state of a [CLS] token from the encoder is used as the patch representation. In some embodiments, a shared DPR may be used to separately encode the query patch Xi 308 and a candidate patch Bj in C as CLSX
ƒϕ(Xi,Bj)=sim(Xi,Bj)=[CLSX
In some embodiments, a shared DPR may be used to separately encode the query patch Xi 308 and a candidate patch Fj in C as CLSX
ƒϕ(Xi,Fj)=sim(Xi,Fj)=[CLSX
While the descriptions herein generally use a similarity between Xi and Bj (e.g., using ƒϕ(Xi, Bj)) for retrieval, it is noted that the similarity used for retrieval may include similarity between Xi and Bj (e.g., using ƒϕ(Xi, Bj)), similarity between Xi and Fj (e.g., using ƒϕ(Xi, Fj)), and/or a combination thereof.
In some embodiments, the semantic based retriever (e.g., DPR) is further trained, using a training dataset including pairs of a buggy patch and a fixed patch. In an example, the codebase 302 including the bug-fix pairs may be used, by considering the buggy code Bj as the query and the corresponding fixed code Fj as the key. This may be performed based on the assumption that the buggy patch and its fixed patch often shares similar semantics (e.g., identifiers, data flow, and code structures). This technique may be used to avoid the massive manual annotation efforts needed to curate a bug-to-bug search dataset.
In an example where bug-fix pairs are used as query and corresponding key, contrastive learning with in-batch negatives method 314 is used for training the semantic based retriever, where in-batch negatives are used to optimize a contrastive loss (e.g., an InfoNCE contrastive loss) as the following:
where M is the current minibatch, and N denotes the number of positive training examples in the minibatch. This objective aims to maximize the similarity between positive examples while minimizing the similarity between negative examples. Each positive example may have |M|−1 negative samples. It is noted that various contrastive learning techniques, e.g., in-batch negatives strategy, hard negative mining strategy, etc. may be used, while in some embodiments, the contrastive learning with in-batch negatives as described above provides better performance than the hard negative mining strategy for noisier training data.
In some embodiments, at the inference stage, given a query buggy patch Xi 308, the semantic-based retriever (e.g., DPR) retrieves a relevant bug-fix pair (Bj, Fj) by computing the similarity between Xi (query) and Bj (key). In some embodiments, the semantic-based retriever may retrieve a relevant bug-fix pair based on the similarity between Xi and Fj, and/or a combination with the similarity between Xi (query) and Bj (key).
Hybrid Retriever. As shown in the example of
In the example of
In some embodiments, a patch generator 306 includes a code-aware programming language model pretrained on a large-scale source code corpora. In an example, the sequence generator uses CodeT5, which is a unified pretrained Transformer-based encoder-decoder model that achieves state of art (SoTA) results in multiple code intelligence tasks such as defect detection and code refinement. It may be pretrained on 8.3 million functions in 8 different programming languages (including JavaScript and Java) collected from GitHub. CodeT5 may employ identifier-aware pretraining objectives to incorporate the code-specific knowledge into the language model. It may provide a code-specific Byte-Pair Encoding (BPE) tokenizer optimized for code, and may be able to avoid Out-of-Vocabulary (OoV) problems. CodeT5 may be used in the patch generator 306, which may provide powerful code understanding capability.
As shown in the example of
In various embodiments, the RAP-Gen framework 300 leverages the general code understanding knowledge encoded via pretraining on a large-scale code corpus (e.g., using CodeT5). For example, the source input sequence 312 may be generated by concatenating the original buggy code patch 308 and the top ranked bug-fix pair 310 from patch retrievers 304. In some embodiments, the augmented source input buggy patch 312 may be generated by concatenating the top-k (e.g., k=2,3,5) retrieved bug-fix pairs to the input buggy patch 308.
At step 402, a patch retriever including a retriever encoder is provided. In the example of
At step 406, a patch generator including a sequence generator neural network model is provided. In the example of
At step 410, a RAP-Gen framework (e.g., RAP-Gen framework of
The two-stage training process includes step 414, at which a second stage training is performed by training the patch generator using a fourth training dataset, using the patch retriever trained by the first stage training. In an example, a teacher forcing algorithm is used to minimize the language modeling loss, when input to the patch generator is generated using an original input buggy code patch and the top ranked bug-fix pair from the trained patch retriever.
During the second stage training, in an example where the fourth training set is generated from the bug-fix pair codebase, the patch retriever (already trained using the first stage training) is not allowed to access the ground-truth bug-fix pair, otherwise the training loss would easily drop close to zero as the patch generator may directly copy the retrieved fix as the target output. In that example, each sample of the fourth training set is a buggy patch of a corresponding bug-fix pair (also referred to as the ground-truth bug-fix pair) from the codebase, and the corresponding ground-truth is the fixed patch of the corresponding bug-fix pair. For each sample buggy patch input, another bug-fix pair (not the ground-truth one) is retrieved by the patch retriever from the codebase. The retrieved bug-fix pair is appended to the buggy patch input to generate an augmented sequence input for the patch generator. Note that the requirement of no access to ground-truth bug-fix pair only applies to the second stage of training when the codebase is used to provide the fourth training set, and does not apply to the first stage of training the patch retriever when the codebase is used to provide the third training set.
How the third and fourth datasets are generated? Recall that we have bug-fix pairs for each downstream dataset, which are exactly the third training set.
Referring to
At step 452, a first buggy patch is received by the trained retrieval-augmented patch generation framework. In the example of
At step 454, one or more bug-fix pairs are provided based on the first buggy patch. In the example of
At step 456, a first augmented buggy patch is generated based on the first buggy patch and the retrieved one or more bug-fix pairs. In the example of
At step 458, a first fixed patch for the first buggy patch is generated using the first augmented buggy patch. In the example of
Referring to
TFix. Specifically, TFix is a large-scale program repair dataset comprising JavaScript code patch pairs curated from 5.5 million GitHub commits. It comprehensively covers 52 unique error types detected by a static analyzer ESLint. In addition to error types, it provides rich error annotations such as error message and localized error line so that there is no need for fault localization like prior work. In TFix, they approach the APR task as a text-to-text generation problem with T5-large. In the source input sequence, they combine all error information together with the buggy code patch into a single piece of text:
During data processing, a duplication issue inside data splits and between data splits is observed. Specifically, there are 114, 2, and 4 duplicates in the train, validation, and test split respectively. For inter-split duplicates, there are 28, 34, and 4 duplicates between train and test, train and test, validation and test splits respectively. Those duplicates (243) are filtered, and a deduplicated version TFix (Dedup) is shown in Table 1 of
Code Refinement. Tufano et al. released two code refinement datasets containing bug-fix pairs at function level, which are collected from public GitHub Archive (https://www.gharchive.org/) between March 2011 and October 2017. They use Google BigQuery APIs to identify all Java commits having a message containing the patterns: (“fix” or “solve”) and (“bug” or “issue” or “problem” or “error”) to ensure the quality of the collected bug-fix function pairs. They normalized the functions via obfuscating identifiers with indexed tokens such as TYPE1, VAR1, METHOD1, etc. One data example can be found in
In some embodiments, the RAP-Gen framework 300 may be fine-tuned (e.g., for 30 epochs) with a sequence-to-sequence generation loss for each benchmark, e.g., using an AdamW optimizer (Ilya Loshchilov and Frank Hutter DecoupledWeight Decay Regularization, ICLR, 2019). Grid search may be conducted for hyper-parameter tuning, with various batch sizes (e.g., 16, 32, 64) and learning rates (e.g., 1e-4, 5e-5, 2e-5). For example, a batch size of 64 with a learning rate of 1e-4 may be used for TFix, and a batch size of 32 with a learning rate of 5e-5 for Code Refinement. In an example, the training time of RAP-Gen-base on each benchmark with one A100 GPU is within 2 days. During inference, beam search may be employed with a beam size of five to produce a ranked list of synthesized fixed patches.
In some embodiments, bug-fix pairs in the training set are adopted as a search codebase to build the patch retriever 304. For lexical-based retrievers, an example open-sourced Python library (e.g., https://pypi.org/project/rank-bm25 of BM25) may be used. As a sparse term-based retriever, the choice of tokenizer would largely affect the retrieval performance. In an experiment, the CodeT5 tokenizer, from which is a code specific BPE tokenizer optimized for code tokenization, is adopted. A BM25 search engine on benchmarks TFix and Code Refinement is applied on a machine of 95 CPUs with 600G memory. Each experiment is finished within one hour with multi-processing.
In the experiment, for semantic-based retrievers, the DPR initialized CodeBERT is used to encode each patch into dense vectors for semantic matching. Separately, a DPR model is fine-tuned on each benchmark for 50 epochs using the InfoNCE contrastive loss. A batch size of 64 and a learning rate of 2e-5 is used to fine-tune on one A100 GPU with 40G memory. The training time for TFix and Code Refinement are around 9 and 5 GPU hours respectively.
For hybrid retrievers, the ranking scores of BM25 and DPR are calculated, and these normalized scores are linearly combined with equal weights to build a hybrid retriever, namely “Hybrid”. For all retrievers, the CodeT5 tokenizer is used to encode the patch with a maximum sequence length of 256.
Evaluation Metrics. For evaluation metrics, the smoothed BLEU-4 (Chin-Yew Lin and Franz Josef Och, ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation, COLING, 2004) scores and Exact Match (EM) accuracy are used to evaluate program repair performance (e.g., following Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi, CodeT5:Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation, EMNLP, Association for Computational Linguistics, 8696-8708). BLEU-4 is a looser metric to evaluate the degree of sub-word overlapping, while EM is a stricter metric requiring the prediction is identical to the ground-truth patch in a real commit. As a buggy program might have different ways to repair, Error Removal metric (e.g., as used in TFix) is used to take various forms of fixes into account. The prediction is counted as correct for Error Removal if the existing error is removed and no new error is introduced after the fix. For all metrics, results are presented in a scale of 0-100(%), and a higher score represents a better performance.
Baseline Models. The RAP-Gen framework is compared with learning based models in two program repair benchmarks. CoCoNuT is a context-aware neural machine translation framework based on convolutional encoder-decoder model. SequenceR is an LSTM-based sequence-to-sequence generation model with copy mechanism. In addition, the RAP-Gen framework is compared with pretrained programming language models based on Transformer architecture. One group of these models is the encoder-only models such as RoBERTa (code), CodeBERT, and GraphCodeBERT. These encoder-only models require a randomly initialized decoder for program repair tasks.
Furthermore, the RAP-Gen framework is compared with encoder-decoder Transformer models. PLBART is a unified pretrained model with denoising objectives including token masking, token deletion, and token infilling. TFix is initialized with T5-large checkpoints and continue to fine-tune on TFix dataset. CoTexT is another T5-based models pretrained on both text and code. NSEdit is a language model with encoder and decoder initialized from CodeBERT and CodeGPT respectively. It is fine-tuned to generate the fix via a neural-symbolic editing sequence and ranks as the current SoTA model on the Code Refinement benchmark. Results from all baseline models are obtained from their original paper.
The experiments validate that the retrieval-augmented patch generation is an effective approach for program repair. Comprehensive experiments were conducted to compare the RAP-Gen with prior learning-based methods on two benchmarks. First, CodeT5 models are evaluated on TFix, and its evaluation is improved by providing a deduplicated version of dataset and a more reasonable metric, and by additionally introducing a looser metric of BLEU-4 score which is aligned with exact match. Results show that CodeT5-base establishes a new SoTA performance on this task, improving T5-large's 49.70 to 53.57 in EM and 76.98 to 78.85 in BLEU-4. Further, RAP-Gen models are evaluated using both TFix and Code Refinement datasets. It is observed that RAP-Gen with lexical and semantic-based retrievers significantly boost the performance. Specifically, RAP-Gen-base with “Hybrid” improves the exact match over the best performing baseline (49.70→54.15) in TFix, while RAP-Gen-base with “Hybrid” boosts the exact match (24.04→24.80) in the small set and (14.18→15.84) in the medium set of the Code Refinement benchmark. All these results validate that retrieval-augmented patch generation (RAP-Gen) is an effective approach for APR.
The experiments illustrate that retrieval-augmented patch generation with CodeT5 is an effective approach for program repair. First, CodeT5 is compared with traditional APR techniques on TFix benchmark, improved with a deduplicated version of the data and a more appropriate evaluation metric. Then RAP-Gen framework integrated with two sizes of CodeT5 is evaluated on TFix and Code Refinement benchmarks. Further, the experiments illustrate that the patch retriever finds relevant patches in terms of lexical and semantic similarity. In addition, case studies are provided to illustrate how retrieved bug fix patterns help in program repair. In addition, as shown by the experiments, the RAP-Gen framework provides improved performance for various error types and fix patterns. Detailed performance breakdown for 52 error types are listed, and types of error that do not benefit from the retrieval-augmentation in RAP-Gen are examined. Furthermore, how models perform with one trivial but dominating fix pattern of error line removal that simply removes the error line from the buggy code is studied.
Experiments illustrate that retrieval-augmented patch generation with CodeT5 is an effective approach for program repair. First, it provides improved TFix evaluation. The original TFix benchmark employs the direct average of exact match (EM) accuracy across 52 error types as the main evaluation metric. However, as shown in Table 7 of
As shown in Table 2 of
Next ablation study observation is described. On the deduplicated TFix dataset, the performance across various metrics consistently drops a little bit. This is an expected phenomenon as duplications (34 instances) between the train and test splits in the original data would lead to a data leakage issue and improperly increase the performance. If the error information including error type and error message is removed, both CodeT5-small and CodeT5-base models witness a consistent performance downgrade, revealing that it is helpful to inform which types of error they need to fix for program repair models.
Referring to Table 3 of
In some embodiments, there can be multiple ways to fix a bug. As such, exact match with one ground-truth patch would be a too strict metric to consider other forms of correct fixes. To deal with this, a looser evaluation with error removal metric following TFix is used. Under this metric, a fixed patch is regarded as correct as long as it resolves the errors in source buggy patch and does not bring new errors (detected by the static analyzer ESLint). When it is tried to reproduce this metric on 10,465 test instances, there are two difficulties: (1) Applying ESLint requires the full file contexts for each code patch, but we found 95 code files are no longer available to retrieve. (2) There are parser errors when applying ESLint with the released configuration (https://github.com/eth-sri/TFix) on some of data samples. As a result, a filtered subset of 6,793 instances is curated by excluding those unavailable code files and samples with parser errors, where it is also spotted that generated fixes from TFix tends to have more parser errors. Referring to Table 4 of
Referring to Table 5 of
From the RAP-Gen model comparison, it is observed that RAP-Gen with various retrievers consistently boost the performance over their CodeT5 counterparts. The best model establishes new SoTA results on two subsets (24.80 EM for small and 15.84 EM for medium), especially surpassing the NSEdit for around 2 absolute points on the more challenging medium set. This again confirms that retrieved fix patterns provide helpful signals to guide the program repair. Among various retrievers, DPR gives better results than BM25 for both RAP-Gen-small and RAP-Gen-base, revealing that semantic information might play a more important role than semantic information for this benchmark. Besides, “Hybrid” outperforms BM25 and DPR, implying the hybrid ensembling method is a more robust retriever to balance both semantic and semantic information for this benchmark.
In summary, comprehensive experiments are performed to compare RAP-Gen with prior learning-based methods on two benchmarks. First evaluate CodeT5 models on TFix and improve its evaluation via providing a deduplicated version of dataset and a more reasonable metric, and additionally introducing a looser metric of BLEU-4 score which is aligned with exact match. Results show that CodeT5-base establishes a new SoTA performance on this task, improving T5-large's 49.70 to 53.57 in EM and 76.98 to 78.85 in BLEU-4. We then evaluate RAP-Gen models both TFix and Code Refinement datasets and observe that RAP-Gen with lexical and semantic-based retrievers significantly boost the performance. Specifically, RAP-Gen-base with “Hybrid” improves the exact match over the best performing baseline (49.70→54.15) in TFix, while RAP-Gen-base with “Hybrid” boosts the exact match (24.04→24.80) in the small set and (14.18→15.84) in the medium set of the Code Refinement benchmark. All these results validate that retrival-augmented patch generation with CodeT5 (RAP-Gen) is an effective approach for APR.
Next, experiments are performed to assess whether the patch retriever is able to find relevant fix patterns to benefit program repair. First, an automatic evaluation to measure the relevance in terms of lexical and semantic similarity between the query and retrieved patches is provided. Further, specific cases are provided to understand how the retrieved fix patterns contribute to better APR.
Referring to Table 6 of
Refinement, where the latter employs obfuscated identifiers (e.g., VAR1, VAR2, . . . ) that hinders the performance of the lexical-based BM25 retriever. The hybrid retriever achieves the best lexical matching on all datasets, revealing the semantic information can complement to the lexical matching.
For semantic matching, DPR achieves the best results on all datasets, which is not surprising as it is optimized towards the identical objective. Notably, the hybrid retriever achieves slightly lower results than DPR but much better results than BM25, implying it may balance both lexical and semantic information and be more robust than the lexical-based retrievers, which are sensitive to the choices of identifier naming.
Referring back to
As such, both quantitative (Table 6 of
Referring to
Further, the effects of retrieval-augmentation in RAP-Gen compared to CodeT5 models for various error types are analyzed. As shown in Table 7 of
To explore why retrieval-augmentation sometimes hinders the exact match performance in RAP-Gen models, a case study for the “no-console” error type is provide in
Next, it is analyzed what fix patterns are performed by the models using the TFix benchmark. After manually inspecting the bug-fix pairs, it is observed that a large proportion of fix consists of deletion operations compared to the code insertion and replacement operations. The bug fix operations consist of code insertion (12.5%), replacement (8.1%), deletion (47.9%), insertion and replacement (6.9%), insertion and deletion (8.2%), replacement and deletion (7.2%), and all three manners (9.2%). Earlier studies also reflect that the deletion operation is one of the most common fix patterns. Among the deletion operations, one dominating bug fix pattern is error line removal, which is to simply remove the error line from the buggy code (such as the example shown in
In summary, the difficulty of program repair varies from error type to error type. The best RAP-Gen-base in experiments may repair more 456 buggy programs than the best performing baseline T5-large. An error analysis is conducted to analyze why retrieval-augmentation sometimes downgrades the performance and a case study is provided to illustrate that it might be due to the limitations of the exact match metric. Moreover, one high-frequent fix pattern of error line removal is investigated to show that RAP-Gen-base gives the best precision score and RAP-Gen-small achieves the best recall and F1 scores in dealing with this pattern.
This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the embodiments of this disclosure. Like numbers in two or more figures represent the same or similar elements.
In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.
Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.
This application the claims benefit of U.S. Provisional Application No. 63/343,264, filed May 18, 2022, which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63343264 | May 2022 | US |
