Patent Application
20130236860
The present embodiment relates to the field of test engines. More specifically, it relates to a system and method for testing a test-taker's programming skills.
Information technology companies assess a potential test-taker for a job based on numerous skill sets like logical ability, quantitative ability, communication skills, or programming skills.
While there are many online tests available for testing a person's Logical, Verbal, Quantitative, and other aptitude skills, there are no standardized tests available for the comprehensive testing of a person's skill for the job readiness in Information Technology sector, especially positions requiring real time programming.
Testing a test-taker on real time programming ability is a crucial benchmark to gauge On-Job performance, yet no standard procedure has been followed with regards to employability assessment of technology professionals Rather, most of the online tests available test the programming ability through a series of objective questions. While this methodology can test the theoretical knowledge about a programming language, it falls short of assessing practical programming ability in a real life scenario. The theoretical questions mostly have singular correct response while the programming problems can have numerous correct solutions. More so, writing a computer program is normally an open ended problem, with different programmers solving the same problem in a different manner, with a different coding style, level of optimality etc, and checking a program for correctness, on all parameters becomes difficult, if not impossible. Thus, the key reason for the lack of such automated assessment tests is the difficulty in designing an assessment engine which can exhaustively determine the correctness, brevity, and benchmarks a human evaluator will use to determine the quality of the response to a programming problem.
The present embodiment relates to a test engine for capturing and analyzing a candidate's response to a set of objective and subjective questions relating to programming skills of the candidate. The subjective questions for the computer language testing are powered by a strong backend engine, namely test engine, which captures without limitation, the candidate's programming style, compile time and run-time errors made during the coding, ability to write a program towards a desired objective and the like. The test engine is integrated to an adaptive engine, to determine the next question, based on the correctness of the response gauged by the test engine. The test engine is powered to providing snippets of pre-coded headers and footers to the test-taker, so as to restrict him to write the code snippet solving the desired programming problem. Further, the test engine is capable of determining the correctness of the code written in response to the question, by comparing it to for example, a finite state machine, a base algorithm etc. written to capture the intent of the program.
Accordingly, the present embodiment discloses a test engine to evaluate a test taker's programming skills. The test engine includes at least one partial program logic module that defines a set of predefined conditions in relation to the solution to a problem, an input module that receives test taker input relating to the problem and an evaluation module that combines the predefined conditions and the test taker input to form a complete functionality/program to be evaluated and assigns a score based upon at least the correctness of the complete functionality/program. The predefined conditions may be static conditions or editable conditions.
In another embodiment, a method of evaluating a test-taker's performance is disclosed. The method formulates a test problem and defines at least one set of predefined conditions relating to the test problem. It then receives the inputs from the test taker relating to the problem and combines the predefined conditions and the test taker inputs to form a complete functionality/program. Lastly, it evaluates the complete functionality/program and assigns a score based upon at least the correctness of the complete functionality/program.
As required, detailed embodiments of the present embodiment are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the embodiment, which can be embodied in various forms. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present embodiment in virtually any appropriately detailed structure. Further, the terms and phrases used herein are not intended to be limiting; but rather, to provide an understandable description of the embodiment.
The terms “a” or “an”, as used herein, are defined as one or more than one. The term plurality, as used herein, is defined as two or more than two. The terms including and/or containing, as used herein, are defined as comprising (i.e., open language). The term coupled/communicates, as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically.
Test engine 108 includes a partial program logic module 112 and an evaluation module 114 which in turn may include one or more of a compiler 116, a graph flow generator (GFG) 118, a code analyzer (CA) 120 and a test module 122. In various embodiments, test engine 108 may include a single partial program logic module 112 or multiple partial program logic modules. The partial program logic modules 112 may also be referred to as precoded header/pre-code and/or precoded footer/post-code. The partial program logic module 112 includes a set of predefined conditions in relation to the solution to a problem. The predefined conditions include without limitation one or more of a function or a part of a function, a behavior, a class definition or a part of class definition, a source code, a control flow graph, a finite state machine, an object code, an interpreted code (IC), snippets of code inside function, parts of code distributed across files, or across pre-compiled software libraries. The partial program logic module 112 may be stored on memory 104 or another storage device like external memory provided with system 100. The problem includes, for example, a programming problem to be provided to the test-taker whose programming skills are to be tested.
The evaluation module 114 combines the predefined conditions, and the test-taker's input to form a complete functionality/program to be evaluated and assigns a score based upon at least the correctness of the complete functionality While assigning the score, the evaluation module 114 may further take into account brevity/quality of the test-taker's inputs, number of compilation attempts, and the like. The test-taker provides the input via the input device 110. The evaluation module 114 includes a transformation module 124 that transforms the test-taker's input to one or more of a compiled code, an interpreted code, a flow graph, an object code or an executable code. The evaluation module 114 combines this transformed form with the corresponding transformed form of the predefined conditions to form a complete functionality For example, if the test taker input is transformed to a compiled code, the evaluation module 114 combines this compiled code with the compiled version of the predefined conditions. Thereafter, the evaluation module 114 may use the compiler 116 or the graph flow generator 118 or the code analyzer 120 or the test module 122 to test the complete functionality/program. For example, compiler 116 may test the types of compile time errors, number of compile time errors made in the complete program, etc. These parameters may then be used to assign a score for the problem. Alternatively, the evaluation module 114 may use the graph flow generator 118 to generate a flow graph of the complete program, by either generating the complete flow graph of the program anew or combining the flowgraph of the test taker input with the flowgraph for the pre-code and post-code stored in the evaluation module, and then compare it with the predefined flow graph for the problem for checking the isomorphism of the complete program. Thereafter, a score to the complete program is assigned.
In yet another embodiment, the evaluation module 114 uses the code analyzer 120 to analyze the test-taker's input against a benchmark and thereafter assigns a score. Alternatively, the evaluation module 114 uses the test module 122 to test the test-taker's inputs by using the test specification and set of test functions defined for the problem. It is to be noted that the evaluation module 114 may use one or more of the four modules (116, 118, 120 or 122) to assign the score. For example, the evaluation module 114 may use the compiler 116 and the test module 122 to assign a score.
The test engine 108 may be provided with the memory 104 or an external memory that is integrated with system 100 and is executed by processor 102.
The pre-code 202 and post-code 204 are predetermined fragments of code that form a part of the function body that comprises the response to the programming problem in question. Also, the pre-code 202 and post-code 204 might not necessarily be a part of a programming function, but might also be a complete file or a set of files, or precompiled program libraries. This emulates closely the real life, on-job, scenario that a programmer faces where he/she has to make modifications, and insert piece of code, when the rest of the programming logic exists across multiple files, and/or pre-compiled libraries, so as to ensure that the complete program/functionality function behaves in a desired manner In various embodiments, the pre-code 202 and/or the post-code 204 may be static, thereby not editable by the test-taker or the pre-code 202 and/or the post-code 204 may be editable by the test-taker. The corresponding flow-graph and/or compiled/interpreted code of the pre-code 202 and the post-code 204 are stored in the memory 104 (or an external memory) along with problem statement. This imposition helps to restrict the variable part of the program, dynamically entered by the test-taker or the test-taker to minimal, while the crucial aspects of solving a programming problem are tested. Hence the present embodiment, through the concept of pre-code and post-code spread across a single or multiple files (and/or functions/procedures/classes/entities, based on the language and context of the program), helps emulate a real time programming problem more closely, and makes the test-taker solve a complex programming problem, which would otherwise not be possible in a limited testing framework.
In an embodiment, the problem provided to the test-taker may require multiple source-code files. In such a scenario, the pre-code 202 and/or the post-code 204 may comprise multiple source code files. While the pre-code/post-code may be static conditions, in an embodiment of the present embodiment they can be editable. Thus, one or more of the multiple source code files may be editable, while the rest of the files may be static (non-editable for the test-taker). The evaluation module 114 compiles all the files as a single program, with all the dependencies intact, thereby simulating a real-time programming environment.
The introduction of pre-code 202 and post-code 204 in the program helps make easy the evaluation of the correctness of the program on various parameters like functionality, brevity etc. This functionality enables the test engine 108 to, automatically, or through evaluator's intervention, test the required skill.
Similarly, imposing predefined conditions on the program body through pre-code and post-code, makes the evaluation of the test taker input for parameters like efficiency, execution time, memory footprints etc, counting the number of iterations of a loop etc, easier and more deterministic. This again helps the test engine 108 assess the test taker input easily, more accurately, in a better manner, and a wider range of programming problems.
The pre-code 202 and post-code 204, along with their flow-graph and/or compiled code already available, the evaluation module 114 is able to determine the isomorphism of flow-graph, and correctness of the behavior of the solution function, much more easily, and exhaustively, for a wider range of programming problems, with a lesser restriction on range and number of test-functions.
If at 404, the method finds that the transformed version is not a flow-graph, the method checks if the transformed version is an interpreted code at 408. If the transformed version is an interpreted code, the method compares the transformed version with the corresponding base version of interpreted code at 410. The method evaluates the transformed version by running the executable in a suitable environment on a set of test-cases and checks for the desired properties and sends it for execution at 418. However, if the transformed code is not an interpreted code, the method checks if the transformed version is a compiled code at 412. If the transformed version is a compiled code, the method extracts the requisite data during compilation and sends the data and the compiled code to be checked for the desired properties at 414. The method analyzes the code using a code analyzer to find the desired properties in the code at 416. The method then proceeds to 418 for executing the program. If the transformed version is not a compiled code, the method passes the flow to the test engine to take appropriate step based on the type of the program and the transformed version at 418. The flow from 406, 418 and 420 passes to 422 where the method evaluates the correctness and does the grading of the test-taker's input.
While the test engine is designed to assess the programming ability of a test-taker, it is possible to comprehensively test the aptitude or other objective ability of the test-taker. An exemplary scenario as implemented using the present embodiment is depicted in
However, if the compilation is successful, the method checks if the complete functionality has runtime behaviour close to acceptable submission state, namely, as expected for the particular programming problem that was presented to the test-taker at 620. If the complete functionality is not in acceptable submission state, the method presents the previously provided input to the test-taker for correction at 622. The method joins step 614. If the complete functionality is in acceptable submission state, the method sends the complete functionality to the evaluation module and queries the test engine for the next level of difficulty question at 624. The method then goes to 604.
After 606, the method assigns a score and stores it at 626. The method finally stops at 628.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2561/DEL/2010 | Oct 2010 | IN | national |
The present application is a National Phase Application for PCT/IN2011/000739 which claims priority from Indian Application Number 2561/DEL/2010, filed on 26th Oct., 2010, the disclosure of which is hereby incorporated by reference herein.
