Patent Application
20240241703
Code is a set of instructions that direct operation of a computing system. It is often important to be able to confirm what code is doing. This may be important when authoring code to be certain that the code is directing the computing system as intended by an author. Debugging tools allow coders to be able to verify operation of code, to thereby correct behavior of the code through editing of that code. However, it is also helpful in some contexts to simply understand what the code does, regardless of whether that code is intended to be edited. Such might be helpful when choosing whether or not to use the code with understanding of what contexts in which the code would be most helpful. Thus, proper understanding of what code is doing is important to correct the code and/or understand how the code can be applied.
The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments describe herein may be practiced.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In accordance with the principles described herein, a computer-implemented mechanism is provided to assist in showing how code is operating with different sets of input. This is done by providing organized visibility on intermediate values. When code operates upon an input value set, the code generates an output value set. However, along the way, usually unseen, the code also generates one or more intermediate values. After accessing the code that is to be evaluated along with the multiple input sets for that code, the computing system generates a multi-dimensional array of values. This is done by, for each of at least some of the multiple input sets, generating a corresponding intermediate value set of one or more intermediate values that are generated as the code operates upon the corresponding input set.
Then, the computing system causes a multi-dimensional array of values to be visualized using a multi-dimensional representation. In this multi-dimensional visualization, input sets are represented in at least one dimension against at least one intermediate value of the corresponding intermediate value set in at least another dimension. Thus, a user can scan through how the intermediate values differ by each set of input values. In one embodiment, the computing system even groups the input sets by common characteristics of the intermediate values or corresponding output values, allowing the user to focus in on different scenarios of execution. In further embodiments, the intermediate values and/or groupings may be labelled with natural language.
Additional features and advantages will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the teachings herein. Features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.
In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments and are not therefore to be considered to be limiting in scope, embodiments will be described and explained with additional specificity and details through the use of the accompanying drawings in which:
In accordance with the principles described herein, a computer-implemented mechanism is provided to assist in showing how code is operating with different sets of input. This is done by providing organized visibility on intermediate values. When code operates upon an input value set, the code generates an output value set. However, along the way, usually unseen, the code also generates one or more intermediate values. After accessing the code that is to be evaluated along with the multiple input sets for that code, the computing system generates a multi-dimensional array of values. This is done by, for each of at least some of the multiple input sets, executing the code using that input set. As the code as executed on the input set, the execution not only generates an output value set, but also a corresponding intermediate value set along the way.
Then, the computing system causes a multi-dimensional array of values to be visualized using a multi-dimensional representation. In this multi-dimensional visualization, input sets are represented in at least one dimension against at least one intermediate value of the corresponding intermediate value set in at least another dimension. Thus, a user can scan through how the intermediate values differ by each set of input values (i.e., by each execution against each input set). In one embodiment, the computing system even groups the input sets by common characteristics of the intermediate values or corresponding output values, allowing the user to focus in on different scenarios of execution. In further embodiments, the intermediate values and/or groupings may be labelled with natural language.
The code 111 may be any code that is capable of receiving input and generating output. The code may be declarative code (such as a spreadsheet, database or any other declarative code), imperative code, or any other type of code. The principles described herein are not limited to the code type or language of the code 111, nor even whether the code 111 is source code, intermediate language code, or machine code.
In this value array, input sets are represented by the letter A with a numerical suffix (e.g., 1, 2, 3 and so forth) to distinguish one input set from another. In this example, there are five different input sets A1, A2, A3, A4 and A5, each used in respective executions 1 through 5 of the same code. The numerical suffix may also be seen as representing a respective execution of the code that uses the respective input set of that same representation. For instance, execution 1 of the code uses input set A1, execution 2 of the code uses input set A2, execution 3 of the code uses input set A3, execution 4 of the code uses input set A4, and execution 5 of the code uses input set A5. Of course, this is just an example, the code under evaluation may have any number of executions using any number of input value sets. But this description will refer to the value array 200 of
The value array 200 also includes output sets that are represented by the letter C with a numerical suffix. Each output set represents one or more output values that are output by the code by the corresponding execution. For instance, execution 1 of the code uses input set A1 to generate output set C1, execution 2 of the code uses input set A2 to generate output set C2, execution 3 of the code uses input set A3 to generate output set C3, execution 4 of the code uses input set A4 to generate output set C4, and execution 5 of the code uses input set A5 to generate output set C5.
The value array 200 also includes intermediate value sets that are represented by a numerical suffix and a lowercase letter. The numerical suffix identifies which execution of the code generated the intermediate value. The lowercase letter suffix distinguishes one intermediate value from another in that execution of the code. Each intermediate value set represents in this case four intermediate values that are generated by the respective execution of the code. For instance, execution 1 of the code uses input set A1 to generate output set C1 and along the way generates intermediate values B1a, B1b, B1c and B1d. Execution 2 of the code uses input set A2 to generate output set C2 and along the way generates intermediate values B2a, B2b, B2c and B2d. Execution 3 of the code uses input set A3 to generate output set C3 and along the way generates intermediate values B3a, B3b, B3c and B3d. Execution 4 of the code uses input set A4 to generate output set C4 and along the way generates intermediate values B4a, B4b, B4c and B4d. Execution 5 of the code uses input set A5 to generate output set C5 and along the way generates intermediate values B5a, B5b, B5c and B5d.
However, the multi-dimensional value array 121 generated in
The method 300 includes accessing code (act 301) and a plurality of input sets of input for the code (act 302). For instance, in
Then, a multi-dimensional array of values is generated (act 303) by for each of at least some of the input sets, generating a corresponding intermediate value set of one or more intermediate values that are generated as the code operates upon the corresponding input set to generate a corresponding output value set. For instance, with respect to
Thereafter, the array generation component causes the multi-dimensional array of values to be visualized using a multi-dimensional representation (act 304). In this multi-dimensional representation, a visualization of at least some of the plurality of input sets are represented in at least one dimension against at least one intermediate value of the corresponding intermediate value set in at least another dimension. As an example, regardless of the code, the multi-dimensional representation may be a two-dimensional representation, a three-dimensional representation, or even a higher-dimensional representation. However, a two-dimensional representation, such as a table, is an intuitive way to review value arrays regardless of the code type. Furthermore, tables are particularly advantageous when the code is a spreadsheet program as such code is already suitable for presenting tables.
In the case of a table, the input sets may be represented with each input set assigned a row (e.g., each execution of the code assigned a row), and with associated intermediate values assigned a column. Alternatively, input sets may be represented with each input set assigned a column (e.g., each execution of the code assigned a column), and with associated intermediate values assigned a row.
To explain this general principle, a specific example will now be provided. In this example, the code is declarative code within a spreadsheet program. Furthermore, in this example, the input set includes 10 input sets, each assigned a row in a table. In this example, each input set includes information about a (fictional) person including 1) first name, 2) middle name, 3) last name, and 4) date of birth. In addition, there is an index number that uniquely identifies the input set. Table 1 below shows a table that represents multiple input sets in which each input set is represented as a row in the table. In the subject example, the table is a spreadsheet table. For purposes of this application, each name and date of birth is fictional and generated using a random name generator. Any similarity or correlation with any real person is strictly coincidental.
Table 1 represents an example of the multiple input sets 112 of
The rightmost column of the Table 2 represents the output set, which for each input set, is the abbreviation. However, somehow, the code has two separate behaviors. The first behavior is that indeed the abbreviated value includes the concatenation of first name, middle name, and last name. This first behavior occurred for all input sets, except one. That is, for the input set corresponding to Charlie Evans, the output set was empty, which represents a second behavior. This second behavior seems to relate to the middle name of the input set being empty.
To verify this hypothesis, the user might select to see some intermediate values in the context of the Table 2. Table 3 below illustrates two intermediate value columns being added to the table, one for last name, and one for the concatenation of the first and middle names.
The columns labelled Text Concatenation and 1st Letter of Last Name are automatically generated and show intermediate values generated in the execution of the code against the corresponding input data to generate the corresponding Abbreviation output set. From this visualization, the second intermediate value is the first letter of the last name. The code correctly generates this first letter of the last name for all input sets. However, the code did not properly generate the concatenation of the first name and the middle name for the input set for Charlie Evans. Instead, the code generates an empty result for that concatenation. The user may further inspect the intermediate values that led to that incorrect intermediate value resulting in Table 4.
Now there are four intermediate value columns shown. The user can see that for Charlie Evans, the empty value for the concatenation is the result of the 1st letter of the middle name intermediate value being empty. Thus, the user can see that code correctly determined the first letter of the first name for all input sets. However, the user can also see that the 1 st letter of the middle name shows empty for the input set of Charlie Evans, and that this error propagated from there resulting in no output for the abbreviation for the input set of Charlie Evans. The user can then try to fix the code, or at least has a better understanding of what the code is able to do. That is, the code can correctly generate initials for names of people that have a first name, a middle name, and a last name, but cannot be relied upon if the input set is missing a middle name.
In
Table 2 may represent a new spreadsheet that is generated by executing that generated code against the input sets of
The user may then navigate the executions of the code to find out what happened. To do this, the user may then go into the “Inspect Columns” portion 410 (e.g., by selecting the drop down 411 to view the code navigation field 412 and thus see how the natural language prompt was translated into code. Here, the code navigation field includes the abbreviation icon 431. The icon is titled with the same name as the new column, letting the user know that the corresponding code relates to the actual output of the execution.
A natural language description of the code that was generated by the language model is illustrated in the field 412. Thus, by reviewing the field 412 in
The user can then decompose the code by selecting the abbreviation icon 431. This abbreviation icon 431 represents the output column called “Abbreviation”. This results in two helper columns (e.g., called “1st letter of Last Name” and “text concatenation”) being displayed in Table 3, and representative icons 432 and 433 of these the two helper columns is also shown in the inspect columns field. Also, a natural language description of the code for the decomposition are displayed as natural language description labelling the helper columns. For instance, the helper column called “1st letter of Last Name” is generated by executing the code “the first character from ‘Last Name’. In addition, the text concatenation column was generated using the code that is described in natural language as “the first character from ‘First Name’+the first character from ‘Middle Name”. The icon 433 may be selected to further decompose who the text concatenation column value was generated resulting in Table 4.
In one embodiment, the input sets are grouped according to a common property or value of one or more corresponding intermediate values of the corresponding intermediate value set. For example, in
Here, the groupings are grouped in the same dimension as the input sets are aligned. Thus, it appears to the user then that the input sets are grouped according to one or more properties and/or values of the input set, intermediate set and/or output set. This allows the user to more quickly see the different behaviors that the code executed depending on the input set. The user may even configure what the common property or values are that are to be used to perform the groupings. The label of the grouping may be used to automatically add a sort filter option in the spreadsheet, as in the example of
As the principles described herein may be performed in the context of a computing system, a computing system will now be described with respect to
Computing systems are now increasingly taking a wide variety of forms. Computing systems may, for example, be handheld devices, appliances, laptop computers, desktop computers, mainframes, distributed computing systems, data centers, or even devices that have not conventionally been considered a computing system, such as wearables (e.g., glasses). In this description and in the claims, the term “computing system” is defined broadly as including any device or system (or a combination thereof) that includes at least one physical and tangible processor, and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by a processor. The memory may take any form and may depend on the nature and form of the computing system. A computing system may be distributed over a network environment and may include multiple constituent computing systems.
As illustrated in
The computing system 600 also has thereon multiple structures often referred to as an “executable component”. For instance, the memory 604 of the computing system 600 is illustrated as including executable component 606. The term “executable component” is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods (and so forth) that may be executed on the computing system. Such an executable component exists in the heap of a computing system, in computer-readable storage media, or a combination.
One of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such structure may be computer readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term “executable component”.
The term “executable component” is also well understood by one of ordinary skill as including structures, such as hard coded or hard wired logic gates, that are implemented exclusively or near-exclusively in hardware, such as within a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other specialized circuit. Accordingly, the term “executable component” is a term for a structure that is well understood by those of ordinary skill in the art of computing, whether implemented in software, hardware, or a combination. In this description, the terms “component”, “agent”, “manager”, “service”, “engine”, “module”, “virtual machine” or the like may also be used. As used in this description and in the case, these terms (whether expressed with or without a modifying clause) are also intended to be synonymous with the term “executable component”, and thus also have a structure that is well understood by those of ordinary skill in the art of computing.
In the description that follows, embodiments are described with reference to acts that are performed by one or more computing systems. If such acts are implemented in software, one or more processors (of the associated computing system that performs the act) direct the operation of the computing system in response to having executed computer-executable instructions that constitute an executable component. For example, such computer-executable instructions may be embodied on one or more computer-readable media that form a computer program product. An example of such an operation involves the manipulation of data. If such acts are implemented exclusively or near-exclusively in hardware, such as within a FPGA or an ASIC, the computer-executable instructions may be hard-coded or hard-wired logic gates. The computer-executable instructions (and the manipulated data) may be stored in the memory 604 of the computing system 600. Computing system 600 may also contain communication channels 608 that allow the computing system 600 to communicate with other computing systems over, for example, network 610.
While not all computing systems require a user interface, in some embodiments, the computing system 600 includes a user interface system 612 for use in interfacing with a user. The user interface system 612 may include output mechanisms 612A as well as input mechanisms 612B. The principles described herein are not limited to the precise output mechanisms 612A or input mechanisms 612B as such will depend on the nature of the device. However, output mechanisms 612A might include, for instance, speakers, displays, tactile output, virtual or augmented reality, holograms and so forth. Examples of input mechanisms 612B might include, for instance, microphones, touchscreens, virtual or augmented reality, holograms, cameras, keyboards, mouse or other pointer input, sensors of any type, and so forth.
Embodiments described herein may comprise or utilize a special-purpose or general-purpose computing system including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computing system. Computer-readable media that store computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: storage media and transmission media.
Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other physical and tangible storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computing system.
A “network” is defined as one or more data links that enable the transport of electronic data between computing systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computing system, the computing system properly views the connection as a transmission medium. Transmission media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computing system. Combinations of the above should also be included within the scope of computer-readable media.
Further, upon reaching various computing system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then be eventually transferred to computing system RAM and/or to less volatile storage media at a computing system. Thus, it should be understood that storage media can be included in computing system components that also (or even primarily) utilize transmission media.
Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general-purpose computing system, special-purpose computing system, or special-purpose processing device to perform a certain function or group of functions. Alternatively, or in addition, the computer-executable instructions may configure the computing system to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computing system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAS, pagers, routers, switches, datacenters, wearables (such as glasses) and the like. The invention may also be practiced in distributed system environments where local and remote computing system, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
Those skilled in the art will also appreciate that the invention may be practiced in a cloud computing environment. Cloud computing environments may be distributed, although this is not required. When distributed, cloud computing environments may be distributed internationally within an organization and/or have components possessed across multiple organizations. In this description and the following claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.
For the processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in differing order. Furthermore, the outlined operations are only provided as examples, and some of the operations may be optional, combined into fewer steps and operations, supplemented with further operations, or expanded into additional operations without detracting from the essence of the disclosed embodiments.
The present invention may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicate by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
