Patent Application
20220374799
The present invention relates to a display control device and a display control method.
In general, the approach frequently taken in business improvement processes of a company includes first grasping actual business situations by hearings using man power, work amount estimation by observation/time measurements, and the like, and then considering improvement plans after finding problematic portions. However, grasping the actual business situations through hearings or the like requires a significant amount of work. In addition, it may be biased toward the subjectivities and procedures of some analysts, and there are problems in accuracy, comprehensiveness, and the like.
Therefore, there is a conventionally proposed method in which an operational log of a terminal is acquired and visualized so that actual business situations can be grasped more efficiently, in a wide range, and finely in granularity.
Among operational log visualization methods, the timeline in which the time axis is set in an x-axis direction on a screen, each row in a y-axis direction is an element (e.g., user), and the duration of the element is expressed by a rectangle is known as an effective method for grasping operational situations on a terminal in the business analysis. For example, Non-Patent Literature 1 proposes a method for hierarchically arranging an operational log with information of the same granularity such as user, application, window, operation, and the like, and visualizing a timeline. In addition, another method for displaying an operation on the timeline and a captured image while associating them with each other is proposed (e.g., refer to Non-Patent Literature 2).
However, in the existing timeline display method (Non-Patent Literature 1), it was impossible to overview another hierarchy (e.g., operation) at the same time when observing a specific hierarchy (e.g., application or window). Further, in the method for displaying the operation on the timeline and the captured image while associating them with each other (Non-Patent Literature 2), it was difficult to overview the details of the operation although individual operational contents and the operational flow in the selected time zone can be observed from the captured image and the animation display of an operation sequence in the selected range.
As mentioned above, according to the conventional methods, an operation entity (e.g., application or window) concerned by an analyst and other operation entities cannot be displayed at the same time. Therefore, according to the conventional method, it is necessary to repeat hierarchy expansion operations, hierarchy contraction operations, individual image confirmation operations, and the like in order to find problematic portions and pursue the causes of the problems, which makes it difficult to perform the analysis efficiently.
The present invention has been made in view of the foregoing, and intends to provide a display control device and a display control method capable of efficiently analyzing actual business situations based on an operational log of a terminal.
To solve the above-mentioned problems and achieve the purpose, a display control device according to the present invention is characterized by including a generation unit configured to read an operational log file and generate a first object indicating operation information visualized by a predetermined visual representation of a first operation entity, about the first operation entity, and second objects each indicating operation information of a second operation entity that is different in granularity from the first operation entity, a visualization unit configured to perform drawing based on the first object and perform display on a screen, and a summary information generation unit configured to generate summary information in which second objects corresponding to a selected part are grouped, when any part of the first object displayed on the screen is selected, wherein the visualization unit displays the summary information while superimposing or associating the summary information on or with a designated area.
Further, a display control method according to the present invention is a display control method that is executed by a display control device. The method is characterized by including a process for reading an operational log file and generating a first object indicating operation information visualized by a predetermined visual representation of a first operation entity, about the first operation entity, and second objects each indicating operation information of a second operation entity that is different in granularity from the first operation entity, a process for performing drawing based on the first object and performing display on a screen, a process for generating summary information in which second objects corresponding to a selected part are grouped, when any part of the first object displayed on the screen is selected, and a process for displaying the summary information while superimposing or associating the summary information on or with a designated range.
According to the present invention, actual business situations can be efficiently analyzed based on an operational log of a terminal.
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to the embodiments. Further, in the description of the drawings, the same parts are indicated by the same reference numerals.
First, a display control device according to an embodiment will be described. The display control device according to the present embodiment visualizes an operational log indicating operational contents for a window of a terminal (not illustrated) screen in a timeline format, in order to analyze actual business situations of a user using the terminal. For example, in the timeline, the time axis is taken in the x-axis direction on the screen, elements configuring the data are arranged in the y-axis direction, and a time zone in which each element corresponding to the row on the y axis appears is displayed as a horizontally long rectangle.
Then, when an analyst selects a part of the timeline, the display control device according to the present embodiment superimposes and displays an operation history image, in which the operation history corresponding to the selected part is displayed on a captured image that indicates a window screen in which the above operation has been performed while the operation history is superimposed or associated on or with the captured image, as a summary image, on the timeline. As mentioned above, the present embodiment can streamline the analysis on the actual business situation by the analyst by making it possible to overview the details of the operation at the same time as the outline of the operation.
[Configuration of Display Control Device]
The operational log file includes information logs of a plurality of operation entities. The operational log is, for example, information indicating terminal information, login user information, application information, window information, and operational contents, and occurrence time. The window information is, for example, window title, URL/file path, window handle, and the like. The operational contents are, for example, operation target, operation type, value, captured image, and the like, and are recorded when an operation for an object in the window occurs.
The display control device 10 can be realized, for example, by a computer or the like that includes a read-only memory (ROM), a random-access memory (RAM), a central processing unit (CPU), or the like, when a predetermined program is read and the CPU executes the predetermined program. Further, the display control device 10 has a communication interface that transmits and receives various information to and from other devices connected via a network or the like. For example, the display control device 10 has a network interface card (NIC) or the like, and communicates with other devices via an electric telecommunication line such as a local area network (LAN) or the Internet. The display control device 10 has a display state management unit 11, a display processing unit (generation unit) 12, a visualization unit 13, and a summary information generation unit 14.
The display state management unit 11 manages display state information 111 that indicates an arrangement structure of row objects. The display state information 111 is information generated by the display processing unit 12, and is updated upon generation of new display state information by the display processing unit 12. The row object (first object) is a constituent component of the timeline, and is an object that collects records in which data items other than the time stamp match in the user's operational log on the terminal screen.
The display processing unit 12 has display setting information 121. The display setting information 121 is setting information related to the timeline display. The display setting information 121 is information for setting items to be displayed, information indicating a hierarchical structure, and objects to be displayed. The display setting information 121 is generated based on the display setting file indicating the initial screen display settings, for example, according to analyst instruction information from the user input device 20.
Here, the hierarchy is a collection of elements belonging to the same data item or an entity (e.g., group or period) set by the user among elements configuring user's operational contents indicated in the operational log or the like. The group is a collection of arbitrary elements focused on, of the user's operational contents.
The display processing unit 12 reads the operational log, and generates objects configuring the timeline according to the display setting information 121. The display processing unit 12 generates row objects. The row object is an object indicating operation information visualized by a predetermined visual representation, more specifically, two-dimensional visual representation, of a first operation entity, about the first operation entity. The row objects correspond to records that match in user name, operation target window title, as well as application name and window handle used in this window.
The display processing unit 12 reads the operational log, and generates operation objects (second objects) each indicating information of an operation entity. The operation object is an object indicating operation information of a second operation entity different in granularity from the first operation entity. The operation objects are information indicating operating user name, operation target window title, operation type, operation time, operation occurrence time (operation time), and the like.
The display processing unit 12 creates the display state information 111 in which generated row objects are hierarchically arranged, based on hierarchy settings described in the display setting information 121. The display state information 111 is information indicating the arrangement position of row objects representing window-based operation information, of the operational log, on the screen of the screen output unit 30. The display state information 111 includes information on each row object (e.g., data item, value, or the like that are keys of the row object), in addition to the entire row object hierarchical structure. The display state information 111 includes operation object arrangement information. The display processing unit 12, when it generates the display state information 111, transmits the display state information 111 to the visualization unit 13.
The visualization unit 13 determines the arrangement of the row objects on the y axis of the timeline, based on the display state information 111 generated by the display processing unit 12, and further determines attribute values such as hue and transparency of the rectangles expressing the row objects, and then generates the objects (including rectangles, labels, axes, and the like) configuring the timeline. Then, the visualization unit 13 outputs the generation results to the screen output unit 30 such as a display. Specifically, based on the row objects and the hierarchical structure, the visualization unit 13 draws rectangles configuring the hierarchically arranged timeline and displays them on the screen.
Further, the visualization unit 13 generates plots indicating the operations based on occurrence times of the operation objects. Then, the visualization unit 13 superimposes and displays the plots on the rectangles expressing the row objects so as to correspond to the occurrence times when the operations have been performed.
When the range of any one of the row objects displayed on the screen is selected, the summary information generation unit 14 generates summary information in which the operation objects corresponding to the selected part are grouped.
When any one of the rectangles displayed on the screen is selected, the summary information generation unit 14 acquires records collected by the row object expressed by this rectangle from the read operational log. Then, based on window information included in the acquired records, the summary information generation unit 14 groups objects included in the start-to-end time of the selected range, of operation objects executed in the window. Then, the summary information generation unit 14 generates, as summary information, an operation history image including a directed graph in which each operation target is indicated by a node and the operation order is indicated by links between the nodes, being superimposed and displayed on a captured image indicating a window screen in which the operation has been performed. Here, the grouping is to collect the operation objects included in the user's selected range. The operation objects of the same group correspond to the operation history image, and each node corresponds to a result of aggregating objects (records) whose operation targets are the same, in the grouped operation objects. The visualization unit 13 superimposes and displays this summary information in the designated range on the displayed timeline.
[Flow of Display Control Processing]
Subsequently, the flow of display control processing in the display control device 10 will be described.
The display processing unit 12 reads the operational log illustrated in
As a result, as illustrated in
Then, based on the read operational log, the display processing unit 12 generates operation objects indicating the operation entity from the record containing operational contents. For example, the display processing unit 12 generates operation objects indicating information of input operations or click operations by the User A for an order details window in the duration from 2019/7/11 09:15:20 to 2019/7/11 09:18:55. The generated operation objects indicate an input operation 1 executed on 2019/7/11 09:15:38, a click operation 2 executed on 2019/7/11 09:15:43, an input operation 3 executed on 2019/7/11 09:17:05, an input operation 4 executed on 2019/7/11 09:17:58, and a click operation 5 executed on 2019/7/11 09:18:49.
Then, based on the operation objects generated by the display processing unit 12, the visualization unit 13 generates plots 511a to 511e indicating the input operation 1, the click operation 2, the input operations 3 and 4, and the click operation 5, respectively. The visualization unit 13 superimposes and draws the plots 511a to 511e so as to correspond to their occurrence times, on the rectangle 511. As a result, the plots each indicating the execution of the operation are displayed in the timeline. The display control device 10 may distinguish the operation type, the operation target, and the like according to the shape and color of the plot. Further, the plot display can be switched between display and non-display by an GUI operation or the like. In the plot display, the switching between display and non-display can be performed in a lump, or the switching between display and non-display can be performed for each user, each operational content, or the like.
Subsequently, when a mouse over, a click operation, or the like is executed on the timeline according to an analyst operation on the user input device 20, the display control device 10 displays an operation history image of the corresponding part as summary information. An exemplary case of
In this case, as illustrated in
In this case, the display control device 10 acquires a captured image of the window screen operated in the period of the selected rectangle 511 part and the operation history, from the operational log. Then, the display control device 10 generates and displays an operation history image in which the operation history at the time corresponding to the selected rectangle 511 is superimposed on the acquired captured image. According to the example illustrated in
The operation history image 521 is generated by the summary information generation unit 14. Here, the processing for generating the operation history image 521 by the summary information generation unit 14 will be described.
First, upon receiving an input of a rectangle selected on the timeline, the summary information generation unit 14 identifies the start time, the end time, and the hierarchy of the selected rectangle and identifies an operation sequence included in this range from the operation objects. The summary information generation unit 14 identifies the number of operation target windows included in the identified operation sequence. Of the operation sequence, the summary information generation unit 14 regards the operations different in window information as being different in the operation target window. Then, regarding the operations having the common window information, the summary information generation unit 14 regards images included in different clusters due to the clustering of captured images as being different in the operation target window, for example.
Subsequently, the summary information generation unit 14 aggregates the time, the number of operations, and the like required in the operation for the same operation target, for each operation target window. The summary information generation unit 14 generates an operation history image including a superimposed and displayed directed graph in which the same operation target is indicated by a node and the operation order is indicated by links, for the captured image corresponding to each operation target window.
For example, the summary information generation unit 14 can visually express time-consuming operations, in addition to the operation order, by distinguishing the detention time with the node size and displaying the operation order with links. The summary information generation unit 14 may distinguish and display the number of operations with the hue, brightness, or the like of the node so as to visually express time-consuming operations.
The summary information generation unit 14 calculates the detention time by subtracting the previous operation time from the next operation time. Further, the summary information generation unit 14 generates the operation history image 520 using information of operation time—captured image, of the operational log illustrated in
The operation history image 521′ includes a directed graph superimposed on the captured image. In the directed graph, nodes Na′ to Ne′ indicate respective operations, the size of each of the nodes Na′ to Ne′ indicates the detention time, the color of each of the nodes Na′ to Ne′ indicates the operation type, links E1′ to E4′ indicate the operation order, and the thickness of each of the link E1′ to E4′ indicates the transition time. As mentioned above, the summary information generation unit 14 can visually express time-consuming operations, longer transition time operations, or the like, by distinguishing the detention time with the node size, distinguishing and displaying the operation type with the hue, brightness, or the like of the node, and displaying the transition time with the thickness of the link.
The summary information generation unit 14 calculates the detention time by subtracting the start time from the end time. Further, the summary information generation unit 14 calculates the transition time by subtracting the end time of the previous operation from the start time of the next operation.
According to the examples of
[Processing Procedure of Display Control Processing]
Next, with reference to
As illustrated in
Then, according to the display setting information 121, the display processing unit 12 generates row objects configuring a timeline and performs display processing for generating operation objects (step S2). The visualization unit 13, the display processing unit 12 perform drawing based on the row objects and the operation objects generated by the display processing unit 12, and perform visualization processing for displaying the timeline on the screen (step S3).
The summary information generation unit 14 determines whether there is a summary information display instruction (step S4). When any rectangle expressing the row objects displayed on the screen is selected, the summary information generation unit 14 determines to be instructed to generate summary information being operation history information during the period indicated by this rectangle. When the summary information display instruction is not present (No in step S4), the display control device generates no summary information and continues to display the timeline.
When the summary information display instruction is present (Yes in step S4), the summary information generation unit 14 performs summary information generation processing for generating summary information in which the operation objects corresponding to the selected part are grouped (step S5). In step S6, the summary information generation unit 14 generates, as summary information, an operation history image indicating the history of operations performed in the displayed window during the period indicated by the selected rectangle. The visualization unit 13 performs summary information visualization processing for superimposing and displaying the summary information at a designated area on the displayed timeline (step S6). In step S6, the visualization unit 13 displays the operation history image in association with the area in the vicinity of the selected rectangle.
As mentioned above, the display control device 10 according to the embodiment displays summary information of another hierarchy so as to be superimposed or associated on or with the hierarchy that is concerned by an analyst on the timeline. Specifically, the display control device 10 displays the operation history included in a predetermined range, as a captured image-based operation history image, so as to be associated with the application or the timeline displayed on the window basis, for example. Alternatively, the display control device 10 plots the occurrence times of respective operations on the timeline, based on the information of an operation entity (occurrence time, operation target, operation type, value, captured image, or the like).
As a result, the analyst can grasp the summary information of another hierarchy while displaying the hierarchy that is concerned by the analyst on the timeline, and therefore it is easy to narrow down the portion to be deeply analyzed. Therefore, the display control device 10 can reduce troublesome works by the analyst such as hierarchy expansion operations, contraction operations, individual image confirmations, and can improve the efficiency in business analysis. Accordingly, using the display control device 10 enables the analyst to efficiently analyze actual business situations based on an operational log of a terminal.
[System Configuration of Embodiment]
Each constituent component of the display control device 10 illustrated in
Further, all or some of the pieces of processing to be performed in the display control device 10 may be realized by a CPU, a graphics processing unit (GPU), and programs that are analyzed and executed by the CPU and the GPU. Further, each piece of processing performed in the display control device 10 may be realized as wired logic hardware.
Further, of the pieces of processing described in the embodiment, all or some of them described as being automatically performed can be manually performed, or all or some of them described as being manually performed can be automatically performed by a known method. In addition, processing procedures, control procedures, and specific names, and information including various data and parameters in the above description and drawings can be arbitrarily changed unless otherwise specified.
[Program]
The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. For example, a detachable storage medium such as a magnetic disk or an optical disc can be inserted into the disk drive 1100. The serial port interface 1050 is, for example, connected to a mouse 1110 and a keyboard 1120. The video adapter 1060 is, for example, connected to a display 1130.
The hard disk drive 1090 stores an operating system (OS) 1091, an application program 1092, a program module 1093, and a program data 1094, for example. That is, the program that defines each piece of processing of the display control device 10 is implemented as the program module 1093 in which codes are described so as to be executed by the computer 1000. The program module 1093 is, for example, stored in the hard disk drive 1090. For example, the program module 1093 for executing processing similar to the functional configuration of the display control device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by a solid state drive (SSD).
Further, the setting data to be used in the processing of the above-mentioned embodiments is stored, as the program data 1094, in the memory 1010 or the hard disk drive 1090, for example. Then, the CPU 1020 reads, into the RAM 1012, the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 when needed, and executes it.
The program module 1093 and the program data 1094 are not limited to those stored in the hard disk drive 1090, and may be stored, for example, in a detachable storage medium so that the CPU 1020 can read out the program module 1093 and the program data 1094 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network such as a local area network (LAN) or a wide area network (WAN). Then, the CPU 1020 may read out the program module 1093 and the program data 1094 from another computer via the network interface 1070.
Although the embodiments to which the present invention made by the present inventor is applied have been described above, the present invention is not limited by the description and drawings serving as a part of the disclosure of the present invention based on the embodiments. That is, the present invention encompasses all of other embodiments, examples, operational techniques, and the like made by those skilled in the art based on the present embodiments.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/042690 | 10/30/2019 | WO |
