This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2021-181739 filed Nov. 8, 2021.
The present invention relates to an information processing apparatus, a non-transitory computer readable medium storing a program, and an information processing method.
An application is operated on various information processing apparatuses such as a personal computer (PC), a tablet, and a smartphone, and a user's operation is performed on a Graphical User Interface (GUI) provided by the application. Various settings of the application, execution of processes, or the like are performed by the user's operation.
JP5931806B discloses an automatic operation apparatus that records a user's operation on the GUI of an operation target application operating on a computer as a scenario, and replays the user's operation on the GUI of the operation target application on the computer by playing the scenario.
JP2020-017233A relates to an electronic device including a screenshot execution unit that acquires an image displayed on a display unit, and discloses a configuration in which in a case where an image requiring user authentication is displayed on the display unit, the acquisition of the image by the screenshot execution unit is prohibited, and in a case where an image that does not require user authentication is displayed on the display unit, the acquisition of the image by the screenshot execution unit is permitted.
In a case where a failure occurs in an application or an apparatus controlled by the application, or in a case where a desired result is not obtained as a result of processing by the application or the apparatus, there is a request to check later which operation was performed by the user on the GUI provided by the application and investigate the cause and factor of the failure and processing result. Such a user's operation may be checked from the log information (history information) recorded in the computer, but the user's operation cannot be checked easily in many cases.
Aspects of non-limiting embodiments of the present disclosure relate to an information processing apparatus, a non-transitory computer readable medium storing a program, and an information processing method that facilitate later checking an operation performed by a user with respect to an application operating in an information processing apparatus, as compared with a case where only log information is used.
Aspects of certain non-limiting embodiments of the present disclosure overcome the above disadvantages and/or other disadvantages not described above. However, aspects of the non-limiting embodiments are not required to overcome the disadvantages described above, and aspects of the non-limiting embodiments of the present disclosure may not overcome any of the disadvantages described above.
According to an aspect of the present disclosure, there is provided an information processing apparatus including a processor configured to: acquire a copy image, that is a copy of an image displayed on a display device, every time a predetermined timing occurs; and store the acquired copy image in a storage device in association with information indicating the predetermined timing that triggers the acquisition.
Exemplary embodiment(s) of the present invention will be described in detail based on the following figures, wherein:
Hereinafter, each exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. The configuration described below is an example for explanation, and can be appropriately changed according to the specifications of the system and the apparatus. Further, in a case where a plurality of exemplary embodiments, modification examples, and the like are included in the following, it is assumed from the beginning that the characteristic portions are appropriately combined and used. The same elements are designated by the same reference numerals in all drawings, and duplicate description is omitted.
The printing system 10 includes a print control apparatus 12 and a printing apparatus 14. The print control apparatus 12 is communicably connected to the printing apparatus 14. The print control apparatus 12 is communicably connected to another computer via a network 16. The network 16 is, for example, a LAN, an intranet, the Internet, or the like.
The printing apparatus 14 is an apparatus that forms an image on a medium such as paper, and is also called an image forming apparatus. The printing apparatus 14 is a digital printing machine such as an electrophotographic printing apparatus and an inkjet printing apparatus, and in this exemplary embodiment, the printing apparatus 14 is a production printer capable of printing at high speed. By selectively adding modules, the printing apparatus 14 can have a function of post-processing (punching (hole punching) processing, staple processing, saddle stitching processing, folding processing, cutting processing, or the like). According to the printing apparatus 14 having the post-processing function, it is possible to perform post-processing after printing on paper, and output a printed matter such as a booklet.
The print control apparatus 12 is a computer called a Digital Front End (DFE) or a print server. An operating system and a printing application operating on the operating system are installed in the print control apparatus 12. The print control apparatus 12 provides various print-related functions depending on the operation of the printing application.
The print control apparatus 12 receives a print job from another computer via the network 16, and manages the print job. The print job is a unit of processing in the printing system 10, and includes print setting information and document and image data to be printed. The print control apparatus 12 receives an operation from the operator to make various settings for the print job, or performs Raster Image Processer (RIP) processing on the image data of the print job to generate a raster image. Further, the print control apparatus 12 receives a print instruction from the operator and supplies a print processing command including various parameters, a raster image, or the like, corresponding to the print job, to the printing apparatus 14, so that a printed matter is output from the printing apparatus 14. The print control apparatus 12 also functions as an apparatus for managing the state of the printing apparatus 14 or making various settings for the printing apparatus 14. The print control apparatus 12 also functions as an apparatus for controlling the operation of the printing apparatus 14.
The storage device 22 is a ROM, a RAM, a flash memory, a hard disk, or the like. The storage device 22 stores the program 60, control data 62, a print job 64, a raster image 66, screenshot (SCS) setting information 67, screenshots 72A, 72B, 72C, and the like. The program 60 and the control data 62 include programs and control data for both the operating system and the printing application, respectively.
The program executed by the processor 20 and the control data to be read can be provided not only via a network such as the Internet but also stored in a computer-readable recording medium such as an optical disk or a USB memory.
The screenshots 72A, 72B, and 72C are electronic files (hereinafter, also referred to as copy images) of images showing all or a part of the display object (display screen image) displayed on the display device 26 of the print control apparatus 12. That is, the screenshots 72A, 72B, and 72C are copy images that are copies of all or a part of the image displayed on the display device 26 of the print control apparatus 12. In this exemplary embodiment, the screenshots 72A, 72B, 72C are electronic files (copy images) of an image showing the entire display object displayed on the display device 26. Hereinafter, the screenshot is also referred to as “SCS”.
The screenshot 72A is acquired at a predetermined time interval (also referred to as periodic), and is hereinafter referred to as a periodic SCS (72A). The screenshot 72B is acquired at the timing when the user performs a predetermined operation (also referred to as a user event) on the GUI of the print control apparatus 12, and is hereinafter referred to as a user event SCS (72B). The screenshot 72C is acquired at the timing when a predetermined event (also referred to as a device event) of the printing application and the operating system or a predetermined event (device event) of the printing apparatus 14 controlled by the printing application occurs, and is hereinafter referred to as a device event SCS (72C). Hereinafter, in a case where it is not necessary to distinguish between the periodic SCS (72A), the user event SCS (72B), and the device event SCS (72C), the periodic SCS (72A), the user event SCS (72B), and the device event SCS (72C) are referred to as screenshot 72 or SCS (72).
The storage device 22 stores a periodic folder 70A, a user event folder 70B, and a device event folder 70C. The periodic SCS (72A) is stored in the periodic folder 70A. The user event SCS (72B) is stored in the user event folder 70B. The device event SCS (72C) is stored in the device event folder 70C.
The input device 24 is a keyboard 24-1, a mouse 24-2 (see
On the print information portion 42, the print progress information of the print job being printed by the printing apparatus 14 is displayed. On the tray information portion 43, paper information (paper type, paper size, paper color, basis weight, or the like) set for each of the plurality of paper trays of the printing apparatus 14 is displayed. The paper information on each paper tray is set by the user. The paper actually put in the paper tray needs to match the paper shown in the tray information portion 43, but in a case where the papers are different due to a setting error or the like, the intended print result may not be obtained.
On the toner information portion 44, information on the remaining amount of the toner of each of CMYK colors is displayed. On the print job list information portion 45, information on each print job is displayed in a form in which with respect to the print jobs stored in the print control apparatus 12, the print jobs are classified according to the print job states (pending, processing, and completed). On the print job detailed information portion 46, detailed information on the print job designated by the cursor 40 in the print job list information portion 45 is displayed. In a case where the display image 28-1 as shown in
The production printer has various functions, and detailed settings can be made for each function. For example, in the function item portion 48 of
Here, in the printing application, in a case where the user performs various settings and instructions, a failure such as an error may occur in the printing application or the printing apparatus 14 controlled by the printing application. Further, the processing result of the printing application (for example, a raster image after RIP processing) or the printed matter output from the printing apparatus 14 may not be the desired result. In the present exemplary embodiment, in preparation for such a case, the SCS (72) is acquired every time a predetermined timing occurs, and what type of operation is performed by the user on the printing application from the SCS (72) can be checked later, so that the causes and factors for failures and processing results can be investigated. For example, the user or the user's employer passes the periodic folder 70A, the user event folder 70B, and the device event folder 70C to the manufacturing company of the printing system 10, the company that provides after-sales service, or the like, and the companies verify the causes and factors for failures and processing results from SCS (72) in each folder.
In particular, since various functions can be set in detail in the printing application of a production printer, as shown in
Further, as described below, in the present exemplary embodiment, the file name of the SCS (72) and information indicating a predetermined timing that triggers the acquisition of the SCS (72) are stored in the storage device. This makes it easier to find out which SCS (72) to check in order to investigate the causes and factors for failures and processing results in the printing application and the printing apparatus.
The SCS (72) is acquired, for example, as follows. First, the printing application issues a capture instruction to acquire a display screen image at a predetermined timing (periodic, user event, or device event), and the operating system receives the capture instruction. Upon receiving the capture instruction, the operating system copies the display screen image displayed on the display device 26 to the clipboard and sends a copy completion notification to the printing application. Upon receiving the copy completion notification, the printing application stores the image on the clipboard in the storage device 22 as an electronic file (copy image) with a file name described below. Thus, the SCS (72) is acquired.
As the acquisition timing of the SCS (72), a timing when the periodic time elapses, a timing when the screen on the display device 26 changes, a timing when the state of the operating system/printing application/printing apparatus/print job changes, a timing when an operation on a print job is performed, a timing when an error has occurred, or the like is considered. In order to deal with these, in the present exemplary embodiment, the SCS (72) is acquired at the periodic, user event, and device event timings. The SCS (72) may be acquired at the timing of one or two of the periodic, user event, and device event.
The file name of the user event SCS (72B) in
“SCAPR” or “SCSYS” is added to the file name of the user event SCS (72B) in
The file name of the device event SCS (72C) in
“SCAPR” or “SCSYS” is added to the file name of the device event SCS (72C) in
Next, the SCS mode will be described.
The mode setting portion 81 is a portion for designating whether or not to acquire the SCS (72), and for designating the frequency of acquiring the SCS (72) in a case where the SCS (72) is to be acquired. The mode setting portion 81 has options of “no acquisition”, “low frequency acquisition mode”, “medium frequency acquisition mode”, “high frequency acquisition mode”, and “detailed setting mode”, and one of the options is designated by the user using the radio button. “No acquisition” means that the SCS (72) is not to be acquired. “Low frequency acquisition mode”, “medium frequency acquisition mode”, and “high frequency acquisition mode” are modes prepared for switching the acquisition frequency of the periodic SCS (72A), and the number of periodic SCSs (72A) acquired per unit time increases in the order of the “low frequency acquisition mode”, the “medium frequency acquisition mode”, and the “high frequency acquisition mode”. For example, the “low frequency acquisition mode” is a mode for acquiring a periodic SCS (72A) with the period T=30 seconds, the “medium frequency acquisition mode” is a mode for acquiring a periodic SCS (72A) with the period T=10 seconds, and the “high frequency acquisition mode” is a mode for acquiring a periodic SCS (72A) with the period T=1 second.
In addition, in the order of “low frequency acquisition mode”, “medium frequency acquisition mode”, and “high frequency acquisition mode”, the number of “predetermined operations” that trigger the acquisition of the user event SCS (72B) is increased, and the acquisition frequency of the user event SCS (72B) may be changed by switching between these modes. Similarly, in the order of “low frequency acquisition mode”, “medium frequency acquisition mode”, and “high frequency acquisition mode”, the number of “predetermined events” that trigger the acquisition of the device event SCS (72C) is increased, and the acquisition frequency of the device event SCS (72C) may be changed by switching between these modes.
The “detailed setting mode” of the mode setting portion 81 is a mode for finely setting the acquisition specifications of the SCS (72), and is a mode for acquiring the SCS (72) according to the specifications designated in the detailed setting window 90 (see
In the detailed setting window 90, in a case where the radio button of the user event SCS is turned on, the radio button of each of “event for job (print job)”, “event not limited to specific job (print job)”, and “event in OS (operating system)” is set. In a case where the radio button of “event for job” is turned on, the user event SCS (72B) is acquired as a result of the user's operation related to the setting or processing of the print job. Further, in a case where the radio button of “event not limited to specific job” is turned on, the user event SCS (72B) is acquired as a result of an operation not limited to the specific print job of the user. Further, in a case where the radio button of “event in OS” is turned on, the user event SCS (72B) is acquired as a result of the user's operation on the operating system.
In the detailed setting window 90, in a case where the radio button of the device event SCS is turned on, the radio button of each of “event for job (print job)”, “event not limited to specific job (print job)”, and “event in OS (operating system)” is set. In a case where the radio button of “event for job” is turned on, the device event SCS (72C) is acquired as a result of the event related to the processing of the print job. Further, in a case where the radio button of “event not limited to specific job” is turned on, the device event SCS (72C) is acquired as a result of the event not limited to the specific print job. Further, in a case where the radio button of “event in OS” is turned on, the device event SCS (72C) is acquired as a result of the event in the operating system.
In the window 80 of
By pressing the “OK” button 84 in the window 80 of
According to the exemplary embodiment described above, it becomes easier to later check the operation performed by the user on the printing application and the operating system operated by the print control apparatus 12, as compared with the case where only the log information is used.
In the exemplary embodiment described above, in a case where the SCS (72) is acquired as a result of the event related to the print job, identification information on the print job is added to the file names of the user event SCS (72B) and the device event SCS (72C). In this case, the attribute information on the print job may be further added to the file names of the SCSs (72B, 72C). For example, in a case where the print job is a security job (a password is required to be input for editing and processing the print job), the wording indicating “security job” may be added to the file names of SCSs (72B, 72C). Thus, it becomes easier to find the desired SCSs (72B, 72C).
Further, in the exemplary embodiment described above, the user sets the SCS mode in the mode setting portion 81 of the window 80 shown in
Further, for example, as shown in
Next, verification using SCS (72) will be described.
The verification window 100 includes a search condition input portion 102, a search range setting portion 104, a search result display portion 106, and an image display portion 108. The search condition input portion 102 includes text boxes to which “start date and time”, “end date and time”, “job name”, and “keyword” are respectively input. The “start date and time” and “end date and time” are required to be entered, and the “job name” and “keyword” are arbitrarily entered. In a case where the user inputs the “start date and time” and the “end date and time” and presses the “search” button 110, the SCSs (72) having dates and times between the “start date and time” and the “end date and time” in the file name are searched from the periodic folder 70A, the user event folder 70B, and the device event folder 70C, and are displayed in the search result display portion 106. The search range setting portion 104 has a radio button for each of the periodic SCS, the user event SCS, and the device event SCS, and the SCS with the radio button turned on (black circle is checked) is the search target. That is, whether or not the periodic folder 70A, the user event folder 70B, and the device event folder 70C are search targets is determined by turning the radio button on/off.
Further, in a case where the user inputs the “job name” in the search condition input portion 102 and presses the “search” button 110, the SCSs (72) having the input job name (print job identification information) in the file name are searched and displayed in the search result display portion 106. That is, only the SCSs (72) related to the print job designated in the search condition are displayed in the search result display portion 106.
Further, in a case where the user inputs a “keyword” in the search condition input portion 102 and presses the “search” button 110, the SCSs (72) having the input keyword (for example, an event name) in the file name are searched and displayed in the search result display portion 106. That is, only the SCSs (72) related to the keyword designated in the search condition are displayed in the search result display portion 106.
In the search result display portion 106, the file names of the SCSs (72) that match the search conditions are displayed in chronological order (from the oldest to the newest date and time included in the file names). In a case where the file name of one SCS (72) is designated by the cursor 40 in the search result display portion 106, the image of the SCS (72) is displayed in the image display portion 108.
According to the verification window 100, in a case where a failure or a desired processing result is not obtained, it is possible to check only SCS (72) narrowed down by a period (start date and time to end date and time) according to the occurrence date and time of an error or the like, a job name, and a keyword in the search conditions. Therefore, there is an increased possibility that the cause or factor of the failure or obtaining the undesired processing results can be quickly found from the SCS (72).
Here, the verification window 100 includes a “player” button 111. In a case where the user presses the “player” button 111, the player window 120 (see
The plurality of buttons 130 to 140 include a play/pause button 130, a stop button 132, a fast forward button 134, a skip button 136, a rewind button 138, and a reverse skip button 140. In a case where the user presses the play/pause button 130, the SCSs (72) in the search result display portion 124 are designated in order from top to bottom (the backgrounds of the SCS file names are grayed out in order), and the image of the designated SCS (72) is displayed on the image display portion 122. That is, the image of the SCS (72) is displayed by frame advance on the image display unit 122. Further, in a case where the user presses the play/pause button 130 again, the frame advance is paused.
The stop button 132 is a button for stopping frame advance, and in a case where the stop button 132 is pressed, the SCS (72) at the top of the search result display portion 124 is designated. The fast forward button 134 is a button for increasing the speed of frame advance. The rewind button 138 is a button for returning again and checking the image of the SCS (72) that has already been frame-advanced. The skip button 136 is a button for jumping the designation of the SCS (72) in the search result display portion 124 to the next user event SCS (72B) or device event SCS (72C). The reverse skip button 140 is a button for returning the designation of the SCS (72) in the search result display portion 124 to the user event SCS (72B) or the device event SCS (72C) that have already been frame-advanced.
By using the player window 120 as shown in
In the exemplary embodiment described above, the printing application of the print control apparatus 12 acquires the periodic SCS (72A), the user event SCS (72B), or the device event SCS (72C). However, an exemplary embodiment of acquiring at least one of a periodic SCS (72A), a user event SCS (72B), and a device event SCS (72C) can also be applied to various applications operating on information processing apparatuses such as PCs, tablets, and smartphones. By checking these SCSs (72), in a case where a failure or an unintended processing result occurs in an application operating on a PC, a tablet, a smartphone, or the like, or an apparatus controlled by the application, it is easier to verify the cause and factor thereof.
In the embodiments above, the term “processor” refers to hardware in a broad sense. Examples of the processor include general processors (e.g., CPU: Central Processing Unit) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, and programmable logic device).
In the embodiments above, the term “processor” is broad enough to encompass one processor or plural processors in collaboration which are located physically apart from each other but may work cooperatively. The order of operations of the processor is not limited to one described in the embodiments above, and may be changed.
The foregoing description of the exemplary embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations will be apparent to practitioners skilled in the art. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, thereby enabling others skilled in the art to understand the invention for various embodiments and with the various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents.
Number | Date | Country | Kind |
---|---|---|---|
2021-181739 | Nov 2021 | JP | national |