Patent Application
20250182008
The technology of the disclosure relates to a work recognition device, a work recognition method and a work recognition program.
Japanese Patent Application Laid-Open (JP-A) No. 2020-144830 discloses a work analysis device that generates determination data for determining whether or not correct work is being carried out. This work analysis device includes: an imaging unit that repeatedly images predetermined work in which plural individual tasks are performed by a worker in a predetermined sequence; a setting unit that, on the basis of predetermined actions specified in advance for each of the individual tasks, sets break information for breaks between the individual tasks from detection timings of the predetermined actions in a work video imaged by the imaging unit; and a generation unit that generates the determination data so as to include the work video imaged by the imaging unit and the break information set by the setting unit.
JP-A No. 2021-106338 discloses a motion recognition system for recognizing motions of a moving body relating to an object. The motion recognition system includes: a position obtaining unit that obtains position information of the object; a parameter obtaining unit that obtains a parameter for controlling a camera that corresponds to the position information of the object on the basis of first information associating the position information with the parameter; a camera control unit that sends the parameter obtained by the parameter obtaining unit to the camera; an image obtaining unit that obtains images of the object imaged by the camera; an area information obtaining unit that obtains area information corresponding to the position information of the object on the basis of second information associating the position information with area information representing an area for recognizing motion of the object; and an output unit that outputs the images obtained by the image obtaining unit and the area information acquired by the area information obtaining unit.
It is unusual for work in a factory to be completed at a single workbench; in many situations, a worker moves between plural workbenches and performs plural tasks.
The technology recited in JP-A No. 2020-144830 addresses only work that is completed at a single workbench, and is difficult to apply to a situation of moving between plural workbenches and performing plural tasks.
The technology recited in JP-A No. 2021-106338 uses plural cameras disposed over a wide area to obtain a general position of a worker, and perceives hands of the operator and an object in a small area with a camera zoomed in on the hands and the object.
However, in the technology recited in JP-A No. 2021-106338, respective areas to be viewed by the cameras perceiving the wide-area position of the worker and the camera zoomed in on the hands of the worker and the object must be set manually, which is troublesome.
The present technology is made in consideration of the problem described above, and an object of the technology is to provide a work recognition device, a work recognition method and a work recognition program that may easily set breaks between tasks when a worker moves between plural workbenches and performs plural tasks.
A first aspect of the disclosure is a work recognition device including: a movement information acquisition unit that acquires movement information relating to movements of a worker; a work procedure acquisition unit that acquires work procedure information relating to a predefined work procedure of a series of tasks to be performed by the worker at a plurality of workbenches; a movement/standstill determination unit that determines when the worker is in movement and in standstill on the basis of the movement information; a break setting unit that sets breaks between the tasks of the series of tasks on the basis of the work procedure information and results of determining when the worker is in movement and in standstill; and an output unit that outputs break information relating to the set breaks between the tasks.
In the first aspect, when the movement/standstill determination unit determines that a state of the worker has switched from standstill to movement, the break setting unit may record an end time of one of the tasks in the break information for setting a break, and when the movement/standstill determination unit determines that a state of the worker has switched from movement to standstill, the break setting unit may record a start time of one of the tasks in the break information for setting a break.
In the first aspect, the movement information may be a captured image imaging the worker, and the movement/standstill determination unit may determine when the worker is in movement and in standstill on the basis of positions of joints of the worker that are identified on the basis of the captured image.
In the first aspect, the movement/standstill determination unit may determine when the worker is in movement and in standstill on the basis of movement amounts of a central position of a circumscribed area circumscribing all the identified joints of the worker.
In the first aspect, the movement/standstill determination unit may determine when the worker is in movement and in standstill on the basis of movement amounts of a position of, among the identified joints of the worker, a joint representing the torso.
A second aspect of the disclosure is a work recognition method including a computer executing processing that includes: acquiring movement information relating to movements of a worker; acquiring work procedure information relating to a predefined work procedure of a series of tasks to be performed by the worker at a plurality of workbenches; determining when the worker is in movement and in standstill on the basis of the movement information; setting breaks between the tasks of the series of tasks on the basis of the work procedure information and results of determining when the worker is in movement or in standstill; and outputting break information relating to the set breaks between the tasks.
A third aspect of the disclosure is a work recognition program executable by a computer to execute processing that includes: acquiring movement information relating to movements of a worker; acquiring work procedure information relating to a predefined work procedure of a series of tasks to be performed by the worker at a plurality of workbenches; determining when the worker is in movement and in standstill on the basis of the movement information; setting breaks between the tasks of the series of tasks on the basis of the work procedure information and results of determining when the worker is in movement or in standstill; and outputting break information relating to the set breaks between the tasks.
Advantageous Effects of Invention
According to the technology of the disclosure, breaks between tasks may easily be set when a worker is moving between plural workbenches and performing plural tasks.
Below, an example of an embodiment of the present disclosure is described with reference to the drawings. In the drawings, the same reference symbols are assigned to structural elements and portions that are the same or equivalent. Proportional dimensions in the drawings may be exaggerated to facilitate description and may be different from actual proportions.
The work recognition device 20 recognizes details of tasks performed by a worker W on the basis of captured images imaged by the camera 30.
The worker W, for example, pulls up a work object M placed on a workbench TB and performs a predetermined task in a task space S.
More specifically, as illustrated in
As shown in
Below, the plural workbenches may simply be referred to as the workbench(es) TB when not being distinguished. Note that positions of plural workbenches TB, numbers of workbenches TB, types of tasks and numbers of tasks are not limited by the example in
The camera 30 is, for example, an imaging device capable of imaging RGB color video images. The camera 30 is disposed at a position that facilitates recognition of movements of the worker W and all of the workbenches TB1 to TB8. More specifically, in the present exemplary embodiment a situation is described in which, as in the example illustrated in
In the present exemplary embodiment, a configuration with one of the camera 30 is described, but a plural number of the camera 30 may be provided.
As shown in
The input/output interface 21D is connected to a console section 22, a display section 23, a communications section 24 and a memory section 25.
The console section 22 includes, for example, a mouse and a keyboard.
The display section 23 is structured as, for example a liquid crystal display or the like.
The communications section 24 is an interface for communicating data with external equipment such as the camera 30 and the like.
The memory section 25 is structured by a non-volatile external memory apparatus such as a hard disc or the like. As shown in
The CPU 21A is an example of a computer. The meaning of the term “computer” as used herein is intended to refer to broadly defined processors, encompassing general-purpose processors (for example, a CPU) and dedicated processors (for example a graphics processing unit (GPU), application-specific integrated circuit (ASIC), field programmable gate array (FPGA), programmable logic device or the like).
The work recognition program 25A is memorized at a non-volatile, non-transitory memory medium, or may be provided by being distributed via a network and installed as appropriate at the work recognition device 20.
A CD-ROM (compact disc read-only memory), magneto-optical disc, HDD (hard disk drive), DVD-ROM (digital versatile disc read-only memory), flash memory, memory card and so forth are envisaged as examples of a non-volatile, non-transitory memory medium.
The movement information acquisition unit 40 acquires movement information relating to movements of the worker W. More specifically, the movement information acquisition unit 40 acquires captured images, in which the camera 30 images the worker W and the workbenches TB1 to TB8, as the movement information. Movement information of the worker may be acquired by using motion sensors or the like instead of the camera 30.
By reading the work procedure information 25B from the memory section 25, the work procedure acquisition unit 41 acquires the work procedure information 25B relating to a predefined work procedure, which is a series of tasks to be performed at plural workbenches by the worker W. As shown in
The walking/stationary determination unit 42 determines when the worker W is walking and stationary on the basis of the movement information. The term “stationary” refers to any state that is not walking. Thus, the meaning of “determines when the worker W is walking and stationary” is, specifically, determining whether or not the worker W is walking and determining that the worker W is stationary when not determined to be walking. The meaning of the term “stationary” refers to a state in which the body of the worker W in general is stationary. Thus, a state in which the worker W is working at one location is regarded as a state in which the worker W is stationary. Note that walking is an example of being in movement and being stationary is an example of being in standstill. The meaning of the term “in movement” is intended to encompass states of jogging and the like in addition to walking. The meaning of the term “in standstill” is intended to encompass states in which the worker W is not in movement, including states in which the worker is moving at a location as well as states in which the body of the worker is not moving at all.
The walking/stationary determination unit 42 identifies positions of joints of the worker W on the basis of captured images serving as the movement information acquired by the movement information acquisition unit 40. The walking/stationary determination unit 42 then determines whether the worker W is walking or stationary on the basis of the identified positions of the joints of the worker W.
More specifically, the walking/stationary determination unit 42 determines when the worker W is walking and stationary on the basis of movement amounts of a central position of a circumscribed area circumscribing all of the identified joints of the worker W, in other words, a bounding box. In the present exemplary embodiment, the bounding box is a quadrilateral shape such as a rectangle, a square or the like circumscribing all of the joints of the worker W. However, shapes of the bounding box are not limited thus.
A publicly known technique known as “OpenPose”, which is described in Reference Document 1 mentioned below, may be used as a method for identifying the joints of the worker W. With OpenPose, skeletal information of a worker W may be detected from a captured image. More specifically, the skeletal information includes a location of the body of the worker W and coordinates of characteristic points such as joints and the like, link information defining links that connect the characteristic points, and labels representing body parts of the characteristic points. For example, the characteristic points include parts of the face of the worker W, such as the eyes, nose and the like, and joints such as the neck, shoulders, elbows, wrists, waist, knees, ankles and so forth.
OpenPose employs a trained model for which a learning model that inputs captured images and outputs skeletal information is trained using numerous captured images as teaching data. A publicly known technique such as, for example, CNN (regions with conversational neural networks) or the like can be used as a training method to provide the trained model.
Reference Document 1: “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, Zhe Cao, Student Member, IEEE, Gines Hidalgo, Student Member, IEEE, Tomas Simon, Shih-En Wei, and Yaser Sheikh, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE.
The walking/stationary determination unit 42 may determine when the worker W is walking and stationary on the basis of movement amounts of, of the identified joints of the worker W, the positions of joints representing the torso of the worker W.
The break setting unit 43 sets breaks between the tasks in the series of tasks on the basis of the work procedure information 25B acquired by the work procedure acquisition unit 41 and results of determining when the worker W is walking and stationary by the walking/stationary determination unit 42. The meaning of the term “breaks between tasks” includes start times and end times of the respective tasks.
The output unit 44 outputs the break information 25C relating to the breaks set between the tasks. The break information 25C is table data representing associations between task details, task start times and task end times. The break information 25C is described in detail below.
Now, work recognition processing that is executed by the CPU 21A of the work recognition device 20 is described with reference to the flowchart shown in
In step S100, the CPU 21A acquires a video image in which the camera 30 images a situation in which the worker W is performing the series of tasks 1 to 14 at the workbenches TB1 to TB8.
In step S101, the CPU 21A acquires the work procedure information 25B memorized at the memory section 25 by reading the work procedure information 25B.
In step S102, the CPU 21A identifies a start time of the series of tasks on the basis of the video image acquired in step S100. For example, an operator may replay the video image and assign a start time at which the worker W starts task 1 at workbench TB1, or the CPU 21A may identify a start time at which the worker W starts task 1 at workbench TB1 by image recognition processing.
In step S103, the CPU 21A determines when the worker W is walking and stationary on the basis of the video image acquired in step S100. That is, the CPU 21A determines whether the worker W is in a walking state or a stationary state. More specifically, the CPU 21A reads a plural number of chronologically successive frame images from the video image acquired in step S100 and, by executing processing using the above-mentioned OpenPose, identifies positions of the joints of the worker W in each of the plural frame images that are read. The CPU 21A identifies whether or not the worker W is walking in the plural frame images on the basis of movement amounts of a central position of bounding boxes circumscribing all of the identified joints of the worker W in the respective plural frame images. In the present exemplary embodiment, as mentioned above, a state in which the worker is not walking is treated as a stationary state.
More specifically, a situation is described below in which, as shown in
As illustrated in
As shown in
Then, movement amounts and movement directions of the central position in the X direction between the frame images are found. Specifically, a calculated movement amount in the X direction of the central position between frame images F1 and F2 is M12, a calculated movement amount in the X direction of the central position between frame images F2 and F3 is M23, and a calculated movement amount in the X direction of the central position between frame images F3 and F4 is M34. In the example in
A calculated movement direction in the X direction of the central position between frame images F1 and F2 is D12, a calculated movement direction in the X direction of the central position between frame images F2 and F3 is D23, and a calculated movement direction in the X direction of the central position between frame images F3 and F4 is D34. If the movement direction being the right direction is represented by “R”, the left direction is represented by “L”, and a zero movement amount is represented by “N”, in the example in
To determine whether or not the worker W is walking, a determination is made as to whether walking conditions are satisfied in accordance with the movement amounts and movement directions of the central position in the X direction between the frame images. More specifically, the walking conditions may be that all of the movement amounts M12, M23 and M34 are at least a predefined threshold amount and that all of the movement directions D12, D23 and D34 are “R”, the direction of a workbench at which the next task is to be performed. When the walking conditions are satisfied, the worker W is determined to be walking toward workbench TB2 to perform the next task. The threshold value is set to a value at which the worker W can be judged to be walking, and is set in advance on the basis of, for example, previously observed measurements of a relationship between movements of the worker W and movement amounts in frame images. In the present exemplary embodiment, as an example, the threshold value is set to 2. Thus, the walking conditions are not satisfied in the example in
Thus, whether or not the worker W is walking toward the next workbench TB is determined for each set of four frames.
In step S104, the CPU 21A makes a determination as to whether there has been a switch between walking and stationary in step S103. That is, the CPU 21A makes a determination as to whether the state of the worker W determined in the previous execution of step S103 and the state of the worker W determined in the most recent execution of step S103 have switched from walking to stationary, and whether the states have switched from stationary to walking.
If the state has switched from walking to stationary or has switched from stationary to walking, the CPU 21A proceeds to step S105. On the other hand, if the state remains walking or the state remains stationary, the CPU 21A proceeds to step S106.
In step S105, the CPU 21A sets the time of the switch from walking to stationary or the switch from stationary to walking as a break between tasks. When the timing of a switch from walking to stationary is set as a break, the CPU 21A determines that the worker W has moved to the next workbench.
As shown in, for example,
As shown in
As is shown in
As is also shown in
The timing of t2, at which the state of the worker W switches from walking to stationary, is a timing when the worker W has moved from workbench TB1 to workbench TB2. Therefore, when the timing of t2 is detected, it may be determined that the worker W has moved from workbench TB1 to workbench TB2.
As shown in
Accordingly, when a plural number of tasks are performed at a single workbench, start times and end times of the respective tasks are recorded by dividing a work duration from the start time of the first of the plural tasks to the end time of the last of the plural tasks by the number of tasks.
In the example in
In step S106, the CPU 21A makes a determination as to whether recording to the break information 25C has been completed for all of the series of tasks. More specifically, the CPU 21A refers to the work procedure information 25B, acquires the number of tasks in the series of tasks, and makes a determination as to whether start times and end times have been recorded in the break information 25C for all of the acquired number of tasks. For example, a determination is made as to whether start times and end times have been recorded for all of tasks 1 to 14 as illustrated in
When recording to the break information 25C has been completed for all tasks in the series of tasks, as in the example in
In step S107, the CPU 21A outputs and memorizes the break information 25C recording the start times and end times of all the tasks at the memory section 25.
In step S108, the CPU 21A accepts editing processing by an operator for annotating the video image acquired in step S100, and executes editing processing for editing the video image in accordance with operations by the operator. Using the break information 25C at this time enables easy switching of a replay position of the video image to start times of tasks and the like. Therefore, the burden of work in editing annotations is moderated.
Thus, in the present exemplary embodiment, on the basis of a video image imaging work of a worker W, timings at which the state of the worker W switches from walking to stationary and timings at which the state switches from stationary to walking are set as task breaks. Thus, when the worker W performs work at plural workbenches TB, breaks in the work may be set easily.
In the present exemplary embodiment, a situation is described in which determinations as to whether the worker W is walking are made on the basis of movement amounts in the X direction of the central position of bounding boxes circumscribing all joints of the worker W. However, if movement amounts of the worker W when moving between the workbenches TB are greater in the Y direction than in the X direction, determinations as to whether the worker W is walking may also be made on the basis of movement amounts in the Y direction of the central position of the bounding boxes. Further, centroid positions of the bounding boxes may be calculated, and determinations as to whether the worker W is walking may be made on the basis of movement amounts of the calculated centroid position in the XY plane.
In the present exemplary embodiment, a situation is described in which determinations as to whether the worker W is walking are made on the basis of movement amounts in the X direction of the central position of bounding boxes circumscribing all joints of the worker W, but a method of determining walking is not limited thus. For example, among the joints of the worker W, determinations as to whether a worker W is walking may be made on the basis of movement amounts of positions of joints representing the torso. This is because it can be supposed that when a worker W is performing a task, the parts that make large movements are the arms and hands, and the position of the torso barely moves at all. Joints representing the torso means, for example, joints at the head, neck and waist. Accordingly, it is sufficient to calculate average positions of the joints of the head, neck and waist in the X direction and make determinations as to whether the worker W is walking on the basis of movement amounts of the calculated average positions.
The exemplary embodiment described above is no more than an exemplary description of a structural example of the present disclosure. The present disclosure is not to be limited by the specific mode described above; numerous modifications are applicable within the scope of the technical gist of the disclosure.
The work recognition processing that, in the exemplary embodiment described above, is executed by a CPU reading software (a program) may be executed by various kinds of processor other than a CPU. Examples of processors in these cases include a PLD (programmable logic device) in which a circuit configuration can be modified after manufacturing, such as an FPGA (field-programmable gate array) or the like, a dedicated electronic circuit which is a processor with a circuit configuration that is specially designed to execute recognition processing, such as an ASIC (application-specific integrated circuit) or the like, and so forth. The work recognition processing may be executed by one of these various kinds of processors, and may be executed by a combination of two or more processors of the same or different kinds (for example, plural FPGAs, a combination of a CPU with an FPGA, or the like). Hardware structures of these various kinds of processors are, to be more specific, electronic circuits combining circuit components such as semiconductor components and the like.
The disclosures of Japanese Patent Application No. 2022-038635 are incorporated into the present specification by reference in their entirety. All references, patent applications and technical specifications cited in the present specification are incorporated by reference into the present specification to the same extent as if the individual references, patent applications and technical specifications were specifically and individually recited as being incorporated by reference.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2022-038635 | Mar 2022 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2023/002319 | 1/25/2023 | WO |
