This application claims priority pursuant to 35 U.S.C. § 119(a) to Japanese Patent Application Nos. 2020-051268, filed on Mar. 23, 2020, and 2021-003427, filed on Jan. 13, 2021 in the Japan Patent Office, the disclosure of which are incorporated by reference herein in its entirety.
This disclosure relates to an information processing system, a user terminal, and a method of processing information processing.
Conference assistance or supporting apparatuses are known and used in various scenes. Typically, a conference assistance apparatus converts speech (i.e., voice) spoken by a conference participant into text, detects an important spoken content from the text by applying a plurality of rules used for specifying the important spoken content, extracts a matter indicating the important spoken content, and presents the extracted matter to the conference participant.
Further, tools such as collaboration tools for assisting or supporting collaborative works performable by a group have been developed and used, in which the tools provide various functions, such as communication function and information sharing function, to a plurality of users who are working as one group or team. For example, a plurality of users who are working as one group or team for a collaborative work or collaboration work can use communication function, such as chat function and voice communication function, and information sharing function to hold an online meeting, and in-person meeting where the users gather at the same physical place.
When a conference using the communication function, information sharing function, or other function is held, recorded video data or recorded audio data of a plurality of users who work together in a group or a team may include one or more action item candidates to be assigned and executed by one or more specific users.
However, conventional collaboration tools have no mechanism for displaying the action item candidate included in the recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data of action item candidate.
As one aspect of the present disclosure, an information processing system is devised. The information processing system includes a plurality of user terminals; and a collaboration work assistance system for assisting a collaboration work performable by a plurality of users who operate the respective user terminals. The collaboration work assistance system includes circuitry configured to transmit, to each of the user terminals, an action item candidate based on voice-text data converted from recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data corresponding to the action item candidate. The user terminal includes another circuitry configured to receive the action item candidate and the replay information from the collaboration work assistance system; and display the received action item candidate and the replay information on a display.
As another aspect of the present disclosure, a user terminal capable of communicating with a collaboration work assistance system for assisting a collaboration work performable by a plurality of users is devised. The user terminal includes another circuitry configured to receive, from the collaboration work assistance system, an action item candidate based on voice-text data converted from recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data corresponding to the action item candidate; and display the received action item candidate and the replay information on a display.
As another aspect of the present disclosure, a method of processing information performable by a user terminal capable of communicating with a collaboration work assistance system for assisting a collaboration work performable by a plurality of users is devised. The method includes receiving, from the collaboration work assistance system, an action item candidate based on voice-text data converted from recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data corresponding to the action item candidate; and displaying the received action item candidate and the replay information on a display.
A more complete appreciation of the description and many of the attendant advantages and features thereof can be readily acquired and understood from the following detailed description with reference to the accompanying drawings, wherein:
The accompanying drawings are intended to depict embodiments of this disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted.
A description is now given of exemplary embodiments of the present inventions. It should be noted that although such twos as first, second, etc. may be used herein to describe various elements, components, regions, layers and/or units, it should be understood that such elements, components, regions, layers and/or units are not limited thereby because such terms are relative, that is, used only to distinguish one element, component, region, layer or unit from another region, layer or unit. Thus, for example, a first element, component, region, layer or unit discussed below could be termed a second element, component, region, layer or unit without departing from the teachings of the present inventions.
In addition, it should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present inventions. Thus, for example, as used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Moreover, the terms “includes” and/or “including,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Hereinafter, a description is given of one or more embodiments with reference to the drawings, but the present invention is not limited to embodiment described in this description. In this description, a group or team of users gathering for performing a collaborative work (or collaboration work) is referred to as a project, and each user belonging to the project is referred to as a member of project or project member. Further, in this description, a work or job to be assigned and conducted by each member is referred to as an action item, but is not limited thereto. For example, the action item may also be referred to as a task or follow-up matter. Further, in this description, “file” may indicate “electronic file”.
The communication network 50 is constructed, for example, by the Internet, a mobile communication network, local area network (LAN) or the like. The communication network 50 may include not only wired communication but also a wireless communication network such as 3rd Generation (3G), Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE).
The collaboration system 10 is a system that assists or supports a plurality of persons or users to perform one work together. For example, the collaboration system 10 is a system that is used to perform voice call, video call, chat, screen sharing (projection), information sharing, review of agenda, preparation and compilation of reference materials, schedule setting, determination of action item, or the like, using personal computer (PC), portable terminal, electronic information board (e.g., electronic whiteboard), projector, conference tool, or the like, in a place where persons gather to discuss and share opinions, consultation, or the like for a specific purpose (including a case where persons gather at the same physical space, and a case where persons are physically separated but user terminals are connected to each other via a network).
The collaboration system 10 has the same meaning as the collaboration work assistance system or collaboration work supporting system. The collaboration system 10 provides various collaboration tools to assist or support a collaborative work or collaboration work performable by project members. Each collaboration tool is a unit, function, or tool for assisting or supporting a plurality of persons who are working together to accomplish one work or job. The collaborative work (or collaboration work) means that a plurality of persons perform one work together. The collaboration tools include various functions, such as communication function, scheduling function (schedule management function), information sharing function, and action item management function.
Further, the collaboration tools provided by a plurality of collaboration systems 10 may not be the same functions. Therefore, in an example case of
A sharing assistant server 11 (or sharing support server 11) included in the collaboration system 10 provides various functions, such as communication function, and information sharing function, to each project member.
A schedule management server 12 included in the collaboration system 10 provides a schedule management function to each project member.
A voice-to-text conversion server 13 included in the collaboration system 10 converts recorded video data or recorded audio data recording speeches of project members into text data. As illustrated in the information processing system 1 of
An action item management server 14 included in the collaboration system 10 provides an action item management function to each project member. The action item is a work or job, which is required to be assigned and executed or conducted by one or more users. As illustrated in the information processing system 1 of
The action item management system 20 provides the action item management function to each user who operates the user terminal 40. The action item management function is a function for receiving a registration of action item from each user, and managing the action item. The action item management function includes at least one function or unit provided in the action item management server 14 illustrated in
The user terminal 40, which is an example of information processing terminal, is operated by each user, such as each project member. The user can use the collaboration system 10 and the action item management system 20 from the user terminal 40.
Further, the user terminal 40 is not limited to a PC as long as the user terminal 40 has the communication function. The user terminal 40 may be, for example, projector (PJ), interactive whiteboard (IWB) having electronic board function that can communicate with each other, output device such as a digital signage, head up display (HUD) device, industrial machine, imaging device, sound collecting device, medical device, network appliance, connected car, personal computer, portable telephone, smart phone, tablet terminal, game machine, personal digital assistant (PDA), digital camera, wearable PC, and desktop PC.
Further, the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, and the action item management server 14 can be implemented by, for example, one or more information processing apparatuses. Further, the collaboration system 10 may be one or more computers, in which a part or all of the functions of the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, and the action item management server 14 are integrated.
Further, one or more functions of the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, and the action item management server 14 can be distributed over a plurality of computers.
Further, the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, and the action item management server 14 may be implemented by computers in a cloud environment or computers in an on-premise environment. As described above, the information processing system 1 illustrated in
Further, the collaboration system 10, the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, the action item management server 14, and the action item management system 20 are not limited to PCs as long as these apparatuses have the communication function.
Further, each of the collaboration system 10, the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, the action item management server 14, and the action item management system 20 may be, for example, projector (PJ), interactive whiteboard (IWB), output device such as a digital signage, head up display (HUD) device, industrial machine, imaging device, sound collecting device, medical device, network appliance, connected car, personal computer, portable telephone, smart phone, tablet terminal, game machine, personal digital assistant (PDA), digital camera, wearable PC, and desktop PC.
Further, the collaboration information provided by the collaboration system 10 to each project member by the collaboration tool includes, for example, synchronous collaboration information and asynchronous collaboration information.
The synchronous collaboration information is information that is exchanged in real time in a place (e.g., meeting), in which persons gather to discuss and share opinions or consult for a specific purpose. The synchronous collaboration information is, for example, voice call, video call, and screen shared by PC/portable terminal/electronic whiteboard (including text and diagrams input from PC, portable terminal, and handwritten input contents to a screen of electronic whiteboard).
Further, the asynchronous collaboration information is information that is input or shared on the network. The asynchronous collaboration information includes, for example, information that can be shared and referenced by participants before and after an event (e.g., meeting) where persons gather to discuss and share opinions or consult for a specific purpose. The asynchronous collaboration information includes, for example, chat, message, content, scheduling, action item, content of audio recording/video recording, or the like.
An action item is a work or job to be executed by an execution person who is determined to execute the work or job.
These terms are included in Conference Solutions Report, “Magic Quadrant for Meeting Solution”, published in September 2019, by Gartner, Inc.
The collaboration system 10, the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, the action item management server 14, the action item management system 20, and the user terminal 40 illustrated in
As illustrated in
The CPU 501 controls the operation of the computer 500 entirely in accordance with one or more programs. The ROM 502 stores programs for driving the CPU 501, such as initial program loader (IPL). The RAM 503 is used as a work area of the CPU 501.
The BD 504 stores various data, such as programs. The HDD controller 505 controls reading or writing of various data from or to the HD 504 under the control of the CPU 501.
The display 506 displays various information, such as cursor, menu, window, characters, or image.
The external device connection I/F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a universal serial bus (USB) memory or a printer. The network I/F 509 is an interface for performing data communication using the communication network 16. The bus line 510 is address bus and data bus for electrically connecting each component, such as the CPU 501 illustrated in
The keyboard 511 is a type of input means including a plurality of keys for inputting characters, letters, numbers, and various instructions.
The pointing device 512 is a type of input means for selecting or executing various instructions, selecting processing target, moving a cursor, or the like.
The DVD-RW drive 514 controls reading or writing of various data from or to the DVD-RW 513, which is as an example of removable recording medium. Further, the removable recording medium may be digital versatile disk recordable (DVD-R) in addition to DVD-RW. The media I/F 516 controls reading or writing of data from or to a recording medium 515, such as flash memory.
Further, each of the above described programs may be recorded on a computer-readable recording medium in an installable form or an executable form and distributed. Examples of the recording medium include compact disc recordable (CD-R), digital versatile disk (DVD), Blu-Ray (registered trademark) disc, and secure digital (SD) card. Further, the recording medium can be provided to a country or a foreign country as a program product.
Hereinafter, with reference to
The meeting minutes application 5502a and the browser 5502b are applications used for generating and displaying a conducted event record screen to be described later. The conducted event record screen displays, for example, minutes of event (e.g., online meeting) conducted or performed using various data transmitted from the schedule management server 12. The replay application 5502c replays recorded video data or recorded audio data.
Although
The information processing system 1 according to the first embodiment is implemented by a functional units illustrated in
As illustrated in
The schedule management server 12 stores and manages a plurality of schedulers. Since different organizations may use different schedulers, the management using the access management table illustrated in
The scheduled event ID is identification information identifying each scheduled or reserved event. The scheduled event ID is an example of scheduled event identification information identifying each event to be conducted.
The conducted event ID is identification information identifying each event that was actually conducted or each event that is being conducted among the scheduled or reserved events.
The conducted event ID is an example of conducted event identification information identifying each conducted event or each being-conducted event.
The reservation holder name is a name of a person who has reserved a shared resource, such as a conference room.
The shared resource is an object, service, space (room), place, or information that can be used by a plurality of persons or a group. The conference room is an example of shared resource that is shared by a plurality of users. If the shared resource is a conference room, the reservation holder name is, for example, a name of organizer or host.
The scheduled start time indicates a scheduled use start time of shared resource. The scheduled end time indicates a scheduled use end time of shared resource.
The event name indicates a name of each event scheduled to be conducted by the reservation holder.
Hereinafter, the “shared resource” may be simply referred to as the “resource” to simplify the expression in this description.
The user ID of other participant is identification information identifying each participant other than the reservation holder.
The other participant name is a name of each participant other than the reservation holder.
The file data is file data of reference file to be used in each event corresponding to the scheduled event ID registered by a user. The file data is data that is created using various applications and a given file format. The file format of file data is, for example, a file for presentation software, a file for spread sheet software, or the like.
The project ID is identification information identifying each project (e.g., project identification information). The project represents a group, team, or grouping of users configured to achieve a particular goal, plan, project, or the like. Members belonging to the same project can share conducted event record, such as the minutes of event associated with the project ID.
The project ID is allocated to each project. Further, the project ID may be also referred to as group ID or team ID.
The content is a content of conducted event that was generated in each event (e.g., meeting) in a particular project, or reference materials that was used in each particular event. The type of content processing includes, for example, audio recording, snapshot, voice-to-text conversion, generation of action item, chatting, and sending or transmitting reference materials.
Further, the content processing ID is identification information identifying each content processing that was generated in each event.
The content includes, for example, history information indicating the content of conducted event, and an action item generated by the conducted event. The history information indicating the content of conducted event includes data, such as recorded audio data, recorded video data, snapshot, statement text data, or reference materials.
The voice-text data includes text data converted from the recorded video data or recorded audio data by performing the voice-text conversion.
The statement text data includes, for example, voice-text data converted from the recorded audio data or recorded video data by performing the voice-to-text conversion, and text data input for chat and message.
The snapshot is a process of acquiring a display screen at a certain time point in an event being conducted, as image data. The snapshot is also referred to, for example, capture or image recognition.
If the content processing type is “audio recording”, the material of content (content data) includes, for example, universal resource locator (URL) indicating a storage destination of the recorded audio data.
If the content processing type is “video recording”, the material of content (content data) includes, for example, universal resource locator (URL) indicating a storage destination of the recorded video data.
Further, if the content processing type is “snapshot”, the material of content (content data) includes, for example, URL indicating a storage destination of image data of a screen acquired by the snapshot (capture). The capture means saving of image (e.g., still image, movie image) displayed on the display 506 as image data.
If the content processing type is “voice-to-text conversion”, the material of content (content data) includes, for example, URL indicating a storage destination of text data of the received voice data.
The action item indicates a content of action, which occurs in each event (e.g., meeting) in each project and is required to be performed or executed by one or more persons related to each event. If the content processing type is “action item generation”, the material of content (content data) includes, for example, user ID of execution person of action item, due date for completing action item, and URL indicating a storage destination of text data indicating the content of action item.
The statement text data ID is identification information identifying each statement text data of action item candidate detected from the voice-text data identified by the content processing ID.
The action item candidate is a statement text data that may be registered with higher probability as an action item detected from spoken content or text input by each project member during an event (e.g., online meeting). The action item candidate is, for example, an item including at least two elements of “who”, “what”, and “until when” in the statement text data.
The replay position information indicates a replay position in recorded video data or recorded audio data of each statement text data of action item candidate identified by each statement text data ID (e.g., an elapsed time from the start of recorded video data or recorded audio data). Further, the replay position information may include a time of speaking each action item candidate.
The content indicates a content of each action item candidate.
The action item ID indicates a particular action item when the particular action item candidate has been registered as the action item, and the action item ID is set blank when the particular action item candidate has not been registered as the action item.
Hereinafter, with reference to
The transmission/reception unit 61 of the sharing assistant server 11 illustrated in
The authentication unit 62, which is implemented by the instructions from the CPU 501 illustrated in
The creation unit 63, which is implemented by the instructions from the CPU 501 illustrated in
The generation unit 64, which is implemented by the instructions from the CPU 501 illustrated in
The determination unit 65, which is implemented by the instructions from the CPU 501 illustrated in
The detection unit 66, which is implemented by the instructions from the CPU 501 illustrated in
Further, the detection unit 66 detects replay position information in recorded video data or recorded audio data corresponding to statement text data of action item candidate.
The replay position information indicates given information that is required to start a replay from a particular position (or a just before the position position) of statement text data of action item candidate when replaying the recorded video data or recorded audio data corresponding to the statement text data of action item candidate. For example, the replay position information can be indicated by an elapsed from the start of recorded video data or recorded audio data. Further, the replay position information may also include a time of speaking each action item candidate.
The storing/reading processing unit 69, which is implemented by the instructions from the CPU 501 illustrated in
As illustrated in
The scheduled use start date/time indicates, for example, a scheduled use start date/time of each resource. The scheduled use end date/time indicates, for example, a scheduled use end date/time of each resource. Each date indicates, for example, year, month, day, hour, minute, second, and time zone. Each date illustrated in
The scheduled event start date/time indicates a scheduled start date/time when an event is to be conducted. The scheduled event end date/time indicates a scheduled end date/time when an event is to be conducted.
Each date/time indicates, for example, year, month, day, hour, minute, second, and time zone. Each/time date illustrated in
The conducted event record management table stores or manages content processing ID, type of content processing, material of content (content data), and start date/time and end date/time of content processing in association with each other for each project ID and conducted event ID.
The conducted event record management DB 8008 stores or manages data, which are partially the same as the content management DB 6005. The same data includes, for example, conducted event ID, content processing ID, type of content processing, start date/time of content processing, and end date/time of content processing. The material of content (content data) has the same storage destination only different in expression of storage destination of content data, such as http:// or c://.
The content generation time indicates a time of generation of content in a conducted event, which is elapsed from a start date/time of the conducted event. The content generation time is generated by the generation unit 83 based on the start date/time of event stored in the event management DB 8005 and the start date/time and end date/time of the content processing stored in the conducted event record management DB 8008. The content generation time is an example of time information.
The recorded video data or recorded audio data includes content processing ID and content processing type.
The voice-text data and screen data include, for example, content processing ID, content processing type, and sequence No. The sequence No indicates an occurrence time sequence in which the content processing has occurred on the voice-text data and the screen data, respectively.
Hereinafter, with reference to
The transmission/reception unit 81 of the schedule management server 12 illustrated in
The authentication unit 82, which is implemented by the instructions from the CPU 501 illustrated in
Further, the authentication unit 82 performs authentication by determining whether information (e.g., access ID and access password) transmitted from the sharing assistant server 11 is information registered in the server authentication management DB 8006 in advance.
The generation unit 83, which is implemented by the instructions from the CPU 501 illustrated in
The storing/reading processing unit 89, which is implemented by the instructions from the CPU 501 illustrated in
As illustrated in
Hereinafter, with reference to
The transmission/reception unit 91 of the voice-to-text conversion server 13 illustrated in
The conversion unit 93, which is implemented by the instructions from the CPU 501 illustrated in
The storing/reading processing unit 99, which is implemented by the instructions from the CPU 501 illustrated in
As illustrated in
The storage unit 10000 includes a user authentication management DB 10001 configured by the user authentication management table illustrated in
Further, the state indicates whether each action item has been completed or has not been completed.
Further, based on the content processing ID and replay position information included in the action item management table illustrated in
Hereinafter, with reference to
The transmission/reception unit 101 of the action item management server 14 illustrated in
The authentication unit 102, which is implemented by the instructions from the CPU 501 illustrated in
The creation unit 103, which is implemented by the instructions from the CPU 501 illustrated in
The image data of screen created by the creation unit 63 is used to display an action item of a user who operates the user terminal 40 or a project member to which each user belongs, using the user terminal 40 or the like.
The registration unit 104, which is implemented by the instructions from the CPU 501 illustrated in
The action unit 105 is implemented by the instructions from the CPU 501 illustrated in
Further, when the action unit 105 receives an inquiry request for a state of an action item from, for example, the sharing assistant server 11 or the user terminal 40, the action unit 105 reads out the state of the action item corresponding to the inquiry request from the action item management table illustrated in
With this configuration, the sharing assistant server 11 or the user terminal 40 can confirm whether the action item has been completed or has not been completed.
The storing/reading processing unit 109, which is implemented by the instructions from the CPU 501 illustrated in
As illustrated in
Hereinafter, with reference to
The transmission/reception unit 51, which is implemented by the instructions from the CPU 501 and the network I/F 509 illustrated in
The reception unit 52 is implemented mainly by the instructions from the CPU 501, the keyboard 511, and the pointing device 512 illustrated in
The display control unit 54 is implemented by the instructions from the CPU 501 illustrated in
For example, the display control unit 54 accesses the sharing assistant server 11, the schedule management server 12, or the action item management server 14 by activating and executing the meeting minutes application 5502a or the browser 5502b operable on the OS 5501 illustrated in
Then, the display control unit 54 downloads WebApplication (WebAPP) including at least HyperText Markup Language (HTML), and Cascading Style Sheets (CSS), JAVASCRIPT (registered trademark), or the like, and instructs the display 506 to display various image data generated by the WebAPP.
Further, for example, the display control unit 54 instructs the display 506 to display image data generated by “HTML5” including data expressed using a given format, such as Extensible Markup Language (XML), JavaScript Object Notation (JSON), or Simple Object Access Protocol (SOAP) format.
The display control unit 54 is an example of display control unit.
The request unit 55, which is implemented by the instructions from the CPU 501 illustrated in
The generation unit 56, which is implemented by the instructions from the CPU 501 illustrated in
For example, the generation unit 56 generates image data related to text data, which is content image data, in order to render text data, which is content data, and to display the rendered data. The rendering is a process of interpreting data written in a web page description language (e.g., HTML, CSS, XML), and calculating an arrangement of characters, image data, or the like actually displayed on a screen.
The replay unit 57, which is implemented by the instructions from the CPU 501 illustrated in
The audio control unit 58 which is implemented by the instructions from the CPU 501 illustrated in
The storing/reading processing unit 59, which implemented by the instructions from the CPU 501 and the HDD controller 505 illustrated in
Each of the above described IDs is an example of identification information. The organization ID includes, for example, company name, business office name, department name, region name, or the like. The user ID includes, for example, employee number, driver's license number, and a social number in the Japanese social security and tax number system.
Hereinafter, with reference to
Hereinafter, a description is given of a process that a reservation holder A (e.g., name is Taro Riko) registers his/her own schedule from the user terminal 40 to the schedule management server 12. The reservation holder is a reservation person.
At first, when the user A operates the keyboard 511 of the user terminal 40, the display control unit 54 of the user terminal 40 displays a sign-in screen 530 used for performing the sign-in on the display 506 as illustrated in
The sign-in screen 530 includes, for example, an input field 531, an input field 532, a sign-in button 538, and a cancel button 539.
The input field 531 is used for inputting a user ID and an organization ID of a user. The input field 532 is used for inputting a password.
The sign-in button 538 is to be pressed in a case of performing the sign-in. The cancel button 539 is to be pressed in a case of cancelling the sign-in.
In this example case, the user ID and the organization ID configure an e-mail address of the user A. The user name portion of the e-mail address indicates the user ID, and the domain name portion indicates the organization ID. Further, the input field 531 may be an input field used for separately inputting the user ID and the organization ID instead of the e-mail address.
Then, if the user A inputs his/her user ID and organization ID in the input field 531, inputs his/her password in the input field 532, and then presses the sign-in button 538, the reception unit 52 receives or accepts a sign-in request (step S12).
Then, the transmission/reception unit 51 of the user terminal 40 transmits sign-in request information indicating the sign-in request to the schedule management server 12 (step S13). The sign-in request information includes information received in step S12 such as user ID, organization ID, and password. Then, the transmission/reception unit 81 of the schedule management server 12 receives the sign-in request information.
Then, the authentication unit 82 of the schedule management server 12 authenticates the user A using the user ID, organization ID, and password (step S14). Specifically, the storing/reading processing unit 89 searches a combination of user ID, organization ID, and password corresponding to a combination of the user ID, organization ID, and password received in step S13, in the user authentication management DB 8001 (see
If the corresponding combination exists in the user authentication management DB 8001, the authentication unit 82 determines that the user A of request source is an authenticated user.
On the other hand, if the corresponding combination does not exist in the user authentication management DB 8001, the authentication unit 82 determines that the user A is not an authenticated user (i.e., the user A is an unauthenticated user). If the user A is the unauthenticated user, the transmission/reception unit 81 notifies the user terminal 40 that the user A is the unauthenticated user.
In this example case, the description is given by assuming the user A is the authenticated user.
Then, the transmission/reception unit 81 transmits an authentication result to the user terminal 40 (step S15). Then, the transmission/reception unit 51 of the user terminal 40 receives the authentication result.
Then, in response to receiving the authentication result indicating that the authentication result is valid in step 15, the generation unit 56 of the user terminal 40 generates an initial screen 540 illustrated in
Then, the display control unit 54 of the user terminal 40 instructs the display 506 to display the initial screen 540 as illustrated in
As illustrated in
If the user presses the “register schedule” button 541, the reception unit 52 receives the schedule registration request (step S18).
Then, the transmission/reception unit 51 transmits the schedule registration request information to the schedule management server 12 (step S19). Then, the transmission/reception unit 81 of the schedule management server 12 receives the schedule registration request information.
Then, the storing/reading processing unit 89 of the schedule management server 12 searches the user management DB 8002 (see
Then, the transmission/reception unit 81 transmits schedule input screen information to the user terminal 40 (step S21). Then, the transmission/reception unit 51 of the user terminal 40 receives the schedule input screen information.
The schedule input screen information includes all of user IDs and all of user names read out in step S20. All of user names also include a title (name) of the reservation holder, such as the user A, who has input the information for the sign-in in step S12.
Then, at the user terminal 40, the generation unit 56 generates a schedule input screen 550 using the schedule input screen information received in step S21 (step S22).
Then, the display control unit 54 of the user terminal 40 instructs the display 506 to display the schedule input screen 550 (see
As illustrated in
The input field 551 is used for inputting an event name. The input field 552 is used for inputting a resource ID or a resource name to be used. The input field 553 is used for inputting a scheduled start date/time of execution of event.
The input field 554 is used for inputting a scheduled end date/time of execution of event. The input field 555 is used for inputting a memo such as agenda or the like.
The display area 556 is used for displaying a reservation holder name. The selection menu 557 is used for selecting one or more participant names of participants other than the reservation holder.
The “OK” button 558 is to be pressed in a case of registering a reservation. The “CANCEL” button 559 is to be pressed in a case of canceling information being input or input information.
Further, it is not required to input the resource ID or resource name in the input field 552, and the input field 552 can be left blank when no resources is used. The reservation holder name is a name of a user who has input information for the sign-in to the user terminal 40 in step S12. Further, a pointer p1 of the pointing device 512, such as a mouse pointer, is also displayed on the schedule input screen 550.
Then, the user A inputs given information items into the input fields 551 to 555, selects a name (user name) of a user who is requested to be participated an event from the selection menu 557 using the pointer p1, and then presses the “OK” button 558.
Then, the reception unit 52 receives an input of the schedule information (step S24).
Then, the transmission/reception unit 51 transmits the schedule information to the schedule management server 12 (step S25).
The schedule information includes event name, resource ID (or resource name), scheduled start date/time, scheduled end date/time, user ID of each participant, and memo.
If a resource ID is input in the input field 552 on the schedule input screen 550, the transmission/reception unit 51 transmits the resource ID to the schedule management server 12.
Further, if a resource name is input in the input field 552 on the schedule input screen 550, the transmission/reception unit 51 transmits the resource name to the schedule management server 12.
Further, if a user name is selected in the selection menu 557 on the schedule input screen 550, since the user ID is also received in step S21, the transmission/reception unit 51 transmits the user ID corresponding to the selected user name to the schedule management server 12. Then, the transmission/reception unit 81 of the schedule management server 12 receives the schedule information.
Then, if the schedule management server 12 receives the resource ID (or resource name) in step S25, the schedule management server 12 searches the resource management DB 8003 (see
Then, the storing/reading processing unit 89 stores reservation information for the resource name (or resource ID) read out in step S26 in the resource reservation management DB 8004 (see
Further, the scheduled use start date/time stored in the resource reservation management DB 8004 corresponds to the scheduled start date/time included in the schedule information. Further, the scheduled use end date/time stored in the resource reservation management DB 8004 corresponds to the scheduled end date/time included in the schedule information.
Further, the storing/reading processing unit 89 stores the schedule information (event information) in the event management DB 8005 (see
The schedule information is configured based on the schedule information received in step S25.
Further, the scheduled event start date/time stored in the event management DB 8005 corresponds to the scheduled start date/time included in the schedule information. Further, the scheduled event end date/time stored in the event management DB 8005 corresponds to the scheduled end date/time included in the schedule information.
With this configuration, the user A can register his/her schedule in the schedule management server 12.
Hereinafter, with reference to
At first, when the reservation holder A presses a power switch of the user terminal 40, the reception unit 52 of the user terminal 40 receives a power ON signal from the reservation holder A (step S31).
Then, the display control unit 54 of the user terminal 40 displays a sign-in screen used for performing a sign-in on the display 560 (step S32).
Then, if the user A inputs his/her e-mail address and password on the sign-in screen, the reception unit 52 receives or accepts a sign-in request (step S33).
Then, the transmission/reception unit 51 transmits sign-in request information indicating the sign-in request to the sharing assistant server 11 (step S34). The sign-in request information includes the information received in step S33 such as user ID, organization ID, and password. Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the sign-in request information.
Then, the authentication unit 62 of the sharing assistant server 11 authenticates the user A using the user ID of the user A, organization ID, and password of the user A received in step S34 (step S35). Specifically, the storing/reading processing unit 69 uses the user ID, organization ID, and password of the user A received in step S34 as search keys to search a combination of user ID, organization ID, and password corresponding to a combination of the user ID, organization ID, and password of the user A received in step S34, in the user authentication management DB 6001 (see
If the corresponding combination exists in the user authentication management DB 6001, the authentication unit 62 determines that the user A of request source is an authenticated user.
On the other hand, if the corresponding combination does not exist in the user authentication management DB 6001, the authentication unit 62 determines that the user A of request source is not an authenticated user (i.e., the user A is an unauthenticated user). If the user A is the unauthenticated user, the transmission/reception unit 61 notifies the user terminal 40 that the user A is the unauthenticated user.
In this example case, the description is given by assuming the user A is the authenticated user.
Then, the storing/reading processing unit 69 of the sharing assistant server 11 searches the access management DB 6002 (see
Then, the transmission/reception unit 61 transmits, to the schedule management server 12, reservation request information indicating a request for reservation information of resource, and schedule request information indicating a request for schedule information of user (step S37). The reservation request information and the schedule request information include the user ID, organization ID, and password of the user A received in step S34, and the access ID and access password read out in step S36. Then, the transmission/reception unit 81 of the schedule management server 12 receives the reservation request information and the schedule request information.
Then, the authentication unit 82 of the schedule management server 12 authenticates the sharing assistant server 11 using the access ID and access password (step S38).
Specifically, the storing/reading processing unit 89 searches a combination of the access ID and access password corresponding to a combination of the access ID and access password received in step S37, in the server authentication management DB 8006 (see
If the corresponding combination exists in the server authentication management DB 8006, the authentication unit 82 determines that the sharing assistant server 11 of request source is an authenticated access user.
On the other hand, if the corresponding combination does not exist in the server authentication management DB 8006, the authentication unit 82 determines that the sharing assistant server 11 of request source is not an authenticated access user (i.e., the sharing assistant server 11 is an unauthenticated access user). If the sharing assistant server 11 is the unauthenticated user, the transmission/reception unit 81 notifies the sharing assistant server 11 that the sharing assistant server 11 is the unauthenticated access user.
In this example case, the description is given by assuming the sharing assistant server 11 is the authenticated access user.
Then, the storing/reading processing unit 89 of the schedule management server 12 reads out the corresponding reservation information by searching the resource reservation management DB 8004 (see
Further, the storing/reading processing unit 89 searches the event management DB 8005 (see
In this example case, the storing/reading processing unit 89 reads out the schedule information whose event start schedule date is today. If the schedule management server 12 is located in a country or region different from the user terminal 40, the schedule management server 12 adjusts the time zone in accordance with the country or region where the user terminal 40 is located based on the time zone information.
Then, the storing/reading processing unit 89 searches the project member management DB 8007 (see
Then, the transmission/reception unit 81 transmits, to the sharing assistant server 11, the reservation information read out in step S39, the schedule information read out in step S40, and all of project IDs and all of project names read out in step S41 (step S42). Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the reservation information, schedule information, all of project IDs, and all of project names.
Then, the creation unit 63 of the sharing assistant server 11 creates a reservation list based on the reservation information and the schedule information received in step S42 (step S43).
Then, the transmission/reception unit 61 transmits, to the user terminal 40, reservation list information indicating content of the reservation list, and all of project IDs and all of project names (step S44). Then, the transmission/reception unit 51 of the user terminal 40 receives the reservation list information, and all of project IDs and all of project names.
Then, at the user terminal 40, the display control unit 54 instructs the display 506 to display a reservation list screen 230 illustrated in
Further, the reservation list screen 230 displays event information 235 to 237 indicating each event using a particular resource (e.g., conference room X in this example) of the date of today.
The event information includes, for example, scheduled use start time and end time of resource, event name, and name of person who has reserved each resource (reservation holder name) for each event. The event information includes, for example, start buttons 235s to 237s to be pressed when a user specifies or designates an event to be started.
Then, in
Then, the display control unit 54 displays a project list screen 240, illustrated in
As illustrated in
Then, in
Then, the transmission/reception unit 51 of the user terminal 40 transmits, to the sharing assistant server 11, the scheduled event ID indicating the scheduled event selected in step S51, and the project ID indicating the project selected in step S53 (step S54). The processing of step S54 is a transmission request processing of conducted event identification information. Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the selected scheduled event ID and the selected project ID.
Then, at the sharing assistant server 11, the generation unit 64 generates a unique conducted event ID (step S55).
Then, the storing/reading processing unit 69 stores or manages the conducted event ID generated in step S55, the scheduled event ID, user ID and organization ID of reservation holder, and the event information received in step S54 in association with each other (step S56).
The user ID and organization ID of the reservation holder and the event information are IDs and information based on the reservation information and the schedule information received in step S42. At this stage, information is not yet input in a column of participation or absence of the reservation management table (see
Then, the storage/readout processing unit 69 stores the project ID received in step S54 and the conducted event ID generated in step S55 in association with each other (step S57).
Then, at the sharing assistant server 11, the transmission/reception unit 61 transmits, to the schedule management server 12, file data transmission request information indicating a transmission request of file data registered in the schedule management server 12 (step S58). The file data transmission request information includes the scheduled event ID received in step S54, the user ID and organization ID of the user A received in step S34, and the access ID and access password read out in step S36. Then, the transmission/reception unit 81 of the schedule management server 12 receives the file data transmission request information.
Then, the storing/reading processing unit 89 of the schedule management server 12 searches the event management DB 8005 (see
Then, the transmission/reception unit 81 transmits the file data read out in step S59 to the sharing assistant server 11 (step S60). Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the file data.
Then, the storing/reading processing unit 69 of the sharing assistant server 11 stores and manages the file data received in step S60 in the schedule management DB 6003 (see
Then, the transmission/reception unit 61 transmits the conducted event ID generated in step S55 and the file data received in step S60 to the user terminal 40 used by the user A (step S62). Then, the transmission/reception unit 51 of the user terminal 40 used by the user A receives the conducted event ID and the file data.
Further, the transmission/reception unit 61 transmits a participation request notice to the user terminal 40 used by other participant. The participation request notice for this particular event includes, for example, various information used for participating the event started by the user A.
With this configuration, the other participant can use the participation request notice for the particular event to notify his/her participation to the particular event from the user terminal 40 used by the other participant to the sharing assistant server 11 to participate the particular event, such as the online meeting started by the user A.
Then, at the user terminal 40, the storing/reading processing unit 59 stores the conducted event ID and the file data in the storage unit 5000 (step S63). The file data transmitted from the sharing assistant server 11 is stored in a specific storage area of the storage unit 5000. With this configuration, the user terminal 40 accesses the specific storage area during the execution of the event, and the display control unit 54 instructs the display 506 to display the file data stored in the specific storage area.
The specific storage area is a temporary storage destination of data provided for each event that is being conducted, and is a storage area specified by an arbitrary path (text string) indicating a location in the storage unit 5000.
Further, the specific storage area is not limited to the configuration provided inside the user terminal 40. For example, the specific storage area may be provided in an external storage device connected to the user terminal 40 or in a local server which exists in an on-premise environment and can communicate with the user terminal 40.
Then, as illustrated in
The event detailed information screen 250 includes, for example, a display area 251 used for displaying event name, a display area 252 used for displaying scheduled event execution time (scheduled start time and scheduled end time) of event, and a display area 253 used for displaying name of reservation holder.
Further, the event detailed information screen 250 includes a display area 256 used for displaying content of memo, a display area 257 used for displaying name of scheduled participant, and a display area 258 used for displaying identification information (e.g., file name) identifying file data stored in the specific storage area of the storage unit 5000.
The display area 257 displays the name of reservation holder and the name of selected other participant indicated in
Further, the display area 258 displays a file name of file data stored in the specific storage area of the storage unit 5000, that is a file name of file data downloaded from the sharing assistant server 11, and also displays a file name of file data being downloaded from the sharing assistant server 11.
Further, the event detailed information screen 250 includes a “close” button 259 at the lower right portion of the event detailed information screen 250, which is to be pressed when the event detailed information screen 250 is closed.
Then, the other participant, who has received the participation request notice, sends or transmits a participation notification to the event from the user terminal 40 used by the other participant to the sharing assistant server 11, and then participates the event, such as an online meeting started by the user A (step S66). Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the user ID of the other participant who has participated the event, such as the online meeting started by the user A.
Then, the sharing assistant server 11 inputs a mark of participation or absence of each participant in the column of participation or absence in the schedule management DB 6003 (see
With this configuration, the user A and the other participant can start the event (e.g., online meeting) using the resource (e.g., conference room X) and the user terminal 40. Then, the display control unit 54 instructs the display 506 to display an event execution screen.
Hereinafter, with reference to
When an event such as an online meeting has started, the user A and other participant who are members of a project for collaboration work can hold the online meeting using a voice call function, a video call function, a chat function, or the like. If the user A or other participant use functions, such as audio-recording/video-recording function, chat function, or message function, the user terminal 40 performs various types of content processing corresponding to each function (step S71).
For example, in a case of the audio-recording/video-recording function, the user terminal 40 performs the content processing, in which the recorded audio data or recorded video data is generated as content. In a case of the chat function or message function, the user terminal 40 performs the content processing, in which the input text data is generated as content.
Then, the transmission/reception unit 51 transmits registration request information indicating a registration request of the generated content to the sharing assistant server 11 (step S72). In step S72, the transmission/reception unit 51 automatically transmits the registration request information every time the content is generated. The content may be transmitted in a file format or in a stream format. The registration request information includes the conducted event ID, user ID of content transmission source, content data, and content processing type information. Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the registration request information.
Then, the determination unit 65 of the sharing assistant server 11 determines the type of the received content processing based on the type information of the content processing included in the registration request information received by the transmission/reception unit 61 (step S73).
If the determination unit 65 determines that the type of content processing is “audio-recording function”, the transmission/reception unit 61 of the sharing assistant server 11 transmits recorded audio data as content data to the voice-to-text conversion server 13 (step S74).
Further, if the determination unit 65 determines that the type of content processing is the “video-recording function”, the transmission/reception unit 61 of the sharing assistant server 11 transmits the recorded video data as content data to the voice-to-text conversion server 13 (step S74).
Then, the transmission/reception unit 91 of the voice-to-text conversion server 13 receives the recorded audio data or recorded video data.
If the content processing type is other than “audio-recording function/video-recording function”, the sharing assistant server 11 does not perform the processing of steps S74 to S76, but proceeds the sequence to step S77.
Then, the conversion unit 93 of the voice-to-text conversion server 13 converts the recorded audio data or recorded video data received by the transmission/reception unit 91 into text data set with time stamp information indicating an elapsed time from the start of recorded audio data or recorded video data (step S75). The processing of step S75 may be performable as a real-time process or a batch process.
Hereinafter, with reference to
At first, the conversion unit 93 acquires or obtains information indicating the date and time when the recorded audio data or recorded video data was received by the transmission/reception unit 91 (step S75-1). The information acquired in step S75-1 may be information indicating date and time when the sharing assistant server 11 has received the recorded audio data or recorded video data, or date and time when the sharing assistant server 11 has transmitted the recorded audio data or recorded video data. In this case, in step S74, the transmission/reception unit 91 of the voice-to-text conversion server 13 receives the recorded audio data or recorded video data, and the information indicating the date and time transmitted from the sharing assistant server 11.
Then, the conversion unit 93 performs the text conversion processing on the recorded audio data or recorded video data received by the transmission/reception unit 91 (step S75-2).
Then, if the conversion unit 93 has completed the text conversion processing on the recorded audio data or recorded video data (step S75-3: YES), the conversion unit 93 proceeds the sequence to step S75-4. Further, the conversion unit 93 repeats the processing of step S75-2 until the conversion unit 93 has completed the text conversion processing on the recorded audio data or recorded video data.
If the conversion unit 93 determines that the text conversion processing has completed (step S75-3: YES), the conversion unit 93 generates voice-text data set with time stamp information, converted from the recorded audio data or recorded video data (step S75-4), with which, the voice-to-text conversion server 13 can convert the recorded audio data or recorded video data transmitted from the sharing assistant server 11 into the voice-to-text data.
The information processing system 1 repeats the above described processing of steps S71 to S76 to repeat the text conversion processing on the recorded audio data or recorded video data generated at the user terminal 40.
If the conversion unit 93 has completed the text conversion on the recorded audio data or recorded video data (step S75), the transmission/reception unit 91 of the voice-to-text conversion server 13 transmits the voice-text data set with the time stamp information to the sharing assistant server 11 (step S76). Then, the transmission/reception unit 61 of the sharing assistant server 11 receives the voice-text data set with the time stamp information transmitted from the voice-to-text conversion server 13.
Then, the generation unit 64 generates a unique content processing ID for identifying the content processing generated by the event (step S77).
The generation unit 64 generates a URL of the content data indicating material of the content (step S78).
The storage/read processing unit 69 manages the content management DB 6005 (see
Then, the detection unit 66 detects, for example, the statement text data of action item candidate set with replay position information from the voice-text data set with the time stamp information received in step S76 (step S80).
At first, the detection unit 66 divides the voice-text data received in step S76 into each one sentence of statement text data (step S80-1). Further, the replay position information is added to the statement text data based on the time stamp information of the first character or word included in one sentence of statement text data.
Then, the detection unit 66 reads out one sentence of statement text data from the statement text data divided into each one sentence and added with the replay position information (step S80-2).
Then, the detection unit 66 performs a morphological analysis on the read-out one sentence of statement text data, and decomposes the one sentence of statement text data in consideration of extracting three elements, such as “who”, “what”, and “until when” (step S80-3). Although this example case may assume the morphological analysis on Japanese, the morphological analysis can be applied to any language, such as Chinese and Arabic, in which one sentence continues for a given length, and the one sentence is required to be divided into each sentence.
For example, as to the element [who], a noun representing [person] can be extracted.
Further, as to the element [until when] that can be configured with A-part and B-part, A-part is extracted using a phrase representing a date or a position on a calendar, such as [date], [day of week], [number+x days later], and [word indicating date (e.g., tomorrow, the day after tomorrow)], and B-part is extracted using a phrase representing a limit, such as [by (until)] and [by due date].
Further, as to the element [what] that can be configured with A-part and B-part, A-part is extracted using a phrase representing [noun]+(particle and auxiliary), and B-part is extracted using a phrase representing a request, such as [verb]+[do it], [please], and [finish it off]. In this case, the order of A-part and B-part may be reversed and spoken.
Then, based on an analysis result of the morphological analysis, the detection unit 66 determines whether the elements having a given pre-set combination pattern, which is likely to become an action item, exists in the one sentence of statement text data (step S80-4).
For example, an action item is typically composed of three elements of “who”, “what”, and “until when.” However, in some cases, some of these three elements may not be included in one single sentence of statement text data, which may become an action item. Therefore, for example, if the elements having any one of the following combination patterns exist in the one single sentence of statement text data, the detection unit 66 detects the statement text data as the action item candidate.
Two elements of [who]+[what], such as “Mr. Tanaka, please call Company Z.”
Two elements of [until when]+[what], such as “Call company Z by Friday.
Three Elements of [who]+[until when]+[what], such as “Mr. Tanaka, please call company Z by Friday.”
In the above examples of the combination patterns of elements, the element [what] is set as an essential piece to configure the action item.
If the one sentence of statement text data includes the elements having any one of the above combination patterns (step S80-4: YES), the detection unit 66 detects the one sentence of statement text data read out in step S80-2 as an action item candidate (step S80-5).
If the one sentence of statement text data includes no elements having any one of the above combination patterns (step S80-4: NO), the detection unit 66 skips the processing in step S80-5 and does not detect the one sentence of statement text data read out in step S80-2 as the action item candidate.
The detection unit 66 repeats steps S80-2 to 580-6 until all of the statement text data divided into each one sentence at step S80-1 have been read out.
By performing the processing of detecting the action item candidate illustrated in
Further, the “elements having given pre-set combination patterns that are likely to become an action item” used for detecting the action item candidate used in steps S80-4 to S80-5 (see
For example, in some cases, an action item candidate, which is detected using the above described “elements having given pre-set combination patterns that are likely to be an action item” by applying a basic rule, may not be registered as an action item by a project member or the like.
Further, in some cases, one sentence of statement text data, in which the above described “elements having given pre-set combination patterns that are likely to be an action item” cannot be detected by applying a basic rule, may be registered as an action item by a project member.
In such cases, the precision of detecting an action item candidate from the statement text data can be improved by applying the machine learning to the action item candidate not registered as the action item by the project member, and to one sentence of statement text data registered as the action item by the project member.
The machine learning is a technology for making a computer acquire learning ability like human, in which the computer autonomously generates an algorithm necessary for determining data identification or the like from learning data acquired in advance, and performs a prediction on new data by applying the generated algorithm.
The learning methods used for the machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, and may be a combination of these learning methods.
The description returns to
Then, the user terminal 40 updates the conducted event record screen (step S82).
Further, if text data input for chat or message is generated as the content in step S71, the transmission/reception unit 61 transmits to the user terminal 40, the statement text data detected as the action item candidate in addition to the input text data.
With this configuration, the user terminal 40 used by the project member can receive, from the sharing assistant server 11, the content input by the project member during an event (e.g., online meeting) by speaking opinions and inputting text, and the statement text data to be settable as action item candidate detected from the content input by the project member.
Then, the display control unit 54 of the user terminal 40 used by the project member instructs the display 506 to display, for example, a conducted event record screen 260 illustrated in
The “display memo using full-screen display” button 262 is used for displaying a content (meeting memo) spoken or input by a project member in voice or text during an event (e.g., online meeting) using a communication function using a full-screen display.
The statement text data 264 is detected as an action item candidate from the content of meeting memo.
The meeting memo is, for example, a speech record obtained by converting voice data into text data using the contents spoken by each project member by applying known voice recognition technologies. Further, the meeting memo may also include text input by each project member as a record of content.
As illustrated in
If the statement text data 264 of action item candidate is text data converted from the recorded video data or recorded audio data, a replay link 265 is displayed, for example, next to the statement text data 264.
The replay link 265 is a link used for starting a replay of the statement text data 264 of action item candidate from a particular position (time offset) of the recorded video data or recorded audio data detected by performing the voice recognition, based on the replay position information of the statement text data 264. As described above, the replay link 265 is incorporated with seek information for starting a replay of the statement text data 264 of action item candidate from the particular position (time offset) of the recorded video data or recorded audio data detected by performing the voice recognition. The seek information indicates information indicating a replay position (replay portion) of the recorded video data or recorded audio data in the entire replay time.
If the replay link 265 is to start a replay of the statement text data 264 of action item candidate from the particular position (time offset) of the recorded video data or recorded audio data detected by performing the voice recognition, a user may not understand the context when the statement text data 264 occurred during an event. In view of this situation, the replay may be started, for example, about 30-seconds before position, which is a given position set before a position indicated by the replay position information. Further, the replay link 265 may also include a time of speaking the action item candidate (e.g., 9:57). Further, as illustrated in
If the replay link 265 is clicked, the replay unit 57 of the user terminal 40 replays the recorded video data or recorded audio data associated with the replay link 265 from the position based on the replay position information. For example, the replay unit 57 may newly open a web page storing the recorded video data or recorded audio data associated with the replay link 265. Then, a user can replay the recorded video data or recorded audio data, from which the statement text data 264 is detected by performing the voice recognition, from the position based on the replay position information, on the newly opened web page.
Further, as illustrated in
Further, a volume of replay data can be increased or decreased, and the replay data can be played using fast-forward/rewind in the video control 276.
The recorded video data or recorded audio data associated with the replay link 265 is acquired from the storage destination indicated in the item of “content data” of the content management table (see
Further, the speaking time displayed with the replay link 265 (see
Further, the speaking time can be displayed in line with a display timing of the video control 276 after receiving a selection of the replay link 265 performed by a user. By displaying the current time at the same timing (step) of displaying the video control 276, even after the user moves the seek bar in the video control 276 to left and right, the user can easily recognize the time when the action item candidate linked to the replay link 265 was spoken, and the user can easily and quickly return to the replay position information using the seek bar.
Further, for example, as illustrated in
Since the statement text data 264 of action item candidate of “Mr. Yamashita modifies and submits document 1 by tomorrow” and “Call Company A” are the text data converted from the recorded video data or recorded audio data, the conducted event record screen 260 displays the replay link 265, for example, next to the statement text data 264 as illustrated in
On the other hand, since the statement text data 264 of action item candidate of “Mr. Riko creates document 2 by end of this month” is the text data not converted from the recorded video data or recorded audio data, the conducted event record screen 260 does not display the replay link 265, for example, next to the statement text data 264 as illustrated in
For example, if the statement text data 264 is created based on the content entered by a participant as a memo during the meeting, the replay link 265 is not displayed.
Further, if the statement text data 264 is created based on text data included in the reference material used in the meeting, a link to a storage destination where the reference material is stored may be displayed.
For example, each project member (hereinafter, registered member), who wants to register the statement text data 264 of action item candidate displayed on the conducted event record screen 260 (
Further, the statement text data 264 of action item candidate displayed on the conducted event record screen 260 is not registered, for example, in the action item management function of the action item management server 14 unless the statement text data 264 of action item candidate is registered as the action item. Since the statement text data 264 is displayed as the speech record, the statement text data 264 may be mixed in other speech, in which an operation of registering the statement text data 264 as the action item may be forgotten. Therefore, the statement text data 264 of action item candidate not registered as the action item may be configured to be posted again as a new speech. The posting of the statement text data 264 again can be performed, for example, by a bot that is an application software that executes an automated task.
As similar to
At first, for example, the registered member places a pointer of the pointing device 512 at a position of the statement text data 264 of action item candidate (e.g., mouse-over) to instruct the conducted event record screen 260 to display action button groups 266 and 268 as illustrated in
In an example case of
If the registered member selects a “set execution person” button 270 of the action button group 268 (see
It should be noted that each project member (a member who has participated the online meeting and a member who has not attended the online meeting) who has been summoned to the online meeting is preferentially displayed as the candidate of person who is to execute the action item. The execution person candidate of action item is not limited to the project member. For example, the execution person candidate of action item can be narrowed down or selected, and displayed in accordance with browsing authority levels (e.g., team, channel) set for the collaboration system in addition to the project member.
For example, a person who makes a reservation such as a meeting organizer (e.g., group organizer) and a member having a higher authority among project members (e.g., user having a higher authority participating a group) can be narrowed down or selected by referring to the above described various tables as the execution person candidate of action item. The execution person of action item is not limited to one person, but can be assigned to a plurality of persons.
The registered member can set the execution person of action item from the execution person input field 272 (see
By performing the above described processing, the statement text data of action item candidate set with the execution person and input with the due date is registered as the action item of the set execution person to the action item management function of the action item management server 14 of the same collaboration system 10, from the user terminal 40 used by the registered member.
Further, the registration request of action item to the action item management server 14 can be performed, for example, by using an application programming interface (API) released to the public by the action item management server 14.
The above description has been given as an example, in which the registered member sets the execution person and inputs the due date, but is not limited thereto. For example, if all of three elements of “who”, “what”, and “until when” exist (e.g., action item candidate of “Mr. Yamashita modifies and submits document 1 by tomorrow” illustrated in
For example, the request unit 55 of the user terminal 40 may determine “who” based on the spoken content related to “who” included in the statement text data 264 of action item candidate.
Further, if the statement text data 264 of action item candidate includes only one “person name” and the only one user can be identified in the information processing system 1, the request unit 55 may automatically determine “who”. The following cases consider the determination priority order. In the following cases, if the execution person of action item cannot be set automatically, the registered member may set he execution person of action item manually.
Priority order 1: “who” corresponds to “Tanaka” and there are users having “Tanaka” as a part of first and last names are participating an online meeting.
Priority order 2: “who” corresponds to “Tanaka” and there are users having “Tanaka” as a part of first and last names are absent from an online meeting.
Priority order 3: “who” corresponds to “Tanaka” and there are users having “Tanaka” as a part of first and last names exist in the information processing system 1.
Further, for example, the request unit 55 of the user terminal 40 may determine “until when” based on the spoken content related to “until when” included in the statement text data 264 of action item candidate.
Further, the request unit 55 of the user terminal 40 may determine “until” from a schedule of the next meeting determined from the meeting attribute. For example, a periodic interval of meetings can be acquired from the schedule management server 12, and then date of “next time” can be specified on a calendar. Similarly, the request unit 55 of the user terminal 40 can specify the date of “tomorrow” or “the day after tomorrow” on the calendar. Further, a phrase indicating “past” included in phrases representing time or period may not be used as a target of specifying “next time”
Further, the registered member who makes the registration request of action item may be limited to, for example, a speaker of the statement text data 264 of action item candidate, a reservation holder such as a meeting organizer, and a member having a higher authority among project members.
The registration unit 104 of the action item management server 14 that has received the action item registration request stores or manages the action item in the action item management table illustrated in
Further, when the action unit 105 of the action item management server 14 receives an inquiry request from the user terminal 40, the action unit 105 responds to the user terminal 40, the content of the action item corresponding to the inquiry request, and the state of the action item.
With this configuration, the display control unit 54 of the user terminal 40 instructs the display 506 to display, for example, a task management screen 271 illustrated in
As to the above described information processing system 1, the recorded video data or recorded audio data of a scene, in which the statement text data 264 of the action item candidate or the action item 273 was generated, can be easily replayed. Therefore, as to the above described information processing system 1, the action item 273 can be registered or executed after confirming the scene, in which the statement text data 264 of the action item candidate or the action item 273 was generated.
Hereinafter, with reference to
If the user presses the “view conducted event record” button 543 on the initial screen 540 (see
Then, the transmission/reception unit 51 transmits conducted event record viewing request information indicating a request for viewing the conducted event record to the schedule management server 12 (step S119). Then, the transmission/reception unit 81 of the schedule management server 12 receives the conducted event record viewing request information.
Then, the storing/reading processing unit 89 of the schedule management server 12 searches the project member management DB 8007 using the user ID and organization ID received in step S113 as search keys to read out the corresponding project ID and project name (step S120).
Then, the transmission/reception unit 81 transmits the project ID and the project name to the user terminal 40 (step S121).
Then, the generation unit 56 of the user terminal 40 generates a project list screen using the project ID and project name received in step S121 (step S122).
Then, at the user terminal 40, the display control unit 54 instructs the display 506 to display the project list screen generated by the generation unit 56 (step S123). The project list screen displays the same contents as the project list screen 240 (see
Then, in
Then, the transmission/reception unit 51 of the user terminal 40 transmits the project ID of the project selected in step S124 to the schedule management server 12 (step S125). Then, the schedule management server 12 receives the project ID.
Then, at the schedule management server 12, the storing/reading processing unit 89 searches the conducted event record management DB 8008 using the project ID received in step S125 as a search key to read out the corresponding conducted event ID (step S126).
In step S126, the storing/reading processing unit 89 reads out all of the conducted event IDs associated with the project ID received in step S125.
Then, the storing/reading processing unit 89 searches the conducted event management DB 8009 using all of the conducted event IDs read out in step S126 as search keys to read out the corresponding conducted event start-end information (step S127). The conducted event start-end information includes, for example, conducted event ID, event name, start date/time, and end date/time.
Then, the transmission/reception unit 81 transmits the conducted event ID, event name, start date/time, and end date/time read out in step S127 to the user terminal 40 (step S128). Then, the transmission/reception unit 51 of the user terminal 40 receives the conducted event ID, event name, start date/time, and end date/time.
Then, the generation unit 56 of the user terminal 40 generates a conducted event list screen 570 as illustrated in
Then, at the user terminal 40, the display control unit 54 instructs the display 506 to display the conducted event list screen 570 generated by the generation unit 56 (step S130). As illustrated in
For example, the information 571 to 573 indicating the conducted event includes, for example, conducted event name, start date/time, and end date/time of event. The information 571 to 573 indicating the conducted event is an example of history information selection area for receiving a selection for displaying history information indicating the content of the conducted event. Further, the conducted event list screen 570 includes, for example, a “close” button 575 at the lower right portion to be pressed when to close the conducted event list screen 570.
Further, the conducted event list screen 570 includes, for example, an “action item” button 577 at the lower left portion to be pressed when the action item is to be viewed. The “action item” button 577 is an example of action item selection area for receiving a selection for displaying an action item.
Then, the reception unit 52 of the user terminal 40 receives a selection of particular conducted event included in the conducted event list screen 570 (step S131). Specifically, if a user selects particular information indicating the particular conducted event included in the conducted event list screen 570, the reception unit 52 receives a selection of particular information indicating the particular conducted event.
Then, the transmission/reception unit 51 of the user terminal 40 transmits a particular conducted event ID of the particular conducted event selected by the reception unit 52 to the schedule management server 12 (step S132). Then, the transmission/reception unit 81 of the schedule management server 12 receives the particular conducted event ID.
Then, the storing/reading processing unit 89 of the schedule management server 12 searches the conducted event record management DB 8008 using the particular conducted event ID received in step S132 as a search key to read out the corresponding particular conducted event record information (step S133). The conducted event record information includes, for example, content processing ID, type of content processing, start date/time of content processing, and end date/time of content processing.
Further, the storing/reading processing unit 89 of the schedule management server 12 searches the related information management DB 8010 using the particular conducted event ID received in step S132 as a search key to read out the corresponding particular related information (step S134). The related information includes, for example, content generation time, content processing ID, type of content processing, and sequence No. The content generation time included in the related information is an example of time information.
Then, the storing/reading processing unit 89 reads out the content data indicating the content of the conducted event record from a storage destination of content data indicating the content of the particular conducted event record read out in step S133 (step S135).
Then, the transmission/reception unit 81 transmits the read-out particular related information and content data to the user terminal 40 (step S136). Then, the transmission/reception unit 51 of the user terminal 40 receives the particular related information and content data.
Then, the request unit 55 of the user terminal 40 requests or inquires about the state or status of the action item to the action item management server 14 of the collaboration system 10 being in the sign-in state (step S137).
If the action unit 105 of the action item management server 14 receives the request or inquiry of the state or status of the action item from the user terminal 40, the action unit 105 reads out the state or status of the action item corresponding to the request or inquiry, from the action item management table illustrated in
Then, the request unit 55 of the user terminal 40 receives the state information of the action item from the action item management server 14 (step S140).
Then, the generation unit 56 of the user terminal 40 generates the statement text data 264 to be set on the conducted event record screen 260 using the related information and the content data received in step S136, and generates the icon 280 or the icon 282 to be set on the conducted event record screen 260 using the state information of the action item received in step S140.
Then, the display control unit 54 of the user terminal 40 can display the conducted event record screen 260 (see
As illustrated in
As to the statement text data 264 of action item candidate displayed on the conducted event record screen 260, a particular member who operates the user terminal 40 can switch between one action item of a project assigned to other project member and another action item of the project assigned to the particular member by displaying the one action item and another action item.
As to the information processing system 1 according to the embodiment, when an action item candidate is detected from contents spoken by members of a project using the communication function provided by the collaboration tools, and is then registered using the action item management function, a situation when the action item candidate is spoken can be confirmed by using the assistance or support of the information processing system 1.
The information processing system 1 illustrated in
Therefore, various requests, such as a registration request of action item and an inquiry request of action item state to the action item management server 14, which are performed from the request unit 55 in
Further, image data of screen such as a conducted event record screen, to be displayed under the control of the display control unit 54 of the user terminal 40, is created by the creation unit 63 of the sharing assistant server 11 (see
Further, as to the information processing system 1 illustrated in
The above described embodiment is also implemented as an information processing apparatus, capable of communicating with a plurality of user terminals, for assisting a collaboration work performable by a plurality of users operating the respective user terminals. The information processing apparatus includes circuitry configured to create image data of a screen for displaying, on a display, an action item candidate based on voice-text data converted from recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data corresponding to the action item candidate; and transmit the image data of the screen to one or more of the user terminals.
The above described embodiment is also implemented as a non-transitory computer readable storage medium storing one or more instructions that, when executed by one or more processors, instructs the one or more processors to perform a method of assisting a collaboration work performable by a plurality of users operating the respective user terminals using an information processing apparatus capable of communicating with a plurality of user terminals. The method includes creating image data of a screen for displaying, on a display, an action item candidate based on voice-text data converted from recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data corresponding to the action item candidate; and transmitting the image data of the screen to one or more of the user terminals.
The above described embodiment is also implemented as a non-transitory computer readable storage medium storing one or more instructions that, when executed by one or more processors, instructs the one or more processors to perform a method of processing information performable by a user terminal capable of communicating with a collaboration work assistance system for assisting a collaboration work performable by a plurality of users. The method includes receiving, from the collaboration work assistance system, an action item candidate based on voice-text data converted from recorded video data or recorded audio data, and replay information of the recorded video data or recorded audio data corresponding to the action item candidate; and displaying the received action item candidate and the replay information on a display.
As to the above described embodiment of this disclosure, an information processing system capable of displaying action item candidates and recorded video data or recorded audio data of action item candidates, and replay information of recorded video data or recorded audio data can be provided.
Numerous additional modifications and variations are possible in light of the above teachings. It is therefore to be understood that, within the scope of the appended claims, the disclosure of this specification can be practiced otherwise than as specifically described herein. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above. For example, a configuration that combines the first embodiment and the modification can be applied. The information processing system 1 described in the above described embodiment are just one example, and there are various system configurations depending on applications and purposes.
Each of the functions of the above-described embodiments can be implemented by one or more processing circuits or circuitry. Processing circuitry includes a programmed processor, as a processor includes circuitry. A processing circuit also includes devices such as an application specific integrated circuit (ASIC), digital signal processor (DSP), field programmable gate array (FPGA), and conventional circuit components arranged to perform the recited functions.
The group of apparatuses or devices described in the above embodiment only represents one of a plurality of computing environments for implementing the embodiment disclosed herein.
In one embodiment, the sharing assistant server 11 and the schedule management server 12 may include a plurality of computing devices, such as server clusters. The plurality of computing devices are configured to communicate with each other over any type of communication link, including the communication network 50, shared memory, or the like, to perform the processing disclosed herein.
Further, the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, the action item management server 14, and the user terminal 40 can be configured to share the processing steps disclosed herein in various combinations. For example, a process performable by a given unit may be performable by other unit. Similarly, the function performable by a given unit may be performable by other unit. Further, each of the functional units of the sharing assistant server 11, the schedule management server 12, the voice-to-text conversion server 13, and the action item management server 14 may be integrated into one apparatus or may be divided into a plurality of apparatuses.
Number | Date | Country | Kind |
---|---|---|---|
2020-051268 | Mar 2020 | JP | national |
2021-003427 | Jan 2021 | JP | national |