Patent Application
20230386162
The present disclosure generally relates to electronic devices with video capturing capabilities and in particular to electronic devices with video capturing capabilities that are utilized within a video communication session.
Modern smartphones are equipped with some of the best digital cameras in the consumer market. Smartphones are often utilized to enable video communication sessions with a second device, where the front camera is used to focus on the user while the display presents the video received from the second device. However, given the limitations in screen size, persons engaged in video communication sessions, such as a video conference, often prefer to use their laptops or desktop devices that present a much larger display for viewing the received video within the communication application interface that can also present one of more communication application settings icons/menu items.
Unlike the high-quality inbuilt cameras of the smartphones, the built-in webcams of laptops and external universal serial bus (usb)-connected webcams are very basic and provide limited video options for the user who utilizes their personal computer (PC) or laptop to engage in a video communication session. Because of the low quality of the video images captured by these webcams, recent solutions have been provided for smartphone users to utilize the better-quality cameras available with their smartphones to capture the video that is transmitted as the video feed in video communication sessions with other devices. The smartphones are wired or wirelessly connected to the PC, which provides a larger display device for video review and viewing.
Many smartphones include video tracking capabilities and support use of air gestures as inputs to make a selection or control a functionality of a phone. However, when the smartphone camera is being utilized as the camera during a video communication session, the user's actions and/or gestures are clearly seen by all the remote participants on the video session. This can be very distracting, particularly if the user is the presenter at the time the actions are being performed.
The description of the illustrative embodiments can be read in conjunction with the accompanying figures. It will be appreciated that for simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements are exaggerated relative to other elements. Embodiments incorporating teachings of the present disclosure are shown and described with respect to the figures presented herein, in which:
According to a first aspect of the disclosure, an electronic device, a method, and a computer program product provides segmenting of a video image into a primary and secondary segments and presentation of a touchless action center in a display area of one or more of the secondary segments to support touchless user interfacing and control during a preview of the video image.
In a first embodiment of the first aspect, an electronic device includes at least one image capturing device that captures video of a local scene and an interface by which the electronic device connects to a display device. The electronic device also includes a memory having stored thereon a video image segmentation and control module (VISCM) that provides program instructions for spatially delineating video images into two or more segments that can be individually presented within or excluded from a video preview and/or video feed. The electronic device also includes at least one processor communicatively coupled to the display device, to each of the at least one image capturing device, and to the memory. The at least one processor executes the program instructions of the VISCM, which causes the at least one processor to identify a primary region of interest (ROI) within a video image captured by the at least one image capturing device, the primary region of interest being a smaller area than the video image. The at least one processor delineates the video image into two or more segments, including a primary segment that encompasses the primary ROI and at least a secondary segment. The at least one processor then associates a touchless action center with a virtual interface with a location of the secondary segment in the video image. The virtual interface includes at least one feature that can be selected via one of air gestures and screen touches during presentation of the primary segment of the video image. The at least one processor then presents a preview of the delineated segments of the video image on the display device with at least the primary segment and the secondary segment of the video image and the virtual interface being visible within the display device.
According to one or more embodiments, the processor is further configured to detect, within a captured video image, one or more air gestures proximate to a location of a specific feature among the at least one feature within the virtual interface, the one or more air gestures representing a selection of the specific feature. The at least one feature comprises at least one of image capturing device configuration and settings, video image presentation settings, local device settings, and application settings for a video conference application enabling a video communication session with at least the second device. The processor is further configured to, in response to identifying the selection of the specific feature, enable a corresponding function associated with the specific feature concurrently with a capturing and transmission of at least the primary segment of the video image.
According to one or more embodiments, the processor is further configured to identify selection of the specific feature, which activates a function that modifies one or more characteristics of at least the content within the primary segment of the video image, apply the function to an original content within at least the primary segment to generate a resulting modified content, and locally present and remotely transmit the resulting modified content within the primary segment in place of original content.
According to one or more embodiments, the processor is further configured to transmit only content within the primary segment to a communicatively-connected second device to which the video image is to be shared. The secondary segment(s) and the associated virtual interfaces of the touchless action center are only presented during the video preview on the local display device.
Additional embodiments provide a method and a computer program product that provide similar functions as the described embodiments of the electronic device.
According to a second aspect of the disclosure, an electronic device, a method, and a computer program product provides granular transmission of select segments of a segmented video image along with a touchless action center to at least one second device among a plurality of second devices that are communicatively connected by a video communication session to enable the at least one second device to control one or more features or functions associated with the video feed being transmitted by the electronic device.
In a first embodiment of the second aspect, the electronic device also includes a communication interface that enables the electronic device to communicatively connect and exchange video data with a plurality of second devices during a video communication session. The at least one processor is communicatively coupled to the display device, to each of the at least one image capturing device, to the communication interface, and to the memory. The processor processes instructions of the VISCM, which causes the at least one processor to identify a primary region of interest (ROI) within a video image captured by the at least one image capturing device, the primary region of interest being a smaller area than the video image. The processor delineates the video image into two or more segments, including a primary segment that encompasses the primary ROI and at least one secondary segment. The processor links/presents/associates an action center with a virtual interface within a location of one or more of the at least one secondary segment in the video image. The virtual interface includes at least one feature that can be selected via one of air gestures and screen touches during presentation of the primary segment of the video image. The processor assigns control features of the virtual interface with at least one selected second device from among the plurality of second devices. The processor transmits, to each of the plurality of second devices, at least the primary segment for displaying at a second display of a corresponding second device during the video communication session. The processor concurrently transmits, to each of the at least one selected second device, the control features of the virtual interface assigned with the at least one selected second device to trigger the at least one selected second device to present the virtual interface on a second display device, concurrently with presenting the primary segment as parts of a combined video stream from the electronic device. Accordingly, each of the plurality of second devices can be transmitted a different combination of segmented video images comprising at least the primary segment with zero or more virtual interfaces remotely presented on a respective second display for selection at a respective at least one selected second device.
In one or more embodiments, to associate the virtual interface with the at least one selected second device from among the plurality of second devices, the at least one processor presents a conference participant interface that receives a selection ranging from between none (zero) to one or more virtual interfaces to provide to a particular second device identified via an associated participant name. And, in response to receiving the selection, the at least one processor associates a selected one or more virtual interfaces with the particular second device. The at least one processor presents, on the local display device, a local preview window including a visual indication of which of the one or more virtual interfaces are associated for transmission to each of the plurality of second devices designated to receive at least one virtual interface within a transmitted video stream.
According to one or more embodiments, the at least one processor detects selection of the particular second device as an administrative device and auto-selects specific virtual interfaces having functions controllable by the administrative device. The at least one processor then transmits the auto-selected virtual interfaces concurrently with the primary segment to enable virtual interfaces to be visually presented within the second display of the particular second device concurrently with the primary segment. The at least one processor further monitors for receipt of an indication of a remote selection or activation of at least one control function from within the virtual interfaces at the particular second device. And, in response to receiving an indication of a remote selection that is associated with a modification of a characteristic of the video images within the primary segment, the at least one processor applies the modification to video image content within the primary segment at the electronic device and transmits the modification of the video image content to each of the plurality of second devices participating in the video communication session.
According to a third aspect of the disclosure, an electronic device, a method, and a computer program product provides an electronic device that receives the primary segment of a segmented video image along with a touchless action center from a video communication session and controls features of the primary segment at an image stream transmitting (IST) device via the virtual interface of a touchless action center that is also transmitted by the IST device to the electronic device. The device includes a communication interface that enables the electronic device to communicatively connect with and receive video data from the IST device during a video communication session. The at least one processor receives the segmented video stream with the primary segment presenting a region of interest captured at the IST device and the at least one secondary segment, each presenting a respective virtual control interface. The processor presents, on the local display device, a control enabling view of the segmented video stream, including both the primary segment and the at least one secondary segment with the respective virtual control interface. Each virtual control interface further presents at least one selectable feature that can be selected by an input from among a screen touch and an air gesture during local presentation of the received segmented video stream. The processor monitors for movement captured by the at least one image capturing device that is indicative of the input occurring within a spatial location proximate to the selectable feature within a corresponding secondary segment. And, in response to detecting the input, identify a function associated with the selectable feature and activate the function to affect one or more characteristics of a video image presented within the primary segment.
In a first embodiment of the second aspect, the selectable feature modifies at least one characteristic of content visibly presented within the primary segment, and the processor modifies the characteristic of the content in response to the input. In one or more embodiments, the processor modifies the characteristic of the content at the IST device by transmitting a corresponding content settings update indication to the IST device via the video communication session. In one or more embodiments, the processor modifies the characteristic of the content for at least one of: the electronic device only; and all connected devices participating in the video communication session that receive the primary segment from the IST device.
According to a fourth aspect of the disclosure, an electronic device, a method, and a computer program product provides transmission of select segments of a segmented video image along with a touchless action center to a second device that is communicatively connected by a video communication session to enable the second device to control one or more features or functions associated with the video feed being transmitted by the electronic device.
In a first embodiment of the fourth aspect, the electronic device includes a memory having stored thereon a video image segmentation and control module (VISCM) and a demonstration object preview module (DOPM). The VISCM includes first program instructions for spatially delineating video images into two or more segments that can be individually presented or hidden within a video feed, and the DOPM includes second program instructions for selectively reversing image mirroring to previews of individual segments of a video feed. The processor processes the first and second program instructions of the VISCM and the DOPM, which configures the at least one processor to identify a demonstration object within a video image captured by the at least one image capturing device, the demonstration object being in a defined area of the video image. The processor spatially segments the video image into two or more segments, including a primary segment that encompasses the defined area with the demonstration object and at least one secondary segment encompassing a remaining portion of the video image. The processor then presents a preview of the delineated segments of the video image on the local display device, with the remaining portion of the video image mirrored within the preview and at least the primary segment presented without mirroring in a correct spatial location relative to the remaining portion of the video image. The person who is presenting the demonstration object receives a non-mirrored, preview presentation of the demonstration object.
In a first embodiment of the fourth aspect, the processor spatially segments and reorients the primary segments relative to the mirrored view of the secondary segment in response to activation of a demonstration object preview (DOP) option within an application supporting presentation of a camera image preview. In one or more embodiment, the processor divides the remaining portion into at least a second and a third segments, with the secondary segment encompassing a primary region of interest and the third segment encompassing sections of the video image than can be selectively cropped from the remaining portion of the video image. The processor presents a virtual interface on the display device, visibly adjacent to or integrated as an overlay on top of the third segment. The processor presents, within the virtual interface, the demonstration object preview (DOP) option, which is selectable via the virtual interface by one or more of an air gesture and a touch of a spatial area of the virtual interface on the display device.
According to one or more embodiments, the processor reorients at least one of an exterior boundary of the primary segment and a view angle of primary segment to present substantially seamlessly fit of the primary segment into a space from which the primary segment is remove within the remaining portion of the video image during a segmentation and preview generation process.
Each of the described features and functions of the primary, second, third, and fourth aspects, which are presented as operations performed by the processor(s) of electronic devices are also described as features and functions provided by a plurality of corresponding methods and computer program products, within the various different embodiments presented herein. In the embodiments presented as computer program products, the computer program product includes a computer readable storage device having stored thereon program instructions or code that, when processed by at least one processor of an electronic device, such as described above, enables the electronic device to complete the functionality of a respective one of the above-described processes.
The above contains simplifications, generalizations and omissions of detail and is not intended as a comprehensive description of the claimed subject matter but, rather, is intended to provide a brief overview of some of the functionality associated therewith. Other systems, methods, functionality, features, and advantages of the claimed subject matter will be or will become apparent to one with skill in the art upon examination of the figures and the remaining detailed written description. The above as well as additional objectives, features, and advantages of the present disclosure will become apparent in the following detailed description.
One motivation for implementation of the segmenting of a video feed/image and presentation of the virtual action center within a local preview is the recognition of the unmet need with conventional video conferencing applications that utilize the camera of a smartphone that is tethered to a larger display for image capturing. With these video conferencing applications, the smartphone provides video of the entire local scene that is captured by the phones' camera sensors within the transmitted video feed to the remote participants of the video communication session. While tracking of the user can be performed by the sophisticated devices, if the user is not within arms-length of the phone, the user is unable to control the camera or application settings unless the user comes closer to the smartphone and uses his/her hands to taps the settings icon on the smartphones local display. Oftentimes, the cameras being utilized in these scenarios are the rear cameras, which are typically better cameras for video capture, and the devices display is then not located on the side that is readily accessible to the user. While in active video communication, every action taken by the user to adjust or move the camera is captured within the field of view of the rear camera and is transmitted to all of the participants receiving the video feed of the local user.
In the following description, specific example embodiments in which the disclosure may be practiced are described in sufficient detail to enable those skilled in the art to practice the disclosed embodiments. For example, specific details such as specific method orders, structures, elements, and connections have been presented herein. However, it is to be understood that the specific details presented need not be utilized to practice embodiments of the present disclosure. It is also to be understood that other embodiments may be utilized and that logical, architectural, programmatic, mechanical, electrical and other changes may be made without departing from the general scope of the disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and equivalents thereof.
References within the specification to “one embodiment,” “an embodiment,” “embodiments”, or “one or more embodiments” are intended to indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The appearance of such phrases in various places within the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Further, various features are described which may be exhibited by some embodiments and not by others. Similarly, various aspects are described which may be aspects for some embodiments but not other embodiments.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another.
It is understood that the use of specific component, device and/or parameter names and/or corresponding acronyms thereof, such as those of the executing utility, logic, and/or firmware described herein, are for example only and not meant to imply any limitations on the described embodiments. The embodiments may thus be described with different nomenclature and/or terminology utilized to describe the components, devices, parameters, methods and/or functions herein, without limitation. References to any specific protocol or proprietary name in describing one or more elements, features or concepts of the embodiments are provided solely as examples of one implementation, and such references do not limit the extension of the claimed embodiments to embodiments in which different element, feature, protocol, or concept names are utilized. Thus, each term utilized herein is to be provided its broadest interpretation given the context in which that term is utilized.
Those of ordinary skill in the art will appreciate that the hardware components and basic configuration depicted in the following figures may vary. For example, the illustrative components within communication device 110 (
Within the descriptions of the different views of the figures, the use of the same reference numerals and/or symbols in different drawings indicates similar or identical items, and similar elements can be provided similar names and reference numerals throughout the figure(s). The specific identifiers/names and reference numerals assigned to the elements are provided solely to aid in the description and are not meant to imply any limitations (structural or functional or otherwise) on the described embodiments.
Referring now to the figures and beginning with
Returning to
Communication device 110 transmits segmented video feed 128 comprised of at least the primary segment through communications network 130, which includes wireless communication network 132 and associated physical networking components and wide area network 134 and associated physical networking components. Wide area network 134 provides or supports connection by video communication application server(s) 135, which is one or more physical components or logical partitions within servers that process server-level functions for video communication session 136. Segmented video feed 128 is forwarded by video communication application server(s) 135 to each session-connected second communication device 140a-140n, which presents segmented video feed 128 on respective remote displays 142a-142n for viewing by their respective remote participants 144a-144n.
It is appreciated that the term segmented video feed can mean different things within the context of the various different aspects of the disclosure. The received video feed is a complete video feed that appears seamless to the remote participants 144a-144n, but can comprise only one of (or less than a full completement of) multiple segments into which the original video stream captured at the local ICD 115 is divided/delineated.
Turning now to
ICD. It is appreciated that communication device 110 can be other types of electronic devices that include at least one front facing camera or one rear facing camera and which supports both video and non-video communication with one or more second communication devices.
Communication device 110 includes processor 202 (or processor integrated circuit (IC) chip), which includes processor resources such as central processing unit (CPU) 203a, communication signal processing resources such as digital signal processor (DSP) 203b, and graphics processing unit (GPU) 203c. Processor 202 can, in some embodiments, include high quality camera image signal processors (ISPs) (not shown) and dedicated artificial intelligence (AI) engines 205. Collectively, processor 202 supports computing, classifying, processing, transmitting and receiving of data and information, and presenting of graphical images within a display. Processor 202 is communicatively coupled to storage device 204, system memory 220, input devices, introduced below, output devices, including integrated display 230, and image capture device (ICD) controller 234. According to one or more embodiments, ICD controller 234 performs or supports functions such as, but not limited to, selecting and activating an active camera from among multiple cameras, adjusting the camera settings and characteristics (e.g., shutter speed, f/stop, ISO exposure, zoom control, etc.) of the active camera, etc. ICD controller 234 can perform these functions in response to commands received from processor 202, which processes instructions of camera control module (296,
For simplicity in describing the features presented herein, the various camera selection, activation, and configuration functions performed by the ICD controller 234 are described as being provided generally by processor 202. Similarly, manipulation of captured images and videos are typically performed by GPU 203c, and certain aspects of device communication via wireless networks are performed by DSP 203b with support from CPU 203a. However, for simplicity in describing the features of the disclosure, the functionality provided by one or more of CPU 203a, DSP 203b, and GPU 203c are collectively described as being performed by processor 202.
Throughout the disclosure, the term image capturing device (ICD) is utilized interchangeably to be synonymous with and/or refer to any one of front or rear facing cameras 232, 233. Front facing cameras (or image capture device (ICD)) 232 and rear facing cameras 233 are communicatively coupled to ICD controller 234, which is communicatively coupled to processor 202. Both sets of ICDs 232, 233 includes image sensors that can capture images that are within the field of view (FOV) of respective ICD 232, 233. Communication device 110 can includes multiple cameras having different functionality, such as a main camera capturing standard view, wide angle camera that captures a wide angle FOV, and telephoto ICD, which captures a telephoto FOV (zoom or magnified). In one or more embodiments, a single camera can be provided with camera control options to change the single camera lends to allow for wide angle and telephoto image capture.
System memory 220 may be a combination of volatile and non-volatile memory, such as random access memory (RAM) and read-only memory (ROM). System memory 220 can store program code or similar data associated with firmware 222, an operating system 224, and/or applications 226. During device operation, processor 202 loads and executes/processes program code of the various applications, modules, OS and firmware, that are stored in system memory 220.
In accordance with one or more embodiments, applications 226 include, without limitation, video image segmentation and control module (VISCM) 250, video communication session application (VCSA) 252, demonstration object preview module (DOPM) 254, and communication module 256. As the names suggest, each module and/or application (250-256) provides program instructions that are processed by processor 202 to cause processor 202 and/or other components of communication device 110 to perform specific operations, as described herein. Specifically, VCSA 252 and communication module 256 include program instructions that supports communication device 110 establishing a communication session with other external devices and systems, such as VCA server 135 and second communication devices 140a-140n. VISCM includes program instructions the support processor 202 delineating parts of a video feed into segments that can be separately presented and/or transmitted and instructions that support processor 202 presenting the touchless action centers with virtual interfaces and functionality associated therewith.
In one or more embodiments, communication device 110 includes removable storage device (RSD) 236, which is inserted into RSD interface 237 that is communicatively coupled via system interlink to processor 202. In one or more embodiments, RSD 236 is a non-transitory computer program product or computer readable storage device. RSD 236 may have a version of one or more of the applications (250, 252, 254) stored thereon. Processor 202 can access RSD 236 to provision communication device 110 with program code that, when executed/processed by processor 202, the program code causes or configures processor 202 and/or generally communication device 110, to provide the various different functionality described herein.
Communication device 110 includes an integrated display 230 which incorporates a tactile, touch screen interface 231 that can receive user tactile/touch input. As a touch screen device, integrated display 230 allows a user to provide input to or to control communication device 110 by touching features presented within/below the display screen. The tactile, touch screen interface 231 can be utilized as an input device. In some implementations, display 230 is integrated into a front surface of communication device 110, while the higher quality ICDs are located on a rear surface. Communication device 110 is placed in an orientation with the higher quality ICDs facing the scene being captured and integrated display located away from the subject of interest.
As one aspect of the disclosure, communication device 110 also includes external display 120, which is communicatively coupled to communication device 110 via a physical interface 265a or a wireless interface 265b. Display 230 can be one of a wide variety of display screens or devices, such as a liquid crystal display (LCD) and an organic light emitting diode (OLED) display.
Wireless interface 265b can be a short-range wireless communication device providing Bluetooth, near field communication (NFC) and/or a wireless fidelity (Wi-Fi) connections. In one embodiment, communication device 110 can receive internet or Wi-Fi based calls via wireless interface 265b. In one embodiment, communication device 110 can communicate wirelessly with externally provided WiFi router via wireless interface 265b. In an embodiment, WCS 242, antenna(s) 248 and wireless interface 265b collectively provide communication interface(s) of communication device 110. These communication interfaces enable communication device 110 to communicatively connect to at least one second communication device 140 (
Physical interface 265a of communication device 110 can serve as a data port and can be coupled to charging circuitry 235 and device battery 243 to enable recharging of device battery 243. Enabling the audio communication aspects for video communication session, communication device 110 further includes microphone 208, one or more output devices such as speakers 244, and one or more input buttons 207a-207n. Input buttons 207a-207n may provide controls for volume, power, and image capture device 232, 233. Microphone 208 can also be referred to as an audio input device. Microphone 208 and input buttons 207a-207n can also be referred to generally as input devices.
Communication device 110 further includes wireless network communication subsystem (WCS) 242, which can represent one or more front end devices (not shown) that each coupled to one or more antennas 248. In one or more embodiments, WCS 242 can include a communication module with one or more baseband processors or digital signal processors, one or more modems, and a radio frequency (RF) front end having one or more transmitters and one or more receivers. WCS 242 and antennas 248 allow communication device 110 to communicate wirelessly with a wireless communication network 132 (
Wireless communication network 132 further allows communication device 110 to wirelessly communicate with second communication devices 140a-140n, which can be similarly connected to wireless communication network 132. Communication device 110 can also communicate wirelessly with wireless communication network 132 via communication signals transmitted by short range communication device(s) (264) to and from an external WiFi router, which is communicatively connected to network 132. In one or more embodiment, wireless communication network 132 can include one or more servers (e.g., 135) that support exchange of wireless data and video and other communication between communication device 110 and second communication device 140a-140n.
Communication device 110 further includes haptic touch controls 245, vibration device 246, fingerprint/biometric sensor 247, global positioning system (GPS) device 260, and motion sensor(s) 262. Vibration device 246 can cause communication device 110 to vibrate or shake when activated. Vibration device 246 can be activated during an incoming call or message in order to provide an alert or notification to a user of communication device 110. In one or more embodiments, vibration device 246 can be used to inform the user when delineation and separation of the segments of a video image occurs and/or when a gesture is detected and accepted. According to one aspect of the disclosure, integrated display 230, speakers 244, and vibration device 246 can generally and collectively be referred to as output devices.
Biometric sensor 247 can be used to provide biometric data, such as fingerprints, to identify or authenticate a user. GPS device 260 can provide time data and location data about the physical location of communication device 110 using geospatial input received from GPS satellites.
Motion sensor(s) 262 can include one or more accelerometers 263 and gyroscope 264. Motion sensor(s) 262 can detect movement of communication device 110 and provide motion data to processor 202 indicating the spatial orientation and movement of communication device 110. Accelerometers 263 measure linear acceleration of movement of communication device 110 in multiple axes (X, Y and Z). For example, accelerometers 263 can include three accelerometers, where one accelerometer measures linear acceleration in the X axis, one accelerometer measures linear acceleration in the Y axis, and one accelerometer measures linear acceleration in the Z axis. Gyroscope 264 measures rotation or angular rotational velocity of communication device 110. In one or more embodiments, the measurements of these various sensors can also be utilized by processor 202 in the determining of the context of a communication. Communication device 110 further includes housing that contains/protects the components internal to communication device 110.
Referring to
Each of the various modules and applications includes program instructions/code that is processed by processor 202 and in some instances AI engine 205 to configure communication device 110 to perform the various different features of the present disclosure. In one or more embodiments, several of the modules include a representation of AI engine 205′, which can further include a machine learning (ML) engine (not separately shown) and computer vision (CV) engine (not separately shown). AI engine 205′ enables communication device 110 to identify a region of interest (ROI) within a locally captured video stream, track one or more subjects within a video stream, identify gesture provided inputs and associate the inputs with appropriate functions provide within virtual control interfaces, and identify text demonstration objects within a video stream, as well as other features and functions presented herein. In one or more embodiments, execution of the various modules by processor 202 enables/configures communication device 110 to perform the method processes presented in the various different flowcharts, as will be described below.
VISCM 250 includes or utilized a local representation of AI engine 205′ which provides the functions of ROI identification 270 and XY(Z) pixel segmentation 272. As provided within AI engine data 290, AI engine 205′ received video image stream 291 and detects specific subjects, such as a human, that are to be represented within ROI images 292. Once ROI image 292 is detected within video image stream 291, AI engine 205′ determines a primary segment having primary segment XYZ coordinates 293. Once the coordinates of the primary segment are identified and the primary segment delineated, AI engine 205′ identifies secondary segment coordinates 294a-294n, which further delineate each of the secondary segments. AI engine 205′ provides processor 202 with the required pixel coordinates for the primary segment and also for the side, secondary segments that can be utilized for presenting the action centers with virtual user interface functions. In one embodiment, AI engine 205′ also monitors video image stream 291 for images within the physical space proximate to the presented locations of the action center(s) on the display device. AI engine data 290 thus includes camera detected hand gestures/inputs 295A and gesture identifying/interpreting database 295B that can include various different gestures or hand/finger movements that correspond to a selection/activation of an action feature/function within the virtual interface. It is appreciated that one or more of the presented AI engine operations may be performed by other processor components.
According to one feature of the disclosure, in response to detection of multiple potential subjects within a received video image, the processor identifies, via artificial intelligence (i.e., AI engine), a primary subject from among the multiple potential subjects and aligns the primary segment with an ROI that incorporates the primary subject.
VISCM 250 also includes action center virtual interface module 274, which supports generation and presentation of action centers within or proximate to the displayed location of one or more of the secondary segments. In the illustrative example of
VCSA 252 includes graphical user interface 281 for the video communication application in which the video images are presented, video preview control functions 283, video image settings 284, received video data 285, remote second participant control sharing UI 286, and received second participant selections 287. The specific operations and/or functions associated with the remote second participant control sharing UI 286 and received second participant selections 287 will be explain in greater details later. Graphical user interface 281 includes a video presentation screen, a video preview screen, and a settings screen.
As illustrated by
Returning to
Communication module 256 enables communication device 110 to communicate with wireless communication network 132 and with other devices, such as second communication device 140, via one or more of audio, text, and video communications. Communication module 256 can support various communication sessions by communication device 110, such as audio communication sessions, video communication sessions, text communication sessions, communication device application communication sessions, or a dual/combined audio/text/video communication session.
CCM 296 includes camera parameters and settings 266, and AI engine 205, which further includes subject tracking 298 and camera selection modules 299. Other data 200 can also be provided within system memory, and the presented modules are for example only. Camera parameters and settings 266 can include fixed values such as focal length, maximum frame rate, and resolution and variable values such as light level values, directional values, and distances to objects within a camera FOV. Camera parameters and settings 266 are values and characteristics that can change during the operation of ICDs 232 and 233 to capture images by the cameras. Camera parameters and settings 266 can include various settings such as aperture, shutter speed, iso level, white balance, zoom level, directional settings (i.e., region of interest (ROI)), distance settings, focus and others. In one embodiment, camera parameters and settings 266 can be determined by either processor 202 or by ICD controller 234 or AI engine 205. In one embodiment, camera parameters and settings 266 can be adjusted by the local participants use of gestures that are captured by the ICD and associated with one or more of the virtual interface functions within the currently displayed action center. Similarly, other characteristics of video images can be determined and/or modified, such as light levels within the ROI, contrast, blur, etc., can be adjusted by the local participants use of gestures that are captured by the ICD and associated with one or more of the virtual interface functions within the currently displayed action center.
With the above components integrated within communication device 110, a first aspect of the disclosure presents a communication device 110 having at least one image capturing device (ICD) 233 that captures video (e.g., video image stream 291) of a local scene, an interface 264/265 by which the electronic device connects to a local display device 120, and a system memory 220 having stored thereon a video image segmentation and control module (VISCM) 250 comprising program instructions for spatially delineating video images into two or more segments that can be individually presented within or excluded from a video feed. The at least one processor 202 is communicatively coupled to the local display device 120, to each of the at least one image capturing device 132/133, and to the system memory 220. The at least one processor 202 executes or processor the program instructions of the VISCM 250, which causes the at least one processor 202 to perform a series of processes, which are illustrated by
According to one aspect of the disclosure, the at least one processor 202 divides the video image into different segments in response to user selection of a video image segmenting function 420 before or during the video communication session 136. The segmentation feature is triggered is response to the activation occurring while the communication device 110 is communicatively connected to the display device 120 and is being utilized as a camera for capturing the video images to transmit to the video communication session 136.
Referring now to
In accordance with a first aspect of the disclosure, the processor 202 associates an action center 510A/510B having a virtual interface 515 with a location of the secondary segment 445A/445B in the video image 405. The virtual interface 515 includes at least one displayed feature that can be selected via one of air gestures and screen touches during presentation of the primary segment 440 of the video image 405. As shown by each of the figures, the processor 202 presents a preview with the delineated segments of the video image on the display device with at least the primary segment 440 and the one or more other segments (e.g., secondary segments 445A-445B) of the video image 405 and the virtual interface of the action center 510A/510B being visible within the local display device 120. In one or more embodiments, the processor 202 presents the virtual interface on the display device 120, visibly adjacent to or integrated as an overlay on top of (i.e., to at least partially obscure) the closest secondary segment. As an example,
To access the video communication session, the communication device 110 includes a communication interface (e.g., WCS 242) that enables the communication device 110 to communicatively connect with at least the second device 140. The processor is communicatively coupled to the communication interface, and the processor enables the communication device 110 to connect, via the communication interface, to the second device 140 via a video communication session 136 enabling exchange of video, audio, and other data with the second device 140. The processor selectively transmits, via the video communication session to the second device, only the content within the primary segment 440 of the video image 405 in response to the video image being locally presented as a segmented video image preview. The secondary segment(s) 445 and the associated virtual interface 515 (
In one or more embodiments, and as presented by
In one or more embodiments, the processor 202 is further configured to identify selection of the specific feature, which activates a function that modifies one or more characteristics of at least the content within the primary segment of the video image. The processor applies the function to an original content within at least the primary segment 440 to generate a resulting modified content, and the processor locally presents and remotely transmits the resulting modified content within the primary segment 440 in place of original content.
With reference to the illustrative examples of
According to one aspect, because the preview is displaying a mirror image of the local participant, the participant selects items on the left action center with his right hand, and vice versa. The right hand appears adjacent to and/or within the same space as the left action center with preview image mirroring. In instances where no mirroring occurs in the preview image, the correct right or left hand and corresponding right or left action centers are aligned. In an alternate embodiment the selection can be is assumed to be based on the user's view of the mirrored image, and the selection can be further enhanced by allowing an icon to track the user's gestures on the preview screen so that the user can visibly see which action center the user is interfacing with. In one or more embodiments, the selection of right versus left placement of the action center can be a user preference, where the user can selectively assign the location of specific action centers within the settings menu of the video communication application. A left-handed user can thus reverse the relative placement of the action centers. Also, a user can move the action centers with the most commonly utilized features to within the best virtual location for capturing the users gesture inputs.
Referring back to the figures,
In one or more embodiments, selection of the secondary segments occurs independent of the application of a background effect. These embodiments enable the local participant to granularly/selectively choose which of the segments to transmit over the video communication session. Right action center 510B is shown having additional selectable features, including ROI zoom 915 and ROI XY Positioning 920, which enables a local participant (or a remote authorized second participant) to adjust a size and location of the primary segment 440. Once delineated on the user device, primary segment can thus be expanded, made smaller, shifted left or right, up or down, extended horizontally or vertically, etc.
With reference to
Method 1000 includes associating a virtual interface with a location of the secondary segment in the video image, the virtual interface including at least one feature that can be selected via one of air gestures and screen touches during presentation of the primary segment of the video image (block 1010). Method 1000 includes presenting a preview of the delineated segments of the video image on the display device with at least the primary segment and the secondary segment of the video image and the virtual interface being visible within the display device (block 1012).
In presenting the preview, method 1000 includes presenting, on the display device, a preview comprising the primary segment and the one or more other segments of the video image as delineated segments from each other and presenting the virtual interface on the display device, visibly adjacent to or integrated as an overlay on top of the secondary segment. In one or more embodiments, method 1000 can also include providing a visual indication of which segment is the primary segment that is being shared with the second device.
From block 1012, method 1000 transitions to decision block 1014 in
With ongoing reference to
With reference now to
From the above table example, each of the plurality of second devices that have not been assigned to receive an action center virtual interface, e.g., Participant 2 and 6, only receives the primary segment in a received video feed from the IST device. PPSW 1100 also includes host control inset 1110 and video control inset 1115, which respectively present a listing/indication of which participants have been given host control (with capability to modify the local video image for all participants), and a listing/indication of which participants have been given local video control (with capability to modify the local video image for just that participant and perhaps on the IST device (communication device 110).
According to the illustrative embodiment, IST device 110 presents preview image 1240 that is shared with second device 140 and is presented within second device display as received video images 1240′. IST device 110 also provides to at least two other second devices, identified as participant 3 1220A (provided with host-level video image modification privileges) and participant 4 1220B (provided with limited (or individual) video image modification privileges) the features provided by one or more action centers 510A-510B. Received virtual action centers 510A′, 510B′ are presented on the corresponding display device of the second device along with the received video image 1240′. By making selections and interfacing with the provided features of the provided action centers, the receiving second device 140a is able to remotely modify the video image 1240 at IST device 110 and on their corresponding local displays. Participant 3 with host privilege can further modify video image 1240 received by other second devices.
As shown by the participants permission settings window (PPSW) 1100 and also within video preview screen of IST device 110 each second device provided with action center functions are indicated via associated identifiers (1220a-1220b) on the local display. These identifiers can be provided within display participant listing feature (UI 515A,
In one or more embodiments, the at least one processor detects selection of the particular second device 140 as an administrative device (e.g., a device having host-level privileges), auto-selects specific virtual interfaces having functions controllable by the administrative device, and transmits the auto-selected virtual interfaces concurrently with the primary segment to enable the virtual interfaces to be visually presented, concurrently with the primary segment, within the second display of the particular second device.
In one or more embodiments, the processor further enables the device to monitor for receipt of an indication of a remote selection or activation of at least one control function from within the virtual interfaces at the particular second device. In response to receiving an indication of a remote selection that is associated with a modification of a characteristic of the video images within the primary segment, the processor applies the modification to video image content within the primary segment at the electronic device and transmits the modification of the video image content to each of the plurality of second devices participating in the video communication session. In one embodiment, the broadcasting of the modified video image content is triggered only when the participant has host control, such as Participant 3 in PPSW 1100. Participant 4 is capable of also modifying the video image, but the modification only affects the primary segment content presented within the IST device display and/or the remote display of Participant 4.
Method 1400 includes transmitting, to each of the plurality of second devices, at least the primary segment 440 for displaying at a corresponding second display of a second device during the video communication session (block 1416). Method 1400 then transitions to block 1418 in
Returning to
According to another aspect of the disclosure, and is introduced within the description of
In one embodiment, the processor modifies the characteristic of the content at the IST device by transmitting a corresponding content settings update indication to the IST device via the video communication session. Also, the processor modifies the characteristic of the content for at least one of: the electronic device only; and all connected devices participating in the video communication session that receive the primary segment from the IST device. The function modifies one or more of: (i) characteristics of the content within the primary segment, (ii) characteristics of the primary segment, (iii) characteristics of the video communication session, (iv) a local device setting, and (v) a setting of the IST device.
In one embodiment, one of the secondary segments includes a virtual white board and the input activates the white board for content sharing between the electronic device and at least the IST device. In one or more embodiments, the received segmented video stream comprises a preview window identifying which segments from among the primary segment and the at least one secondary segment are received by each second device connected to the video communication session that receives video content from the IST device.
According to another aspect of the disclosure, and as presented within
In one or more embodiments, the electronic device includes system memory 220 (
As presented by
In one or more embodiments, the processor presents a virtual interface (1618) on the display device, visibly adjacent to or integrated as an overlay on top of at least one of the secondary segments 445B. The processor presents, within the virtual interface 1618, the DOP option 1620, which option is selectable via the virtual interface 1618 by one or more of an air gesture and a touch of a spatial area of the virtual interface on the display device. The DOP option enabling selection of a mirrored, normal view and a non-mirrored view to present the demonstration object. The processor presents, within the virtual interface, the DOP option, which option is selectable via the virtual interface 1618 by one or more of an air gesture and a touch (physical or virtual) of a spatial area of the virtual interface 1618 on the display device, the DOP option 1620 enabling selection of a mirrored, normal presentation preview and a non-mirrored presentation preview to present the demonstration object 1610. Accordingly, the processor spatially segments and presents a non-mirrored view of the DOP segment relative to the mirrored view of the primary and secondary segments in response to receiving an activation of a demonstration object preview (DOP) option 1620 within an application supporting presentation of a camera image preview.
In addition to DOP option 1620, example virtual interface 1618 incudes secondary reverse mirroring selectors 1622 that enables the processor to reverse the mirroring of either the demonstration object only or all of the video image. In
In one or more embodiment, the communication interface enables the electronic device to communicatively connect with at least one second device in a video communication session. The processor configures the electronic device to connect, via the communication interface, to the at least one second device via the video communication session enabling exchange of video and other data with the second device, and to transmit, over the video communication session to the at least one second device, an unmirrored original version of the primary segment 440 and the DOP segment 1615 of the video image. The transmitted segments (440, 1615) of the video image are presented in correct (original) non-mirrored orientation adjacent to each other on a second participant display 1650 of the at least one second device 140. In one embodiment, the processor transmits, over the video communication session to the at least one second device, an original (unsegmented) version of the video image with all of the video image presented in correct non-mirrored orientation on a display of the second device.
According to at least one embodiment, method 1700 includes spatially segmenting the video image into two or more segments, including the DOP segment that encompasses the defined area with the demonstration object and at least one secondary segment encompassing a remaining portion of the video image (block 1706). Method 1700 includes delineating a virtual boundary of the demonstration object using x-y-z coordinate pixel locations to identify the DOP segment. Method 1700 includes mirroring/flipping the primary segment to provide the non-mirrored view of the demonstration object within the virtual boundary. Method 1700 includes reorienting, in x-y-z directional coordinates, at least one exterior edge of the primary segment and a view angle of primary segment to fill a space within the virtual boundary with the primary segment. Method 1700 further includes stitching the edges of the primary segment to the virtual boundary to present a substantially seamlessly fit of the primary segment into a space from which the primary segment is remove within the remaining portion of the video image. The reorienting incudes reorienting at least one of an exterior boundary of the primary segment and a view angle of primary segment to present a substantially seamlessly fit of the primary segment into a space from which the primary segment is remove within the remaining portion of the video image during an image stitching process.
Returning to the flow chart, method 1700 includes dividing the remaining portion into at least a primary segment and secondary segments, with the primary segment encompassing a primary region of interest and the secondary segment encompassing sections of the video image than can be selectively cropped from the remaining portion of the video image (block 1708). Method includes presenting a virtual control interface on the display device, visibly adjacent to or integrated as an overlay on top of the secondary segment (block 1710). Method 1700 includes presenting, within the virtual control interface, a DOP option that is selectable via the virtual interface by one or more of an air gesture and a touch of a spatial area of the virtual interface on the display device, the DOP option enabling selection/toggling between a mirrored, normal view and a non-mirrored view to present the demonstration object (block 1712). Accordingly, the spatially segmenting of the video image and presenting of the non-mirrored view of the primary segment relative to the mirrored view of the secondary segment is performed in response to detecting an activation of a demonstration object preview (DOP) option within an application supporting presentation of a video image preview. Method 1700 includes presenting a preview of the delineated segments of the video image on the display device with the remaining portion of the video image mirrored within the preview and at least the primary segment presented without mirroring in a correct spatial location relative to the remaining portion of the video image, wherein a local subject presenting the demonstration object receives a non-mirrored presentation preview of the demonstration object (block 1714).
According to an extension of the described embodiments, method 1700 includes presenting, on the display device, the preview including the DOP segment modified to a non-mirrored view and stitched into a virtual inner boundary of the at least one secondary segment of the video image, to present both segments as a substantially contiguous (i.e., non-segmented) video image. Method 1700 includes providing a visual indication of which segment is the primary segment that is being presented via a non-mirrored presentation within the video image preview. This visual indication can be provided by presenting the dashed outline of the segment as shown in the various figures,
In the above-described methods, one or more of the method processes may be embodied in a computer readable device containing computer readable code such that operations are performed when the computer readable code is executed on a computing device. In some implementations, certain operations of the methods may be combined, performed simultaneously, in a different order, or omitted, without deviating from the scope of the disclosure. Further, additional operations may be performed, including operations described in other methods. Thus, while the method operations are described and illustrated in a particular sequence, use of a specific sequence or operations is not meant to imply any limitations on the disclosure. Changes may be made with regards to the sequence of operations without departing from the spirit or scope of the present disclosure. Use of a particular sequence is therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined only by the appended claims.
Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object-oriented programming language, without limitation. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine that performs the method for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. The methods are implemented when the instructions are executed via the processor of the computer or other programmable data processing apparatus.
As will be further appreciated, the processes in embodiments of the present disclosure may be implemented using any combination of software, firmware, or hardware. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment or an embodiment combining software (including firmware, resident software, micro-code, etc.) and hardware aspects that may all generally be referred to herein as a “circuit,” “module,” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable storage device(s) having computer readable program code embodied thereon. Any combination of one or more computer readable storage device(s) may be utilized. The computer readable storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage device can include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage device may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Where utilized herein, the terms “tangible” and “non-transitory” are intended to describe a computer-readable storage medium (or “memory”) excluding propagating electromagnetic signals, but are not intended to otherwise limit the type of physical computer-readable storage device that is encompassed by the phrase “computer-readable medium” or memory. For instance, the terms “non-transitory computer readable medium” or “tangible memory” are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, RAM. Program instructions and data stored on a tangible computer-accessible storage medium in non-transitory form may afterwards be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which may be conveyed via a communication medium such as a network and/or a wireless link.
The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the disclosure. The described embodiments were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.
As used herein, the term “or” is inclusive unless otherwise explicitly noted. Thus, the phrase “at least one of A, B, or C” is satisfied by any element from the set {A, B, C} or any combination thereof, including multiples of any element.
While the disclosure has been described with reference to example embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the disclosure. In addition, many modifications may be made to adapt a particular system, device, or component thereof to the teachings of the disclosure without departing from the scope thereof. Therefore, it is intended that the disclosure not be limited to the particular embodiments disclosed for carrying out this disclosure, but that the disclosure will include all embodiments falling within the scope of the appended claims.
This application is related to U.S. application Ser. No. 17/826,639, Ser. No. 17/826,683, and Ser. No. 17/826,735, filed on May 27, 2022, the contents of which are fully incorporated herein by reference.
