Patent Application
20180234708
This application claims the benefit of priority of Korean Patent Application No. 10-2017-0018551 filed on Feb. 10, 2017 and Korean Patent Application No. 10-2018-0015578 filed on Feb. 8, 2018, all of which are incorporated by reference in their entirety herein.
The present invention relates to an image processing and a service providing method based on a sticker added to a screen or a photograph captured by a camera.
In recent years, broadcasting through the Internet as well as public TV broadcasting and cable TV broadcasting through broadcasting networks have been activated, thereby allowing viewers to experience various broadcasting contents through an Internet Protocol (IP) network. In particular, although broadcasting of a Broadcast Jockey (BJ) who deals with interesting contents is useful in an advertisement effect for a large number of viewers, it is difficult to consider all of these factors in general personal broadcasting. Accordingly, there is a limitation in that the broadcasting of BJ cannot be technically utilized well. In addition, it is difficult for the general personal broadcasting to utilize various broadcasting special effects available in broadcasting stations.
The present invention provides a live streaming image generating method and apparatus, a live streaming service providing method and apparatus, and a live streaming system, which allow a Broadcast Jockey (BJ) of a live broadcasting service to provide a much richer media broadcast by utilizing various types of real-time animation stickers and effects, A providing method and apparatus, and a live streaming system.
In an aspect, a method for generating a live streaming image in a client terminal includes: acquiring platform information for generating a streaming image; generating a first streaming image based on the platform information; displaying the first streaming image; selecting, by a user input, at least one of stickers and effects for decorating at least a portion of the displayed first streaming image; and generating a second streaming image by applying the selected at least one of the stickers and effects to the first streaming image in real-time.
The generating of the first streaming image may include: photographing a subject through a camera; and generating the first streaming image based on the photographed subject.
The generating of the second streaming image may include: tracking the subject in real-time to acquire tracking information; and generating the second streaming image by applying the selected at least one of the stickers and effects to the subject based on the acquired tracking information.
The tracking information may include information associated with at least one feature point of the subject, and the selected at least one of the stickers and effects may be applied to the subject via the at least one feature point included in the tracking information.
The information associated with the at least one feature point may include motion information of the at least one feature point, and the selected at least one of the stickers and effects may be applied to the subject in response to a motion of the subject based on the motion information of the at least one feature point.
The method may further include transmitting the generated second streaming image to a live broadcasting server providing a service associated with live broadcasting. Here, the second streaming image transmitted to the live broadcasting server is transmitted to a client terminal of a viewer who watches the live broadcasting.
The platform information may include broadcast platform information associated with live broadcasting acquired from a live broadcasting server providing a service associated with the live broadcasting, and the acquired broadcast platform information may include information on at least one of the stickers and the effects.
The sticker may include a sticker that arbitrarily sets position information in the first streaming image according to a user input to cover a portion of the first streaming image.
The effect may include an effect of allowing at least a portion of the sticker previously applied to the first streaming image to disappear by a user input.
In another aspect, an apparatus of generating a live streaming image includes: a communication unit for communicating with a server; a display unit for displaying a first streaming image generated based on platform information for generating a streaming image acquired from the server through the communication unit; and a controller selecting at least one of stickers and effects for decorating at least a portion of the displayed first streaming image and generating a second streaming image by applying the selected at least one of the stickers and effects to the first streaming image in real-time by a user unit.
In another aspect, a method for providing a live streaming service in a server apparatus providing a live streaming service includes: providing platform information associated with live streaming to a first client terminal; receiving a second streaming image, in which at least one of stickers and effects for decorating at least a portion of the first streaming image generated based on the platform information is applied in real-time, from the first client terminal; and providing the second streaming image acquired from the first client terminal to a second client terminal.
The platform information may include information about at least one of stickers and effects.
The live streaming service may include a live broadcasting service based on the first client terminal, and the second client terminal may be a terminal associated with a viewer who enters a broadcasting station established by the first client terminal.
At least one of a user input at the first client terminal for applying the sticker and a user input at the second client terminal for activating an effect of allowing at least portion of the applied stickers to disappear may be associated with cost payment from at least one of a user account associated with the first client terminal and a user account associated with the second client terminal to the server apparatus.
The at least one of the stickers and the effects may include at least one of stickers and effects associated with advertisement.
when sensing that a user associated with the second client terminal performs an arbitrary action associated with the advertisement based on at least one of stickers and effects associated with the advertisement applied to the streaming image when generating, in the first client terminal, the streaming image by applying at least one of the stickers and effects associated with the advertisement, the method may further include checking advertising-related information corresponding to at least one of the stickers and effects associated with the advertisement applied to the streaming image, and sharing at least a portion of a revenue associated with the advertisement with a user associated with at least one of the first client terminal and the second client terminal based on the confirmed advertisement-related information.
The sharing of revenue associated with the advertisement may be performed by estimating the number of times of application of at least one of stickers and effects associated with an advertisement applied to the second streaming image.
The method may further include providing a sticker/effect selection menu including at least one of sticker candidates and effect candidates applicable at the first client terminal based on the platform information.
At least one of the sticker candidates and the effect candidates included in the sticker/effect selection menu may consider at least one of a history, a content, and a service type of a user associated with the first client terminal and may include at least one of sticker candidates and effect candidates corresponding thereto.
In another aspect, an apparatus for providing a live streaming service includes: a communication unit for communicating with a first client terminal and a second client terminal; and a controller transmitting platform information associated with live streaming provided to the first client terminal through the communication unit, and receiving, from the first client terminal, a second streaming image to which at least one of stickers and effects for decorating at least a portion of the first streaming image generated based on the platform information to provide the second streaming image acquired from the first client terminal to the second client terminal.
Since the present invention may be modified into various types and may be implemented into various embodiments, specific embodiments will be illustrated in the drawings and described in this disclosure in detail.
However, the present invention is not limited to a specific implementation type, but should be construed as including all modifications, equivalents, and substitutes involved in the spirit and the technical scope of the present invention.
The terms such as “a first/the first” and “a second/the second” may be used to describe various components, but the components should not be limited by the terms. The terms are used only in order to distinguish one component from another component. For example, a first component may be named a second component without deviating from the scope of the present invention, and similarly, the second component may be named the first component. The term “and/or” includes a combination of a plurality of related items or any one of a plurality of related items.
It should be understood that when an element is referred to as being “connected” or “coupled” to another element, it may be directly connected or coupled to the other element but another element may also be interposed therebetween. On the other hand, when an element is referred to as being “directly connected” or “directly coupled” to another element, it should be understood that there are no other elements in between.
The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. In this disclosure, the terms “include,” “include,” or “have” specify features, numbers, steps, operations, elements or combinations thereof, but do not exclude existence or addition possibility of one or more other features, numbers, steps, operations, elements or combinations thereof.
Unless described otherwise, all terms used herein including technical or scientific terms may include the same meaning as those generally understood by persons skilled in the art to which the present invention belongs. Terms as defined in dictionaries generally used should be construed as including meanings which accord with the contextual meanings of related technology. Also, unless clearly defined in this disclosure, the terms should not be construed as having ideal or excessively formal meanings.
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to help the overall understanding of the present disclosure, the same reference numerals will be used for the same elements in the drawings, and a duplicate description of the same elements will be omitted.
Throughout this specification, it will be apparent to those skilled in the art that all stickers/effects mean digitized forms of information for decorating display images, and even when only the term sticker is used, effect may also be considered together.
The live streaming image generating apparatus may be referred to as a client terminal and a live broadcasting streaming image generating apparatus, and the live streaming service providing apparatus may be referred to as a server apparatus and a live broadcasting service providing apparatus. In addition, provision of a selection menu including sticker candidates and effect candidates for application of stickers/effects, application of stickers/effects according to selection in the selection menu, and generation of streaming images based on sticker/effects application may be performed in a client terminal as well as in a server apparatus.
In an embodiment of this specification, the sticker/effect are based on the use for a broadcast service, but is not necessarily used for the broadcast service. The sticker/effect is also applicable to a general communication method. In the broadcast service or other services, the sticker/effect may be utilized as a form of a virtual advertisement of inserting a virtual image with a computer graphic.
Referring to
The first client terminal 110 may enrich the broadcast contents of the broadcast jockey using various broadcast-related control units existing in the broadcast platform. The units for enriching broadcast contents may include stickers and/or effects. The sticker comprehensively means designs and elements associated with the designs for decorating at least a portion of a subject photographed by a camera and displayed on the display unit, a display image containing the subject, and/or the first streaming image generated based on the platform information. For example, the sticker may include designed objects such as fur hat shape, crown shape, eyeglass shape, headband shape, various figures, animation shapes, etc., which are overlaid on the subject. The sticker may exist at a specific position in the image while not moving in non-association with the subject, or may exist in a form moving in accordance with the movement of the subject in association with the subject. In addition, the sticker may include a moving sticker like an animation and an immobile stationary sticker, and may include a sticker implemented with 3D. The sticker may be used as a form of being continuously applied when the sticker/effect is selected, and may also be used as a form of expressing certain emotions and disappearing. The effect may include at least one of a sound-based audio effect, a subtitle, a text, a graphic, a sound, a functional graphic that can call a specific animation, and a video. The stickers and effects may be applied at the same time, and a plurality of stickers and effects may be applied.
The first client terminal 110 selects a sticker and/or an effect to be applied to the first streaming image from a plurality of sticker candidates and/or effect candidates. Here, the sticker candidates and effect candidates may refer to selectable stickers/effects stored in a sticker storage (or inventory) before being selected as an actually applied sticker/effect. The sticker candidates and effect candidates are displayed on the sticker/effect selection menu screen, and at least one of sticker/effect sets including the displayed sticker candidates and the effect candidate may be selected and applied to at least a portion of the display image (first streaming image). A second streaming image is generated based on the applied sticker and/or effect. The generated second streaming image is provided to the server apparatus 120. The first client terminal 110 may together transmit the basic streaming image (first streaming image) to which the sticker/effect is not applied and the second streaming image to which the sticker/effect is applied to the server apparatus 120.
The server apparatus 120 is a terminal that hosts a live broadcasting service. The server apparatus 120 receives the streaming image from the first client terminal 110 executing the personal broadcasting and broadcasts the streaming image to an unspecified number of the second client terminals 130. The server apparatus 120 may manage basic broadcast platform information for broadcasting as well as transmission of the broadcast data. The broadcasting may be implemented through an application or other software programs, and the broadcast platform information includes installation and management information of the application or software programs. The server apparatus 120 occupies a specific Uniform Resource Locator (URL), and allocates a part of the occupied URLs in response to a request from the first client terminal 110 that intends to perform personal broadcasting to allow a broadcasting station to be established through the corresponding channel. In addition, the server apparatus 120 allows other users (viewers) to enter the established broadcasting station, thereby supporting the smooth broadcasting between the broadcast jockey and viewers.
According to an embodiment of the present invention, the broadcast platform information includes information related to the sticker and/or the effect. The broadcast platform information may also include policy information that is related to the calculation of the advertising revenue associated with the sticker and/or effect and the distribution of the calculated advertising revenue.
The second client terminals 130 are terminals of viewers which are connected to the server apparatus 120 and enter a plurality of personal broadcasting stations (including a broadcasting station opened by the first client terminal 110) managed by the server apparatus 120 to enjoy broadcasting. The second client terminal 130 may enter URL-based personal broadcasting stations provided by the server apparatus 120 and allow viewers to view the streaming images provided by the first client terminal 110. In this case, data may be exchanged between a plurality of viewers and the broadcast jockey using the chat window provided by the server apparatus 120. In addition, the second client terminal 130 may allow a viewer to view various advertising contents derived from the stickers and/or effects applied to the streaming image during broadcasting, and may perform a role of creating advertisement-related profits such as purchase of related products.
In an embodiment, at least one of the first client terminal 110 and the second client terminal 120 user terminal may be referred to as a Mobile Station (MS), a User Equipment (UE), a User Terminal (UT), a wireless terminal, an Access Terminal (AT), a terminal, a fixed or mobile subscriber unit, a Subscriber Station (SS), a cellular phone, a wireless device, a wireless communication device, a Wireless Transmit/Receive Unit (WTRU), a mobile node, a mobile, a mobile station, a Personal Digital Assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, Consumer Electronics (CE) or other terminologies. Various examples of the terminals may include cellular phones, smart phones having a wireless communication function, Personal Digital Assistants (PDAs) having a wireless communication function, wireless modems, portable computers having a wireless communication function, photographing devices such as digital cameras having a wireless communication function, game devices having a wireless communication function, music storage and playback home appliances having a wireless communication function, Internet home appliances capable of wireless Internet access and browsing, and portable units or terminals incorporated with combinations of such functions, but are limited thereto.
According to another embodiment of the present invention, the first client terminal 110 may not use only a subject currently photographed through a camera as a source image, but may also uses an existing streaming or recorded video as a source to apply stickers/effects. For example, a sticker may be applied even when a streaming image being broadcasted by another terminal is relayed.
Also, a sticker/effect may be applied to a terminal of a viewer like the second client terminal 130, and then may be provided to another client terminal.
Referring to
The left drawing of
Referring to the right drawing of
According to an embodiment of the present invention, a specific sticker/effect included in the sticker/effect menu provided through the server apparatus may include a plurality of sticker elements. That is, when one sticker is clicked and applied to a broadcast streaming image, the lip shapes of 10 or more may be displayed on the whole screen. That is, one sticker may include a plurality of sticker elements, and a first sticker element is disposed at a first position and a second sticker element is disposed at a second position. Thus, a sticker in which the sticker elements are disposed on the whole screen may be utilized. The server apparatus may include and hold the information associated with such a sticker/effect in the broadcast platform information, and the first client terminal that acquires the broadcast platform information and performs broadcasting may also hold the information associated with the sticker/effect.
According to an embodiment of the present invention, the broadcast streaming image is generated at the first client terminal. In some cases, with regard to the application of the sticker/effect, the server apparatus may also generate a streaming image to which the sticker/effect is applied. That is, after a basic streaming image is generated by photographing a subject with a camera or is generated based on an image pre-stored or received from other terminals in the first client terminal, when the basic streaming image is transmitted to the server apparatus and a signal related to the selection of a sticker/effect is provided from the first client terminal to the server apparatus, the server apparatus may parse the signal related to the selection of the sticker/effect to apply the selected sticker/effect to the basic streaming image, and may again transmit the applied streaming image to the first client terminal and/or the second client terminal for broadcasting.
According to another embodiment of the present invention, the sticker/effect information included in the sticker/effect selection menu, which is included in the broadcast platform information and provided to the first client terminal, may be differently provided in accordance with the identification information (e.g., a user account of a broadcast jockey) of a user (i.e., broadcast jockey) of the first client terminal. That is, when a first user pursues a personal broadcasting related to music, a sticker/effect set including a plurality of stickers/effects associated with music may be provided as a selection menu. Alternatively, when the first user pursues a personal broadcasting related to food, a menu including a plurality of sticker/effect sets associated with affiliates various foods/desserts and income dividend may be provided. In other words, the sticker/effect information may be differently generated in consideration of the broadcasting characteristics of a user, and the sticker/effect information thus generated may be included in the broadcast platform information and provided to the first client terminal. In addition, different sticker/effect sets may be provided in accordance with IP and service type of the first client. In order to provide these customized stickers/effects, the data related to the user's broadcast or the histories related to the use of the stickers/effects may be stored in the server apparatus, and the server apparatus may provide the customized stickers/effects to the first client terminal through various statistics algorithms based on the stored data.
According to another embodiment of the present invention, a function of analyzing texts used in the chat window and/or comments used by a broadcast jockey during broadcasting and automatically displaying a screen for selecting a sticker/effect corresponding to the analyzed texts and/or comments may be added. This may be accomplished by pre-storing specific texts and comments in the server apparatus and automatically popping up a menu for selecting the sticker/effect corresponding thereto when the stored text and comments are recognized.
Referring to
Referring to
The platform information includes a tracking algorithm. The tracking algorithm includes a face tracking algorithm. That is, the tracking algorithm is an algorithm that acquires information for tracking the motion of a moving subject. The tracking algorithm specifies at least one feature point of a specific subject (e.g., a face, or mouth, nose or the like of face) to which a specific sticker is applied, and includes an algorithm that acquires information (e.g., movement of coordinates) related to the movement of at least one specified feature point. Based on the tracking algorithm, the movement of the moving subject is detected, and the movement distance of the sticker corresponding to the detected movement distance is calculated. Then, the sticker/effect is moved by the calculated moving distance, thereby allowing the sticker/effect to move according to the motion of a subject as well. However, since the tracking may or may not be required according to the characteristics of each sticker, it is desirable that the movement of the sticker according to the tracking information is applied corresponding to the characteristics of the sticker.
The platform information includes various kinds of sticker/effect information. The platform information includes all customized sticker/effect set information for each user and service as well as sticker/effect information stored in the server apparatus. This sticker/effect information is continuously updated as new information associated with the sticker/effect is generated. Also, as the statistical analysis such as the use history of a user is performed, the update is continuously performed. The platform information including the updated sticker/effect information may be periodically provided to each terminal. Also, the sticker/effect information includes information related to the activation of the sticker/effect selection menu corresponding to the detection of a specific text or a specific comment according to voice recognition. In addition, according to another embodiment of the present invention, in regard to the broadcast platform information, the sticker/effect set may be configured based on the advertising sticker/effect associated with or corresponding to the corresponding location based on the location information of the first client terminal. That is, a broadcast jockey at a specific location may transmit broadcast platform information including an advertising sticker/effect set corresponding to the specific location preset in the server to a broadcast jockey such that the location-based sticker/effect can be applied to the streaming image.
Alternatively, according to another embodiment of the present invention, a specific advertising sticker may be applied to a predetermined position based on the location of a viewer, i.e., the second client terminal. The location-based sticker may be directly applied to the broadcast streaming image received at the second client terminal. Alternatively, the location-based sticker may be applied to a streaming image relayed from the second client terminal to a third client terminal.
The broadcast platform information includes various kinds of functional information for broadcasting. The functional information may include information such as chatting, screen layout, and management of viewers of each broadcasting station.
The broadcast platform information includes link sensing function information, which represents information related to a function of sensing actions of a viewer who responds to the sticker/effect and uses advertising products, advertising contents, or advertising links associated therewith. In other words, since the viewer's secondary actions are used as an important factor in the distribution of advertising revenue, an algorithm for sensing the secondary actions may be importantly utilized.
The broadcast platform information includes policy information related to cost sharing. This represents information related to the policy related to the use of the sticker/effect in the broadcasting. For example, the policy information may include detailed policy information on whether or not points can be obtained only by using a specific sticker and whether or not points can be obtained when there is an action of a viewer associated with a specific sticker.
Various kinds of information related to the streaming and/or broadcasting and information related to use of the sticker/effect in streaming and/or broadcasting are included in the platform information and provided to the client terminal 420.
Referring again to
First, in an embodiment of streaming associated with broadcasting, the first client terminal 310 establishes a broadcasting station to start personal broadcasting (S314). In this case, broadcasting establishment information for informing the broadcasting establishment is transmitted to the server 320 together with the broadcasting establishment. The broadcast establishment information may include information such as a user account associated with the first client terminal 310, a total number of available viewers, and a content type. The broadcasting establishment information may be provided to terminals of viewers who desire to enter the broadcasting station, that is, the second client terminal 330 (S318). This may be performed in response to a request from the second client terminal 330. When the information of the currently opened broadcasting station is requested, the information of the opened broadcasting stations is provided in response thereto. The second client terminal 330 may enter a broadcasting station selected from the established broadcasting stations that are provided by itself (S330).
The first client terminal 310 that has opened the broadcasting may photograph a subject using a camera interworking with the terminal or a camera included in the terminal (S320). In this case, the subject may be anything as well as a user himself or herself. Then, the photographed image may be displayed through the display unit of the terminal (S322). The display screen of the photographed subject is streamed and broadcast. If a user desires to apply stickers and/or effects during the broadcasting, then a user clicks an icon that pops up the menu window for selecting stickers and/or effects through the user interface. Then, various stickers/effects included in the broadcast platform information are displayed on the screen. In this case, a sticker/effect set corresponding to information on the user's account, the type of the terminal, and the content type of the established broadcasting station may be generated and provided in a customized manner. The first client terminal 310 selects at least one of the provided sticker/effect sets (S324). The selected sticker/effect is applied to at least a portion of the display image including the moving subject to be generated into a streaming image (S326). The streaming image to which the sticker/effect is applied is transmitted to the server 320 (S328), and the server 320 provides the streaming image to all the viewer terminals 330 that enter the broadcasting station (S334).
Referring to
Referring to
In this case, it is possible to check whether the applied sticker/effect is in association with the advertisement (S630). The association with the advertisement means that the corresponding sticker may be a sticker created by affiliating with a specific business (e.g., a brand). Accordingly, the sticker may include information about the specific business in association with a product of the specific business. That is, when a sticker is clicked, a linking operation may be performed such as linking to a product or a business site of a business, or replaying an associated advertisement video.
If the sticker/effect is in non-association with the advertisement, since the application of the sticker/effect performs a function of giving a viewer fun by applying the sticker/effect, the purpose of the application of the sticker/effect is achieved. However, if there is an association with the advertisement, the revenue distribution according to the advertisement effect may be additionally considered. The association with the advertisement may be confirmed by parsing the individual feature information of each sticker, and in some cases, flags are assigned such that an advertisement-based sticker/effect is indicated as “1”, and a non-advertisement-based sticker/effect is indicated as “0”.
If the applied sticker/effect is in association with the advertisement, the policy information related to the revenue distribution of the advertisement is parsed. In this case, when the revenue is distributed as the same time as the use of the sticker/effect, a broadcast jockey may acquire a portion of the fee only by using the advertisement-based sticker at the first client terminal. In this case, since the advertisement effect is maximized as the number of viewers who enter the broadcasting station increases, the fee may be set in consideration of the number of viewers who are in the broadcasting station at the time of use of the sticker. For example, in the case of a sticker of “10” points, 100 points may be provided to the broadcast jockey as the fee when there are 10 viewers, and 1000 points may be provided to the broadcast jockey when there are 100 viewers.
However, when the fee is not set at the same time as the use as described above and the fee is set based on the secondary action of a viewer of the corresponding broadcasting station, the action of the viewer terminal associated with the applied sticker/effect is sensed (S640). This may be performed in the server apparatus. When an action such as purchasing a product, touring a product, and adding to a shopping cart is performed through a site linked to the brand of the sticker, a compensation corresponding thereto may be offered to the broadcast jockey in accordance with the types of secondary actions. Alternatively, the fee may be set in consideration of the number of click operations. That is, at least a portion of the advertising revenue calculated through various methods as described above is allocated to an account connected to the account of the broadcast jockey (S650). In other words, the broadcast jockey may be allowed to consider the allocation of these fees as an incentive, and to use the advertisement-based stickers as much as possible at the appropriate time, thereby allowing a business associated with the advertisement to enjoy various advertising effects as well.
Referring to
Referring to
Based on these functions, each viewer may acquire different streaming images through different number of times being clicked. This may induce viewers to spend money, and may allow the spent cost to flow to the broadcast jockey (allowing the service provider to acquire a portion of fee in the server apparatus), thereby having an effect of giving the broadcast jockey an incentive for making the broadcasting more enjoyable.
Referring to
The camera 820 is a device for photographing a subject, and an image photographed through the camera 820 is processed in the controller 840. The camera 820 may be present in the live broadcasting streaming image generating apparatus, or may be connected to the outside through an interface such as a High-Definition Multimedia Interface (HDMI).
The display unit 830 outputs the image processed through the controller 830. The display unit 830 may be implemented as a monitor, a touch screen, a TV receiver, or the like.
The controller 840 may include a broadcast application executing unit 842, a tracking information generating unit 844, an application unit 846, and a streaming image generating unit 848.
The broadcast application executing unit 842 installs and executes an application/program associated with the streaming based on the platform information received through the communication unit 810. In the case of broadcast streaming, the broadcast application executing unit 842 installs a broadcast application/program on the basis of the broadcast platform information, and executes the broadcast application/program by an input through the input unit 850. In this case, a program entry operation such as login may be performed.
The tracking information generating unit 844 generates tracking information according to the tracking algorithm included in the platform information and provides the tracking information to the application unit 846.
The application unit 846 performs a function of applying a sticker/effect. The application unit 846 outputs a set of appropriate sticker/effects through the display unit 830 based on at least one of the account of a user of the first client terminal that enters the broadcast application/program, the IP of the first client terminal, and the broadcast service contents and type. Then, the application unit 846 receives an input related to a user's selection through the input unit 850, and applies the selected sticker/effect to one area of the target subject or the display screen (first streaming image). In this case, the sticker/effect may be applied along the movement of the subject based on the tracking information.
The streaming image generating unit 848 processes the image photographed through the camera 820 to generate a basic streaming image. Here, encoding and decoding may be performed. Also, when the sticker/effect is applied, processing of the second streaming image to which the sticker/effect is applied may also be performed.
The input unit 850 is an interface for processing a user input such as a keyboard, a mouse, and a touch pad.
The memory 860 stores instructions (or directives) related to each function performed in the controller 840. The memory 860 includes instructions based on the broadcast platform information, and may store information received from the server apparatus or other devices through the communication unit 810.
The communication unit 910 is a device responsible for communication with the first client terminal and the second client terminal.
The controller 920, which is an element for managing and controlling an application, may include an account managing unit 921, a broadcast platform managing unit 923, a streaming image processing unit 925, an application determining unit 927, and an advertising revenue processing unit 929.
The account managing unit 921 manages information related to an account that logs into an application (e.g., a broadcast application). The account managing unit 921 may manage the personal history of each account as well as the history of broadcasting performed through each account. For example, the account managing unit 921 may manage information such as a broadcasting station opening time, an opening format, a handling content, and a service type opened through a specific user account, information related to admission to a broadcasting station opened by another user, and/or history of stickers/effects that are selected and applied.
The broadcast platform managing unit 923 manages the platform information (including broadcast platform information) of
The streaming image processing unit 925 processes the first streaming image and/or the second streaming image received from the first client terminal. The streaming image processing unit 925 may together receive basic streaming information (first streaming image) to which the sticker/effect is not applied from the first client terminal and streaming image (second streaming image) to which the sticker/effect is applied, and may independently process these first and second streaming images. For example, when the basic streaming image information is received, a signal associated with a request for applying a specific sticker may be received, and the received signal may be parsed, thereby applying the requested sticker to the basic streaming image and directly generating a streaming image to which a sticker is applied.
Also, the streaming image processing unit 925 individually processes a request associated with a sticker from the second client terminal to perform image processing such that different streaming images can be viewed in regard to the area of the sticker in different terminals.
The application determining unit 927 determines whether or not the sticker is applied, and whether or not the sticker for advertisement is applied. In the case of a sticker for advertisement, it is determined which business (or brand) is associated.
The advertising revenue processing unit 929 distributes advertising revenue associated with the sticker. In particular, in the case of a sticker for advertisement, the advertising revenue processing unit 929 may detect arbitrary actions of viewers (viewers admitted to the corresponding broadcasting station) who view the advertising sticker detects an arbitrary action linked to the corresponding brand, and may share a revenue according to the advertisement sales with a broadcast jockey and a live broadcast service providing server (and/or a sticker providing server (see
The input unit 930 is an interface for processing user inputs for changing various user settings associated with the broadcast application.
The memory 940 stores instructions related to each function performed by the controller 920. For example, the memory may include user history information associated with the account, use history information of the sticker, information of the sticker itself (information on whether it is a sticker for advertisement and which business and which link is associated with the sticker for advertisement).
Referring to
The sticker providing server 1040 may provide a face tracking development Software Development Kits (SDKs) to the live broadcasting server 1020, and may apply SDKs to app clients 1010 and 1030 or the web of the live broadcasting service. Then, the first client terminal 1010 opens a sticker inventory to a broadcast service used by a broadcast jockey, such that at least one of SDKs can be selectively used. For various live broadcasting services, various stickers stored in a sticker archive of the sticker providing server 1040 may be provided at the same time.
The sticker providing server 1040 may provide and operate stickers to media desired by advertisers like an ad network. Also, in addition to the advertising sticker (brand-affiliated sticker), the contents such as the background sound sources, the professional effects, the subtitle functions, and the intermediate advertisement used for broadcasting may be provided in real-time. The server 1030 may provide such information to the client terminals 1010 and 1030.
In order to implement this operation, the sticker providing server 1040 provides the face tracking SDK and the user use tracker to the live broadcasting server 1030, and the live broadcasting server 1030 transmits information related to the user use data and the function promotion to the sticker providing server 1040. Then, both may together share advertising revenue.
As described above, the sticker may be applied to both the web and the mobile environment, and the broadcast jockey may maximize the fun of broadcasting by displaying a screen on which a sticker is applied to a viewer who watches broadcasting on his/her broadcasting station. In particular, when a brand-affiliated sticker is utilized, viewers who view the brand-affiliated sticker sense any action that is linked to the corresponding brand, and thus a revenue according to the advertisement sales may be shared by broadcast jockey, a live broadcasting service provider (i.e., server 1020), and the sticker providing server 1040. In this case, a Cost Per Click (CPC) method, a Pay Per Click (PPC) method, a Cost Per Time (CPT) method, a Cost Per Mile (CPM) method, a Cost Per Millenium (CPM) method, a Cost Per Impression (CPI) method, and a Cost Per Conversion (CPCon) method regarding links related to the brand-affiliated sticker may be considered for the distribution of advertising revenue.
According to another embodiment of the present invention, when distributing the advertising revenue, the server apparatus may also distribute the revenue to accounts related to the broadcast performers appearing on the broadcasting through the first client terminal and/or accounts of the broadcast viewers. That is, when the advertising sticker is used for the broadcast performers, the information on the broadcast performer to which the sticker is applied may be acquired, and a portion of the advertising revenue may be distributed to the accounts associated with the acquired information. Also, when a viewer watches the advertising sticker of the streaming image, the viewer may leave a log and share a portion of the advertising revenue.
Referring to
Next, a mode for sticker creation is selected (S1130). Here, the mode for creation may include two modes, i.e., a first mode and a second mode.
The first mode is a mode for recognizing a subject such as a face, and associates a sticker design element generated by a user input with at least one of displayed subjects. In this case, the contour of the subject may be considered as an element determining the relationship with the sticker design element. The contour of the subject may be identified using pre-stored edge templates. For example, when a subject is a face, edge templates including the overall layout information of a plurality of components such as eyes, nose, and mouth in the face are stored in advance, and the contour of the face and the contours of the components in the face displayed on the displayed screen are matched with the pre-stored edge templates to first recognize the face and identify and store the contour information of the recognized face. In this case, information such as brightness, motion, color, and eye position estimation may be used to distinguish the face and the background. In this case, it is effective to analyze the main part of the face, discard the fine error, capture only the large feature, and compare with the stored face template. In the embodiment of the present invention, the edge template is not necessarily related to the face, but may be related to other parts of the human body such as arms, legs, hands, and the like.
In the first mode, due to the association with the subject, the sticker design element generated by a user in response to a motion of the subject and/or a change in the size of the subject may also move and change in size.
The second mode is a mode for designing to display a subject at a specific position on the display screen in consideration of the association with the displayed screen instead of the association with the subject. In the second mode, the sticker design element may be displayed at a fixed size at a predetermined position in the display screen without being affected by the motion of a subject. In some cases, in the second mode, a moving effect such as a change in size may also be given according to the setting of the user.
When the mode selection is completed, a user receives a user input for the sticker design through a user interface (e.g., a touch screen, a keyboard, a mouse, etc.) (S1140). The user input for the sticker design may be a text input, and may be an input through a drawing mode. The drawing mode may be referred to as a doodling mode. A sticker design element may be created by inputting a text input through typing using a keyboard, and a sticker may be created by inputting a line or a figure selected by a user through a mouse or a touch screen in a drawing mode.
According to an embodiment of the present invention, figures and images may be imported in addition to the text and drawings. The figures may include a still figure, an animation graphic, or a 3D subject, and the images may be imported from a camera or album and may be used as a sticker design element. The figures may include solid lines, triangles, squares, arrows, bent lines, half-moon shapes, clouds, hearts, mathematical expression-related figures, flowchart-related figure, and the like. These figures may be stored in a template in advance.
According to another embodiment of the present invention, other effects such as a sound and a functional figure capable of calling a specific animation may be utilized as sticker design elements.
The client terminal identifies in-screen coordinate information of each sticker design element to memorize the inputted sticker design element as a user-created sticker (S1150). The coordinate information is recognized by identifying the coordinates of pixels related to the sticker design element in the displayed screen. The identified coordinate information may be utilized as relationship information indicating an association with a display screen or a subject in accordance with a mode selected by a user.
After identifying the coordinates of the inputted sticker design element, it is determined whether the mode is the first mode or the second mode (S1160). The first mode (subject recognition mode) is allowed to have an association with the contour of the subject, and the second mode is allowed to have an association with the display screen. The switching between the first mode and the second mode is possible at any time before the sticker is stored. Accordingly, upon sensing of a mode change, all of the sticker design elements interpreted with the second mode may be interpreted with the first mode. That is, the sticker design elements analyzed through operation S1165 may be interpreted through operations S1170 to S1190, and vice versa.
If the first mode is selected in operation S1160, first, the contour of a subject is identified using the edge template (S1170). The terminal stores a figure (e.g., a rectangle) including all the sticker design elements as an image, and recognizes a subject in the stored image to identify the contour of the identified subject. In this case, there may be a plurality of subjects in the image. For example, the face may be a subject, and other parts of a person such as hand and arm may be a subject. The client terminal recognizes a subject based on a pre-stored edge template of the subject, and identifies the contour of the subject. In this case, coordinate information of the pixel related to the contour can be checked.
Then, relationship information for associating the contour of the identified subject with the sticker design element is generated based on the coordinate information of the sticker design element identified in operation S1150 (S1180). According to an embodiment of the present invention, the center point of the subject is searched based on the contour of the subject included in the stored image. Then, based on the distance from the center point to the contour of the subject, the relationship between the sticker design element and the subject may be grasped by considering the distance from the contour of the subject or the center point to the sticker design element. Then, the relationship information may be generated based on the grasped relationship.
According to another embodiment, in operation S1180, a point corresponding to the synchronization point is searched on the pre-stored edge template. The synchronization point, which is a point for synchronizing the contour of the actual subject currently displayed, the sticker design element and the pre-stored edge template, may be specified as a specific point on the contour of the subject. For example, a specific point on the contour of the subject having an association with the end of the sticker design element becomes a synchronization point. Alternatively, the start point and/or the end point of the sticker design element and the nearest point on the contour may become synchronization points. In some cases, a point at which the sticker design element and the contour meet each other may become a synchronization point. Thus, there are various algorithms for determining the synchronization point, and this may be received while being included in broadcast platform information received from the server apparatus. The synchronization point on the contour of the actual subject corresponds to a specific point on the edge template that is matched with the subject, and based thereon, may be appropriately applied to another subject to which the edge template is applied. The generated synchronization point may be considered as the relationship information.
When the relationship information is generated, the client terminal stores a user-created sticker in a local area based on the relationship information and the sticker design element (S1190). In this case, a user may instruct to perform storage by pressing a storage icon. That is, if it is determined that a user has created a preferred sticker to such an extent as to register as a sticker while creating user-created stickers as a fun factor through the process from operation S1110 to operation S1180, the user may register and save the sticker that is currently being implemented. That is, the storage of the sticker may also be selectively determined according to the preference of a broadcast jockey. Then, the stored user creation information and the streaming image to which the user creation sticker is applied are transmitted to the server apparatus. The server apparatus that receives the information and the image may broadcast the streaming image to which the user creation sticker is applied to a viewer client.
In this case, the sticker-related information that is stored and transmitted may include the relationship information, the information on the contour of the subject and each sticker design element coordinates, and the information on the sticker design element itself. Alternatively, related edge template information may be further stored. In this case, the relationship information may include synchronization point information (e.g., a specific point on the subject or a point on the edge template corresponding to the specific point) and information (e.g., distance information, etc.) indicating a relationship between the synchronization point and the sticker design element. The information on the sticker design element itself may include the type of the sticker design element (whether the sticker design element is text or drawing), and in case of drawing, may include information on the color, thickness, shape, etc. of the line forming the sticker design element.
If the second mode is selected in operation S1160, the procedure enters the second mode, and a user-created sticker is created and stored in consideration of the position in the display screen based on the coordinate information of the sticker design element (S1165). In this case, information related to the subject is not stored, and only the coordinate information in the display screen may be stored for association with the display screen.
Since the subject may continue to move and change in size during the photographing except a case where pre-stored pictures are used, the time point at which the contour of the subject and the pixel coordinates of the sticker design element are identified may be important. According to an embodiment of the present invention, the time point at which the contour of the subject and the coordinates of the sticker design element are identified may include a time point at which a user input for creating a sticker design element starts or a time point at which a user input is completed. The subject and the sticker design element are identified in accordance with the corresponding time point, and the sticker design element also changes corresponding to the change of the subject after the corresponding time point.
Referring to
Also, when the icon 1204 is selected, a text input is possible, and when the icon 1206 is selected, the user input of the drawing mode is possible. The embodiment of
In the drawing mode, a user may select the thickness and shape of a line through a menu 1220 for selecting a basic line for drawing on the left side of the screen. A user may arbitrarily change the thickness and shape of the line using the menu 1220 during the drawing. For example, in regard to the thickness of line, a menu for selecting a plurality of thicknesses from the thickest line at the top to the thinnest line at the bottom may be provided. In addition, a menu (a menu for selecting a pen type or a brush type) for selecting the shape of line may include options such as a triangular shape line, a rectangular shape line, a circular shape line, a heart shape line, a two-colored line, a line with a solid effect, a line with a shadow effect, and the like.
Also, a menu 1230 for selecting a line color may be provided at the bottom of the screen. A user may perform drawing by selecting one of a plurality of provided colors.
In the embodiment of
Hereinafter, a method of recognizing these sticker design elements 1210-1 to 1210-7 and storing the whole user creation sticker by the client terminal will be described. The client terminal individually recognizes the sticker design elements 1210-4 to 1210-7 and associates each sticker design element with the subjects. That is, the subjects associated with the sticker design elements 1210-4 through 1210-7 may be different. The sticker design element may be recognized as a figure formed of one connected line, or as a text unit typed at a time.
Referring to
Based on the generated relationship information, since the distance between the center point and the contour of the face is changed when the face contour is changed (e.g., the shape of the face changes or the size of the face changes), the distances to each of the sticker design elements 1210-1 to 1210-7 may also be changed.
Referring to
In this embodiment, points 1260-1 and 1260-2, which are points on the face contour 1212 closest to the start point 1250-1 and the end point 1250-2, may be detected as the synchronization points.
After the client terminal detects the synchronization points 1260-1 and 1260-2, the client terminal calculates distances d1′ and d2′ between the face contour 1212 and the start point 1250-1 and the end point 1250-2, and stores the distances d1′ and d2′ as relationship information. That is, even if applied to another face, the created sticker may be reproduced in such a manner that the sticker design element is created at the distances d1 ‘and d2’ away from the synchronization points. In this case, if the size of the face changes, the distances d1′ and d2′ may also be reduced or increased in proportion to the changed size. That is, even if the shape or scale of the face is changed, the sticker design element may be appropriately applied by searching for the coordinates corresponding to the coordinates of the synchronization points of the current face from the changed face.
In addition, the shape and scale of the sticker design element may be appropriately resized by identifying the coordinates of the whole pixels, on which the sticker design elements are displayed, to recognize and store the shape and by responding to changes in the size and shape (tilt, etc.) of the subject later. For example, when the size of the subject is changed, the size as well as the coordinates of the synchronization point may also be changed so as to correspond to the size of the changed subject while maintaining the shape of the sticker design element.
Next, in the case of the sticker design element 1210-3, the relationship with the face contour 1212 may also be considered, but the relationship with the nose contour 1214 is also considered. Accordingly, in this case, a plurality of synchronization points may be detected with respect to the contours 1212 and 1214 of a plurality of subjects, and relationship information with the detected plurality of synchronization points may be generated and stored.
According to another embodiment of the present invention, a sticker design element that covers the whole of the face may be created. In this case, considering the relationship with the face contour 1212, when the face is displayed small, the size of the sticker design element also becomes small enough to cover the reduced face, and when the face is displayed large, the size of the sticker design element also becomes large enough to cover the enlarged face. The size and shape of such a sticker design element and the position change of the synchronization points may be automatically changed in response to a change in the size and shape of the subject being photographed in real-time.
The tracking method based on the relationship information of the subject and the sticker design elements according to an embodiment, which has been described in
According to another embodiment of the present invention, in a method of providing a live broadcasting service, a server stores a hash tag for a sticker applied to a streaming image in a database, and automatically applies the hash tag stored in the database at a time of sharing the corresponding image. Thus, it is possible to easily retrieve the image to which an advertising sticker is applied, thereby estimating the use frequency or the exposure frequency of the advertising sticker. Naturally, a watermark for the corresponding sticker may be inserted into the sticker according to an embodiment of the present invention, and the use frequency of the sticker may also be estimated using the inserted watermark code.
Although the present invention has been described with reference to the drawings and embodiments, it can be understood that the scope of protection of the present invention is not limited by the drawings and embodiments, and it will be understood by those skilled in the art that various modifications and changes can be made without departing from the spirit and scope of the present invention described in the appended claims.
According to the live streaming image generating method and apparatus, the live streaming service providing method and apparatus, and the live streaming system of the present invention, an additional profit can be created by utilizing a brand-affiliated sticker through live broadcasting.
Also, advertisement sales from various brands and advertisers who want to advertise through live broadcasting are effectively shared by a live broadcasting provider, BJ, and a sticker provider.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2017-0018551 | Feb 2017 | KR | national |
| 10-2018-0015578 | Feb 2018 | KR | national |
