Patent Grant
10943404
This application is a National Phase Entry of PCT International Application No. PCT/KR2017/014238, which was filed on Dec. 6, 2017 and claims a priority to Korean Patent Application No. 10-2016-0164913, which was filed on Dec. 6, 2016, the contents of which are incorporated herein by reference.
Various embodiments disclosed herein relate to technology of content output based on object recognition.
With the remarkable development of mobile communication systems, an electronic device may share various types of content with other electronic devices, which makes it possible for users to experience more various types of content. For example, an electronic device may receive or generate and output augmented reality content that is a combination of an actual object in a real world and a virtual image, thereby allowing a user to experience a stereoscopic content.
The output of the augmented reality content may be realized by photographing a specific object for which the augmented reality content is supported based on execution of a specified application program. To output the augmented reality content, it may be required to determine whether or not the augmented reality content is supported for the photographed object. However, the user cannot intuitively determine whether or not the the augmented reality content is supported for an arbitrary object. When the photographed object is not related to the augmented reality content, execution of the above-described application program or object photographing may be meaningless and may cause unnecessary power consumption of the electronic device.
Various embodiments disclosed in the disclosure provide a content output method capable of supporting easy recognition of an object related to content output and an electronic device supporting the same.
In addition, various embodiments disclosed in the disclosure provide a content output method capable of simplifying a functional operation or a user input operation of an electronic device involved in content output and an electronic device supporting the same.
An electronic device according to an embodiment includes a memory that stores information related to content, a display that outputs a screen associated with the content, a camera module that photographs a surrounding environment of the electronic device, and a processor that is electrically connected to the memory, the display, and the camera module.
According to an embodiment, the processor may activate the camera module based on a change in physical quantity acting on the electronic device in a state where at least a part of a screen rea of the display is turned off, and output a first display object associated with an arbitrary object on the display of which the at least a part of the screen area is turned off when information related to the arbitrary object recognized by the activated camera module is present in the memory.
According to the various embodiments, a user may intuitively recognize an object related to the content output based on a display object which is output at low power in outputting content based on object recognition.
In addition, various effects may be provided that are directly or indirectly understood through the disclosure.
Hereinafter, various embodiments of the present disclosure are disclosed with reference to the accompanying drawings. However, the present disclosure is not intended to be limited by the various embodiments of the present disclosure to a specific embodiment and it is intended that the present disclosure covers all modifications, equivalents, and/or alternatives of the present disclosure provided they come within the scope of the appended claims and their equivalents. With respect to the descriptions of the accompanying drawings, like reference numerals refer to like elements.
The term “include,” “comprise,” and “have”, or “may include,” or “may comprise” and “may have” used herein indicates disclosed functions, operations, or existence of elements but does not exclude other functions, operations or elements.
For example, the expressions “A or B,” or “at least one of A and/or B” may indicate A and B, A, or B. For instance, the expression “A or B” or “at least one of A and/or B” may indicate (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.
The terms such as “1st,” “2nd,” “first,” “second,” and the like used herein may refer to modifying various different elements of various embodiments of the present disclosure, but are not intended to limit the elements. For instance, “a first user device” and “a second user device” may indicate different user devices regardless of order or importance. For example, a first component may be referred to as a second component and vice versa without departing from the scope of the present disclosure.
In various embodiments of the present disclosure, it is intended that when a component (for example, a first component) is referred to as being “operatively or communicatively coupled with/to” or “connected to” another component (for example, a second component), the component may be directly connected to the other component or connected through another component (for example, a third component). In various embodiments of the present disclosure, it is intended that when a component (for example, a first component) is referred to as being “directly connected to” or “directly accessed” another component (for example, a second component), another component (for example, a third component) does not exist between the component (for example, the first component) and the other component (for example, the second component).
The expression “configured to” used in various embodiments of the present disclosure may be interchangeably used with “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of” according to the situation, for example. The term “configured to” may not necessarily indicate “specifically designed to” in terms of hardware. Instead, the expression “a device configured to” in some situations may indicate that the device and another device or part are “capable of” For example, the expression “a processor configured to perform A, B, and C” may indicate a dedicated processor (for example, an embedded processor) for performing a corresponding operation or a general purpose processor (for example, a central processing unit (CPU) or application processor (AP)) for performing corresponding operations by executing at least one software program stored in a memory device.
Terms used in various embodiments of the present disclosure are used to describe certain embodiments of the present disclosure, but are not intended to limit the scope of other embodiments. The terms of a singular form may include plural forms unless they have a clearly different meaning in the context. Otherwise, all terms used herein may have the same meanings that are generally understood by a person skilled in the art. In general, terms defined in a dictionary should be considered to have the same meanings as the contextual meaning of the related art, and, unless clearly defined herein, should not be understood differently or as having an excessively formal meaning. In any case, even the terms defined in the present specification are not intended to be interpreted as excluding embodiments of the present disclosure.
An electronic device according to various embodiments of the present disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video telephone, an electronic book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a Motion Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a mobile medical device, a camera, or a wearable device. The wearable device may include at least one of an accessory-type device (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, a head-mounted device (HMD)), a textile- or clothing-integrated-type device (e.g., an electronic apparel), a body-attached-type device (e.g., a skin pad or a tattoo), or a bio-implantable-type device (e.g., an implantable circuit).
In some various embodiments of the present disclosure, an electronic device may be a home appliance. The smart home appliance may include at least one of, for example, a television (TV), a digital video/versatile disc (DVD) player, an audio, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, an air cleaner, a set-top box, a home automation control panel, a security control panel, a television (TV) box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console (e.g., Xbox™ or PlayStation™), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.
In other various embodiments of the present disclosure, an electronic device may include at least one of various medical devices (e.g., various portable medical measurement devices (e.g., a blood glucose measuring device, a heart rate measuring device, a blood pressure measuring device, a body temperature measuring device, or the like), a magnetic resonance angiography (MRA) device, a magnetic resonance imaging (MRI) device, a computed tomography (CT) device, a scanner, an ultrasonic device, or the like), a navigation device, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), a vehicle infotainment device, electronic equipment for vessels (e.g., a navigation system, a gyrocompass, or the like), avionics, a security device, a head unit for a vehicle, an industrial or home robot, an automatic teller machine (ATM), a point of sales (POS) device of a store, or an Internet of things (IoT) device (e.g., a light bulb, various sensors, an electric or gas meter, a sprinkler, a fire alarm, a thermostat, a streetlamp, a toaster, exercise equipment, a hot water tank, a heater, a boiler, or the like).
According to various embodiments of the present disclosure, an electronic device may include at least one of a part of furniture or a building/structure, an electronic board, an electronic signature receiving device, a projector, or a measuring instrument (e.g., a water meter, an electricity meter, a gas meter, a wave meter, or the like). An electronic device may be one or more combinations of the above-mentioned devices. An electronic device according to some various embodiments of the present disclosure may be a flexible device. An electronic device according to an embodiment of the present disclosure is not limited to the above-mentioned devices, and may include new electronic devices with the development of new technology.
Hereinafter, an electronic device according to various embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. The term “user” used herein may refer to a person who uses an electronic device or may refer to a device (e.g., an artificial intelligence electronic device) that uses an electronic device.
Referring to
In one embodiment, the electronic device 100 may receive information related to specified content from the at least one external device 1, 2, 3 and/or 4 based on the network 200. Alternatively, the electronic device 100 may transmit the received information to a specific external device, or may generate specified content and transmit information related thereto to the at least one external device 1, 2, 3 and/or 4. The information related to the content (hereinafter referred to as “content-related information”) may include, for example, additional information on a specific object or a specific external device (e.g., outline, shape, size, color, or depth information of an object or an external device). Alternatively, the content-related information may include two-dimensional or three-dimensional text, images, or videos capable of being output in correspondence with recognition (or identification) of the photographing of the specific object or the specific external device.
In one embodiment, the electronic device 100 may store the content-related information received from the at least one external device 1, 2, 3 and/or 4 in the form of database. The electronic device 100 may refer to the database in operation of recognizing the specific object or the specific external device involved in processing of outputting the content. For example, the electronic device 100 may determine whether information corresponding to the recognized object or external device is present in the database, and, when the information is present in the database, perform outputting of the content associated with the object or the external device according to a series of specified processes. In the above-described operation, the electronic device 100 may control performance of at least a part of a function of a specific internal component so as to reduce power consumption due to functional operation of the specific internal component (e.g., a display). For example, the electronic device 100 may control at least some pixels of the display to process the output of the shooting screen with low power, in the shooting recognition operation on the object or the external device. With regard to the output of content based on recognition of an object or an external device, functional operations of components of the electronic device 100 and various embodiments related thereto will be described.
Referring to
The communication module 110 may support communication between the electronic device 100 and at least one external device (the at least one external device 1, 2, 3, and/or 4 in
The memory 120 may store at least one piece of data related to the operation of the electronic device 100, or may store at least one instruction related to the functional operation of components of the electronic device 100. For example, the memory 120 may store content-related information received from the at least one external device 1, 2, 3 and/or 4. In one embodiment, the content-related information may include additional information for a specific object or a specific external device (e.g., outline, shape, size, color or depth information of an object or an external device) and/or two-dimensional or three-dimensional text, images or videos that may be output corresponding to recognition (or identification) of photographing for the specific object or the specific external device. The memory 120 may build database for the content-related information under the control of the processor 160. For example, the memory 120 may build the database by mapping the object or the external device to the two-dimensional or three-dimensional text, images, or videos corresponding to the object or the external device. In one embodiment, the memory 120 may store at least one application program associated with the output of the content based on recognition of the object. The at least one application program may be installed, for example, in a preloaded form upon manufacturing of the electronic device 100 or may be downloaded in a third party form from an online marketplace.
The camera module 130 may be mounted in a portion of the electronic device 100 to photograph images (e.g., still image or video) of surrounding environment of the electronic device 100. In one embodiment, a plurality of camera modules 130 may be provided, and the plurality of camera modules 130 may be disposed on the electronic device 100 so as to photograph different areas or directions. For example, the plurality of camera modules 130 may be disposed on portions opposite to one another (e.g., portions of the front and rear surface of the electronic device 100) so as to enable photographing of the front and rear side of the electronic device 100. The image photographed by the camera module 130 may be stored in the memory 120 under the control of the processor 160. In various embodiments, the operation of the camera module 130 may be triggered and activated or deactivated by a specific functional operation of the sensor module 140.
The sensor module 140 may perform sensing on the electronic device 100 or the surrounding environment of the electronic device 100. In one embodiment, the sensor module 140 may include at least one of an acceleration sensor, a proximity sensor, or an image sensor. The acceleration sensor may output an acceleration value in the direction of three axes (e.g., X, Y, and Z) in accordance with the movement or positional change of the electronic device 100 and transfer the acceleration value to the processor 160. The proximity sensor may sense an object or an external device adjacent to the electronic device 100 (or approaching the electronic device 100) by emitting, for example, light of a specified wavelength band (for example, infrared light) toward a photographing zone or photographing direction of the camera module 130 and receiving the light reflected and returned. The image sensor may detect at least one object (or an external device) on a photographed image of the camera module 130 based on, for example, an outline filter.
The display 150 may output various screens corresponding to user control or specified scheduling information. For example, the display 150 may display a specified display object (e.g., a text message) to provide a notification to a user when the electronic device 100 receives content-related information from the at least one external device 1, 2, 3 and/or 4. Alternatively, the display 150 ma output a playback screen or a display screen of content when a specified condition (e.g., a condition that an object or an external device related to the content is recognized through image photographing) is satisfied.
The processor 160 may be operatively or electrically coupled to other components of the electronic device 100 to perform control, communication operations, or data processing, or the like on the components. For example, the processor 160 may recognize (or identify) a specific object or a specific external device photographed by the camera module 130 by referring to database on the memory 120, and process the output of the content associated with the specific object or the specific external device based on the recognized result. The processor 160 may also perform turn-on or -off control on at least some of a plurality of pixels included in the display 150. For example, the processor 160 may turn off at least some of the plurality of pixels to support low power operation of the electronic device 100 based on minimal pixel operation when the electronic device 100 is to perform a particular functional operation (e.g., receiving content-related information from the at least one external device 1, 2, 3, and/or 4).
In one embodiment, the processor (160 of
In one embodiment, the first display object 10 may be temporarily eliminated after passage of a time specified in association with the low power operation of the electronic device 100, and may be redisplayed on the screen area 151 of the display 150 in response to, for example, a user operation on a home button area 101 of the electronic device 100. When a user input (e.g., touch) is applied to the first display object 10, the first display object 10 may be completely eliminated and the screen area 151 of the display 150 may be in a black screen state. Although not illustrated, in various embodiments, the first display object 10 may include outline information (e.g., an object name, an object location, or the like) for a specific object (or a specific external device) contained in the content-related information.
Referring to
In one embodiment, the sensor module 140 may further include an illuminance sensor, in connection with the prevention of unintentional activation of the camera module 130. For example, when the electronic device 100 is housed in a user's garment or bag or the like, movement or positional change of the electronic device 100 may occur depending on the movement of the user's body. In this regard, when the movement or the positional change of the electronic device 100 occurs, the illuminance sensor may sense the ambient illuminance of the electronic device 100 and transmit the sensed illuminance value to the processor 160. The processor 160 may exclude the activation of the camera module 130 when the illuminance value is less than the specified magnitude. In various embodiments, the activation condition of the camera module 130 described above may be changed through the system settings of the electronic device 100. For example, the activation condition of the camera module 130 may be replaced by a user gesture (e.g., hovering over a specified time or more) with respect to the electronic device 100.
Referring to
In one embodiment, the sensor module 140 may detect the object 20 on a photographed image (e.g., still image or video) based on the image sensor in response to the photographing of the object 20 of the camera module 130. The processor 160 may obtain information on the detected object 20 (e.g., the outline, shape, size, color, or depth information of the object 20) and determine whether information on the object 20 is present in the database of the memory 120 (
Referring to
Referring to
Referring to
Referring to
In one embodiment, the sensor module (140 in
In one embodiment, the processor 160 may switch a screen including the fourth display object 60 to an output screen of content associated with the photographed object 50, corresponding to specified scheduling information or user control. For example, when a specified time elapses from the output of the fourth display object 60 or a user input (e.g., touch) is applied onto the fourth display object 60, the processor 160 may perform screen switching. Referring to
Referring to
In response to the output of the display object, the user may search for the object 70 corresponding to the display object and allow the camera module (130 in
In one embodiment, information on the object 70 (e.g., shape information or state information of the object 70, or content information related to the object 70) of the object 70 may be included in the signal received from the object 70. The processor 160 may output a fifth display object 80 (e.g., an outline of at least a portion of the object 70) on the screen area 151 of the display 150 based on information received from the object 70. For example, the processor 160 may turn on at least some pixels (e.g., pixels corresponding to the area for outputting the fifth display object 80) included in the display 150 to output the fifth display object 80.
Referring to
In various embodiments, the embodiment described with reference to
An electronic device according to various embodiments may include a memory that stores information related to content, a display that outputs a screen associated with the content, a camera module that photographs a surrounding environment of the electronic device, and a processor that is electrically connected to the memory, the display, and the camera module.
According to various embodiments, the processor may activate the camera module based on a change in physical quantity acting on the electronic device in a state where at least a part of a screen rea of the display is turned off, and output a first display object associated with an arbitrary object on the display of which the at least a part of the screen area is turned off when information related to the arbitrary object recognized by the activated camera module is present in the memory.
According to various embodiments, the processor may be configured to output a specified second display object on the display in which the at least a part of the screen area is turned off, when the information related to the content is received from an external device.
According to various embodiments, the processor may eliminate the second display object when a user input is applied to the second display object and turn off the at least a part of the screen area of the display.
According to various embodiments, the electronic device may further include a sensor module that senses a change in physical quantity of the electronic device.
According to various embodiments, the processor may activate the camera module when the change in the physical quantity of the electronic device is not detected by the sensor module for a specified time after an acceleration value corresponding to the change in the physical quantity of the electronic device is output at a specified first magnitude or greater or when the acceleration value is output at less than the first magnitude.
According to various embodiments, the sensor module may include an illuminance sensor that senses ambient illuminance of the electronic device.
According to various embodiments, the processor may deactivate the camera module when an ambient illuminance value of the electronic device output by the illuminance sensor is less than a specified magnitude.
According to various embodiments, the processor may turn off the at least a part of the screen area of the display when the activated camera module starts to photograph the arbitrary object.
According to various embodiments, the processor may exclude output of the first display object and deactivate the activated camera module when no information on the arbitrary object recognized by the activated camera module is present in the memory.
According to various embodiments, the processor may switch a screen of the display including the first display object to a screen associated with lock release of the electronic device when a specified time elapsed from output of the first display object or a user input is applied to the first display object.
According to various embodiments, the processor may switch a screen of the display including the first display object to a content output screen associated with the arbitrary object when a specified time elapsed from output of the first display object or a user input is applied to the first display object.
Referring to
In operation 603, a processor (160 in
In operation 605, the user may search for an object or an external device in the surrounding environment of the electronic device in response to output of the first display object. In this regard, the user may allow the camera module (130 in
In operation 609, the activated camera module may initiate photographing of an image (e.g., a still image or a video) of the focused object. In this operation, the processor may perform a turn-off control for a plurality of pixels included in the display at the same time as the activation of the camera module or within a specified time from the activation. Accordingly, the screen area of the display may be displayed in a black screen state, and the preview display of the object photographed by the camera module may be excluded.
In operation 611, the sensor module of the electronic device may detect an object on the photographed image based on an image sensor, and the processor may obtain object information (e.g., outline, shape, size, color or depth information of the object). The processor may determine whether information on the object is present in database of a memory.
When the information on the object is present in the database in operation 613, in operation 615, the processor may display, for example, a second display object for the photographed object (e.g., an outline for at least a portion of the object) on the display area of the display. For example, the processor may output the second display object by controlling the display which is in a black screen state (e.g., turning on at least one pixel corresponding to an output area of the second display object).
In operation 617, when a specified time elapses from the output of the second display object or a user input (e.g., touch) is applied to the second display object, the processor may switch a screen including the second display object to an output screen of content associated with the object (e.g., a screen including a real image of the object and a three-dimensional image or video containing at least one virtual image related to the object or a screen including two-dimensional text or image representing object information).
In operation 613, when no information on the object is present in the database, the processor may exclude the output of the second display object and maintain the display screen in a black screen state. In addition, it is possible to deactivate the camera module which is in an active state.
According to the embodiments described above, a content output method in an electronic device may include turning off at least a part of a screen area of a display, activating a camera module based on a change in physical quantity acting on the electronic device, and outputting a first display object associated with an arbitrary object on the display of which the at least a part of the screen area is turned off when information related to the arbitrary object recognized by the activated camera module is present in a memory.
According to various embodiments, the content output method may further include receiving content-related information from an external device.
According to various embodiments, the receiving of the content-related information may include outputting a specified second display object on the display in which the at least a part of the screen area is turned off.
According to various embodiments, the receiving of the content-related information further include eliminating the second display object when a user input is applied to the second display object and turning off at least a part of the screen area of the display.
According to various embodiments, the activating of the camera module may include activating the camera module when a change in physical quantity of the electronic device is not detected for a specified time after an acceleration value corresponding to the change in the physical quantity of the electronic device is output at a specified first magnitude or greater or when the acceleration value is output at less than the first magnitude.
According to various embodiments, the activating of the camera module may include excluding the activation of the camera module when an ambient illuminance value of the electronic device is less than a specified magnitude.
According to various embodiments, the activating of the camera module includes turning off the at least a part of the screen area of the display when the activated camera module starts to photograph the arbitrary object.
According to various embodiments, the outputting of the first display object may include excluding output of the first display object when no information on the arbitrary object recognized by the activated camera module is present in the memory.
According to various embodiments, the outputting of the first display object may include deactivating the activated camera module when no information on the arbitrary object recognized by the activated camera module is present in the memory.
According to various embodiments, the outputting of the first display object may include switching a screen of the display including the first display object to a screen associated with lock release of the electronic device when a specified time elapsed from output of the first display object or a user input related to the first display object is applied.
According to various embodiments, the outputting of the first display object may include switching a screen of the display including the first display object to a content output screen associated with the arbitrary object when a specified time elapsed from output of the first display object or a user input is applied to the first display object.
An electronic device 701 in a network environment 700 according to various embodiments of the present disclosure will be described with reference to
The bus 710 may include a circuit for connecting the above-mentioned elements 710 to 770 to each other and transferring communications (e.g., control messages and/or data) among the above-mentioned elements.
The processor 720 may include at least one of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 720 may perform data processing or an operation related to communication and/or control of at least one of the other elements of the electronic device 701.
The memory 730 may include a volatile memory and/or a nonvolatile memory. The memory 730 may store instructions or data related to at least one of the other elements of the electronic device 701. According to an embodiment of the present disclosure, the memory 730 may store software and/or a program 740. The program 740 may include, for example, a kernel 741, a middleware 743, an application programming interface (API) 745, and/or an application program (or an application) 747. At least a portion of the kernel 741, the middleware 743, or the API 745 may be referred to as an operating system (OS).
The kernel 741 may control or manage system resources (e.g., the bus 710, the processor 720, the memory 730, or the like) used to perform operations or functions of other programs (e.g., the middleware 743, the API 745, or the application program 747). Furthermore, the kernel 741 may provide an interface for allowing the middleware 743, the API 745, or the application program 747 to access individual elements of the electronic device 701 in order to control or manage the system resources.
The middleware 743 may serve as an intermediary so that the API 745 or the application program 747 communicates and exchanges data with the kernel 741.
Furthermore, the middleware 743 may handle one or more task requests received from the application program 747 according to a priority order. For example, the middleware 743 may assign at least one application program 747 a priority for using the system resources (e.g., the bus 710, the processor 720, the memory 730, or the like) of the electronic device 701. For example, the middleware 743 may handle the one or more task requests according to the priority assigned to the at least one application, thereby performing scheduling or load balancing with respect to the one or more task requests.
The API 745, which is an interface for allowing the application 747 to control a function provided by the kernel 741 or the middleware 743, may include, for example, at least one interface or function (e.g., instructions) for file control, window control, image processing, character control, or the like.
The input/output interface 750 may serve to transfer an instruction or data input from a user or another external device to (an)other element(s) of the electronic device 701. Furthermore, the input/output interface 750 may output instructions or data received from (an)other element(s) of the electronic device 701 to the user or another external device.
The display 760 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display 760 may present various content (e.g., a text, an image, a video, an icon, a symbol, or the like) to the user. The display 760 may include a touch screen, and may receive a touch, gesture, proximity or hovering input from an electronic pen or a part of a body of the user.
The communication interface 770 may set communications between the electronic device 701 and an external device (e.g., a first external electronic device 702, a second external electronic device 704, or a server 706). For example, the communication interface 770 may be connected to a network 762 via wireless communications or wired communications so as to communicate with the external device (e.g., the second external electronic device 704 or the server 706).
The wireless communications may employ at least one of cellular communication protocols such as long-term evolution (LTE), LTE-advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM). The wireless communications may include, for example, a short-range communications 764. The short-range communications may include at least one of wireless fidelity (Wi-Fi), Bluetooth, near field communication (NFC), magnetic stripe transmission (MST), or GNSS.
The MST may generate pulses according to transmission data and the pulses may generate electromagnetic signals. The electronic device 701 may transmit the electromagnetic signals to a reader device such as a POS (point of sales) device. The POS device may detect the electromagnetic signals by using a MST reader and restore data by converting the detected electromagnetic signals into electrical signals.
The GNSS may include, for example, at least one of global positioning system (GPS), global navigation satellite system (GLONASS), BeiDou navigation satellite system (BeiDou), or Galileo, the European global satellite-based navigation system according to a use area or a bandwidth. Hereinafter, the term “GPS” and the term “GNSS” may be interchangeably used. The wired communications may include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), plain old telephone service (POTS), or the like. The network 762 may include at least one of telecommunications networks, for example, a computer network (e.g., local area network (LAN) or wide area network (WAN)), the Internet, or a telephone network.
The types of the first external electronic device 702 and the second external electronic device 704 may be the same as or different from the type of the electronic device 701. According to an embodiment of the present disclosure, the server 706 may include a group of one or more servers. A portion or all of operations performed in the electronic device 701 may be performed in one or more other electronic devices (e.g., the first electronic device 702, the second external electronic device 704, or the server 706). When the electronic device 701 should perform a certain function or service automatically or in response to a request, the electronic device 701 may request at least a portion of functions related to the function or service from another device (e.g., the first electronic device 702, the second external electronic device 704, or the server 706) instead of or in addition to performing the function or service for itself. The other electronic device (e.g., the first electronic device 702, the second external electronic device 704, or the server 706) may perform the requested function or additional function, and may transfer a result of the performance to the electronic device 701. The electronic device 701 may use a received result itself or additionally process the received result to provide the requested function or service. To this end, for example, a cloud computing technology, a distributed computing technology, or a client-server computing technology may be used.
Referring to
The processor 810 may drive, for example, an operating system (OS) or an application program to control a plurality of hardware or software components connected thereto and may process and compute a variety of data. The processor 810 may be implemented with, for example, a system on chip (SoC). According to an embodiment of the present disclosure, the processor 810 may include a graphic processing unit (GPU) (not shown) and/or an image signal processor (not shown). The processor 810 may include at least some (e.g., a cellular module 821) of the components shown in
The communication module 820 may have the same or similar configuration to the communication interface 770 of
The cellular module 821 may provide, for example, a voice call service, a video call service, a text message service, or an Internet service, and the like through a communication network. According to an embodiment of the present disclosure, the cellular module 821 may identify and authenticate the electronic device 801 in a communication network using the SIM 828 (e.g., a SIM card). According to an embodiment of the present disclosure, the cellular module 821 may perform at least part of functions which may be provided by the processor 810. According to an embodiment of the present disclosure, the cellular module 821 may include a communication processor (CP).
The Wi-Fi module 822, the BT module 823, the GNSS module 824, the NFC module 825, or the MST module 826 may include, for example, a processor for processing data transmitted and received through the corresponding module. According to various embodiments of the present disclosure, at least some (e.g., two or more) of the cellular module 821, the Wi-Fi module 822, the BT module 823, the GNSS module 824, the NFC module 825, or the MST module 826 may be included in one integrated chip (IC) or one IC package.
The RF module 827 may transmit and receive, for example, a communication signal (e.g., an RF signal). Though not shown, the RF module 827 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, or a low noise amplifier (LNA), or an antenna, and the like. According to another embodiment of the present disclosure, at least one of the cellular module 821, the Wi-Fi module 822, the BT module 823, the GNSS module 824, the NFC module 825, or the MST module 826 may transmit and receive an RF signal through a separate RF module.
The SIM 828 may include, for example, a card which includes a SIM and/or an embedded SIM. The SIM 828 may include unique identification information (e.g., an integrated circuit card identifier (ICCID)) or subscriber information (e.g., an international mobile subscriber identity (IMSI)).
The memory 830 (e.g., a memory 730 of
The external memory 834 may include a flash drive, for example, a compact flash (CF), a secure digital (SD), a micro-SD, a mini-SD, an extreme digital (xD), a multimedia card (MMC), or a memory stick, and the like. The external memory 834 may operatively and/or physically connect with the electronic device 801 through various interfaces.
The security module 836 may be a module which has a relatively higher secure level than the memory 830 and may be a circuit which stores secure data and guarantees a protected execution environment. The security module 836 may be implemented with a separate circuit and may include a separate processor. The security module 836 may include, for example, an embedded secure element (eSE) which is present in a removable smart chip or a removable SD card or is embedded in a fixed chip of the electronic device 801. Also, the security module 836 may be driven by an OS different from the OS of the electronic device 801. For example, the security module 836 may operate based on a java card open platform (JCOP) OS.
The sensor module 840 may measure, for example, a physical quantity or may detect an operation state of the electronic device 801, and may convert the measured or detected information to an electric signal. The sensor module 840 may include at least one of, for example, a gesture sensor 840A, a gyro sensor 840B, a barometric pressure sensor 840C, a magnetic sensor 840D, an acceleration sensor 840E, a grip sensor 840F, a proximity sensor 840G, a color sensor 840H (e.g., red, green, blue (RGB) sensor), a biometric sensor 840I, a temperature/humidity sensor 840J, an illumination sensor 840K, or an ultraviolet (UV) sensor 840M. Additionally or alternatively, the sensor module 840 may further include, for example, an e-nose sensor (not shown), an electromyography (EMG) sensor (not shown), an electroencephalogram (EEG) sensor (not shown), an electrocardiogram (ECG) sensor (not shown), an infrared (IR) sensor (not shown), an iris sensor (not shown), and/or a fingerprint sensor (not shown), and the like. The sensor module 840 may further include a control circuit for controlling at least one or more sensors included therein. According to various embodiments of the present disclosure, the electronic device 801 may further include a processor configured to control the sensor module 840, as part of the processor 810 or to be independent of the processor 810. While the processor 810 is in a sleep state, the electronic device 801 may control the sensor module 840.
The input device 850 may include, for example, a touch panel 852, a (digital) pen sensor 854, a key 856, or an ultrasonic input device 858. The touch panel 852 may use at least one of, for example, a capacitive type, a resistive type, an infrared type, or an ultrasonic type. Also, the touch panel 852 may further include a control circuit. The touch panel 852 may further include a tactile layer and may provide a tactile reaction to a user.
The (digital) pen sensor 854 may be, for example, part of the touch panel 852 or may include a separate sheet for recognition. The key 856 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 858 may allow the electronic device 801 to detect a sound wave using a microphone (e.g., a microphone 888) and to verify data through an input tool generating an ultrasonic signal.
The display 860 (e.g., a display 760 of
The interface 870 may include, for example, a high-definition multimedia interface (HDMI) 872, a universal serial bus (USB) 874, an optical interface 876, or a D-subminiature 878. The interface 870 may be included in, for example, the communication interface 770 shown in
The audio module 880 may convert a sound and an electric signal in dual directions. At least part of components of the audio module 880 may be included in, for example, an input and output interface 750 (or a user interface) shown in
The camera module 891 may be a device which captures a still image and a moving image. According to an embodiment of the present disclosure, the camera module 891 may include one or more image sensors (not shown) (e.g., a front sensor or a rear sensor), a lens (not shown), an image signal processor (ISP) (not shown), or a flash (not shown) (e.g., an LED or a xenon lamp).
The power management module 895 may manage, for example, power of the electronic device 801. According to an embodiment of the present disclosure, though not shown, the power management module 895 may include a power management integrated circuit (PMIC), a charger IC or a battery or fuel gauge. The PMIC may have a wired charging method and/or a wireless charging method. The wireless charging method may include, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic method, and the like. An additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier, and the like may be further provided. The battery gauge may measure, for example, the remaining capacity of the battery 896 and voltage, current, or temperature thereof while the battery 896 is charged. The battery 896 may include, for example, a rechargeable battery or a solar battery.
The indicator 897 may display a specific state of the electronic device 801 or part (e.g., the processor 810) thereof, for example, a booting state, a message state, or a charging state, and the like. The motor 898 may convert an electric signal into mechanical vibration and may generate vibration or a haptic effect, and the like. Though not shown, the electronic device 801 may include a processing unit (e.g., a GPU) for supporting a mobile TV. The processing unit for supporting the mobile TV may process media data according to standards, for example, a digital multimedia broadcasting (DMB) standard, a digital video broadcasting (DVB) standard, or a MediaFLO™ standard, and the like.
Each of the above-mentioned elements of the electronic device according to various embodiments of the present disclosure may be configured with one or more components, and names of the corresponding elements may be changed according to the type of the electronic device. The electronic device according to various embodiments of the present disclosure may include at least one of the above-mentioned elements, some elements may be omitted from the electronic device, or other additional elements may be further included in the electronic device. Also, some of the elements of the electronic device according to various embodiments of the present disclosure may be combined with each other to form one entity, thereby making it possible to perform the functions of the corresponding elements in the same manner as before the combination.
According to an embodiment of the present disclosure, the program module 910 (e.g., a program 740 of
The program module 910 may include a kernel 920, a middleware 930, an application programming interface (API) 960, and/or an application 970. At least part of the program module 910 may be preloaded on the electronic device, or may be downloaded from an external electronic device (e.g., a first external electronic device 702, a second external electronic device 704, or a server 706, and the like of
The kernel 920 (e.g., a kernel 741 of
The middleware 930 (e.g., a middleware 743 of
The runtime library 935 may include, for example, a library module used by a compiler to add a new function through a programming language while the application 970 is executed. The runtime library 935 may perform a function about input and output management, memory management, or an arithmetic function.
The application manager 941 may manage, for example, a life cycle of at least one of the application 970. The window manager 942 may manage graphic user interface (GUI) resources used on a screen of the electronic device. The multimedia manager 943 may determine a format utilized for reproducing various media files and may encode or decode a media file using a codec corresponding to the corresponding format. The resource manager 944 may manage source codes of at least one of the application 970, and may manage resources of a memory or a storage space, and the like.
The power manager 945 may act together with, for example, a basic input/output system (BIOS) and the like, may manage a battery or a power source, and may provide power information utilized for an operation of the electronic device. The database manager 946 may generate, search, or change a database to be used in at least one of the application 970. The package manager 947 may manage installation or update of an application distributed by a type of a package file.
The connectivity manager 948 may manage, for example, wireless connection such as Wi-Fi connection or BT connection, and the like. The notification manager 949 may display or notify events, such as an arrival message, an appointment, and proximity notification, by a method which is not disturbed to the user. The location manager 950 may manage location information of the electronic device. The graphic manager 951 may manage a graphic effect to be provided to the user or a user interface (UI) related to the graphic effect. The security manager 952 may provide all security functions utilized for system security or user authentication, and the like. According to an embodiment of the present disclosure, when the electronic device (e.g., an electronic device 701 of
The middleware 930 may include a middleware module which configures combinations of various functions of the above-described components. The middleware 930 may provide a module which specializes according to kinds of OSs to provide a differentiated function. Also, the middleware 930 may dynamically delete some of old components or may add new components.
The API 960 (e.g., an API 745 of
The application 970 (e.g., an application program 747 of
According to an embodiment of the present disclosure, the application 970 may include an application (hereinafter, for better understanding and ease of description, referred to as “information exchange application”) for exchanging information between the electronic device (e.g., the electronic device 701 of
For example, the notification relay application may include a function of transmitting notification information, which is generated by other applications (e.g., the SMS/MMS application, the e-mail application, the health care application, or the environment information application, and the like) of the electronic device, to the external electronic device (e.g., the first external electronic device 702 or the second external electronic device 704). Also, the notification relay application may receive, for example, notification information from the external electronic device, and may provide the received notification information to the user of the electronic device.
The device management application may manage (e.g., install, delete, or update), for example, at least one (e.g., a function of turning on/off the external electronic device itself (or partial components) or a function of adjusting brightness (or resolution) of a display) of functions of the external electronic device (e.g., the first external electronic device 702 or the second external electronic device 704) which communicates with the electronic device, an application which operates in the external electronic device, or a service (e.g., a call service or a message service) provided from the external electronic device.
According to an embodiment of the present disclosure, the application 970 may include an application (e.g., the health care application of a mobile medical device) which is preset according to attributes of the external electronic device (e.g., the first external electronic device 702 or the second external electronic device 704). According to an embodiment of the present disclosure, the application 970 may include an application received from the external electronic device (e.g., the server 706, the first external electronic device 702, or the second external electronic device 704). According to an embodiment of the present disclosure, the application 970 may include a preloaded application or a third party application which may be downloaded from a server. Names of the components of the program module 910 according to various embodiments of the present disclosure may differ according to kinds of OSs.
According to various embodiments of the present disclosure, at least part of the program module 910 may be implemented with software, firmware, hardware, or at least two or more combinations thereof. At least part of the program module 910 may be implemented (e.g., executed) by, for example, a processor (e.g., a processor 810). At least part of the program module 910 may include, for example, a module, a program, a routine, sets of instructions, or a process, and the like for performing one or more functions.
The term “module” used herein may represent, for example, a unit including one of hardware, software and firmware or a combination thereof. The term “module” may be interchangeably used with the terms “unit”, “logic”, “logical block”, “component” and “circuit”. The “module” may be a minimum unit of an integrated component or may be a part thereof. The “module” may be a minimum unit for performing one or more functions or a part thereof. The “module” may be implemented mechanically or electronically. For example, the “module” may include at least one of an application-specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA), and a programmable-logic device for performing some operations, which are known or will be developed.
At least a part of devices (e.g., modules or functions thereof) or methods (e.g., operations) according to various embodiments of the present disclosure may be implemented as instructions stored in a computer-readable storage medium in the form of a program module. In the case where the instructions are performed by a processor (e.g., the processor 720), the processor may perform functions corresponding to the instructions. The computer-readable storage medium may be, for example, the memory 730.
A computer-readable recording medium may include a hard disk, a floppy disk, a magnetic medium (e.g., a magnetic tape), an optical medium (e.g., CD-ROM, digital versatile disc (DVD)), a magneto-optical medium (e.g., a floptical disk), or a hardware device (e.g., a ROM, a RAM, a flash memory, or the like). The program instructions may include machine language codes generated by compilers and high-level language codes that can be executed by computers using interpreters. The above-mentioned hardware device may be configured to be operated as one or more software modules for performing operations of various embodiments of the present disclosure and vice versa.
While the present disclosure has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be defined as being limited to the embodiments, but should be defined by the appended claims and equivalents thereof.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2016-0164913 | Dec 2016 | KR | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/KR2017/014238 | 12/6/2017 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2018/106019 | 6/14/2018 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 9288390 | Hong et al. | Mar 2016 | B2 |
| 9996150 | Swaminathan et al. | Jun 2018 | B2 |
| 10028227 | Kim et al. | Jul 2018 | B2 |
| 10146412 | Jung et al. | Dec 2018 | B2 |
| 20030043288 | Malloy Desormeaux | Mar 2003 | A1 |
| 20030163703 | Robins | Aug 2003 | A1 |
| 20140168056 | Swaminathan et al. | Jun 2014 | A1 |
| 20140168494 | Hong | Jun 2014 | A1 |
| 20140282220 | Wantland et al. | Sep 2014 | A1 |
| 20160077711 | Jung et al. | Mar 2016 | A1 |
| 20170094156 | Kim | Mar 2017 | A1 |
| 20170214857 | Park | Jul 2017 | A1 |
| 20170280394 | Kim et al. | Sep 2017 | A1 |
| 20180253145 | Swaminathan et al. | Sep 2018 | A1 |
| 20190173987 | Sung | Jun 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 10-2013-0104682 | Sep 2013 | KR |
| 10-2014-0076876 | Jun 2014 | KR |
| 10-2014-0139319 | Dec 2014 | KR |
| 10-2016-0031851 | Mar 2016 | KR |
| 10-2016-0034075 | Mar 2016 | KR |
| 10-2016-0070371 | Jun 2016 | KR |
| 10-2016-0091168 | Aug 2016 | KR |
| Number | Date | Country | |
|---|---|---|---|
| 20200082631 A1 | Mar 2020 | US |
