Patent Application
20230153419
Certain example embodiments may relate to a display apparatus having installed thereon an operating system (OS) that operates in a secure world (SWD) corresponding to a TrustZone, and/or an operation method of the display apparatus.
Electronic devices, such as computers, smartphones, tablet personal computers (PCs), and personal digital assistants (PDAs), are used in close connection with users. As electronic devices that are personally used by users are widely used, the amount of personal information of a user or user-related information input to or stored in an electronic device has increased.
In addition, as users perform various tasks through electronic devices or server devices, the number of cases in which personal information of users or user-related information must be input to electronic devices or server devices has also increased.
Hereinafter, an electronic device or a server device that performs a certain task or a certain operation will be referred to as a ‘computing device’. When personal information of a user or user-related information is input to or stored in a computing device, it is important to protect the user's personal information or user-related information from hacking.
Thus, security technologies for protecting users' personal information or user-related information from hacking have been developed.
A security method using a TrustZone is an example of such security technologies.
In the TrustZone, which is a hardware security technology, hardware such as memory is divided into a secure world (SWD) corresponding to the trustzone and a normal world (NWD), and the NWD is set to be unable to access the resources of the SWD, thereby protecting data received through the SWD and stored in the SWD. In other words, electronic devices may have increased security performance by processing data requiring security processing in the SWD.
The SWD included in an electronic device has a significant limitation in operating performance. For example, in order to establish an SWD in an electronic device, the electronic device needs to have separate hardware resources (e.g., a central processing unit (CPU) and memory) operating in the SWD. Accordingly, there is a problem in that additional manufacturing costs for separate hardware resources (e.g., a CPU and memory) are required, leading to an increase in the manufacturing costs of the electronic device. Therefore, in general, electronic devices are provided by limiting the hardware resources forming an SWD according to a certain limit.
For example, a company's digital TVs use only one CPU core in order to establish an SWD and also use a maximum memory specification limited to several tens of mega bites (MB) or less. In addition, in order to establish an SWD in an electronic device, a driver for each hardware chipset (H/W chipset) operating in the SWD must be developed. For example, a driver applied to an SWD is different from a driver applied to an NWD, and separate development is inevitable.
Moreover, the memory capacity of the SWD is very small compared to that of the NWD. Accordingly, the SWD has limited capacity of storable resources compared to the NWD, and also is limited in the amount and speed of work that can be processed. Furthermore, when a certain operation is performed in the SWD, the SWD lacks programs, tools, and/or resources that are helpful in performing the certain operation, and thus has a low processing speed compared with a case when the certain operation is performed in the NWD.
Therefore, an apparatus and method for overcoming the above-mentioned problems to increase the security performance and achieve a desired security maintenance while using limited SWD resources without increasing the SWD resources are required.
In an example embodiment, there may be provided a display apparatus with improved security performance while using the resources of a secure world (SWD), and/or an operation method of the display apparatus.
For example, there may be provided a display apparatus capable of reducing the use of the resources of an SWD necessary for performing a calculation through a neural network and also increasing security performance, when input data is input to and processed by the neural network, and/or an operation method of the display apparatus.
According to an example embodiment, a display apparatus may include a display, a data obtainer configured to obtain input data, and a controller including at least one processor, comprising processing circuitry, configured to perform one or more instructions to separately run a normal operating system (OS) and a secure OS, which are respectively executed in a normal world (NWD) and a secure world (SWD) corresponding to a TrustZone. The processor may be configured to, upon execution of the one or more instructions, identify whether the input data input to a deep neural network (DNN) including a plurality of hidden layers between an input layer and an output layer corresponds to security data, based on identifying that the input data corresponds to the security data, control the secure OS so that layers from the input layer of the DNN to at least one hidden layer connected to the input layer are driven within the SWD, and, based on identifying that the input data does not correspond to security data, control the normal OS so that the entire DNN is driven in a world other than the SWD.
Based on the input data corresponding to at least one of user input data, content protection data, or personal information of a user, the processor may identify that the input data corresponds to the security data. “Based on” herein covers based at least on.
Based on the input data being identified as corresponding to security data, the processor may differently set a depth of the at least one hidden layer driven in the SWD, based on at least one of the type or characteristics of the input data.
Based on the input data being identified as corresponding to security data, the processor may differently set the depth of the at least one hidden layer, based on a security level that is applied to the input data.
Based on first input data to which digital rights management (DRM) encryption has not been applied and second input data to which DRM encryption has been applied corresponding to a first security level and a second security level, respectively, the processor may be configured to control the depth of the at least one hidden layer driven in the SWD to be a first depth, in the DNN receiving the first input data to which DRM encryption has not been applied, and may control the depth of the at least one hidden layer driven in the SWD to be a second depth higher than the first depth, in the DNN receiving the second input data to which DRM encryption has been applied.
The DNN may be formed as a convolution neural network (CNN) including a plurality of hidden layers. Based on identifying that the input data corresponds to security data, the at least one processor may control the secure OS so that layers from the input layer to at least one hidden layer connected to the input layer from among the plurality of hidden layers included in the CNN are driven within the SWD, and/or may control a calculation corresponding to one hidden layer performing a convolution operation for obtaining a feature map to be performed in the SWD at least once.
The processor may set the depth of the at least one hidden layer driven in the SWD, based on a deformation degree between each of a plurality of intermediate products respectively output by the plurality of hidden layers and the input data.
The processor may set the depth of the at least one hidden layer driven in the SWD, based on a deformation degree increase rate between the plurality of intermediate products respectively output by the plurality of hidden layers.
The processor may train the DNN so that a deformation degree between an intermediate product output by the at least one hidden layer driven in the SWD and the input data increases.
Based on identifying that the input data corresponds to the security data, the processor may control the normal OS so that layers from a subsequent hidden layer subsequently connected to the at least one hidden layer of the DNN to the output layer are driven within the NWD.
Based on identifying that the input data does not correspond to security data, the processor may control the normal OS so that the entire DNN is driven in the NWD.
Based on identifying that the input data does not correspond to security data, the processor may control the communication interface so that the input data is transmitted to the external device so that the entire DNN is driven in the NWD.
According to an example embodiment, an operation method of a display apparatus including at least one processor configured to separately run a normal operating system (OS) and a secure OS, which are respectively executed in a normal world (NWD) and a secure world (SWD) corresponding to a trustzone, may include identifying whether input data input to a deep neural network (DNN) including a plurality of hidden layers between an input layer and an output layer corresponds to security data; based on identifying that the input data corresponds to security data, controlling the secure OS so that layers from the input layer to at least one hidden layer connected to the input layer from among the plurality of hidden layers included in the DNN are driven within the SWD; and, based on identifying that the input data does not correspond to security data, controlling the normal OS so that the entire DNN is driven in a world other than the SWD.
A display apparatus and an operation method thereof, according to an example embodiment, may increase security performance while using the resources of a secure world (SWD).
In detail, in the display apparatus and the operation method thereof, according to an example embodiment, when input data is input to a neural network and processed, use of the resources of an SWD for performing an arithmetic operation through a neural network may be reduced, and/or security performance may be increased.
For example, in the display apparatus and the operation method thereof, according to an example embodiment, when the input data is input to the neural network and processed, security for input data that is processed may be maintained without performing all arithmetic operations that are performed in a neural network.
Moreover, in the display apparatus and the operation method thereof, according to an example embodiment, when input data is input to a neural network and processed, the depth of a deep neural network that is separated and driven within an SWD may be set differently, based on security necessity or a security level of the input data. Thus, security performance may be increased in correspondence with the security necessity of the input data.
The above and other aspects, features, and advantages of certain example embodiments will be more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which:
Throughout the disclosure, the expression “at least one of a, b or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.
Embodiments of the disclosure will now be described more fully with reference to the accompanying drawings such that one of ordinary skill in the art to which the disclosure pertains may easily execute the disclosure. The disclosure may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. In the drawings, parts irrelevant to the description are omitted for the simplicity of explanation, and like numbers refer to like elements throughout. In all of the drawings, the same components are indicated by the same reference numerals or characters.
Throughout the specification, when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or can be electrically connected or coupled to the other element with intervening elements interposed therebetween. In addition, the terms “comprises” and/or “comprising” or “includes” and/or “including” when used in this specification, specify the presence of stated elements, but do not preclude the presence or addition of one or more other elements.
Phrases such as “according to some embodiment” or “according to an embodiment” appearing in various places in this disclosure do not necessarily all refer to the same embodiment.
Some embodiments may be represented by functional block configurations and various processing operations. Some or all of such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, the functional blocks of the disclosure may be implemented by one or more microprocessors, or by circuit configurations for performing an intended function. In addition, for example, functional blocks according to the disclosure may be implemented with any programming or script language. The functional blocks may be implemented as algorithms executed in one or more processors. The disclosure may employ the conventional art for electronic configuration, signal processing, and/or data processing, for example. Terms such as modules and configurations may be used broadly and are not limited to mechanical and physical configurations. Each “module” herein may comprise circuitry.
Furthermore, the connecting lines or connectors between components shown in the various figures presented are intended to represent exemplary functional relationships and/or physical or logical couplings between the components. In an actual device, a connection between components may be represented by various functional connections, physical connections, or circuit connections that are replaceable or added.
Throughout the disclosure, the expression “at least one of A, B or C” indicates only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C.
According to an example embodiment, a display apparatus may refer to any electronic device capable of displaying a screen corresponding to content. Here, the content may be broadcast, game, lecture, movie, home training service content, etc.
According to an example embodiment, the display apparatus may exist in various forms, such as a TV, a digital TV, a smart TV, a digital signage, a smart phone, a tablet personal computer (PC), a personal digital assistant (PDA), a laptop computer, a media player, a digital broadcasting terminal, a mobile phone, a computer, and a notebook computer. The display apparatus may have a form that is movable or portable by a user, as well as a fixed type.
A case in which the display device according to an example embodiment is a TV (e.g., a smart TV or a digital TV) will now be described and illustrated as an example.
Recently, due to the development of display apparatus technology, various applications or programs may be stored and installed in a TV to perform various functions, and various functions and services in addition to a broadcasting content playback function may be provided through the installed applications.
For example, a smart TV may be equipped with an operating system (OS) to provide various functions or services in addition to a broadcast signal reception and reproduction function.
For example, a smart TV may perform various functions, such as game content playback and video call execution, in addition to a broadcast signal reception and reproduction function.
As another example, a TV may provide a home training service. Here, the home training service is a service for allowing a user to follow an exercise motion included in content currently being played-back through a display apparatus while watching the content.
As another example, a TV may receive a user's personal information (e.g., user's gender, age, hobbies, occupation, and viewing history information about the broadcast contents the user has watched during a certain period), and may analyze the user's personal information to provide a list of optimized recommended channels to the user.
As described above, as the functions or services that a TV is able to provide are diversified, the TV may use, analyze, or process data requiring information protection, such as a user's personal information or image data corresponding to content requiring content protection.
The processing of data requiring information protection generated in a TV will now be described below with reference to the examples of
Referring to
For example, a reproduced content may be moving picture content expressing a squat motion. For example, the content may include contents in which a training instructor 101 performs a squat motion. In the above-described example, the display apparatus 100 may reproduce corresponding moving picture content by displaying or outputting images expressing the squat motion through a display 110 in real time. Then, a user 150 may watch the moving picture content reproduced on the display 110, and may follow the squat motion.
The display apparatus 100 may obtain an image of the user 150 taking a certain action by watching content while the home training service is being provided. In detail, the camera 105 included in the display apparatus 100 or electrically connectable to the display apparatus 100 may photograph the user 150 positioned in front of the display 110. Accordingly, the camera 105 may obtain an image of a user who follows the squat motion.
Then, the display apparatus 100 may analyze the image obtained by the camera 105 to classify the posture or motion of the user and obtain information used to coach the posture or motion of the user. The display apparatus 100 may control the obtained information (e.g., information used to coach a user's posture or motion) to be output on the display 110. Here, the ‘information used to coach a user's posture or motion’ may be referred to as ‘coaching information’.
For example, referring to
An image obtained by photographing the user 150 will now be referred to as a ‘user image’. For example, the camera 105 may be located in front of the display apparatus 100 to photograph a certain space 160 including as an object the user 150 following an action and obtain a user image.
The display apparatus 100 needs to obtain the user image and analyze the user image to provide the coaching information 161. Because the user image used to obtain the coaching information 161 is an image having an individual's right of portrait, the user image may be data requiring security processing.
Referring to
For example, the display apparatus according to an example embodiment may capture a game image 200 corresponding to content currently being played back and analyze the captured game image 200, to perform an operation such as identification of whether the game image 200 is a game image or an image corresponding to media content, identification of the type of game by identifying objects included in the game image 200, or extraction of an object that a user may want to enlarge and display in the game image 200 (for example, a mini-map 211).
In general, game content is content for which copyright protection is requested. Accordingly, when a game image that is a reproduction screen image of game content is obtained, a copyright protection request may exist for the captured game image as well. Therefore, security processing for the captured game image may be needed.
Among pieces of data analyzed or used by the display apparatus 100, the user image described in the example of
According to an example embodiment, examples of ‘security data’ may include data under copyright, personal information about users, etc., data related to personal portrait rights, data encrypted for information protection, and the like.
According to recent technology development trends, artificial intelligence (AI) technology is widely used for data processing.
For example, a method of performing data classification, object recognition, object tracking and/or object differentiation, etc. by using AI technology that performs calculation through a neural network has been developed and used. Because the neural network is implemented using Al technology, the neural network may be referred to as an artificial neural network.
For example, the neural network may be a deep neural network (DNN) 220 that includes a plurality of layers to perform multi-stage calculations. For example, a DNN calculation may include a Convolutional Neural Network (CNN) calculation. In detail, a data recognition model for object recognition may be implemented via the above-illustrated neural network, and may be trained using learning data. The above-illustrated neural network may analyze input data, for example, a received image, by using a trained data recognition model to recognize, analyze and/or classify an object in the input image and output a desired result as output data. A CNN is referred to any neural network that performs an algorithm that searches for a pattern by analyzing an image, and may have various types and forms.
For example, the user image described in
As another example, the game image 200 described in
In
For example, the display apparatus 100 according to an example embodiment may provide necessary information to the user by using the output data 230 output by the DNN 220.
For example, when the output data 230 includes the information about the area (e.g., the mini map 211) that needs to be enlarged and displayed within the game image 200, an enlarged image corresponding to the mini-map 211 may be provided to the user, based on the output data 230. Then, the user can view an enlarged and displayed mini-map and enjoy the game more conveniently.
As another example, when the output data 230 includes an exercise (e.g., squat) done by the user, an exercise posture, and information for correction, information used to correct an exercise posture of the user (for example, coaching information) may be provided to the user, based on the output data 230. Then, a user who uses a home training service will be able to more accurately follow a home training operation by seeing the coaching information.
An example embodiment relates to a display apparatus including an operating system (OS) operating in a secure world (SWD) corresponding to a trustzone, in order to process security data requiring security processing, as described above with reference to
An example embodiment may use a DNN based on AI technology to analyze security data, as described above with reference to
A display apparatus capable of reducing the use of the resources of an SWD necessary for performing a calculation through a neural network and also increasing security performance, when input data is input to and processed by the neural network, and an operation method of the display apparatus will now be described in detail with reference to the attached drawings.
Referring to
For example, the controller 310 may include at least one processor 301. Although
According to an example embodiment, the processor 301 controls the following operations to be performed, by executing the one or more instructions. In detail, the processor 301 identifies whether the input data input to a DNN including a plurality of hidden layers between an input layer and an output layer corresponds to security data. When it is identified that the input data corresponds to the security data, the processor 301 controls the secure OS so that layers from the input layer of the DNN to at least one hidden layer connected to the input layer are driven within the SWD. When it is identified that the input data does not correspond to the security data, the processor 301 controls the normal OS so that the entire DNN is driven in a word other than the SWD. Here, being ‘driven’ may refer to performing a calculation defined in the DNN. For example, driving a DNN in an SWD may refer to loading software, a program, or an application corresponding to the DNN into a memory in the SWD and allowing the loaded DNN to perform a calculation according to a neural network in the SWD.
For example, the ‘other world’ may refer to an area other than the SWD. For example, the ‘other world’ may be an NWD included in the display apparatus 300. For example, when it is identified that input data does not correspond to security data, the processor 301 may control the normal OS so that the entire DNN is driven in the NWD.
As another example, the ‘other world’ may refer to an area formed on a separate device or server different from the display apparatus 300. For example, when it is identified that input data does not correspond to security data, the processor 301 may control the normal OS so that the entire DNN is driven in the separate server. In detail, the processor 301 may control the display apparatus 300 to transmit input data to an external server device and receive result data obtained by driving the entire DNN on the external server device.
For example, the OS may be referred to as an operating system, and may refer to software or a program for managing and controlling a computing device. The OS may manage hardware resources such as data input/output and/or memory allocation, and may control execution of programs. For example, the OS may manage or control a memory corresponding to a storage and/or a main memory such as a hard disk, and may manage or control an operation or task to be performed.
According to an example embodiment, an OS operating in the SWD may be identified as the secure OS, and an OS operating in the NWD may be identified as the normal OS.
For example, the processor 301 may perform an operation related to processing of data for which security needs to be kept, under a trusted execution environment (TEE), and may perform operations related to other processing under a rich execution environment (REE). For example, the processor 301 may perform a certain control operation under the REE in order to maintain a high overall control execution speed. Alternatively, when a control operation itself requires security maintenance, the processor 301 may perform a certain control operation under the TEE.
Here, the NWD and the SWD will be described in detail below with reference to
In the display apparatus 300, the controller 310 may control overall operations of the display apparatus 300. In detail, the controller 310 controls an overall operation of the display apparatus 300 and signal transfer among the internal components of the display apparatus 300, and processes data.
The controller 310 may include a memory in which at least one program for performing the above-described operation and an operation to be described later is stored, and at least one processor for executing the stored at least one program.
For example, the controller 310 may include an internal memory (not shown) and at least one processor 301, comprising processing circuitry, for executing stored at least one program. The internal memory (not shown) of the at least one processor 301 may store one or more instructions. The processor 301 may perform a certain operation by executing at least one of the one or more instructions stored in the internal memory (not shown). The processor 301 performing the certain operation may include not only a case where the processor 301 directly perform the certain operation but also a case where the processor 301 controls other components included in the display apparatus 100 (e.g., see display apparatus 300 in
For example, the controller 310 may include RAM (not shown) that stores a signal or data input by an external source of the display apparatus 300 or is used as a memory area for various operations performed by the display apparatus 300, ROM (not shown) that stores a control program for controlling the display apparatus 300, and at least one processor 301. Examples of the ROM (not shown), which is a non-volatile memory included in the controller 310, may include a flash memory.
The at least one processor 301 may include a graphics processing unit (GPU) (not shown) for graphic processing corresponding to video. The at least one processor 301 may be implemented as a system-on-chip (SoC) into which a core (not shown) and a GPU (not shown) are integrated. The at least one processor 301 may include a single core, a dual core, a triple core, a quad core, or a multiple core thereof.
According to an example embodiment, the data obtainer 305 may obtain input data. The input data may refer to data that is to be processed through a neural network, in detail, a DNN. For example, the input data may be data that is input to an input layer of a DNN that is used in an example embodiment.
For example, the input data may be data that is intended to be used, analyzed, and/or processed in an example embodiment.
For example, the data obtainer 305 (comprising circuitry) may obtain the input data by receiving the input data from an external device. For example, the data obtainer 305 may include a communication interface (not shown) (e.g., may correspond to a communication interface 360 of
As another example, the data obtainer 305 may obtain the input data by autonomously generating the input data. For example, the data obtainer 305 may include at least one component for generating the input data. For example, the data obtainer 305 may include a camera (not shown), and may obtain an image captured by the camera as the input data. For example, the camera included in the data obtainer 305 may correspond to the camera 105 described with reference to
As another example, the data obtainer 305 may obtain, as the input data, data collected, stored, and/or processed by the display apparatus 300. For example, the data obtainer 305 may obtain, as the input data, view history information about broadcasting contents collected and stored in the display apparatus 300.
For example, the display 320 may display an image. For example, the display 320 may visually output an image through an internally-included display panel (not shown).
For example, the display 320 may display an image corresponding to video data through the internally-included display panel so that a user visually recognizes moving picture content. In detail, moving picture data that forms content may include a plurality of frame images, and the display 320 may reproduce the moving picture content by consecutively displaying the plurality of frame images under a control by the processor 301.
Hereinafter, an SWD and an NWD formed in the display apparatus 300 will be described in detail with reference to
In processing the security data described with reference to
As one of the security technologies for protection against hacking, a security method using a trustzone may be exemplified.
In the trustzone, which is a hardware security technology, hardware such as memory is divided into an SWD corresponding to the trustzone and an NWD, and the NWD is made unable to access the resources of the SWD, thereby protecting data received through the SWD and stored in the SWD.
For example, the ‘SWD’ may be a world that provides a ‘TEE’ because the SWD corresponds to a trustzone and its security is guaranteed. The ‘NWD’ may be a world that provides or secures an ‘REE’ because the NWD has abundant hardware resources without being limited by hardware resources like in a trustzone.
Hereinafter, for convenience of description, the SWD and a world providing the TEE are collectively referred to as an SWD, and the NWD and a world providing the REE are collectively referred to as an NWD.
According to an example embodiment, the controller 310 may include the at least one processor 301 that executes the one or more instructions to run an OS that is divided into the NWD and the SWD corresponding to the trustzone. In detail, the controller 310 may run the OS by executing the one or more instructions stored in an internal memory (not shown) included in the controller 310 or a memory (not shown) formed separately from the controller 310.
For example, hardware (not shown) forming the controller 310 may be formed by being divided into the NWD and the SWD corresponding to the trustzone. The hardware (not shown) may include a GPU, a central processing unit (CPU), etc. that form at least one processor included in the controller 310. The processor (not shown) may be implemented as an SoC into which a core (not shown) and a GPU (not shown) are integrated. For example, hardware forming the controller 310, for example, an SOC, may be formed by being divided into the NWD and the SWD corresponding to the trustzone.
Software (not shown) forming the controller 310 (e.g., an OS) may be formed or executed differently within the NWD and the SWD corresponding to the trustzone.
Referring to
A case in which an application that receives and processes input data in the form of an image operates in the SWD 430 will now be illustrated. In this case, an external attacker such as malware may steal the input data input to the application through a method such as screen capturing. In order to address security vulnerabilities that may occur in the NWD, the application may be operated in the SWD 430.
According to a trustzone setting technology, the NWD 410 is unable to access resources existing in the SWD 430. In detail, the NWD may be accessed through only a minimum authenticated path with the SWD area, and the NWD and the SWD may be operated independently. Accordingly, the trustzone setting technology may provide a high level of security.
Here, the resources may refer to i) data input to the OS, and ii) various types of data generated, processed, and/or stored in the OS. In detail, in order to address a security problem such as data leakage that may occur while inputting, processing and/or storing data, a processor running the OS may enable at least one operation for inputting, processing and/or storing data to be performed in the SWD 430.
For example, the NWD 410 may include one or more applications 411 and 412, an OS 415 operating in the NWD, and a memory 417. The one or more applications 411 and 412, and the OS 415 operating in the NWD may be each stored and executed in the NWD 410. In
For example, the SWD 430 may include one or more applications 431 and 432, an OS 435 operating in the SWD, and a memory 437. The one or more applications 431 and 432, and the OS 435 operating in the SWD may be each stored and executed in the SWD 430. In
Because the applications, the OSs, and the memories shown in
For example, data to be processed in the SWD 430, for example, input data, may be temporarily stored in the memory 437. The secure OS 435 may manage the input data stored in the memory 437 to be loaded into the application 431, which is to receive and process the input data. The secure OS 435 may control unauthorized programs or data to not access the SWD 430 through a legitimate authentication procedure. Accordingly, the display apparatus including the SWD 430 may secure security performance in data processing for maintaining security.
However, as described above, the memory capacity of the SWD is very small compared to the NWD. Accordingly, the SWD is limited in the capacity of storable resources compared to the NWD, and is also limited in the amount and speed of work that can be processed. Furthermore, when a certain operation is performed in the SWD, the SWD lacks programs, tools, and/or resources that are helpful in performing the certain operation, and thus provides a low processing speed compared with when the certain operation is performed in the NWD.
In addition, the calculation of a DNN using AI technology generally has a large amount of calculations, and accordingly, a large amount of memory allocation is unavoidable to perform a calculation according to the DNN within the SWD. However, due to the above-described limitation of resource capacity, a processing speed may decrease when the calculation according to the DNN is performed in the SWD.
Therefore, an example embodiment separates and drives only a portion of the DNN in the SWD based on the input data, so that a desired operation may be quickly performed without degrading the security performance. Separation and driving of the DNN according to an example embodiment will now be described in detail with reference to
A display apparatus 500 of
Referring to
The memory 350 may store at least one instruction. The memory 350 may store at least one instruction that the processor 301 executes. The memory 350 may store at least one program that the processor 301 executes. Alternatively, the memory 350 may store an application for providing a certain function, a certain operation, and/or a certain service. According to an example embodiment, the memory 350 may store a DNN. The DNN may correspond to the DNN 220 briefly described with reference to
For example, the memory 350 may be formed by being separated into the SWD and the NWD as described above with reference to
For example, the memory 350 may store information or data that is used in an operation of the display apparatus 500. The memory 350 may also store content reproducible by the display apparatus 500. Alternatively, the memory 350 may store input data that is to be input to the DNN.
For example, the memory 350 may include at least one type of storage medium selected from among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, a secure digital (SD) or extreme digital (XD) memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), magnetic memory, a magnetic disk, and an optical disk.
The communication interface 505 may include at least one among from at least one communication module, comprising communication circuitry, transmitting or receiving data to or from an external device, or at least one port. In detail, the communication interface 505 may include a communication interface 360 and an I/O interface 340.
For example, the communication interface 360 performs communication with an external device (not shown) through at least one wired or wireless communication network. According to an example embodiment, the communication interface 360 may communicate with the external device. The external device may be a server, and the communication interface 360 may perform communication with a server (not shown). The server (not shown) may be a content providing server, an Internet server, etc. Alternatively, the server (not shown) may be a server that analyzes and/or processes an image. Alternatively, the server (not shown) may be a server that manufactures and/or distributes an application. Alternatively, the server (not shown) may be a server that performs a calculation according to a neural network by using AI technology.
For example, the communication interface 360 may be formed in a form including at least one communication module comprising communication circuitry, a communication circuit, and/or the like, and may transmit or receive data to or from the external device through the communication module and/or the communication circuit.
For example, the communication interface 360 may include at least one communication module (not shown) that performs communication according to a communication standard such as Bluetooth, Wi-Fi, Bluetooth Low Energy (BLE), near field communication (NFC)/radio frequency identification (RFID), Wi-Fi Direct, ultra wideband (UWB), or ZIGBEE.
For example, the communication interface 360 may further include a server (not shown) for supporting long-distance communication according to a long-distance communication standard, and a communication module (not shown) for performing communication. In detail, the communication interface 360 may include a communication module (not shown) for performing communication via a network for Internet communication. The communication interface 360 may include a communication network following a communication standard such as 3G, 4G, 5G, and/or 6G.
The communication interface 360 may include a communication module, comprising communication circuitry, capable of receiving a control command from a remote controller (not shown) located in a short distance, for example, an infrared (IR) communication module. In this case, the communication interface 360 may receive a control command from the remote controller (not shown). For example, the control command received from the remote controller (not shown) may include a turn-on or turn-off command, and a signal for requesting execution of a certain application.
For example, the I/O interface 340 may include a High-Definition Multimedia Interface (HDMI) port (not shown), a component jack (not shown), a PC port (not shown), or a USB port (not shown). The I/O interface 340 may include a combination of the HDMI port, the component jack, the PC port, and the USB port. In this case, the I/O interface 340 may directly receive moving picture data corresponding to the content to be reproduced on the display apparatus 500 through the HDMI port, the component jack, the PC port, or the USB port.
The user interface 370 may receive a user input for controlling the display apparatus 500. The user interface 370 may include, but is not limited to, a touch panel for sensing a user's touch, a button for receiving a user's push operation, a wheel for receiving a user's rotation operation, a keyboard, and a dome switch.
The user interface 370 may include a voice recognition device (not shown) for voice recognition. For example, the voice recognition device may be a microphone, and the voice recognition device may receive a user's voice command or voice request. Accordingly, the processor 301 may control an operation corresponding to the voice command or voice request to be performed.
The user interface 370 may also include a motion detection sensor (not shown). For example, the motion detection sensor (not shown) may detect a motion of the display apparatus 500 and receive the detected motion as a user input. The voice recognition device (not shown) and the motion detection sensor (not shown) may be included in the display apparatus 500 as a module independent from the user interface 370, rather than being included in the user interface 370.
For example, at least one of the camera 330, the communication interface 360, or the I/O interface 340 may be disposed to be included in the data obtainer 305. For example, the data obtainer 305 is configured to obtain the input data, and may include one or more circuits and/or components for obtaining the input data. As another example, the data obtainer 305 may obtain the input data by receiving the input data from a separate circuit and/or component that generates the input data.
For example, according to an example embodiment, when input data input to the input layer of the DNN is an image, the camera 330 may obtain the input data generated by photographing the image. The processor 301 may control the input data obtained by the camera 330 to be input to the input layer of the DNN.
Referring to
The communication interface 650, the display 615, a camera 662, the I/O interface 670, the processor 680, and the memory 690 of the display apparatus 600 may correspond to the communication interface 360, the display 320, the camera 330, the I/O interface 340, the processor 301, and the memory 350 of
The video processor 610 processes video data that is received by the display apparatus 600. The video processor 610 may perform a variety of image processing, such as decoding, scaling, noise filtering, frame rate transformation, and resolution transformation, on the received video data.
The display 615 displays video included in a broadcasting signal received via the tuner 640 on the screen thereof, under a control by the processor 680. The display 615 may also display content (for example, a moving picture) that is input via the communication interface 650 or the I/O interface 670.
The display 615 may output an image stored in the memory 690 under a control by the processor 680. The display 615 may include a voice user interface (UI) (e.g., including a voice instruction guide) for performing a voice recognition task corresponding to voice recognition or a motion UI (e.g., including a user motion guide for motion recognition) for performing a motion recognition task corresponding to motion recognition.
The audio processor 620 processes audio data. The audio processor 620 may perform a variety of processing, such as decoding, amplification, or noise filtering, on the audio data. The audio processor 620 may include a plurality of audio processing modules, comprising processing circuitry, to process audios corresponding to a plurality of pieces of content. Each processor herein comprises processing circuitry.
The audio output interface 625 outputs audio included in a broadcasting signal received via the tuner 640, under a control by the processor 680. The audio output interface 625 may output audio (for example, a voice or a sound) that is input via the communication interface 650 or the I/O interface 670. The audio output interface 625 may also output audio stored in the memory 690 under a control by the processor 680. The audio output interface 625 may include at least one of a speaker 626, a headphone output port 627, or a Sony/Philips Digital Interface (S/PDIF) output port 628. The audio output interface 625 may include a combination of the speaker 626, the headphone output port 627, and the S/PDIF output port 628.
The power supply 630 supplies power that is input from an external power source, to the internal components 610 through 690 of the display apparatus 600, under the control by the processor 680. The power supply 630 may also supply power that is output by one or more batteries (not shown) located in the display apparatus 600, to the internal components 610 through 690 of the display apparatus 600, under the control by the processor 680.
The tuner 640 may tune and select a frequency band corresponding to a channel that the display apparatus 600 wants to receive, from among many radio wave components that are obtained by, for example, amplifying, mixing, or resonating a wired or wireless broadcasting signal. The broadcasting signal includes audio, video, and additional information (for example, an electronic program guide (EPG)).
The tuner 640 may receive a broadcasting signal in a frequency band corresponding to a channel number (e.g., cable broadcasting No. 506) according to a user input (for example, a control signal received from an external control device (not shown) (e.g., a remote controller), e.g., a channel number input, a channel up-down input, and a channel input on an EPG screen image).
The tuner 640 may receive a broadcasting signal from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner 640 may also receive a broadcasting signal from a source such as analog broadcasting or digital broadcasting. The broadcasting signal received via the tuner 640 is decoded (for example, audio decoding, video decoding, or additional information decoding) and is thus divided into audio, video, and/or additional information. The audio, the video, and/or the additional information may be stored in the memory 690 under the control by the processor 680.
The display apparatus 600 may include a single tuner 640 or a plurality of tuners 640. According to an embodiment, when a plurality of tuners 640 are included, the plurality of tuners 640 may output a plurality of broadcasting signals to a plurality of windows included in a multi-window screen image provided to the display 615.
The tuner 640 may be all-in-one with the display apparatus 600, or implemented as a separate device (for example, a tuner (not shown) that is connected to a set-top box (not shown) and the I/O interface 670) having a tuner that is electrically connected to the display apparatus 600.
The communication interface 650 may connect the display apparatus 600 to an external device (for example, an external camera, a smartphone, or an audio device) under the control by the processor 680. The processor 680 may transmit/receive content to/from the external device connected via the communication interface 650, download an application from the external device, or perform web-browsing. In detail, the communication interface 650 may be connected to a network to receive content from an external device (not shown).
As described above, the communication interface 650 may include at least one of a short-range wireless communication module (not shown), a wired communication module (not shown), or a mobile communication module (not shown).
The communication interface 650 may include a module combination including one or more of the wireless LAN 651, the Bluetooth interface/network 652, and the wired Ethernet network 653. The communication interface 650 may receive a control signal of a control device (not shown) under the control by the processor 680. The control signal may be implemented as a Bluetooth signal, a radio frequency (RF) signal, or a Wi-Fi signal.
The communication interface 650 may further include short-range communication (for example, near field communication (NFC) or Bluetooth low energy (BLE)), in addition to the Bluetooth interface/network 652.
The sensor 660 senses a voice of a user, an image of the user, or an interaction with the user.
According to an example embodiment, the sensor 660 may obtain data for identifying a gesture of the user. In detail, the sensor 660 may include the camera 662, and may obtain the data for identifying a gesture of the user (e.g., an image representing the user's gesture) by using the camera 662. The sensor 660 may further include at least one of a microphone 661 or a light receiver 663.
The microphone 661 receives an uttered voice of the user. The microphone 661 may transform the received voice into an electrical signal and output the electrical signal to the processor 680. The user voice may include, for example, a voice corresponding to a menu or function of the display apparatus 600. For example, a recognition range of the microphone 661 may be recommended to be within 4 m from the microphone 661 to a location of the user, and may vary in correspondence to the magnitude of the voice of the user and a surrounding environment (for example, a speaker sound or ambient noise).
The microphone 661 may be integrated with or separate from the display apparatus 600. The separated microphone 661 may be electrically connected to the display apparatus 600 via the communication interface 650 or the I/O interface 670.
It will be easily understood by one of ordinary skill in the art that the microphone 661 may be excluded according to the performance and structure of the display apparatus 600.
The camera 662 receives an image (for example, consecutive frames) corresponding to a motion of the user including a gesture within a recognition range of the camera 662. For example, the recognition range of the camera 662 may be a distance within 0.1 to 5 m from the camera 662 to the user. The motion of the user may include a part of the body of the user, such as the face, a facial expression, the hand, the fist, and a finger of the user, or a motion or the like of the part of the body of the user. The camera 662 may convert a received image into an electrical signal under the control by the processor 680 and output the electrical signal to the processor 680.
The processor 680 may select a menu that is displayed on the display apparatus 600 by using a result of the recognition of the received motion, or perform control corresponding to a result of the motion recognition. Examples of the control may include channel adjustment, volume adjustment, or indicator movement.
The camera 662 may include a lens (not shown) and an image sensor (not shown). The camera 662 may support optical zoom or digital zoom by using a plurality of lenses and image processing. The recognition range of the camera 662 may be variously set according to the angle of the camera 662 and surrounding environment conditions. When the camera 662 is composed of a plurality of cameras, a three-dimensional (3D) still image or a 3D motion may be received by the plurality of cameras.
The camera 662 may be integrated with or separate from the display apparatus 600. A separate device (not shown) including the separate camera 662 may be electrically connected to the display apparatus 600 via the communication interface 650 or the I/O interface 670.
It will be easily understood by one of ordinary skill in the art that the camera 662 may be excluded according to the performance and structure of the display apparatus 600.
The light receiver 663 receives an optical signal (including a control signal) from the external control device (not shown) via a light window (not shown) of the bezel of the display 615. The light receiver 663 may receive an optical signal corresponding to a user input (for example, touch, pressing, a touch gesture, a voice, or a motion) from the control device (not shown). A control signal may be extracted from the received optical signal under the control by the processor 680.
For example, the light receiver 663 may receive a signal corresponding to a pointing location of the control device (not shown) and transmit the received signal to the processor 680. For example, when a user interface screen image for receiving data or a command from a user has been displayed via the display 615 and the user wants to input data or a command to the display apparatus 600 via the control device (not shown), the user moves the control device (not shown) while touching a touch pad (not shown) provided in the control device, and, at this time, the light receiver 663 may receive a signal corresponding to a motion of the control device (not shown) and transmit the received signal to the processor 680. The light receiver 663 may receive a signal indicating that a certain button provided on the control device has been pressed, and transmit the received signal to the processor 680. For example, when the user presses a button-type touch pad (not shown) provided in the control device with his or her finger, the light receiver 663 may receive a signal indicating that the button-type touch pad has been pressed, and transmit the received signal to the processor 680. For example, the signal indicating that the button-type touch pad has been pressed may be used as a signal for selecting one from among items.
The I/O interface 670 receives video (for example, a moving picture), audio (for example, a voice or music), and additional information (for example, an EPG) from outside the display apparatus 600 under the control by the processor 680. The I/O interface 670 may include a HDMI port 671, a component jack 672, a PC port 673, or a USB port 674. The I/O interface 670 may include a combination of the HDMI port 671, the component jack 672, the PC port 673, and the USB port 674.
It will be understood by one of ordinary skill in the art that the structure and operation of the I/O interface 670 may be variously implemented according to embodiments.
The processor 680 controls an overall operation of the display apparatus 600 and signal transfer among components of the display apparatus 600 and processes data. When there is an input of a user or stored preset conditions are satisfied, the processor 680 may run an OS and various applications that are stored in the memory 690.
A case in which an operation method 700 of the display apparatus and operations to be described later with reference to
Referring to
In the operation method 700 of the display apparatus, it is identified whether input data input to a DNN including a plurality of hidden layers between an input layer and an output layer corresponds to security data (S710). Operation 5710 may be performed by the processor 301. For example, operation 5710 may be performed by the processor 301 operating in the normal OS 410.
The security data may refer to data that requires security processing.
According to an example embodiment, examples of the ‘security data’ may include data under copyright, personal information about users, etc., data related to personal portrait rights, data related to personal characteristics (e.g., voice features of users), data encrypted for information protection, and the like. For example, the user image described in the example of
For example, when a copyright is set on content including at least one of photos, images, audio, or text, at least a portion of content data may be encrypted to protect the copyright of the content, and thus the content may be protected against being reproduced or executed by unauthorized persons. For example, in order to protect the copyright of the content, Digital Rights Management (DRM) encrypting may be performed on the content data. Alternatively, content input through HDMI/Screen mirroring may be protected by High-bandwidth Digital Content Protection (HDCP).
Processing of the security data as described above may be performed in the SWD 430 providing a TEE.
For example, when the input data corresponds to at least one of user input data, content protection data, or personal information of a user, the processor 301 may identify that the input data corresponds to the security data.
For example, in operation S710, identification of the security data may be made by determining whether DRM encryption for copyright protection of content has been applied to the input data.
For example, in operation S710, the identification of the security data may be made by determining whether the input data is data generated through direct utterance or input by the user. As another example, in operation S710, the identification of the security data may be made by determining whether the input data is information created or processed based on the personal information of the user.
According to an example embodiment, when data desired to be processed in the DNN is security data, operation S720 may be subsequently performed to drive a portion of the DNN receiving the security data in the SWD 430 providing a TEE.
The DNN used in an example embodiment will be described later in detail with reference to
AI technology may be composed of machine learning such as deep learning and element technologies using machine learning. AI technology is widely used in technical fields such as classification/identification, linguistic understanding, visual understanding, deduction/prediction, knowledge representation, and operation control.
For example, AI technology may be used in visual understanding and deduction/prediction. In detail, a technology for analyzing and classifying input data by using AI technology may be implemented. In other words, AI technology may be used to implement a method and apparatus capable of analyzing and/or classifying input data to obtain a desired result.
The AI technology may be implemented using an algorithm. Here, the algorithm for implementing the AI technology or a set of the algorithms is referred to as a neural network. For example, the neural network may receive input data, perform a calculation for the aforementioned analysis and classification, and output result data. In this way, in order for the neural network to accurately output the result data corresponding to the input data, the neural network needs to be trained. Here, the ‘training’ may refer to training the neural network so that the neural network discovers or learns by itself a method of receiving various pieces of data and analyzing the received data, a method of analyzing the received data pieces, a method of classifying the received data pieces, and/or a method of extracting a feature necessary for result data creation from the received data pieces. Here, ‘training’ may be expressed as ‘learning’ or ‘training’.
In addition, a set of algorithms for outputting output data corresponding to input data through the above-described neural network, software for executing the set of algorithms, and/or hardware for executing the set of algorithms are referred to as an ‘AI model’ (or an ‘artificial intelligence model’). The AI model may be formed in a form including one neural network, or may be formed in a form in which a plurality of neural networks are combined. For example, the AI model may be used in the same sense as a DNN.
According to an example embodiment, the neural network may be a DNN including a plurality of hidden layers between an input layer and an output layer. For example, when a hidden layer includes a plurality of layers, this may be expressed as having a plurality of depths.
For example, the DNN may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a calculation of the layer through a calculation between a calculation result of a previous layer and the plurality of weight values. For example, the DNN may be a set of algorithms, hardware, and/or a program for learning a method of recognizing an object from an image input to a neural network based on AI technology.
According to an example embodiment, the DNN outputs a result corresponding to the input data by performing a calculation through the plurality of layers.
Referring to
An AI model used in an example embodiment may be formed as a DNN for recognizing and classifying an object included in an input image by analyzing the input image.
For example, a neural network (in detail, a DNN) may learn a method of recognizing an object from an image, based on (i) supervised learning using an image as an input value and/or (ii) unsupervised learning of discovering a pattern or method for recognizing an object from an image, by self-learning the type of data necessary for recognizing an object from an image without supervision. As another example, the neural network may learn a method of recognizing an object from an image, by using reinforcement learning using a feedback regarding whether a result of recognizing an object according to training is correct. As another example, the neural network may learn a method of recognizing an object from an image, by using semi-supervised learning.
For example, the DNN 800 may be a neural network for performing a convolution operation of analyzing and processing an image. Examples of the DNN may include a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Networks, and but the neural network according to an example embodiment is not limited to the above-described examples except in cases where it is specified. The CNN may be classified into a Deep Convolution Neural Network (DCNN) or a Capsnet neural network (not shown). The DNN 800 may be in the form of a combination of a plurality of different types of neural networks.
Referring to
For example, each of the plurality of layers forming the DNN 800 may include one or more nodes. For example, the input layer 811 may include one or more nodes (e.g., 810) that receive data.
For example, the input layer 811 may receive, through a plurality of nodes 810, a plurality of images obtained by scaling the input image 200, respectively. In detail, the plurality of images obtained by scaling the input image 200 for each frequency band may be input to the plurality of nodes 810.
In the DNN 800, the hidden layer 820 may be formed of a plurality of layers.
As shown in
The DNN 800 may be trained based on a plurality of training images and thus may be constructed as an object recognition model for recognizing an object included in an image. In detail, in order to increase the accuracy of a result output through the DNN 800, the DNN 800 may repeatedly perform training in a direction from the output layer 830 to the input layer 811, based on the plurality of training images, and may correct weight values to increase the accuracy of the output result.
The DNN 800 having the finally corrected weight values may be used as an object recognition model. When the DNN 800 is trained to receive an image and recognize an object included in the image, when the DNN 800 receives an image 200, the DNN 800 may analyze the image 200 and output output data 840 corresponding to the image 200.
For example, when the DNN 800 is trained to classify whether an input image is a game image or another type of content image, the output data 840 may include information indicating that the input image 200 may be a ‘game’ image.
As another example, when the DNN 800 is trained to analyze an object included in an input image to identify the name or type of a corresponding game, the output data 840 may include information about the name of a game corresponding to a game image that is the input image 200 and the type of the game.
As another example, when a user image obtained by photographing a user as in the example of
Referring to
Referring to
For example, the DNN 800 may output, as output data 940, a result of classifying the received sound data into at least one class or type, for example, speech, music, and noise.
As another example, when the received sound data is voice data uttered by a user, the DNN 800 may analyze the voice data to output a classification result of identifying an individual user who has spoken.
According to an example embodiment, the display apparatus 500 may receive a user voice and identify an individual user from among a plurality of users to provide a customized service to the individual user. In this case, the display apparatus 500 may input the input data 910 corresponding to a voice uttered by the individual user to the DNN 800. Because the voice uttered by the individual user corresponds to personal information of the individual user, the display apparatus 500 may process a necessary operation in the SWD 430 in order to analyze the voice uttered by the individual user.
According to an example embodiment, when the processor 301 seeks to input security data to the neural network 800 and process the input security data, the processor 301 may separate the neural network 800 and drive the neural network 800 in the SWD 430. For example, the processor 301 may control, based on the input data, at least one layer forming the hidden layer 820 from among a plurality of layers constituting the neural network 800 to be driven within the SWD 430. For example, the processor 301 may identify whether the input data (for example, the game image 200) input to the DNN 800 including the plurality of hidden layers 820 between the input layer 811 and the output layer 830 corresponds to security data, and, when the input data is identified as corresponding to the security data, may control the secure OS so that from the input layer 811 of the DNN 800 to at least one hidden layer (for example, at least some of the plurality of hidden layers 820) connected to the input layer 811 are driven within the SWD 430.
Referring to
The CNN-based neural network may introduce the concept of a filter illuminating only a specific region and convolve pieces of information within the filter with each other to create a new expression (feature, feature map or representation), when a correlation between pieces of information included in an image is local.
Referring to
In addition, when the pieces of data generated while passing through the convolution layer and the pooling layer are processed through a hidden layer formed as a fully connected layer, output data corresponding to a desired result may be obtained.
For example, the DNN 1000 may include an input layer 1010, a first convolution layer 1020, a first pooling layer 1030, a second convolution layer 1040, a second pooling layer 1050, a hidden layer hidden layer 1060, and an output layer 870.
For example, layers from one convolution layer to a next convolution layer may correspond to one hidden layer. For example, a first hidden layer may include the first convolution layer 1020 and the first pooling layer 1030, and a second hidden layer may include the second convolution layer 1040 and the second pooling layer 1050. In the example shown in
In the DNN 1000, respective depths of the convolution layer and the pooling layer may vary, and a depth of the hidden layer may also vary. As the respective depths of the convolution layer and the pooling layer increase, more various pieces of data may be created. For example, as the respective depths of the convolution layer and the pooling layer increase, pieces of information indicating the features of an input image may have more detailed shapes. The depth and shape of the DNN 1000 may be designed in various ways in consideration of the accuracy of a result, the reliability of the result, the processing speed and capacity of a processor, and the like.
When a DNN receiving input data is a CNN (for example, 800 or 1000) including a plurality of hidden layers, and the input data is identified as corresponding to the security data, the processor 301 may control the secure OS so that from an input layer included in the CNN to at least one hidden layer connected to the input layer from among the plurality of hidden layers are driven in the SWD 430. Accordingly, the processor 301 may control a calculation corresponding to one hidden layer performing a convolution operation for obtaining a feature map to be performed in the SWD 430 at least once. For example, the processor 301 may control a calculation corresponding to one hidden layer (for example, the first hidden layer or the second hidden layer) performing a convolution operation once to be performed in the SWD 430 once or more times. For example, when a DNN includes n hidden layers (here, each of the hidden layers is able to perform a convolution operation one time), the processor 301 may separate the DNN and drive the separated DNN within the SWD 430, so that a calculation corresponding to m hidden layers (where n>m, n and m are natural numbers) from among the n hidden layers is performed within the SWD 430.
Referring back to
In the operation method 700 of the display apparatus, when the input data is identified as not corresponding to the security data, the normal OS may be controlled so that the entire DNN (for example, 1000) is driven in a world other than the SWD 430 (S730). For example, the control may be performed by the processor 301. For example, operation 5730 may be performed by the processor 301.
Referring back to
As another example, the processor 301 may control a calculation corresponding to from the input layer 1010 to two hidden layers connected, directly or indirectly, to the input layer 1010 (for example, the first hidden layer including the first convolution layer 1020 and the first pooling layer 1030 and the second hidden layer including the second convolution layer 1040 and the second pooling layer 1050) to be performed within the SWD 430. A calculation corresponding to layers after the second hidden layer may be controlled to be performed in the NWD 410.
The DNN separation and driving operation will now be described in more detail with reference to
A DNN 1110 of
For convenience of explanation, in the DNN 1110, an input layer receiving the input data to at least one hidden layer from among a plurality of hidden layers are referred to as a HEAD part 1111, and from a layer subsequent to the HEAD part 1111 to an output layer re referred to as a BODY part 1112.
For example, when the DNN 1000 of
Referring to
In the operation method 700 of the display apparatus, when security processing for the input data is not required, namely, when the input data is not identified as the security data (S710), the HEAD part 1111 and the BODY part 1112 of the DNN 1110 may operate in the NWD providing an REE (S730). In detail, in the operation method 700 of the display apparatus, when security processing is not required (S710), the HEAD part 1111 of the DNN 1110 may operate in the NWD providing an REE (S731), and the BODY part 1112 may also operate in the NWD providing an REE (S732).
A case in which the operation method 1200 of the display apparatus is performed by the display apparatus 500 of
Referring to
In the operation method 1200 of the display apparatus, operation 5720 or 5730 is performed according to a result of determining whether the input data obtained in operation 5710 corresponds to the security data.
In the operation method 1200 of the display apparatus, output data output from the output layer of the DNN may be obtained according to a result of operation 5720 or 5730 (S740).
Referring to
Subsequent to operation 5720, the normal OS may be controlled to drive the rest of the DNN (for example, the BODY part 1112 which is a part of the DNN excluding the HEAD part 1111) within the NWD 410 (S732). In detail, the processor 301 may control the normal OS to drive the BODY part 1112 of the DNN to be driven within the NWD 410.
In the operation method 1200 of the display apparatus, when the input data is not identified as corresponding to the security data (S710), the entirety of the DNN 1110, namely, both the HEAD part 1111 and the BODY part 1112, may operate in the NWD providing an REE (S730).
Referring back to
Feature data generated while the input data is passing through a hidden layer is different from the input data, and the feature data may have an irreversible feature. For example, a convolution layer, a pooling layer, and/or an activation function forming the hidden layer may generally be an irreversible operation. For example, the input data may not be restored from feature data generated by passing through the hidden layer, which is data obtained by performing an irreversible operation. Thus, the input data may not be restored from intermediate production data of each of the plurality of hidden layers included in the DNN 800 or 1000.
For example, data output through the aforementioned HEAD part 1111 may be data irreversibly deformed from input data that is original data. Therefore, even when the data output from the last end of the HEAD part 1111 is stolen, it is impossible to restore the original data.
According to an example embodiment, even when the entirety of the DNN 800 or 1000 is not driven in the SWD, namely, even when only some of the hidden layers included in the DNN are driven in the SWD, the input data may not be restored based on an intermediate product (for example, pieces of feature data output by the hidden layer) created in at least one hidden layer driven in the SWD. Accordingly, the risk of hacking in which the input data is stolen may be removed when the input data is processed. Moreover, because the rest of the DNN is operated in the NWD, insufficient resources in the SWD when the entire DNN is driven in the SWD may be addressed.
A DNN separation and driving operation will now be described in more detail with reference to
Components of
According to an example embodiment, only a part of a DNN may be separated and driven in the SWD, based on input data.
Referring to
Referring to
As described above, the display apparatus may obtain an intermediate product (or intermediate production data) output by the first hidden layer 851 within the SWD 430, by driving only the HEAD part of the DNN 800 within the SWD 430. The processor 301 may drive neural network components included in the block 1320, which is the rest of the DNN 800, by transmitting the intermediate product created in the SWD 430 to the NWD 410. Accordingly, a calculation corresponding to the components included in the block 1320, which is the rest of the DNN 800, may be performed in the NWD 410. Thus, the result data 840 may be obtained within the NWD 410.
Components of
Referring to
The operation of obtaining the input data within the SWD 430 will now be described in detail with reference to
The DNN 1000 of
In
In
Referring to
For example, an intermediate product created in the SWD (for example, pieces of feature data output by the HEAD part 1111 of the DNN) may be transmitted to and stored in a memory 1530 that is shared between the SWD and the NWD. The intermediate product stored in the memory 1530 may be read and input to the input terminal of the BODY body 1112 of the DNN driven in the NWD. Thus, the BODY part 1112 of the DNN may receive the intermediate product and may perform a calculation corresponding to at least one hidden layer and an output layer that constitute the BODY part 1112 (S1551).
According to an example embodiment, when the input data is identified as corresponding to the security data (S710-YES), the processor 301 may differently set the depth of the at least one hidden layer driven in the SWD, based on at least one of the type or characteristics of the input data. For example, the type of input data may be classified into copyrighted content, user input data, a user image, and the like. Alternatively, the characteristics of the input data may include information indicating whether DRM encryption or HDCP exists in the input data.
For example, the processor 301 may set a security level, based on at least one of the type or characteristics of the input data. The processor 301 may differently set the depth of the at least one hidden layer driven in the SWD, based on the security level.
For example, based on at least one of the type or characteristics of the input data, the processor 301 may set the security level to be high, when security necessity increases, and may set the security level to be low, when security necessity decreases.
For example, the processor 301 may differently set the security level according to whether DRM encryption has been applied to the input data. For example, first input data to which DRM encryption has not been applied and second input data to which DRM encryption has been applied may correspond to a first security level (for example, security level=low) and a second security level (for example, security level=high), respectively. In the above-described case, the processor 301 may control the depth of the at least one hidden layer driven in the SWD to be a first depth, in the DNN receiving the first input data to which DRM encryption has not been applied. In addition, the processor 301 may control the depth of the at least one hidden layer driven in the SWD to be a second depth higher than the first depth, in the DNN receiving the second input data to which DRM encryption has been applied. Here, when the depth of a hidden layer is high, it may indicate that more hidden layers are included. For example, when one hidden layer is included, a depth may be considered 1, and, when two hidden layers are included, a depth may be considered 2.
For example, when the depth of a hidden layer increases in a DNN, this may indicate that the number of steps of hidden layers of the DNN increases.
In the DNN, the degree of abstraction may increase as the hidden layers increase . For example, an intermediate product created by passing through the first hidden layer has a higher degree of deformation than an intermediate product created by sequentially passing through the first hidden layer and the second hidden layer, and thus is created in a more abstract expression. The degree of abstraction increasing may indicate that data is created with more irreversible characteristics. As the degree of deformation increases, it may become difficult to restore the original data based on the intermediate product. Accordingly, as the depth of the hidden layer driven in the SWD increases, the degree of security for input data may be increased. Thus, as security necessity for the input data increases, the processor 301 may set the security level so that the depth of the hidden layer driven in the SWD increases.
Referring to
When security processing for the input data is required (S710-YES), the processor 301 may separate the DNN (for example, 800, 1000, or 1110) and drive the separated DNN in the SWD (S721). For example, the processor 301 may differently set a part (for example, a head part) of the DNN (for example, 800, 1000, or 1110) that is driven in the SWD according to the security level.
For example, referring to operation S1640, when a security level corresponding to the input data is high (security level =high) (S1641), the processor 301 may separate and drive the DNN (for example, 800, 1000, or 1110) in the SWD so that a depth from an input layer of the DNN (for example, 800, 1000, or 1110 is the first depth (S1642). The rest of the DNN (for example, 800, 1000, or 1110) excluding a part having the first depth may operate in the NWD (S1643).
For example, referring to operation S1630, when the security level corresponding to the input data is low (security level =low) (S1631), the processor 301 may separate and drive the DNN (for example, 800, 1000, or 1110) in the SWD so that a depth from the input layer of the DNN (for example, 800, 1000, or 1110 is the second depth (S1632). The rest of the DNN (for example, 800, 1000, or 1110) excluding a part having the second depth may operate in the NWD (S1633). The second depth may have a smaller value than the first depth. For example, when the security level is high (S1641), a part from the input layer to the third hidden layer in the DNN (for example, 800, 1000, or 1110) may be driven in the SWD, and, when the security level is low (S1631), a part from the input layer to the first hidden layer in the DNN (for example, 800, 1000, or 1110) may be driven in the SWD.
Compared with
Referring to operation S1640, when a security level corresponding to the input data is high (security level=high) (S1641), the processor 301 may separate and drive the DNN (for example, 800, 1000, or 1110) in the SWD so that a depth from an input layer of the DNN (for example, 800, 1000, or 1110 is the first depth (S1642). The rest of the DNN (for example, 800, 1000, or 1110) excluding a part having the first depth may operate in the NWD (S1643).
Referring to operation S1635, when the security level corresponding to the input data is medium (security level=medium) (S1636), the processor 301 may separate and drive the DNN (for example, 800, 1000, or 1110) in the SWD so that a depth from the input layer of the DNN (for example, 800, 1000, or 1110 is the second depth (S1637). The rest of the DNN (for example, 800, 1000, or 1110) excluding a part having the second depth may operate in the NWD (S1638).
Referring to operation S1630, when the security level corresponding to the input data is low (security level=low) (S1631), the processor 301 may separate and drive the DNN (for example, 800, 1000, or 1110) in the SWD so that a depth from the input layer of the DNN (for example, 800, 1000, or 1110 is the third depth (S1632). The rest of the DNN (for example, 800, 1000, or 1110) excluding a part having the third depth may operate in the NWD (S1633).
In
When the security level is low (security level=low) (S1631), a part from the input layer to the first hidden layer in the DNN (for example, 800, 1000, or 1110) may be driven in the SWD.
For example, input data 1810 may be input to the DNN (for example, 800, 1000, or 1110). As the input data 1810 passes through the layers included in the DNN (for example, 800, 1000, or 1110, in detail, as the input data 1810 passes through a hidden layer, the degree of abstraction of the input data 1810 may increase.
For example, the DNN (for example, 800, 1000, or 1110) may be a neural network that receives an image and recognizes an object included in the image. For example, the DNN (for example, 800, 1000, or 1110) may receive an image and may output, as output data, information indicating that a ‘female person’ is included in the image.
Referring to
For example, the processor 301 may set the depth of the at least one hidden layer driven in the SWD, based on a deformation degree between each of a plurality of intermediate products respectively output by a plurality of hidden layers included in the DNN and the input data.
For example, the processor 301 may set the depth of the at least one hidden layer driven in the SWD, based on whether the deformation degree between each of the plurality of intermediate products respectively output by the plurality of hidden layers included in the DNN and the input data is equal to or greater than a certain value.
For example, the deformation degree may be calculated based on a similarity between input data and an intermediate product. For example, the deformation degree may be obtained based on a value indicating a similarity or quality difference between a converted image and the original image, such as a structural similarity index measure (SSIM) or a peak signal-to-noise ratio (PSNR). For example, the deformation degree is a value calculated based on the value of the SSIM, the PSNR, or the like, and may be a value indicating a low degree of similarity to input data.
For example, the processor 301 may calculate the deformation degree between each of the plurality of intermediate products respectively output by the plurality of hidden layers included in the DNN and the input data, and may control from the input layer to a hidden layer outputting an intermediate product of which deformation degree is equal to or greater than a certain limit value to be driven in the SWD. For example, when the deformation degree of the intermediate product 1830 created by passing through the third hidden layer is equal to or greater than the certain limit value, the processor 301 may perform a calculation corresponding to layers from the input layer to the third hidden layer in the SWD, and may perform a calculation corresponding to layers from the fourth hidden layer to the output layer in the NWD.
As another example, the processor 301 may set the depth of the at least one hidden layer driven in the SWD, based on a deformation degree increase rate between the plurality of intermediate products respectively output by the plurality of hidden layers included in the DNN.
For example, when the deformation degree of the intermediate product created by passing through the third hidden layer increases the most, the processor 301 may perform a calculation corresponding to layers from the input layer to the third hidden layer in the SWD, and may perform a calculation corresponding to layers from the fourth hidden layer to the output layer in the NWD. The deformation degree increase rate being large may indicate that, when passing through a corresponding hidden layer, the input data has a greatly-increased deformation degree and thus irreversibility increases. Thus, the increase in the irreversibility of the input data may make it more difficult to restore the input data by using an intermediate product created in the corresponding hidden layer. Therefore, according to an example embodiment, based on the deformation degree increase rate, a DNN may be separated and driven with respect to a hidden layer of which the deformation degree increase rate increases beyond a limit value.
In addition, in order to increase security performance, training of the DNN may be performed in a direction in which the degree of deformation of an intermediate product corresponding to a part of the DNN that is driven in the SWD increases.
For example, the processor 301 may train the DNN so that a deformation degree between an intermediate product output by at least one hidden layer driven in the SWD and input data increases.
For example, the DNN (for example, 800, 1000, or 1110) may be trained in a direction of improving a deformation degree of each layer of the DNN (for example, 800, 1000, or 1110), based on a loss function.
For example, an arbitrary layer may be selected in a training stage of the DNN (for example, 800, 1000, or 1110), and a deformation degree of the arbitrary layer may be additionally reflected in the loss function. In this case, the arbitrary layer may be provided in plural. For example, the degree of deformation of a corresponding layer may correspond to a similarity loss 1801 corresponding to an intermediate product, and the degree of deformation of the entire DNN may correspond to a Model's own loss 1802 of the model of a final output layer. Loss, which is the loss function corresponding to the degree of deformation of the corresponding layer included in the DNN, may be expressed as Loss=(Model's own loss)(1802)+similarity loss(1801).
As another example, in a training-completed DNN (for example, 800, 1000, or 1110), each of the plurality of hidden layers may be additionally re-trained (tuned). A part of the tuned DNN may be separated and driven in the SWD. For example, the degree of deformation of each layer may be first calculated, and then a layer having a high degree of deformation may be selected. The loss function of the DNN may be designed so that a result of evaluating a deformation degree between an output (for example, an intermediate result) of the selected layer and the input data input data is reflected. Therefore, the degree of deformation may be made considered during back propagation for the DNN. Moreover, at the completion of training, a final evaluation score for each layer may be stored as a data table, and may be used to designate a head part to be driven in the SWD.
As described above, in the training stage of the DNN, when training is performed so that the degree of deformation is increased based on the loss function of each stage, the irreversibility of an intermediate product generated in a hidden layer driven in the SWD may be increased. Accordingly, security performance may be further increased.
Referring to
The server 1950 may perform a calculation according to a neural network. For example, the server 1950 may store the neural network (or an AI model corresponding to the neural network), and may drive the neural network. As a result of performing a calculation according to the neural network, the display apparatus 1900 and the server 1950 may transmit or receive an AI model obtained by training the neural network, data necessary for training the AI model, and data required for the AI model to output a result, and the like.
Referring to
The DB 2010 may include a memory (not shown), and the server 1950 may store, in the memory, at least one of at least one instruction, program, or data required to perform a certain operation.
The DB 2010 may store pieces of data required by the server 1950 to perform the calculation according to the neural network. For example, the DB 2010 may store an AI model corresponding to a DNN.
The communication interface 2020 may include at least one component that enables communication with the display apparatus 1900. A detailed configuration of the communication interface 2020 is the same as that of the communication interface 360 of
The controller 2030 controls all operations of the server 1950. For example, the controller 2030 may control the DB 2010 and the communication interface 2020 by executing the programs stored in the DB 2010 of the server 1950. The controller 2030 may drive the DNN by executing the program stored in the DB 2010.
The controller 2030 may perform training of the AI model, output of a result of the AI model, an operation necessary for transmitting the result output by the AI model to the display apparatus 1900, or an operation necessary for data transmission/reception with the display apparatus 1900.
In
According to an example embodiment, when the input data is identified as not corresponding to the security data, the processor 301 included in the controller 310 may control a communication interface so that the input data is transmitted to an external device, for example, the server 1950, so that the entire DNN is driven within the external device. For example, when the input data is identified as not corresponding to the security data, the processor 301 included in the controller 310 may control the communication interface 360 to transmit the input data to the communication interface 2020 of the server 1950 so that the entire DNN is driven within the serve 1950. Therefore, the server 1950 may receive the input data, and may input the received input data to the DNN so that the received input data is processed, thereby obtaining output data. The server 1950 may transmit the output data to the display apparatus 1900. Accordingly, the processor 301 included in the controller 310 may obtain output data corresponding to the input data via the communication interface 360.
In general, a high-performance processor may be required in order to perform an operation corresponding to a DNN, because a calculation according to the DNN has a large amount of data calculation. Accordingly, when there is no need for security, output data corresponding to input data may be quickly and easily obtained by transmitting the input data to the external server 1950 having a large memory capacity and including a processor with a guaranteed high performance and processing the input data.
An operation method of a display apparatus according to an example embodiment may be embodied as program commands executable by various computer means and may be recorded on a non-transitory computer-readable recording medium. An example embodiment may include a non-transitory computer readable recording medium having embodied thereon at least one program including instructions which perform the operation method of the display apparatus, according to an example embodiment.
Each embodiment herein may be used in combination with any other embodiment(s) described herein.
The non-transitory computer-readable recording medium may include program commands, data files, data structures, and the like separately or in combinations. The program commands to be recorded on the non-transitory computer-readable recording medium may be specially designed and configured for the disclosure or may be well-known to and usable by one of ordinary skill in the art of computer software. Examples of the non-transitory computer-readable recording medium include a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape, an optical medium such as a compact disk-read-only memory (CD-ROM) or a digital versatile disk (DVD), a magneto-optical medium such as a floptical disk, and a hardware device specially configured to store and execute program commands such as a ROM, a random-access memory (RAM), or a flash memory. Examples of the program commands are advanced language codes that can be executed by a computer by using an interpreter or the like as well as machine language codes made by a compiler.
A machine-readable storage medium may be provided as a non-transitory storage medium. The ‘non-transitory storage medium’ is a tangible device and only means that it does not contain a signal (e.g., electromagnetic waves). This term does not distinguish a case in which data is stored semi-permanently in a storage medium from a case in which data is temporarily stored. For example, the non-transitory storage medium may include a buffer in which data is temporarily stored.
According to an example embodiment, a method according to various disclosed embodiments may be provided by being included in a computer program product. Computer program products are commodities and thus may be traded between sellers and buyers. Computer program products are distributed in the form of device-readable storage media (e.g., compact disc read only memory (CD-ROM)), or may be distributed (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or between two user devices (e.g., smartphones) directly and online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be stored at least temporarily in a device-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server, or may be temporarily generated.
While the disclosure has been illustrated and described with reference to various embodiments, it will be understood that the various embodiments are intended to be illustrative, not limiting. It will further be understood by those skilled in the art that various changes in form and detail may be made without departing from the true spirit and full scope of the disclosure, including the appended claims and their equivalents. It will also be understood that any of the embodiment(s) described herein may be used in conjunction with any other embodiment(s) described herein.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2021-0156053 | Nov 2021 | KR | national |
This application is a continuation of International Application No. PCT/KR2022/017797, filed on Nov. 11, 2022, designating the United States, and claiming priority to KR 10-2021-0156053, filed Nov. 12, 2021, the disclosures of which are all hereby incorporated by reference herein in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/KR2022/017797 | Nov 2022 | US |
| Child | 18093111 | US |
