Patent Application
20250182237
This disclosure generally relates to projected displays.
Electronic display screens are often used to view content. For example, a TV is typically wall mounted or set on a piece of furniture, and the TV includes a screen that displays content selected by a user viewing the content. The content is typically delivered via over-the-air connection (e.g., using an antenna), a wired connection (e.g., a coaxial, optical fiber, or HDMI cable), or a wireless connection (e.g., using a Wi-Fi receiver to stream content).
A display typically shows only one source of content (e.g., one program or video) at a time. A display device that is capable of simultaneously receiving, processing, and displaying multiple sources of content can show content from multiple sources by dividing the available display-screen area among the sources of content. For example, a TV that has picture-in-picture capabilities can show a first program on a first portion of the TV screen and a second program on a second portion of the TV screen. However, this often reduces the viewing quality (size, resolution, etc.) of each content.
Embodiments of this disclosure relate to projecting one or more content items on a surface. The content items may be projected in connection with, and may be based on, content displayed on a display screen, such as a TV screen. In contrast to picture-in-picture approaches, which force users to reduce viewing quality in order to view multiple content items, embodiments of this disclosure make use of available non-screen surfaces near a display screen, thereby allowing users to view multiple content items without suffering reductions in viewing quality. As explained more fully herein, embodiments of this disclosure identify a usable projection area surrounding a user's displays, determine the best locations for specific content within the usable projection area, determine the most appropriate content to display for a user based on their preferences, and detect the optimal position of a projector for the projected user content (e.g., by giving a user feedback on how to move the projector to avoid a skewed image or to avoid objects).
Projectors are often smaller, lighter, and more flexible (e.g., more mobile) than TV screens, making them ideal for use in a variety of environments. When projecting onto an imperfect surface (e.g., a crowded wall with pictures, mirrors, and shelves), users may need to spend a significant amount of time adjusting the projection site, for example so that projected content doesn't overlap with objects on a wall. Additionally, users may have to re-setup the projector whenever using it in a new environment, which can be frustrating. Any particular projection site is often very different from other projection sites (e.g., different objects are present, in different sizes and locations), and optimizing a projection site can be a time-consuming and difficult process, and may require manually reorganizing the projection site (e.g., moving or removing pictures or shelving, etc.)
Step 110 of the example method of
In particular embodiments, accessing an image of the scene can include capturing an image of the scene. The image may be captured by a camera that is part of a projector. For instance, in the example of
Step 120 of the example method of
In particular embodiments, an image may be skewed relative to a display device and a surface near the display device. For instance, the center of a camera's field of view may not be parallel to the surface normal, resulting in a slant between the image's perspective and the surface. A skewed image can result in a skewed projection from a projector, and therefore particular embodiments may deskew an accessed image. For example, geometric features in the scene (as captured by the image or by another image) may be used to deskew the image. For example, at least a portion of an object such as a picture frame or the edges of the display screen (e.g., a corner of the display screen) may be detected in an image (e.g., using objection-recognition techniques). The geometric shape of the detected object may be compared to a rectangular shape, and the image may be deskewed (i.e., transformed) so that the detected object's shape in the deskewed image is rectangular. As another example, an image may displayed on the display screen (e.g., a predetermined geometric shape such as a circle, a square, etc.) and the accessed image may be deskewed so that the displayed image on the display screen is unskewed (e.g., a displayed circle appears as a circle rather than as an ellipse, etc.). The transformation to deskew an image can then be used to generate an image that represents the scene as if the camera was perpendicular to the scene. As a result, a camera (e.g., camera 255 in projector 250) need not be oriented perpendicular to a scene in order to capture a usable image of the scene, improving the placement options of the camera and/or the projector.
In particular embodiments, step 120 of the example method of
In particular embodiments, determining portions of the surface captured in the image that are not suitable for projection includes determining an exclusion area of the surface that is not suitable for projection. For example, the exclusion area may include one or more objects on the surface or occluding the surface, one or more colors or textures that are unsuitable for projection onto, or a brightly-lit area that would make the projection difficult to discern. For example, using the image accessed in step 110, objects may be identified in the maximum projection area of the image. The areas occupied or occluded by these objects are excluded from the potential projection area.
Images 510, 520, and 530 illustrate different potential layouts for the resulting projection based on the number of displays to project. Image 510 illustrates a single projected display 518 for a first content. Image 520 illustrates two projected displays: projection 522 for a first content and projection 524 for a second content. Image 530 illustrates three projected displays. In each instance, the projected displays are sized and placed on the region of wall 512 that is not part of the exclusion area, as determined by the detected objects that obscure the surface. In addition, if portions of wall 512 were deeply colored (e.g., a deep red color) or strongly textured (e.g., a stucco wall), then those portions may be excluded from the possible projection area, for example if other areas do not contain those strong visual elements.
In particular embodiments, once an available projection area is determined, the available projection area may be divided into potential projection regions. Each region may be of a particular shape. For example, the available projection area may be divided into the largest contiguous rectangular regions that make up the available projection area. For example, image 510 of
The available content to project may be determined by a user or automatically, e.g., based on user preferences. In particular embodiments, the available content to project may be based on the content displayed on a display screen, e.g., a TV. For example, a program such as a sporting event may have associated available projection content that includes information about the participants (e.g., a roster list or a currently playing list, statistical information, etc., which may focus on select participants that are most relevant to a user), social media posts about the event, current standings, related maps (e.g., a map of a race course or a golf course), and so on. As another example, gaming content may have associated available content that includes a map of the current level, scores or other statistical information, videos such as walkthroughs or views from the perspective of other participants, etc. As another example, exercise or fitness content may have associated available projection content that includes health-related information such as user vitals (e.g., heart rate, breathing, etc), statistics regarding the workout and/or the user's workout goals, a current soundtrack, etc. As another example, displayed content related to smart devices (e.g., displayed content from a video camera) may have associated available projection content that includes feeds from other cameras (e.g., other security cameras), information from smart devices such as a thermostat, etc. This disclosure, however, is not limited to the specific examples of displayed and projected content described above.
In particular embodiments, the available content may be determined by searching the internet or data on one or more connected devices for content related to the content displayed on the display. The content displayed on the display may be determined based on metadata regarding the content (e.g., metadata accompanying a streamed program) or based on content recognition (e.g., based on detecting text or images in the content). Potential projected content may be provided by one or more applications or third parties. In particular embodiments, a user may specify content available for projection, and that specification may be associated with particular programs or types of programs.
In particular embodiments, a user may select which particular content they would like projected in any particular instance. For example, potential content may be determined for a particular content displayed on a TV, and that potential content may be identified to a user (e.g., in a list, etc.). The user may then select which particular content they would like projected. In particular embodiments, a user may select particular content for projection based on a type of content. For example, a user may specify that video games should include certain types of projected content (e.g., a map of the level, etc.), while other types of content (e.g., statistics) should be projected when watching a sporting event. In particular embodiments, a user may specify types of content to project when the display screen is off (e.g., project an image of pictures, a clock, a calendar, the weather, etc.). In particular embodiments, a user may specify types of content to project based on a time of day and/or on the identity of the user.
In particular embodiments, a user may specify a priority for content available for projection. The priority may be used to determine which content to project, if the available projection area is not sufficiently large to accommodate all selected projections. In particular embodiments, a user may specify preferred aspect ratios, sizes, resolutions, and/or other display aspects, both for projected content generally and for particular types or instances of projected content. For example, for a racing event, a user may specify that the current driver standings may be a first priority, while a track map may be a second priority. As explained below, a priority may be used to determine how and whether to project content to a user (e.g., where the content should be projected, at which size the content should be projected, whether the content should be projected at all, etc.). In particular embodiments, certain video settings such as aspect ratio or resolution may be limited by or specified by the content source. In particular embodiments, a user's content preferences may be determined from the user's historical choices regarding content preferences.
In particular embodiments, step 120 of the example method of
In particular embodiments, a machine-learning model may use a deep-learning model that includes an encoder and a layout score predictor. The encoder may be a convolutional neural network that extracts features from the image. The extracted features may be represented by a 3-dimensional tensor. The layout score predictor may be a light-weight neural network. The inputs of the score predictor include the candidate layout (i.e., each projection site's position, aspect ratio, and size) and the accessed image of the projection environment. The output from the deep-learning model is the corresponding score for each layout. In particular embodiments, a set of one or more heuristic rules may be used as post-processing to refine the layout scores provided by the deep-learning model. For example, one heuristic rule may be that the score assigned to a layout is set to negative if the proposed projection site overlaps with detected objects (e.g., pictures on the wall).
Step 424 includes predicting a layout score for a candidate layout. In particular embodiments, step 424 is performed by a trained neural network. For example, the neural network may be trained on labelled ground-truth data that includes encoded image features, a particular layout, and a corresponding ground-truth score. A candidate layout may consist of the number of projected displays (which may be determined based on the user's preferences or selections), the size of each display, the location of each display, and an aspect ratio of each display. Different candidate layouts may have different parameters for the size, location, and/or aspect ratio of one or more of the number of projected displays.
In particular embodiments, in order to predict a layout score, a neural network receives encoded image information and encoded layout information. Each of these encodings may be represented as a three-dimensional tensor. The neural network may include layers for integrating the two tensors. The trained neural network may include layers that predict the layout score for a potential layout based on the encoded inputs.
Determining the highest score may be performed by any of a number of different approaches. For example, a greedy-search scoring approach may start with a particular projected display (e.g., a projected display corresponding to a user's highest assigned priority) and determine the parameters (e.g., size and location) of the display that maximizes the score of that display. However, this approach may not lead to the globally optimal solution for all the projected displays; for example, the first projected display's highest score may occur when that display takes up most or all of the available projection area, leaving insufficient room for other projected displays. Another scoring option is to test all possible layout combinations, but this approach can be computationally intensive and time-consuming. Another approach may be to use a beam search algorithm, which picks the “k” best sequences so far based on the total score of each screen. Here, k is a fixed hyperparameter; in general, the quality of the results increase as k increases, but the speed of the search algorithm decreases as k increases.
In particular embodiments, a scoring approach may both score a particular projected display, based on the encodings of that's display in the layout, and also may predict a future score for the remaining displays in the layout.
Column 620 illustrates the next iteration. In this iteration, the layout for display 1 is already determined in the previous iteration, and thus its score is fixed (as its layout doesn't change). Display 2 is placed in a variety of different layouts, and the score for each layout is determined. In addition, the iteration predicts the layout score for the third layout, given the layout of the first two displays (e.g., if the first two displays take up all the available space, they may receive a high score, but the third display would have a low predicted score, regardless of which particular possible layout parameters it took). In column 620, the scores shown in parentheses are, from left to right, the scores of the first display, the second display, and the predicted score for the third display. The two highest scoring layouts are selected, and in the next iteration (column 630) the third display is placed in a variety of layouts, and each layout is scored. In this example, the two highest scoring layouts are selected for presentation to the user.
In particular embodiments, a neural network may use two branches (two sets of layers): one for scoring a current projected display under consideration and one for predicting the future score of the remaining displays that have not yet been scored. In particular embodiments, a single branch may perform both scoring and predicted scoring. In a two-branch embodiment, however, the neural network is more interpretable, as a projected display score and a predicted future score for remaining displays are produced separately, and therefore troubleshooting and training can be performed by considering the output of each branch separately.
Returning to the example of
While the example of
In particular embodiments, projected content may be shown with one or more visual enhancements, and such visual enhancements may be based on visual aspects of the environment, e.g., as captured by the accessed image. For example, a visual enhancement may be a color scheme or background pattern, or both, to project along with the projected content.
Step 130 of the example method of
In particular embodiments, some or all of the method of
As described throughout this disclosure, while particular embodiments of the projection techniques described herein may result in projecting content in association with a display screen, the techniques described herein are also used to project content when no display screen is present (e.g., to determine the appropriate layout for projecting a variable number of content items onto a surface).
This disclosure contemplates any suitable number of computer systems 800. This disclosure contemplates computer system 800 taking any suitable physical form. As example and not by way of limitation, computer system 800 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 800 may include one or more computer systems 800; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 800 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems 800 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 800 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
In particular embodiments, computer system 800 includes a processor 802, memory 804, storage 806, an input/output (I/O) interface 808, a communication interface 810, and a bus 812. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.
In particular embodiments, processor 802 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 802 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 804, or storage 806; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 804, or storage 806. In particular embodiments, processor 802 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 802 including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor 802 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 804 or storage 806, and the instruction caches may speed up retrieval of those instructions by processor 802. Data in the data caches may be copies of data in memory 804 or storage 806 for instructions executing at processor 802 to operate on; the results of previous instructions executed at processor 802 for access by subsequent instructions executing at processor 802 or for writing to memory 804 or storage 806; or other suitable data. The data caches may speed up read or write operations by processor 802. The TLBs may speed up virtual-address translation for processor 802. In particular embodiments, processor 802 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 802 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 802 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 802. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.
In particular embodiments, memory 804 includes main memory for storing instructions for processor 802 to execute or data for processor 802 to operate on. As an example and not by way of limitation, computer system 800 may load instructions from storage 806 or another source (such as, for example, another computer system 800) to memory 804. Processor 802 may then load the instructions from memory 804 to an internal register or internal cache. To execute the instructions, processor 802 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 802 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 802 may then write one or more of those results to memory 804. In particular embodiments, processor 802 executes only instructions in one or more internal registers or internal caches or in memory 804 (as opposed to storage 806 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 804 (as opposed to storage 806 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 802 to memory 804. Bus 812 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 802 and memory 804 and facilitate accesses to memory 804 requested by processor 802. In particular embodiments, memory 804 includes random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 804 may include one or more memories 804, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.
In particular embodiments, storage 806 includes mass storage for data or instructions. As an example and not by way of limitation, storage 806 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage 806 may include removable or non-removable (or fixed) media, where appropriate. Storage 806 may be internal or external to computer system 800, where appropriate. In particular embodiments, storage 806 is non-volatile, solid-state memory. In particular embodiments, storage 806 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 806 taking any suitable physical form. Storage 806 may include one or more storage control units facilitating communication between processor 802 and storage 806, where appropriate. Where appropriate, storage 806 may include one or more storages 806. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.
In particular embodiments, I/O interface 808 includes hardware, software, or both, providing one or more interfaces for communication between computer system 800 and one or more I/O devices. Computer system 800 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system 800. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 808 for them. Where appropriate, I/O interface 808 may include one or more device or software drivers enabling processor 802 to drive one or more of these I/O devices. I/O interface 808 may include one or more I/O interfaces 808, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.
In particular embodiments, communication interface 810 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 800 and one or more other computer systems 800 or one or more networks. As an example and not by way of limitation, communication interface 810 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 810 for it. As an example and not by way of limitation, computer system 800 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 800 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system 800 may include any suitable communication interface 810 for any of these networks, where appropriate. Communication interface 810 may include one or more communication interfaces 810, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.
In particular embodiments, bus 812 includes hardware, software, or both coupling components of computer system 800 to each other. As an example and not by way of limitation, bus 812 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 812 may include one or more buses 812, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.
Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.
Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.
The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend.
