REMOTELY CONTROLLING A SELF-PROPELLED DEVICE IN A VIRTUALIZED ENVIRONMENT

Abstract

A computing device operating as a controller can obtain image data from a camera component. The computing device can determine a location of the self-propelled device relative to the camera based on the image data. A virtual content may be generated on the computing device based at least in part on the location of the self-propelled device.

Description

TECHNICAL FIELD

Examples described herein relate to remote control of devices, and more specifically, to implementing remotely controlling devices in a virtualized environment.

BACKGROUND

Various kinds of remotely controllable devices exist. For example hobbyists often operate “RC” devices in the form of cars, trucks, airplanes and helicopters. Such devices typically receive commands from a controller device, and after movement (e.g., direction or velocity) based on the input. Some devices use software-based controllers, which can be implemented in the form of an application running on a device such as a smart phone or a tablet.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for implementing a remotely controlling a self-propelled device in context of a virtual environment, according to an embodiment.

FIG. 2 illustrates a method for controlling a self-propelled device for use in a virtual environment, according to an embodiment.

FIG. 3 illustrates a method for implementing a virtual environment for use with a remotely controlled self-propelled device, according to an embodiment.

FIG. 4 illustrates an example of a virtual environment on a device that controls movement of another device.

FIG. 5 illustrates an example hardware diagram a computing device on which a controller for a self-propelled device can be implemented.

FIG. 6A and FIG. 6B illustrate examples of self-propelled devices, under some embodiments.

DETAILED DESCRIPTION

According to some embodiments, a computing device is operated to process image data in order to track a movement or position of a self-propelled device.

Still further, in some embodiments, a self-propelled device is tracked, and position information determined from tracking the self-propelled device is integrated with a virtual environment. The virtual environment can include facets that are based on the position and/or state of the object, as well as on other aspects of the real-world environment as determined from tracking and/or communicating with the self-propelled device.

In an embodiment, image data is generated by a camera component. The camera component can be provided as part of, or alternatively coupled to, a computing device that acts as a controller for a self-propelled device. From the image data, a location of the self-propelled device can be determined (e.g., relative to the computing device). Content can be generated based at least in part on the location of the self-propelled device as the self-propelled device is moved or otherwise operated through the controller.

In some variations, sensor input is received from the self-propelled device. In particular, the sensor input can be indicative of the position of the self-propelled device. By way of example, the self-propelled device can include a gyroscope, an inertial mass unit (IMU), a GPS, an accelerometer, a light sensor, and/or proximity sensor. The sensor information communicated from the self-propelled device can include readings or output from the sensors. Additionally, the sensor information communicated from the self-propelled device can be in raw form, or in processed form (e.g., numerical values determined from a combination of sensors).

According to another embodiment, a computing device operating as a controller can obtain image data from a camera component. The computing device can determine a location of the self-propelled device relative to the camera based on the image data. A virtual content may be generated on the computing device based at least in part on the location of the self-propelled device.

As used herein, the term “substantially” means at least almost entirely. In quantitative terms, “substantially” means at least 80% of a stated reference (e.g., quantity of shape).

In similar regard, “spherical” or “sphere” means “substantially spherical.” An object is spherical if it appears rounded, contoured or elliptical to a user. Objects which include, for example, half-domes, quarter-domes, elliptical (dimension of one axis larger than another) can be considered spherical as used herein.

DETAILED DESCRIPTION

FIG. 1 illustrates a system for remotely controlling a self-propelled device in context of a virtual environment, according to an embodiment. In FIG. 1, system 50 includes a controller and a self-propelled device (“SPD”) 10. The controller 100 can correspond to, for example, a mobile computing device, such as a voice/telephony device for cellular communications, or a roaming device that can use local wireless or similar communications to communicate with network nodes and/or other devices. By way of example, controller 100 can correspond to a smart phone, tablet, netbook are other mobile computing device that utilizes application layer logic (e.g., downloaded application or “app”) or other programming to control the SPD 10. In variations, the controller 100 can be implemented as a dedicated or specialized device that controls self-propelled devices.

As described by various embodiments, SPD 10 can be operated to move under control of another device, such controller 100. In some embodiments, SPD 10 is configured with resources that enable one or more of the following: (i) maintain self-awareness of orientation and/or position relative to an initial reference frame after the device initiates movement; (ii) process control input programmatically, so as to enable a diverse range of program-specific responses to different control inputs; (iii) enable another device to control its movement using software or programming logic that is communicative with programming logic on the self-propelled device; and/or (iv) generate an output response for its movement and state that it is software interpretable by the control device.

With further reference to FIG. 1, controller 100 includes a user interface 110 an object detector 120, and an SPD interface 130. As described in greater detail, the user interface 110 can be operated to generate content that is based partially on the SPD 10, and/or the location of the SPD 10 relative to the controller 100. In particular, the user interface 110 can be used to generate content that provides a virtual (or augmented reality) context for the SPD 10. The user interface 110 can also include a command interface 112 that translates user input and/or other events into commands 119 for controlling the SPD 10 via the SPD interface 130.

In operation, controller 100 may include a camera interface 134, which obtains image data 113 depicting a real-time state of the SPD 10 in a scene that is captured by the camera of the controller 100. In one embodiment, the camera interface 134 receives image data that represents a series of images (or a video) of the scene in which the SPD 10 is present (e.g., moving under control of the controller 100). The object detector 120 can include image processing logic to analyze the image data 113, and to identify objects of interest from the image data 113. In particular, the object detector 120 can process the image data 113 in order to identify a depiction of the SPD 10, and optionally, a depiction of objects or landmarks in the scene that are deemed to be of interest.

In some implementations, the object detector 120 includes image analysis logic for analyzing image data 113 for presence of spheres, half spheres or portions thereof (e.g., quadrant of sphere). Thus, object detector 120 can include image processing logic for locating a shape that is indicative of SPD 10 (e.g., detect circle or portion thereof that is indicative of a spherical aspect of the SPD 10). Other characteristics such as illumination (e.g., LED illumination) or structural features of the SPD 10 can also be used to both detect the SPD 10, or to detect an orientation or other aspect of the SPD 10 in its environment. The object detector 120 can also include image processing resources that are configured to detect other specific kinds of objects other than spheres, such as objects typically encountered in the real world environment of SPD 10. These can include, for example, wall structures, table legs, or surfaces (e.g., carpet, grass etc.).

In one implementation, object detector 120 uses dimensional analysis to determine, from the image data 113, a relative position or distance of the SPD 10 from controller 100. In particular, one embodiment provides for the SPD 10 to be spherical. The object detector 120 can use dimensional analysis by comparing a dimension of the depiction for the SPD 10 (e.g., circle or portion thereof) with a reference dimension for the spherical housing of the self-propelled device.

In one variation, the controller 100 can also use local sensor information 109, provided from sensor devices resident on the computing device of controller 100. For example, local sensor information 109 can be provided by an accelerometer, gyroscope, magnetometer, or Global Positioning System (GPS) device that is resident on the computing device of the controller 100. As an addition or alternative, some implementations provide for the controller 100 to use the relative height of the controller 100 (e.g., distance from ground), and/or the orientation of the controller 100 with respect to a horizontal plane, in order to determine position information for the SPD 10.

As an addition or alternative, SPD 10 can communicate sensor input 11 to the controller 100. The sensor input 11 can correspond to, for example, information determined from an inertial mass unit (“IMU”), gyroscope, accelerometer, magnetometer, or GPS. The sensor input 11 can be either raw data, or data processed on the SPD 10 before being communicated to the controller 100. In variations, the controller 100 includes sensor logic 128, either with the object detector 120, or as a separate logical component (e.g., plug-in), to handle device sensor input 111 (corresponding to sensor input 11 provided from the SPD 10). The sensor logic 128 can determine sensor location information 115, corresponding to information that is indicative, probative or otherwise relevant to a position, relative location, or physical state of the SPD 10. Sensor location information 115 can be used by the object detector 120, in combination with image data 113, to determine a relative position for the SPD 10. The information determined for the SPD 10 can include, for example, a distance of the SPD 10 from the controller or reference point in either 1-, 2- or 3-dimensions, or a coordinate of the SPD 10 within a particular reference frame. The information determined from the SPD 10 can also be indicative of a physical state of the device (e.g., LED on device is on, device is going downhill or has hit obstruction, etc.).

In other variations, the position information for the SPD 10 can be communicated by the SPD to the controller 100 via a wireless link (e.g., Bluetooth). For example, the SPD 10 may be self-aware by way of its own geo-aware resources. In particular, the SPD 10 can include sensors and devices such as an accelerometer, a Global Positioning System (GPS) unit, a gyroscope and/or a magnetometer.

The object detector 120 can communicate raw position information 117 to one or more content output components of the controller 100. In one implementation, a coordinate mapping component 122 maps the raw position information 117 to coordinate positions 121 of an alternative reference frame that is specific to a particular virtual environment. The alternative reference frame can be generated and maintained through, for example, the content generation component 124, as part of a virtual environment. The content generation component 124 may independently use content input from, for example, a content library 125 in order to generate aspects of a virtual environment.

In some variations, the content generation component 124 can obtain virtual environment parameters 129 for use in creating specific virtual environments. For example, specific games or gaming scenarios may carry alternative virtual parameters 129. The virtual environment parameters 129 can, for example, (i) map raw position information 117 to virtual coordinates, (ii) convert detected objects into graphical representations (e.g., transfer real-world object into an anime gaming feature), and (iii) provide rules for a physics engine (e.g., application of Newtonian using virtual reference frame). Using the virtual environment parameters 129 and the content library 125, content generation component 124 can update the virtual environment using the raw position information 117 determined from tracking the position of the SPD 10. For example, a graphic representation of the SPD 10 can be reflected in the virtual environment, based on the position information and the alternative coordinate system provided by the coordinate mapping component 122.

As an addition or alternative, the content generation component 124 can include logic to (i) flag certain coordinates (e.g., determined from, or corresponding to raw position information 117) as landmarks or points of interest for future use in a virtual environment, and/or (ii) access a data store of historical content (“historical content store 127”) that is based on prior landmarks or points of interest.

The content generation component 124 can provide content output 123 for user interface 110. The user interface 110 can create a presentation that depicts the content output 123. The user interface 110 can also include or modify the content output 123 to allot for input from the user that can affect operation of the SPD 10, as well as to permit manipulation of the content presented. For example, the user interface 110 can include a framework of graphic features that the user can interact with in order to (i) alter a virtual aspect provided from content generation component 124, and/or (ii) adjust performance or operation (e.g., speed up, change direction, stop, execute sequence, spin etc.) of the SPD 10.

In more detail, the user interface 110 can include the command interface 112 to enable the user to control the SPD 10. For example, the command interface 112 can correspond to one or more features or aspects of the virtual environment that allow the user to enter input for purpose of controlling movement, operations and other aspects of the SPD 10. As described elsewhere, subsequent control of SPD 10 (e.g., movement) can also affect the virtual environment.

Methodology

FIG. 2 illustrates a method for controlling a self propelled device for use in a virtual environment, according to an embodiment. FIG. 3 illustrates a method for implementing a virtual environment for use with a self controlled device, according to an embodiment. Examples such as described with FIG. 2 and FIG. 3 can be implemented using components of the system such as described with FIG. 1. Accordingly, reference may be made to elements of FIG. 2 and FIG. 3 for purpose of illustrating suitable components or elements for performing a step or sub-step being described.

With reference to FIG. 2, image data can be generated for a particular scene in real time, with respect to the operation or movement of a self-propelled device (210). For example, controller 100 can be operated on a computing device that includes a camera. A user can point the camera to capture at the SPD 10 (e.g., operate the camera in video mode) in order to receive image data 113.

A location of a self-propelled device is detected from the scene (220). According to some embodiments, the image data is used to detect a location of the self-propelled device within a given scene. For example, in order to locate the SPD 10, the image processing logic can include object-type specific detectors to locate (i) shapes, such as circles, semi-circles, or portions thereof, in implementations in which the SPD 10 is a sphere, (ii) surface color (e.g., white), (iii) surface ornamentation or mark (e.g., visual code), or (iv) specific characteristic structural features. The image processing logic can also include object-type specific detectors to identify, from the image data objects that are likely to be found in the environment of the SPD 10. For example, object detector 120 can implement image processes to detect walls, rugs, specific kinds of floorings, pet dishes, steps, or lighting conditions that may be present in the environment of the SPD 10.

As an addition or alternative, controller 100 can use sensor data 109, communicated from the SPD 10, in order to determine the position of the SPD in a given scene. For example, SPD 10 can include resources for being aware of its position relative to, for example, controller 100. U.S. patent application Ser. No. 13/342,853, which is incorporated by reference herein, describes various sensor inputs and logic for enabling a device such as SPD 10 be self-aware of its location.

Content is generated based on the location of the SPD 10 (230). In particular, various kinds of virtual content can be generated that are based on movement, physical presence of other aspects of SPD 10 in the real-world environment. In one implementation, the content provided includes a map or other depiction of physical space (232). For example, the SPD 10 can be operated in a room, and a graphic map of the room, or of an alternative environment (e.g. gaming environment, such as a race track) can be generated that is based on the position of the SPD 10.

As an addition or alternative, the generated content can correspond to a virtual environment (234), such as, for example, in environment in which the SPD 10 has a virtual representation, and the surrounding environment is based partly on the real-world environment of the SPD 10. A set of virtual parameters can be determined as part of generating content. For example, a virtual map can be structured, and/or used as a framework or basis for conducting other events and aspects of a virtual environment. The virtual map can be determined from, for example, a path of the SPD, identified landmarks and other events, then translated into an alternative reference frame for the specific virtual environment.

With respect to FIG. 3, the location of a self-propelled device is determined from image data (310). In one implementation, the self-propelled device can have a geometry or shape that is detectable from the surrounding environment when viewed through image data. Dimensional analysis can be performed on the detected shape within the image data in order to determine a relative position (e.g., distance) of the self-propelled device with respect to a particular reference, such as the location where the image is captured (e.g., controller 100). As an example, the self-propelled device can be implemented as a sphere, and image processing logic of the controller can be configured to detect circles are ellipses (or portions thereof) that can correspond to the sphere. Based on the dimensions of the circle or ellipse, the relative position of the self-propelled device can be determined. The relative position can be determined as a coordinate (e.g., in 2- or 3-D), or through single linear dimension such as distance from the point where the image is captured.

In an embodiment, the location of the self-propelled device is recorded as the object moves about its scene (320). Thus, the self-propelled device can be tracked as it moves about. A path can be determined for the self-propelled device based on its past locations in a given timeframe.

The locations of the self-propelled device can be recorded over a given duration (330). For example, the device can be operated in a given time frame, and the movement of the device can be sampled by way of image processing. The self-propelled device can also be tracked so that the device's path can be determined for the given time period. The path or recorded locations of the self-propelled device can then be integrated into a virtual environment. More generally, the position of the self-propelled device can be integrated with content that is otherwise independent of the real-world environment of the self-propelled device.

In particular, some embodiments provide for recording landmarks that the self-propelled device experiences (332). Landmarks can include, for example, a starting point, an obstruction such as a wall or table leg, variations of the underlying surface in which to self-propelled device operates (e.g., type of flooring), variations in lighting (e.g., well lighted place versus dark place), and variations in the gradient of the underlying surface.

As another addition our variation, the recorded path of the self-propelled device can be used to generate a map (334). The map can define geographic parameters that are translated into a virtual environment. For example, the walls of the room can be translated into a track. The path of the object can also be translated into a virtual path. The map can also include the landmarks, as determined in (332).

EXAMPLE

FIG. 4 illustrates an example of a virtual environment on a device that controls movement of another device. In an example of FIG. 4, a mobile device 400 operates as a controller for a self-propelled device 410. In particular, the mobile device 400 can execute software (e.g., downloaded application) which creates virtual environment that is based on the movements of the self-propelled device 410. For example, mobile device 400 can execute different applications, of which each generate separate virtual environments, and each virtual environment can have a corresponding content, theme, simulated physics, and user involvement (e.g., how and what the user can respond to, input features etc.).

A field of view for a camera component 404 of the device 400 can monitor the region 411 for movement or changes to the physical state of the self-propelled device 410. The region 411 can also extend beyond the field of view. For example, the region 411 can encompass multiple rooms or an extended path traversed by the self-propelled device.

In the example of FIG. 4, the mobile device 400 renders a virtual environment in which the self-propelled device is reflected as a vehicle moving about obstacles. Thus, the self-propelled devices depicted in alternative, graphic form. The mobile device 400 may execute an application that coordinates a virtual environment with the real-world events related to the movement of the self-propelled device 410 in a given area. In an example of FIG. 4, the self-propelled device 410 is depicted as an automobile 412 on the mobile device 400. An application, running on the mobile device 400, renders the self-propelled device 410 as a vehicle. A remainder of the virtual environment can reflect hazards or other facets that tie into what the self-propelled device 410 experiences.

According to some embodiments, the self-propelled device can be moved in the real world amongst obstacles, and landmarks or points of interest. A region 411 in which the self-propelled device 410 is moved in the real-world can also be mapped into the virtual environment, and some landmarks or points of interest within the region 411 can be reflected in alternative forms in the virtual environment. For example, real-world obstruction such as walls 415 can be reflected as an obstacle 422 (e.g., one that is smaller in size) in the virtual environment. As another example, a charging dock 418 can be used as a landmark for the self-propelled device in maintaining a point of reference for future use. For example, the location of the charging dock 418 can be recorded and mapped to a virtual garage 428.

In some embodiments, mobile device 400 uses a camera to track the object in a given real-world region. The position of the object in the given region can be reflected in the corresponding virtual environment. Each virtual environment in use on the mobile device 400 can map to the real world based on a set of transformations. Thus, for example, mobile device 400 can use its camera to track the self-propelled device 410, and the position of the self-propelled device (as well as other physical information) can be used to coordinate the position of a virtual object of interest in a corresponding virtual environment.

Additionally, the self-propelled device 410 can be shaped so that it is readily detectable from image data. Still further, the self-propelled device 410 can be shaped so that the relative depth or distance from the mobile device 400 can be determined based on dimensional analysis of the detected shape versus a reference. For example, as shown by an example of FIG. 4, the self-propelled device 410 can be spherical, so that the detected size of the object in the image data can be correlated to a physical distance measure.

Optionally, the mobile device 400 can use sensor input from the self-propelled device, as an alternative or addition to using image data. The self-propelled device can communicate, for example, information determined from the IMU of the device. This information can enable the device to determine its own location relative to a particular reference frame. The device 400 can signal the information to the mobile device 400 using a wireless link.

Controller Hardware Diagram

FIG. 5 illustrates an example hardware diagram a computing device on which a controller for a self-propelled device can be implemented. In particular, a computing device 500 can configured to implement functionality described with, for example, controller 100 as described by some embodiments of FIG. 1. In some implementations, the computing device 500 is a mobile device, such as a cellular telephony messaging device that (e.g., IPHONE model device manufactured by APPLE INC.) or tablet (e.g., IPAD model device manufactured by APPLE INC.) which executes one or more applications for controlling movement and other operations of a self-propelled device. For example, in the context of FIG. 1, system 50 may be implemented by memory and processor resources as described in FIG. 5.

In one embodiment, the computing device 500 includes one or more processors 510, memory resources 520, a display device 530, one or more communication sub-systems 540 (including wireless communication sub-systems), and one or more sensors 560. According to different implementations, the communication sub-systems 540 enables the computing device 500 to exchange wireless communications with a self-propelled device using wireless communication mediums and protocols (e.g., WI-FI, BLUETOOTH, Infrared). In some embodiments, the computing device 500 can also include one or more input mechanisms 550 (e.g., a button, a switch, a touch-sensitive input device).

The processor 510 can be configured with software and/or other logic to perform one or more processes, steps and functions described with the various examples described herein, such as described by FIG. 1 through FIG. 3. Accordingly, the processor 510 can be configured, with instructions and data stored in the memory resources 520, to implement functionality described with controller 100 (as described with FIG. 1). The memory resources 520 may store instructions (e.g., applications) used by the processor 510 in implementing the functionality of the controller 100. For example, the memory resources 520 may store instructions for enabling the processor to (i) determine position information for the self-propelled device, (ii) implement a virtual reality where a graphic representing the self-propelled device is presented, and (iii) communicate and command the self-propelled device.

In performing operations of the controller, the processor 510 may utilize various forms of input. In particular, processor 510 can receive user input via the input mechanisms (e.g., touch sensor integrated with display, buttons, voice input etc.). Furthermore, the processor 510 can also receive sensor input from one or more sensors 560 that are included with the computing device 500. Examples of sensors 560 include accelerometers, proximity sensors, capacitive sensors, light sensors, magnetometers, inertial mass unit (or IMU) or gyroscopes. As described with an example of FIG. 1, processor 510 may also receive image data from image capture device 570, such as a charged-coupled device (CCD) camera that is able to capture video and images.

In one embodiment, the processor 510 can control the display device 530 to provide virtual or “augmented reality” content. As described with other examples, such content may include alternative graphic representations of a self-propelled device, as well as virtual elements or facets which are affected by real-world events, particularly relating to the position, movement and state of the self-propelled device that is under control of the computing device 500.

Self-Propelled Device

FIG. 6A and FIG. 6B illustrate examples of self-propelled devices, under some embodiments. With reference to FIG. 6A, a self-propelled device 600 can be spherical in shape, and include internal components 622 such as a drive system, sensors (e.g., gyroscope, inertial mass unit (IMU), GPS, accelerometer, light sensor, proximity sensor, etc.), processing resources and communication interface for communicating with a controller using a wireless communication medium (e.g., Bluetooth, Wi-Fi etc.). Numerous examples for the construction and operation of the self-propelled device, in accordance with an example such as provided in FIG. 6A and elsewhere in this application, are disclosed in, for example, U.S. patent application Ser. No. 13/342,863 and U.S. patent application Ser. No. 13/261,647; both of which are hereby incorporated in its entirety by reference.

In some examples, the self-propelled device 600 includes a housing 610 that is substantially spherical. In variations, the housing can be elliptical, semi-spherical, or include spherical portions. Other suitable geometric shapes may also be used. However, one advantage provided by using a spherical shape for the housing 610 is that the shape of the device remain symmetrical when viewed from various angles. This facilitates using image analysis for purpose of locating the object by relative position or distance.

With reference to FIG. 6B, a combination device 650 includes a second device 660 and a spherical housing element 670. The spherical housing element 670 may be modularized so as to be assembled onto the second device 660 as a self-contained unit, either during manufacturing or after-sale. For example, the second device 660 can correspond to a radio-frequency controlled device (e.g., car) that is augmented with the spherical housing element 670 to facilitate detection/position determination with use of image processing. In particular, the exposed spherical aspect of the housing element 670 can be imaged and subjected to dimensional or geometric analysis in order to approximate position information (e.g., distance from camera). As with other examples, the combined device can also include a variety of sensors 672, such as gyroscope, inertial mass unit (IMU), GPS, accelerometer, light sensor, or proximity sensor. For example, the housing element 670 can use wireless communications to signal additional sensor data to a controller device. The sensor data communicated from the sensors can also be indicative or determinative of the position of the housing element 670.

It is contemplated for embodiments described herein to extend to individual elements and concepts described, independently of other concepts, ideas or system, as well as to combinations of elements recited anywhere in this application. Although numerous examples are described in detail with reference to the accompanying drawings, it is to be understood that the embodiments extend beyond the described examples. Furthermore, it is contemplated that a particular feature described either individually or as part of an embodiment can be combined with other individually described features, or parts of other embodiments, even if the other features and embodiments make no mention of the particular feature.

Claims

1. A computer-implemented method for operating a computing device, the method comprising: programmatically detecting, from image data generated by a camera component of the computing device, a location of a self-propelled device relative to the computing device;receiving input from the self-propelled device, the input including sensor information that is obtained on the self-propelled device;generating content based on the location of the self-propelled device as the self-propelled device moves.