Patent Application
20240362659
This invention relates generally to a spectator sport system and method that displays different views of the event and, in particular, utilizes a user's selected source position to assist in displaying information about the target.
GPS systems have been used as navigation systems to location destinations. GPS systems have also been used in sports by participants in contests where position, location and distance to features are important. For example, U.S. Pat. No. 5,364,093 describes a GPS system and method for allowing a golfer to tell distance to a hole or other feature, and permits the course to track and manage golfers on the course. NASCAR with Sportsline has developed a GPS system mounted to cars for TV viewers to monitor a race.
GPS systems are becoming much more accurate, inexpensive and robust. GPS antennas and engines are fairly inexpensive and accurate with WAAS to less than 2 meters. At a sporting arena, the accuracy of GPS can be improved to centimeters, depending on the accuracy required, latency constraints, processing and bandwidth available, etc. Further, communication links are becoming very inexpensive and high bandwidth. For example, WiFi (802.11g) has modems with network signals approaching a 1 mile range, cost less than $5, with bandwidth of 54M bit/sec. Wi-max (802.16) has network signals approaching 30 miles with data rates as high as 70M bit/sec, but is more relevant to fixed installations Future versions of WiFi or other radio technology might be less than $1 with 10-100× bandwidths within a few years (as used herein WiFi refers to current and future versions of wireless local area networks (WLAN) based on the IEEE 802.11 specifications). Other radio technologies are also promising in many applications, such as Zigbee and Ultrawideband.
What has not been done in sports is an integrated GPS system for spectators to more fully enjoy a sport particularly at a remote location. For example, at a NASCAR race, the TV or radio viewing location limits his view of the race and is not his own unique perspective. While watching a race, the spectator might listen to a radio or watch a portable TV, but the perspective is the announcer's or TV angle. Such divergent perspectives—announcer versus personal—can be confusing. Further, a remote spectator might be most interested in the cars he is interested in—the ones near the 3rd turn. Other sports would benefit from a system that allows a spectator to more fully integrate the contest information with his desired viewing perspective. In addition to auto racing, football, yachting, horse racing, golf, hockey or any motor sport are candidates for the system and method hereof, especially as size and weight of GPS and radios accompanying a participant decreases.
U.S. Pat. No. 6,744,403 describes a GPS system for tracking objects, such as cars, at a sporting event. See also, U.S. Pat. No. 6,195,090; U.S. Patent Application Publication No. 2006/0105857; U.S. Patent Application Publication No. 2005/0259002. High data rate packet transmission is known, such as U.S. Pat. Nos. 6,894,994; 6,909,738; 6,885,652; 6,917,644; and 6,801,516. Examples of user interfaces, such as PDAs, cell phones, headsets, and the like are described, for example, in U.S. Pat. Nos. 7,053,780; 6,879,443; and 6,115,177. All references cited herein are incorporated by reference.
In navigation and locator GPS-based systems, what is lacking is an integrated GPS system for an individual user to gain situational awareness and to easily identify destinations or other areas of interest. That is, while a user might possess a GPS-enabled cell phone that transmits his position and gives text based directions to a destination, this information gives a very incomplete understanding to visually identify a destination. Such a user might have an overhead view of a map showing the position of the destination on the map, but leaves it up to the user to find and identify the destination.
A particular problem in the area of personal navigation is identifying a place of interest in a confusing environment, such as a crowded street. For example, a cell phone having a GPS might be enabled to identify that a destination is near, but the user cannot locate the destination because of the clutter or environment, e.g. a crowded street or neighborhood or obstructions to the user's line of sight. Users also have difficulty relating how a small mark identifying a place on a map correlates to their position or their view of the environment.
U.S. Pat. No. 7,002,551 describes augmented reality approaches based on digitized video camera or optical as follows:
Augmented Reality (AR) enhances a user's perception of, and interaction with, the real world. Virtual objects are used to display information that the user cannot directly detect with the user's senses. The information conveyed by the virtual objects helps a user perform real-world tasks. Many prototype AR systems have been built in the past, typically taking one of two forms. In one form, they are based on video approaches, wherein the view of the real world is digitized by a video camera and is then composited with computer graphics. In the other form, they are based on an optical approach, wherein the user directly sees the real world through some optics with the graphics optically merged in. An optical approach has the following advantages over a video approach: 1) Simplicity: Optical blending is simpler and cheaper than video blending. Optical see-through Head-Up Displays (HUDs) with narrow field-of-view combiners offer views of the real world that have little distortion. Also, there is only one “stream” of video to worry about: the graphic images. The real world is seen directly through the combiners, which generally have a time delay of a few nanoseconds. Time delay, as discussed herein, means the period between when a change occurs in the actual scene and when the user can view the changed scene. Video blending, on the other hand, must deal with separate video streams for the real and virtual images. Both streams have inherent delays in the tens of milliseconds. 2) Resolution: Video blending limits the resolution of what the user sees, both real and virtual, to the resolution of the display devices, while optical blending does not reduce the resolution of the real world. On the other hand, an optical approach has the following disadvantages with respect to a video approach: 1) Real and virtual view delays are difficult to match. The optical approach offers an almost instantaneous view of the real world, but the view of the virtual is delayed. 2) In optical see-through, the only information the system has about the user's head location comes from the head tracker. Video blending provides another source of information, the digitized image of the real scene. Currently, optical approaches do not have this additional registration strategy available to them. 3) The video approach is easier to match the brightness of real and virtual objects. Ideally, the brightness of the real and virtual objects should be appropriately matched. The human eye can distinguish contrast on the order of about eleven orders of magnitude in terms of brightness. Most display devices cannot come close to this level of contrast.
AR displays with magnified views have been built with video approaches. Examples include U.S. Pat. No. 5,625,765, titled Vision Systems Including Devices And Methods For Combining Images For Extended Magnification Schemes; the FoxTrax Hockey Puck Tracking System, [Cavallaro, Rick. The FoxTrax Hockey Puck Tracking System. IEEE Computer Graphics & Applications 17, 2 (March-April 1997), 6-12.]; and the display of the virtual “first down” marker that has been shown on some football broadcasts.
U.S. Pat. No. 6,919,867 describes the state of the art in augmented reality approaches as follows:
U.S. Pat. Nos. 7,855,638; 8,275,397; 8,249,626; 8,207,843, and U.S. Publication No. 2008/0198230 (all incorporated herein by reference) relate generally to viewing people, places, and events, such as sporting events, using positioning and artificial reality to improve the event viewing experience. While a plan view of a map is useful for event planning, such views can be confusing and typically have limited functions to interact with the event.
Commercial applications of augmented reality exist such as Layar, Wikitude, Junaio, Sekai Camera and others which use augmented reality to aid finding information about points of interest. Sec, e.g., www.layar.com, www.wikitude.org/en/, and www.junaio.com.
Products or services that are tailored to the user are prevalent, such as advertising models from Google based on search terms or advertising based on personal information of a user. For example, Apple postulates displaying advertising to a mobile customer using one of its devices based on marketing factors. To compute marketing factors the Apple system captures not only the machine identity, but search history, personal demographics, time of day, location, weather, loyalty program membership, media library, user opinion or opinions of friends and family, etc. (collectively, referred to as “marketing factors”). See, e.g., U.S. Publication Nos. 2010/0125492; 2009/0175499; 2009/0017787; 2009/0003662; 2009/0300122, and U.S. Pat. No. 7,933,900 (all incorporated herein by reference). Links to and use of social media, such as Facebook and Twitter, sometimes paired with location, are also possible indicators of a user behavior and user demographics. Sec e.g., U.S. Publication Nos. 2009/0003662; 2011/0090252, and U.S. Pat. Nos. 7,188,153; 7,117,254; 7,069,308 (all references cited herein are incorporated herein by reference).
Various attempts have been made at sharing sports related event information. For example, select information can be shared or published with groups as more fully described in U.S. Pat. No. 7,478,078 and U.S. Publication Nos. 2008/0036653; 2011/0142016; 2010/0070758; 2011/0098928; and 2009/0144392. Related applications, including U.S. Pat. No. 8,249,626 and U.S. Publication No. 2007/0117576 describe how locations, interest, social information, are created and published to groups or within defined groups. Groups can be formed based on certain criteria. Groups can be formed ahead of time or be dynamic. Social networking participants can view shared views and AR messages.
The present invention contemplates a system that provides target area information relevant to the viewer's selected perspective or location or is selectable to view the target area to another location or target. Preferably, the object is a target location, such as a destination location and the target is conveyed in the context of its background environment, such as a row of buildings on a city street. The user preferably has a portable viewing device that accepts the user's derived position and selectively displays a view of the target from the user's perspective or location or selectively from another location. Preferably, the user's portable viewing device includes an orientation indicator so that the user has feedback indicating the device is pointed towards the desired target. That is, the user can selectively view and identify a target from different locations, views, and magnification and the user can tell if the viewing device is oriented correctly. Even remote users can use a device with a network information feed to identify a target. In a preferred form, a geo-referenced billboard is included on or near the target and conveys information to the user. For example, messages such as advertisements or the name of the target can be conveniently displayed on the billboard.
For example, in a NASCAR race, the cars are all equipped with a GPS engine and a communication link to a central server. Each spectator has an internet device has a communication link to the central server. The internet device logs in with the central server, optionally authenticating and telling the server the spectator's selected location and optionally a target location or orientation. During the race, the positions of the cars are broadcast to the spectator's internet device. In one mode, the internet device displays information most relevant to the spectator's selected source location. For example, the position and vital information of the cars nearest the spectator's selected location. In another mode, the internet device has the processing power to take the positions of the cars and the location of the spectator's selection and render a depiction of the cars in real time on the track. The spectator can select the view. For example, the spectator might select “finish line,” “overhead,” “car 3 driver's view,” or “my view.”
A spectator might designate the 3rd turn as “my view” selected to see the perspective of the rendering on the internet device to match a view from the 3rd turn—i.e. the source location including elevation. This adds to a much greater enjoyment of the contest because visual data is added to the display which matches the spectator's desired observation. Importantly, the spectator can not only switch views and source locations, but can also tilt or pan the perspective or observation point or zoom. That is, from “my view” the spectator might rotate a toggle up incrementally up (or down) from the horizontal view from the spectator's selected location of the car positions to a vertical view of the contest. Preferably, the toggle would also allow left/right pan at any time.
In addition to the view of the vehicles (or contestants) the spectator can selectively view appended important information. For example, in one mode the user might select no information, in a second mode, the user might select car identification only, while in another mode, the user might select identification plus speed, plus acceleration of one or more selected cars. Other information from car sensors are possible (or derived from GPS information) such as linear or radial G, engine or tire condition, fuel state, etc. Preferably, the spectator could go from a view mode to other modes, such as a display of the current information of the cars in tabular form, a view from a particular camera (streaming video), pit video or rendering, etc. Preferably, the internet device could include a radio (such as WiFi, Zigbee, Ultrawideband, or WiMax) to relay other information such as audio for monitoring car or pit communications or radio broadcasts, or even streaming video of TV broadcasts in, e.g. a separate window.
In “my view,” for example, the portable device might display only information to the 3rd turn selected position for cars nearest the 3rd turn. Alternatively, the spectator might want to follow a particular car continuously, e.g. follow car number 8, with selectable views (overheard, turns, stands, head, driver's view). In any of these modes, the spectator could zoom, pan or tilt as described above, freeze, slow motion, replay, etc. In many instances, the spectator might want to view the event from a position of a participant.
While one embodiment is described in the context of a car race such as NASCAR, IRL, NHRA, CART, Formula One, it is easily seen how the system and method of the present invention is applicable to a wide variety of spectator sports. Auto, aircraft, motorcycle, bicycle, boat racing, horse racing, football, soccer and hockey are just a few examples. Even hunting or fishing tournaments or TV shows would benefit from the system and method hereof. For example, a golf spectator (on the course or at home) might use the internet device to follow a particular golfer during a round. Information on the position of the player's golf ball can be supplied from a variety of sources-currently the PGA uses triangulation to acquire the data during play and provides the golf ball position information on its web site. However, the information could easily be gathered by a portable GPS unit accompanying the golfer's caddy or other dedicated data gatherer accompanying the golfer. Once the golf ball position information of each player is gathered, the information is distributed to the spectator based on the spectator's desires. As may be surmised from the NASCAR description above, the golf spectator might select the source viewing position and the target or orientation, determine the angle or view of the graphic rendering, the tilt, pan or zoom of the graphic depiction, the format of the presentation, i.e. graphic of a hole or a tabular summary of all golfers, or one golfer, statistics for a location or hole, etc.
While one limitation to the applicability of the present invention to a spectator sport is the ability to determine the position of a participant, those limitations are becoming less meaningful as technology progresses. For example, with current technology a football player can be fitted with a GPS device and radio that weighs less than ½ pound and is accurate within 2 meters. Foreseeable technology will enable such a device weighing several ounces (mostly battery weight), accurate with WAAS to less than 1 meters and with post-processing to centimeter accuracy. Accuracy to GPS positioning in an outdoor, unobstructed, clutter-free environment will soon be better than one meter. Post-processing at the server can occur in real time (or near real time) to achieve the desired participant accuracy. While the preferred embodiment contemplates obtaining participant location information via GPS, other types of location determination sensors are possible, such as proximity sensors, radar or radio triangulation. See, e.g., U.S. Pat. No. 5,564,698 for tracking hockey pucks, other objects, or people.
While the internet device of one embodiment is a gaming device with GPS and a radio (e.g., WiFi or Zigbee) other types of personal computers, gaming, PDA, and cell phones may equally be used. Further, although one embodiment contemplates broadcasting participant location information to spectators by radio, another embodiment broadcasts information to spectators over the internet. The rendering load of the data might be distributed, i.e., some of the graphics pipeline for the rendering could be accomplished at the server before transmission; other rendering can be performed on the internet device. However, rendering technology is rapidly advancing and becoming increasingly realistic with advances in game technology and as the processing power of the internet device increases and the rendering technology develops, it is anticipated that most of the graphics rendering can be performed at the internet device.
High bandwidth, wireless networks are becoming commonplace, as is the computing power of mobile devices. Further rendering engines are becoming readily available for wide ranging applications of artificial reality. Viewing an event, such as a sporting event, using a mobile device adds greatly to the user experience. Many sporting events, such as golf, can be enhanced using a mobile device and artificial reality. U.S. Pat. No. 7,855,638 describes several examples of a system and method for viewing such events. In such event viewing systems, the background can be a real world image (e.g. photograph) or a virtual world rendering, but in a preferred case, artificial reality is used to enhance the perspective viewing experience.
In creating such environments for the venue of the event, such as a golf course, bike race, ski competition, or the like, it is desirable to insert virtual objects into the environment, such as messages, tips, targets, critiques, warnings, etc. The virtual objects overlay a background environment, preferably as viewed from a user selected position. Typically, the user selected position is a participant present position as determined by GPS. Thus, in a preferred embodiment the sport participant (golfer, skier, biker, etc.) is presented with a perspective view of the event from the participant's current position (i.e. “viewpoint”) with virtual objects visually presented to the participant.
The present system and methods also address many sport related functions that can be used in such an artificial reality or mixed reality environment. For example, a basic function in golf is displaying in perspective view the golf hole from the golfer's location and identifying distances to hazards or the green. However, other functions exist, such as displaying suggested playing strategy based on the golfer's personal data or profile, a similar handicap golfer, or as suggested by tips from a pro. Further, games can be incorporated into the system and method such as corporate outing type experiences-closest to the pin, longest drive, accurate drive, as measured from an artificial reality target, etc. Other functions such as contests and betting can also be incorporated.
In the present application, the term “message” is used to encompass any artificial reality or virtual object, such as distance messages, advertisements, other facts, targets, shot distribution diagrams, event information, warnings, announcements and other types of alpha numeric displays. However, the message could also be a graphic, logo or brand. It shall be understood that other objects or graphics may also be enhanced and the term “message” is understood to include other objects.
In the present application, the term “social network” is used to refer to any process or system that tracks and enables connections between members (including people, businesses, and other entities) or subsets of members. The connections and membership may be static or dynamic and the membership can include various subsets within a social network. For example, a person's social network might include a subset of members interested in golf and the person shares a golf outing only with the golf interest subset. Further, a social network might be dynamically configured. For example, a social network could be formed for “Pebble Beach” for August 3 and anyone interested could join the Pebble Beach August 3 social network. Alternatively, anyone within a certain range of the event might be permitted to join. The permutations involving membership in a social network are many and not intended to be limiting.
A social network that tracks and enables the interactive web by engaging users to participate in, comment on and create content as a means of communicating with their social graph, other users and the public. In the context of the present invention, such sharing and social network participation includes participant created content and spectator created content and of course, jointly created content. For example, the created content can be interactive to allow spectators to add content to the participant created event.
The term “geo-referenced” means a message fixed to a particular location or object. Thus, the message might be fixed to a venue location, e.g., golf course fence or fixed to a moving participant, e.g., a moving golf car or player. An object is typically geo-referenced using either a positioning technology, such as GPS, but can also be geo-referenced using machine vision. If machine vision is used (i.e. object recognition), applications can be “markerless” or use “markers,” sometimes known as “fiducials.” Marker-based augmented reality often uses a square marker with a high contrast. In this case, four corner points of a square are detected by machine vision using the square marker and three-dimensional camera information is computed using this information. Other detectable sources have also been used, such as embedded LED's or special coatings or QR codes. Applying AR to a marker which is easily detected is advantageous in that recognition and tracking are relatively accurate, even if performed in real time. So, in applications where precise registration of the AR message in the background environment is important, a marker based system has some advantages.
In a “markerless” system, AR uses a general natural image instead of a fiducial. In general, markerless AR uses a feature point matching method. Feature point matching refers to an operation for searching for and connecting the same feature points in two different images. A method for extracting a plane using a Simultaneous Localization and Map-building (SLAM)/Parallel Tracking and Mapping (PTAM) algorithm for tracking three-dimensional positional information of a camera and three-dimensional positional information of feature points in real time and providing AR using the plane has been suggested. However, since the SLAM/PTAM algorithm acquires the image so as to search for the feature points, computes the three-dimensional position of the camera and the three-dimensional positions of the feature points, and provides AR based on such information, a considerable computation is necessary. A hybrid system can also be used where a readily recognized symbol or brand is geo-referenced and machine vision substitutes the AR message.
In the present application, the venue for the sporting event can be a real view or depicted as a photo background environment or a virtual environment, or a mixture, sometimes referred to as “mixed reality.” A convenient way of understanding the messages of the present invention is as a layer of artificial reality or “augmented reality” messages overlaid the sporting event venue background. There are different methods of creating the sporting event venue background as understood by one of ordinary skill in the art. For example, an artificial background environment can be created by a number of rendering engines, sometimes known as a “virtual” environment. See, e.g., Nokia's (through its Navteq subsidiary) Journey View which blends digital images of a real environment with an artificial 3D rendering. A real environment can be the background as seen through glasses of
Turning to the drawings, cars 10 traverse a track 12. In
Of course, other data can be incorporated into the views of
Preferably, the position of the “source” can be specified, e.g. my position or participant's position (driver of car 3). Additionally, the target or direction can be specified. A particular useful sensor augmentation to the GPS information is a “direction” or orientation finder for the device 20. GPS always tells a bearing to a waypoint from a GPS determined location or source. That is, device 20 can give the user a bearing from the current user location (source) to the meeting point location or rendezvous location for one or more friendly participants. GPS information can give direction of travel when the device 20 is moving. When stopped, the orientation of the device 20 relative to the location of another participant or rendezvous location is useful information. So, in addition to the bearing to the rendezvous location the user could know if the device 20 was oriented in the correct direction—i.e., pointed to the rendezvous location or another participant or target area.
Methods are available for determining orientation of a device using GPS, but usually such GPS methods employ multiple antennas. Other means for determining direction are believed to be preferred, such as a fluxgate compass, INS, gyrocompass, digital compass, fluxgate magnetometers, or multi-axis accelerometers to name a few. Preferably, device 20 includes a direction-determining mechanism which is selectable to another location or a designated participant, and tells the user when device 20 is oriented or pointed correctly to a selected target. Alternatively, the user can select the source (e.g. current location) and the target (e.g. car #3). The user can then additionally deviate from a direct line from the source to the target by, for example, moving the elevation.
While the device 20 of
Similarly, the graphic device 220 of
As used herein, GPS is meant to include all of the current and future positioning systems that include satellites, such as the U.S. Navistar, GLONASS, Galileo, EGNOS, WAAS, MSAS, QZSS, etc. The accuracy of the positions, particularly of the participants, can be improved using known techniques, often called differential techniques, such as WAAS (wide area), LAAS (local area), Carrier-Phase Enhancement (CPGPS), Wide Area GPS Enhancement (WAGE), or Relative Kinematic Positioning (RKP). Even without differential correction, numerous improvements are increasing GPS accuracy, such as the increase in the satellite constellation, multiple frequencies (L1, L2, L5), modeling and AGPS improvements, software receivers, and ground station improvements. Of course, the positional degree of accuracy is driven by the requirements of the application. In the NASCAR embodiment of the preferred embodiment, two meter accuracy provided by WAAS would normally be acceptable.
In
An illustrative embodiment uses a mobile device, such as smart phone 300 of
In more detail,
Mobile device 300 includes a touch-sensitive graphics display 102. The touch-sensitive display 302 can implement liquid crystal display (LCD) technology, light emitting polymer display (LPD) technology, or some other display technology. The touch-sensitive display 302 can be sensitive to haptic and/or tactile contact with a user.
The touch-sensitive graphics display 302 can comprise a multi-touch-sensitive display. A multi-touch-sensitive display 302 can, for example, process multiple simultaneous touch points, including processing data related to the pressure, degree and/or position of each touch point. Such processing facilitates gestures and interactions with multiple fingers, chording, and other interactions. Other touch-sensitive display technologies can also be used, e.g., a display in which contact is made using a stylus or other pointing device. An example of a multi-touch-sensitive display technology is described in U.S. Pat. Nos. 6,323,846; 6,570,557; 6,677,932; and U.S. Patent Application Publication No. 2002/0015024, each of which is incorporated by reference herein in its entirety. Touch screen 302 and touch screen controller can, for example, detect contact and movement or break thereof using any of a plurality of touch sensitivity technologies, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 302.
Mobile device 300 can display one or more graphical user interfaces on the touch-sensitive display 302 for providing the user access to various system objects and for conveying information to the user. The graphical user interface can include one or more display objects 304, 306. Each of the display objects 304, 306 can be a graphic representation of a system object. Some examples of system objects include device functions, applications, windows, files, alerts, events, or other identifiable system objects.
Mobile device 300 can implement multiple device functionalities, such as a telephony device, as indicated by a phone object; an e-mail device, as indicated by the e-mail object; a network data communication device, as indicated by the Web object; a Wi-Fi base station device (not shown); and a media processing device, as indicated by the media player object. For convenience, the device objects, e.g., the phone object, the e-mail object, the Web object, and the media player object, can be displayed in menu bar 318.
Each of the device functionalities can be accessed from a top-level graphical user interface, such as the graphical user interface illustrated in
Upon invocation of particular device functionality, the graphical user interface of mobile device 300 changes, or is augmented or replaced with another user interface or user interface elements, to facilitate user access to particular functions associated with the corresponding device functionality. For example, in response to a user touching the phone object, the graphical user interface of the touch-sensitive display 302 may present display objects related to various phone functions; likewise, touching of the email object may cause the graphical user interface to present display objects related to various e-mail functions; touching the Web object may cause the graphical user interface to present display objects related to various Web-surfing functions; and touching the media player object may cause the graphical user interface to present display objects related to various media processing functions.
The top-level graphical user interface environment or state of
The top-level graphical user interface is shown in
Mobile device 300 can include one or more input/output (I/O) devices and/or sensor devices. For example, speaker 322 and microphone 324 can be included to facilitate voice-enabled functionalities, such as phone and voice mail functions. In some implementations, loud speaker 322 can be included to facilitate hands-free voice functionalities, such as speaker phone functions. An audio jack can also be included for use of headphones and/or a microphone.
A proximity sensor (not shown) can be included to facilitate the detection of the user positioning mobile device 300 proximate to the user's ear and, in response, disengage the touch-sensitive display 302 to prevent accidental function invocations. In some implementations, the touch-sensitive display 302 can be turned off to conserve additional power when mobile device 300 is proximate to the user's ear.
Other sensors can also be used. For example, an ambient light sensor (not shown) can be utilized to facilitate adjusting the brightness of the touch-sensitive display 302. An accelerometer (not shown) can be utilized to detect movement of mobile device 300, as indicated by the directional arrow. Accordingly, display objects and/or media can be presented according to a detected orientation, e.g., portrait or landscape.
Mobile device 300 may include circuitry and sensors for supporting a location determining capability, such as that provided by the global positioning system (GPS) or other positioning system (e.g., Cell ID, systems using Wi-Fi access points, television signals, cellular grids, Uniform Resource Locators (URLs)). A positioning system (e.g., a GPS receiver) can be integrated into the mobile device 10 or provided as a separate device that can be coupled to the mobile device 10 through an interface (e.g., port device 332) to provide access to location-based services.
Mobile device 300 can also include a front camera lens and sensor 340. In a preferred implementation, a backside camera lens and sensor is located on the back surface of the mobile device 300. The cameras can capture still images and/or video. The camera subsystems and optical sensors may comprise, e.g., a charged coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) optical sensor, can be utilized to facilitate camera functions, such as recording photographs and video clips. Camera controls (zoom, pan, capture and store) can be incorporated into buttons 334-336 (
While the preferred embodiment contemplates most processing occurring at mobile devices, different amounts of preprocessing of the position data can be processed at the server 44. For example, the participant information can be differentially corrected at the server (using e.g. either WAAS or a local area differential correction) or even information post-processed with carrier phase differential to achieve centimeter accuracy. Further, it is anticipated that most of the graphics rendering can be accomplished at the portable device, but an engineering choice would be to preprocesses some of the location and rendering information at the server 44 prior to broadcast.
Mobile devices 20, 120, 220, 300 preferably accompanies some of golfers in attendance at the course. Devices 20, 120, 220, 300 communicate over one or more wired and/or wireless networks 46 in data communication with server 44. In addition, the devices can communicate with a wireless network, e.g., a cellular network, or communicate with a wide area network (WAN), such as the Internet, by use of a gateway. Likewise, an access point associated with Radio 46, such as an 802.11b/g/n wireless access point, can provide communication access to a wide area network.
Both voice and data communications can be established over the wireless network of
Mobile devices 20, 120, 220, 300 can also establish communications by other means. For example, wireless device 20, 120, 220, 300 can communicate with other wireless devices, e.g., other wireless devices 100, cell phones, etc., over a wireless network. Likewise, mobile devices can establish peer-to-peer communications, e.g., a personal area network, by use of one or more communication subsystems, such as the Bluetooth™ communication device. Other communication protocols and topologies can also be implemented.
In use in the play of golf, it is believed preferable to use a real environment as the background, such as a digital image captured by backside camera of
In other embodiments, a virtual environment may be used as the background. In such cases, server 44 preferably uses the OTOY, Gaikai, or OnLive video compression technology to transmit the participant position information the virtual background environment, as well as the AR objects, such as each car 54. OTOY (and Gaikai and OnLive) are cloud based gaming and application vendors that can transmit real time photorealistic gaming to remote gamers. Such companies that render photorealistic 3D games for realtime remote play are Otoy, see, e.g., www.otoy.com; OnLive, see, e.g., en.wikipedia.org/wiki/OnLive; and Gaikai, see, e.g., technabob.com/blog/2010/03/16/gaikai-cloud-based-gaming. Onlive, for example, advertises that with 5 mbps it can transfer 220 frames per second with 12-17 ms latency, employed advanced graphics—ajax, flash, Java, ActiveX.
The goal is high performance game systems that are hardware and software agnostic. That is, a goal is intense game processing performed on a remote server and communicated to the remote user. Using such cloud based gaming technology, smart phones 300 can run any of the advanced browsers (e.g. IE9 or Chrome) running HTML5 that support 3D graphics. However, other AR specific browsers can alternatively be used, such as available from Layar, Junaio, Wikitude, Sekai Camera or Mixare (www.mixare.org). While OTOY (and Gaikai and OnLive) promise no discernable latency in their gaming environment, server 44 for the golf event of
Therefore, the amount of processing occurring at server 44 versus devices 20, 120, 220, 300 is a design choice based on the event, the background, the radio network available, the computational and display capability available at the device or other factors.
In addition the content of the advertisement messages can be changed based on context. Such smart phones 300 have not only machine ID's, but also search history, location history, and even personal information. Further, the user might be identified based on social media participation—e.g. Facebook or Twitter accounts. Such information is considered “context” in the present application, along with the typical demographics of an event and “marketing factors” as previously discussed. That is, the event might have its own context which indicates the demographic profile of most of the spectators at the event. A golf match might have a context of golf spectators with adequate disposable income to purchase a vehicle. Therefore, advertising a Buick makes advertising sense. See, U.S. patent application Ser. No. 13/152,476 (incorporated by reference).
5 The graphics generated on the screen 22 (
Preferably, however, the graphics generated on screen 22 are 3D. OpenGL and Direct3D are two popular APIs for the generation of real-time imagery in 3D. (Real-time means that image generation occurs in ‘real time’, or ‘on the fly’) Many modern graphics cards provide some degree of hardware acceleration based on these APIs, frequently enabling the display of complex 3D graphics in real-time. However, it's not necessary to employ any one of these to actually create 3D imagery. The graphics pipeline technology is advancing dramatically, mainly driven by gaming applications enabling more realistic 3D synthetic renderings of
3D graphics have become so popular, particularly in computer games, that specialized APIs (application programmer interfaces) have been created to ease the processes in all stages of computer graphics generation. These APIs have also proved vital to computer graphics hardware manufacturers, as they provide a way for programmers to access the hardware in an abstract way, while still taking advantage of the special hardware of this-or-that graphics card. These APIs for 3D computer graphics are particularly popular:
There are also higher-level 3D scene-graph APIs which provide additional functionality on top of the lower-level rendering API. Such libraries under active development include:
Photo-realistic image quality is often the desired outcome, and to this end several different, and often specialized, rendering methods have been developed. These range from the distinctly non-realistic wireframe rendering through polygon-based rendering, to more advanced techniques such as: scanline rendering, ray tracing, or radiosity. The rendering process is computationally expensive, given the complex variety of physical processes being simulated. Computer processing power has increased rapidly over the years, allowing for a progressively higher degree of realistic rendering. Film studios that produce computer-generated animations typically make use of a render farm to generate images in a timely manner. However, falling hardware costs mean that it is entirely possible to create small amounts of 3D animation on a small processor, such as in the device 20. Driven by the game studios, hardware manufacturers such as ATI, Nvidia, Creative Labs, and Ageia have developed graphics accelerators which greatly increase the 3D rendering capability. It can be anticipated that in the future, one or more graphics rendering chips, such as the Ageia Physx chip, will be added to the device 20.
While full 3D rendering is not possible with the device 20 described herein, advances in processing and rendering capability will enable greater use of 3D graphics in the future. In a particular application, such as NASCAR, a car object and a track object (e.g., Taladega) can be rendered in advance and stored, making realistic 3D graphics possible. In 3D computer graphics, the terms graphics pipeline or rendering pipeline most commonly refer to the current state of the art method of rasterization-based rendering as supported by commodity graphics hardware. The graphics pipeline typically accepts some representation of a 3D scene as an input and results in a 2D raster image as output.
Special requests from spectators 48 can be made to the server 44, such as for streaming video of a particular scene or audio of a particular car 10, refreshment orders, memorabilia purchases, etc. This function is shown at 50, 52 in
While one embodiment has been described in the context of a spectator in physical attendance at a sporting event with information broadcast by radio, the use of the graphic devices 20, 120, 220, 300 at remote locations is equally feasible. In another embodiment more suited for remote locations, for example, the portable device 20, 120, 220, 300 can be used at home while watching a sporting event on TV, with the participant location and other information streaming over the internet. WiFi in the home is a preferred mode of broadcasting the information between the portable device and the network.
Using graphic devices 20, 120, 220, 300 at home while watching the same sporting event on TV is believed to be a preferred embodiment for use at remote locations. However, other examples of remote location of a sporting event viewing might not be accompanied by watching TV. That is, the views of
Some spectators 48 may be remote from the sporting event. In this case, server 44 can transmit the desired information over internet connection 49 to the club house, home computer or television remote from the event. While one embodiment has been described in the context of a spectator in physical attendance at the golf course with information broadcast by radio, the use of devices 20, 120, 220, 300 at remote locations is equally feasible. In another embodiment more suited for remote locations, for example, portable device 20, 120, 220, 300 can be used at home while watching a golf event on TV, with the participant location and other information streaming over the internet. WiFi in the home is a preferred mode of broadcasting the information between the portable device and the network.
One function of the server 44 is to allow observation of a round by a golfer, either in real time or post play. That is, the views can be posted to the server 44 and observed by a spectator 48 using any graphic device, including a personal computer, tablet, or a cell phone. Similar to using graphic device 20, 120, 220, 300 coupled to the internet, a personal computer spectator can select the source or position of origination of the desired view, and the target or orientation from the source or target. Elevations, zoom, pan, tilt, etc. may be selected by the remote spectator as desired to change the origin viewpoint or size.
In “my view,” for example, the remote location graphic device might display only information to the 3rd turn spectator for cars nearest the 3rd turn. Alternatively, the remote location spectator might want to follow a particular car continuously, e.g. follow car number 8 (or particular golfer, etc.), with selectable views (overheard, turns, stands, head, driver's view). In any of these modes, the remote location spectator could zoom, pan or tilt as described above, freeze, slow motion, replay, etc. to obtain a selected view on the graphic device.
Messages can be conveyed using more conventional designations such as sidebar advertisements, pop-up balloons with advertisements or messages, click info, banner ads, etc. However, a billboard area may be convenient as an area that does not interfere with other features of the target. Other examples of billboard can be large windows, open facades, real or artificial signs, doorways, or the like. Because the billboard is geo-referenced and the information is in the digital domain, the message on the billboard stays in the same location on the building as the user's view changes.
This application is a continuation of U.S. patent application Ser. No. 17/397,130, filed Aug. 9, 2021, which is a continuation of U.S. patent application Ser. No. 16/668,143 (now U.S. Pat. No. 11,087,345), filed Oct. 30, 2019, which is a continuation of U.S. patent application Ser. No. 15/097,711 (now U.S. Pat. No. 10,512,832), filed Apr. 13, 2016, which is a continuation of U.S. Patent Application No. 14,037,642 (now U.S. Pat. No. 9,498,694) filed Sep. 26, 2013, which is a continuation of U.S. patent application Ser. No. 13/605,174 (now U.S. Pat. No. 8,589,488) filed Sep. 6, 2012 which is a continuation of U.S. patent application Ser. No. 13/182,930 (now U.S. Pat. No. 8,933,967) filed Jul. 14, 2011 and Ser. No. 13/152,476 (now abandoned) filed Jun. 3, 2011. Application '930 is a continuation-in-part of U.S. patent application Ser. No. 12/111,688 (now U.S. Pat. No. 9,445,225) filed Apr. 29, 2008 which is a continuation-in-part of U.S. patent application Ser. No. 11/875,414 (now U.S. Pat. No. 8,249,626) filed Oct. 19, 2007 which is a continuation-in-part of U.S. patent application Ser. No. 11/624,998 (now U.S. Pat. No. 8,275,397) filed Jan. 19, 2007 which is a continuation-in-part of U.S. patents application Ser. No. 11/456,715 (now U.S. Pat. No. 7,855,638) and Ser. No. 11/456,723 (now U.S. Pat. No. 7,518,501) filed Jul. 11, 2006 both of which claim priority to U.S. Provisional Application No. 60/699,205 filed Jul. 14, 2005 (sometimes referred to collectively as “related applications” herein). All of which are incorporated herein by reference, and the benefit of 35 U.S.C. § 120 is claimed.
| Number | Date | Country | |
|---|---|---|---|
| 60699205 | Jul 2005 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17397130 | Aug 2021 | US |
| Child | 18768624 | US | |
| Parent | 16668143 | Oct 2019 | US |
| Child | 17397130 | US | |
| Parent | 15097711 | Apr 2016 | US |
| Child | 16668143 | US | |
| Parent | 14037642 | Sep 2013 | US |
| Child | 15097711 | US | |
| Parent | 13605174 | Sep 2012 | US |
| Child | 14037642 | US | |
| Parent | 13182930 | Jul 2011 | US |
| Child | 13605174 | US | |
| Parent | 13152476 | Jun 2011 | US |
| Child | 13182930 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12111688 | Apr 2008 | US |
| Child | 13182930 | US | |
| Parent | 11875414 | Oct 2007 | US |
| Child | 12111688 | US | |
| Parent | 11624998 | Jan 2007 | US |
| Child | 11875414 | US | |
| Parent | 11456715 | Jul 2006 | US |
| Child | 11624998 | US | |
| Parent | 11456723 | Jul 2006 | US |
| Child | 11456715 | US |
