Patent Application
20240095418
The disclosure generally relates to a system and method for an augmented-virtual reality driving simulator using a vehicle.
A virtual reality simulation may include a headset wearable by a user, wherein the headset includes a display screen. A virtual reality simulation replaces the view of the actual surroundings of the user. A user standing in his or her living room may see through the headset a view of a beach or a view skydiving from a plane. An alternate, virtual reality is graphically presented to the user through the headset.
An augmented reality mixes objects and features from the actual surroundings of the user with virtual renderings upon a headset. A camera upon the headset captures images of the actual surroundings of the user, and computerized programming is used to render an object, features, or a character in the context of the actual surroundings. For example, an empty chair may actually be in front of the user wearing a headset. Computerized programming may render an animated fictional character, with the graphics being configured for making the fictional character appear to be sitting on the actual chair.
The headset may include programming to track movement of the user's head and change a viewpoint or perspective of graphics illustrated to the user based upon the movement. As the user turns his or her head to the left, the graphics rendered upon the headset pan to the left in coordination with the actual movement of the head. Additionally, a camera upon the headset and/or a camera in a surrounding environment of the user may capture movement of the user. Virtual hands may be graphically rendered in the view provided by the headset, with movement of the virtual hands mimicking the actual movement of the user's hands. Actual movement of the user's hands may be utilized as a control input to simulation programming, for example, with virtual movement of the user's hands being used to simulate and provide a judged score for a dance routine.
A vehicle includes a plurality of control devices or input devices. A steering wheel is used to provide left-right directional control of a vehicle. An accelerator pedal enables a user to maintain or increase a speed of the vehicle. A brake pedal enables a user to slow or stop the vehicle. A transmission gear selector enables a user to select a driving mode of the vehicle (park, reverse, neutral, drive, etc.)
A system for an augmented-virtual reality simulator using a vehicle is provided. The system includes the vehicle including a control feature configured for selective use in an actual operation mode and in a simulation mode. In the actual operation mode, the control feature is utilized to control actual operation of the vehicle. In the simulation mode, the control feature is utilized as an input device to the augmented-virtual reality simulator. The system further includes a computerized simulation controller. The controller includes programming to monitor activation of the augmented-virtual reality simulator, command activation of the simulation mode based upon the activation of the augmented-virtual reality simulator, monitor inputs to the control feature, and operate the augmented-virtual reality simulator based upon the inputs to the control feature. The system further includes a headset configured for being worn by a user providing a graphical rendering to the user based upon the augmented-virtual reality simulator.
In some embodiments, the control feature includes one of a steering wheel, an accelerator pedal, a brake pedal, or a transmission gear selector.
In some embodiments, the vehicle includes a plurality of control features configured for selective use in an actual operation mode and in a simulation mode. In the actual operation mode, the control features are utilized to control actual operation of the vehicle. In the simulation mode, the control features are utilized as input devices to the augmented-virtual reality simulator. The plurality of control features include a steering wheel, an accelerator pedal, a brake pedal, and a transmission gear selector.
In some embodiments, the computerized simulation controller further includes programming to monitor a transmission mode of the vehicle and enable activation of the augmented-virtual reality simulator based upon the transmission mode indicating that the vehicle is in park.
In some embodiments, the augmented-virtual reality simulator includes virtual reality. The graphical rendering replaces visible features in an environment of the user.
In some embodiments, the augmented-virtual reality simulator includes augmented reality, wherein the graphical rendering includes a mixture of images of visible features in an operating environment of the user and computerized graphics rendered based upon a three-dimensional model.
In some embodiments, the augmented-virtual reality simulator includes a driving simulation.
In some embodiments, the driving simulation includes a race.
In some embodiments, the driving simulation includes a virtual test drive.
In some embodiments, the driving simulation includes driving instruction for a student driver.
In some embodiments, the augmented-virtual reality simulator includes a simulation of operating a boat or an airplane.
In some embodiments, the augmented-virtual reality simulator includes operation of one of vibration outputs, haptic outputs, climate control system outputs, or audio outputs.
In some embodiments, the augmented-virtual reality simulator includes limited-edition non-fungible tokens.
According to one alternative embodiment, a system for an augmented-virtual reality simulator using a vehicle is provided. The system includes the vehicle including a control feature configured for selective use in an actual operation mode and in a simulation mode. In the actual operation mode, the control feature is utilized to control actual operation of the vehicle. In the simulation mode, the control feature is utilized as an input device to the augmented-virtual reality simulator. The system further includes a computerized simulation controller. The controller includes programming to monitor activation of the augmented-virtual reality simulator, command activation of the simulation mode based upon the activation of the augmented-virtual reality simulator, monitor inputs to the control feature, and operate the augmented-virtual reality simulator including a driving simulator configured to simulate a virtual test drive based upon the inputs to the control feature. The system further includes a headset configured for being worn by a user providing a graphical rendering to the user based upon the augmented-virtual reality simulator. The augmented-virtual reality simulator enables the user to select from a plurality of virtual vehicles.
In some embodiments, the augmented-virtual reality simulator enables the user to select from a plurality of maps upon which to operate the virtual test drive.
In some embodiments, the augmented-virtual reality simulator identifies a bumpy road condition and controls a vibration unit in a seat of the user based upon the bumpy road condition.
According to one alternative embodiment, a method for an augmented-virtual reality simulator using a vehicle is provided. The system includes operating the vehicle including a control feature configured for selective use in an actual operation mode and in a simulation mode. In the actual operation mode, the control feature is utilized to control actual operation of the vehicle. In the simulation mode, the control feature is utilized as an input device to the augmented-virtual reality simulator. The method further includes, within a computerized processor, monitoring activation of the augmented-virtual reality simulator, commanding activation of the simulation mode based upon the activation of the augmented-virtual reality simulator, monitoring inputs to the control feature, and operating the augmented-virtual reality simulator based upon the inputs to the control feature. The method further includes, within a headset configured for being worn by a user, providing a graphical rendering to the user based upon the augmented-virtual reality simulator.
In some embodiments, monitoring inputs to the control feature includes monitoring inputs to a steering wheel.
In some embodiments, the method further includes, within the computerized processor, monitoring a transmission mode of the vehicle and enabling activation of the augmented-virtual reality simulator based upon the transmission mode indicating that the vehicle is in park.
The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description of the best modes for carrying out the disclosure when taken in connection with the accompanying drawings.
A simulation or a computerized simulation is a computerized program configured for providing a computerized approximation of a real-world experience. For example, a vehicle simulator may provide a user with an ability to utilize an electronic steering wheel attached to a desk to provide inputs to the vehicle simulator, and an output display such as a liquid crystal display monitor may provide a rendered street surface. A user may turn the electronic steering wheel to control the output upon the monitor. The computerized simulation may apply rules and context to the approximation, for example, providing graphical images of trees and lampposts to avoid.
A system and method for an augmented-virtual reality driving simulator using a vehicle is provided. The system and method may be described as an augmented-virtual reality simulator or as a computerized simulation useful to render either augmented reality graphics or virtual reality graphics. A headset configured for displaying augmented reality and/or virtual reality graphics to a user sitting in a driver's seat of the vehicle may be utilized. The actual vehicle may be confirmed to be in park when the simulation is initiated. The controls of the vehicle may be disengaged from their normal functions, such that the user may utilize the steering wheel, the pedals, the transmission gear selector, and other control features of the actual vehicle as control inputs to the simulation. A graphical representation of the steering wheel may be provided to the user, and as the user actually turns the steering wheel of the vehicle, the graphical representation of the steering wheel may be controlled to appear to turn. Additionally, hand movements of the user may be monitored to interact with virtual controls. For example, virtual police sirens may be presented for the user to activate by touching a representation of a button. No physical feature or control may be where the user's hand actually goes to activate the virtual button, but the act of moving the hand to the actual location corresponding to the virtual button may be utilized to activate sirens in the simulation.
The simulation may be a racing simulation. The user may be able to compete against time or against virtual computerized racers on a track or on virtual city streets. The vehicle may be configured for wireless communication with a remote server device or directly with another vehicle in vehicle-to-vehicle communications. Users in two separate vehicles may virtually compete with each other, with the computerized programs of each vehicle coordinating with each other regarding location, trajectory, engine speed, etc. of the vehicles. In one embodiment, a remote server device may operate leagues, competitions, services to match racers with similar records, and updating content such as virtual races corresponding with actual races being held in real life.
The simulation may be a sales tool, useful for a car dealership to enable potential customers to virtually test drive vehicles that are currently for sale. For example, a user that currently owns a five year old gasoline powered vehicle manufactured by company A may utilize a simulation published by company B to test drive a new electric vehicle. The user may see graphical renderings of the interior surfaces of the vehicle. The user may virtually drive the vehicle, experience the acceleration of the virtual vehicle, hear sounds from the virtual vehicle played over the sound system of the actual vehicle, and experience other features and options provided by the new vehicle.
The simulation may be a training tool, with a student driver getting time virtually training in the family vehicle prior to actually driving the vehicle upon a roadway.
The simulation may be of a vehicle of a different type from the actual vehicle utilized in the simulation. For example, the simulation may provide a virtual boating experience, with the transmission gear selector of the actual vehicle being utilized to simulate a throttle control in the boat. An electronic seat may be utilized to rock and sway the user to simulate wave action upon the boat. In another example, an airplane simulation may be operated, with a telescoping steering wheel being utilized to simulate a steering wheel/control stick in the simulated plane. As the user pushes the actual steering wheel forward, the simulated plane may be pushed forward into a dive. As the user pulls back on the actual steering wheel, the simulated plane may be shown to climb. The brake pedal and the accelerator pedal may be utilized to simulate left and right rudder controls, respectively, for the virtual airplane.
As electric vehicles charge at a charging station, the user of the vehicle may have to wait in a stationary vehicle for thirty minutes or more as the vehicle charges. The disclosed system and method may be utilized to entertain users and increase user satisfaction while the vehicle is being charged. A company owning a brand of charging stations may offer a program that may be operated locally while at a location operated by the company to build customer loyalty. A company may reward users with non-fungible token (NFT) rewards for buying a vehicle during a promotion, for example, enabling the user to operate a virtual limited-edition Corvette® in simulations or granting a virtual leather wrapped wheel with an insignia to be utilized by the driver in simulations as a reward. NFT rewards may be owned by a specific person utilizing blockchain methods.
Operation of the simulation may include a variety of feedbacks to the user. For example, vibrating mechanisms in the base and back of the user's seat may be useful to provide haptic feedback to the user. Force feedback may be operated through the steering wheel, with servo motors being used to make the steering wheel vibrate or pulse in a direction. Sounds of an engine revving or of a seagull flying by may be played over the sound system of the vehicle. The climate control system of the vehicle may be utilized to provide a puff of air during virtual rapid acceleration or a flow of hot air as virtual sunlight is visible shining upon the user. Voice and/or textual information may be provided contextually, for example, with a virtual salesperson being rendered sitting in the seat next to the user answering questions about the virtual vehicle being test driven or with current sales incentives being scrolled across a view of the user as the simulation is operated.
The disclosed simulations may be operated as virtual reality simulations, with the view of the user being based upon a programmed environment separate from the user's actual environment. In the alternative, augmented reality may be utilized in actual vehicle, for example, with a parked and immobilized vehicle at a shopping center or sporting event enabling users to experience virtually driving the vehicle. The user may see images captured by the headset of the actual control surfaces and the hands of the user upon the control surfaces, with augmented reality being utilized to simulate the vehicle driving down the road or over a forest trail.
The disclosed system and method may include receiving a plurality of control inputs from a variety of actual vehicle controls disconnected from their nominal vehicular functions. The disclosed system and method may include receiving a single control input from a single actual vehicle control, such as the steering wheel or the accelerator pedal. Additional virtual controls may be provided to the user in the simulation, for example, with a virtual steering wheel being controlled just by the position of the user's hands in the air. In another embodiment, the disclosed system and method may include simulating operation of the actual vehicle utilizing virtual controls, where the user may sit in the car seat and feel the actual interior of the vehicle around the user, with the inputs of the simulation being provided by virtual controls presented graphically to the user.
The disclosed system and method may be capable of operating a virtual reality simulator. The disclosed system and method may be capable of operating an augmented reality simulator. The disclosed system and method may be capable of operating either a virtual reality simulator or an augmented reality simulator. The disclosed system and method may be capable of operating a mixture of virtual and augmented reality programming, where a user may be able to select how much of the graphic rendering of the headset is virtual and how much is augmented. In one example, the controls of the actual vehicle to be utilized by the user may be captured in images and graphically presented to the user as augmented reality features, while a remainder of a view of the user may be virtual.
Referring now to the drawings, wherein like reference numbers refer to like features throughout the several views,
The exemplary steering wheel 30 is attached to a steering column 34 and a steering wheel mode selection device 32. The steering wheel mode selection device 32 includes mechanical and/or electronic mechanisms useful for selectively connecting and disconnecting the steering wheel 30 to a steering system of the vehicle 10 configured to turn actual wheels of the vehicle 10 left and right. Based upon control signals provided by the computerized simulation controller 20, the steering wheel 30 may be selectively disconnected from the steering system of the vehicle 10, and an actual rotation of the steering wheel 30 may be utilized as a control input to a simulation operated by the computerized simulation controller 20. The steering wheel 30, the steering column 34, and/or the steering wheel mode selection device 32 may additionally include a telescoping feature. During normal operation of the vehicle 10, the user 5 may utilize the telescoping feature to position the steering wheel 30 a comfortable distance from the user 5. In a simulation mode, the telescoping feature may be utilized as an additional input to the computerized simulation controller. Additionally, the steering wheel 30, the steering column 34, and/or the steering wheel mode selection device 32 may include servo motors or other features useful to make the steering wheel 30 vibrate, pulse in a direction, or otherwise provide haptic feedback to the user 5 during operation of a simulation.
The transmission gear selector 40 includes a transmission gear selector mode selection device 42. The transmission gear selector mode selection device 42 includes mechanical and/or electronic mechanisms useful for selectively connecting and disconnecting the transmission gear selector 40 to a transmission system of the vehicle 10 configured to select a mode of travel of the vehicle 10. Based upon control signals provided by the computerized simulation controller 20, the transmission gear selector 40 may be selectively disconnected from the transmission system of the vehicle 10, and an actual position of the transmission gear selector 40 may be utilized as a control input to a simulation operated by the computerized simulation controller 20.
The wireless short-range communication device 70 is useful to provide communication between the computerized simulation controller 20 and the headset 60. The wireless short-range communication device 70 is further useful to provide communication between the computerized simulation controller 20 and a computerized cellular device of the user 5 such as a smartphone. The wireless short-range communication device 70 may include Wi-fi communications and/or Bluetooth® communications. In one embodiment, the wireless short-range communication device 70 may be substituted with a data cable connecting the headset 60 to the vehicle 10, eliminating the usefulness of wireless communication between the vehicle 10 and the headset 60.
The climate control system 80 may include a blower unit configured to create a flow of air through air ducts 82. The climate control system 80 may further include a heater core configured to heat the flow of air and an evaporator core configured to cool and remove humidity from the flow of air. The climate control system 80 may be controlled by the computerized simulation controller 20 during operation of a simulation to provide a sensation of a flow of air to user 5 based upon contextual operation of the simulation.
The wireless long-range communication device 12 may include cellular communications and may provide an ability for the computerized simulation controller 20 to update, download content, and interact with other users in other vehicles remotely.
The vehicle 10 is further illustrated including an interior camera device 50 useful to capture images of the user 5 and interior features of the vehicle 10. The interior camera device 50 is optional and may be used to replace or augment data collected from the camera device 62. A plurality of camera devices 50 may be present within the vehicle 10 and may capture different perspectives within the vehicle 10. The vehicle 10 is further illustrated including a speaker 96 of an audio system of the vehicle, wherein the speaker 96 may be utilized to emit audio content of the simulation to the user 5. The headset 60 may additionally or alternatively include audio headphone outputs to the user 5. The vehicle 10 is further illustrated including the seat 14 which may be equipped with vibration unit 90 in a base of the seat 14 and vibration unit 92 in a back of the seat 14. The seat 14 may further include electronic height, tilt, recline angle, and lumbar control features which may be controllable by the computerized simulation controller 20.
The accelerator pedal 120 and the brake pedal 122 may each include mechanical and/or electronic mechanisms useful for selectively connecting and disconnecting the accelerator pedal 120 and the brake pedal 122 to a propulsion system and a braking system, respectively, of the vehicle 10 configured to control an actual speed of the vehicle. Based upon control signals provided by the computerized simulation controller 20, the accelerator pedal 120 and the brake pedal 122 may each be selectively disconnected from the systems of the vehicle 10, and an actual depression of each of the accelerator pedal 120 and the brake pedal 122 may be utilized as control inputs to a simulation operated by the computerized simulation controller 20.
Details or outputs of the simulation being operated by the computerized simulation controller 20 may be displayed within the context of the representation of the windshield 210. A graphical representation of a road surface 212 is illustrated. A graphical representation of another vehicle 214 is additionally illustrated.
Text, emojis, status bars, or other information may be provided within the context of the graphical rendering 205. An exemplary text box 216 is provided within the representation of the windshield 210, which may provide a current lap time in a simulated race or may provide a feature of the vehicle being virtually test driven. An exemplary text box 218 rendered upon the steering wheel 230 may provide an identity of a vehicle being virtually test driven or may include a limited-edition version of the steering wheel 230, for example, awarded for having a fastest lap time in a previous race. The representation of the touchscreen display 235 may provide an interactive list of information, for example, with a hand motion detected by the user 5 to the representation of the touchscreen display 235 being used to select or change information provided to the user 5. The representation of the instrument panel 250 may include a virtual speedometer 252 providing an output of the computerized simulation controller 20, describing a speed of the vehicle being simulated. The details of the graphical rendering 205 are exemplary, and the disclosure is not intended to be limited to the particular examples provided herein. Features, graphics, vehicle models, and other portions of the graphical rendering may include limited-edition NFT items, where ownership of the NFT items may be established through blockchain methods.
The processing device 310 may include memory, e.g., read only memory (ROM) and random-access memory (RAM), storing processor-executable instructions and one or more processors that execute the processor-executable instructions. In embodiments where the processing device 310 includes two or more processors, the processors may operate in a parallel or distributed manner. The processing device 310 may execute the operating system of the computerized simulation controller 20. Processing device 310 may include one or more modules executing programmed code or computerized processes or methods including executable steps. Illustrated modules may include a single physical device or functionality spanning multiple physical devices. In the illustrative embodiment, the processing device 310 also includes a simulation mode authorization and management module 312, a simulation operation module 314, and a virtual/augmented reality rendering module 316, which are described in greater detail below.
The communications device 320 may include a communications/data connection with a bus device configured to transfer data to different components of the system and may include one or more wireless transceivers for performing wireless communication.
The input output coordination device 330 includes hardware and/or software configured to enable the processing device 310 to receive and/or exchange data with on-board sensors of the host vehicle and to provide control of switches, modules, and processes throughout the vehicle based upon determinations made within the processing device 310.
The memory storage device 340 is a device that stores data generated or received by the computerized simulation controller 20. The memory storage device 340 may include, but is not limited to, a hard disc drive, an optical disc drive, and/or a flash memory drive.
The vehicle 10 of
The simulation operation module 314 may include programming to operate the desired simulation within the vehicle 10. For example, the simulation operation module 314 may include a program configured for operating a race, a program configured for operating a test drive, a program configured for training a student driver, a program configured to simulate operation of a boat, or a program to simulate operation of an airplane. The simulation operation module 314 may include programming to monitor inputs to actual vehicle controls and/or to virtual controls graphically rendered to the user 5. The simulation operation module 314 may include programming to monitor images of hand gestures of the user 5 and determine desired commands in the simulation corresponding to the hand gestures. The simulation operation module 314 may include rules of operation of the simulation, for example, monitoring simulated operation of the virtual vehicle upon the roadway, determining whether the virtual vehicle is maintaining position within a lane, and generating an output based upon the determination. The simulation operation module 314 may determine a location of the virtual vehicle relative to other virtual vehicles, for example, applying rules that prevent both virtual vehicles from occupying a same virtual location. The simulation operation module 314 may include rules related to operating a race, for example, monitoring a ranking of one virtual vehicle versus a second vertical vehicle. The simulation operation module 314 may include operations related to providing a virtual test drive, for example, monitoring an audio system input to a microphone device, determining whether the user 5 is asking a question about the virtual vehicle, and referencing a look-up table to provide an answer to the question. The simulation operation module 314 may include rules related to training a student driver, for example, monitoring compliance with basic driving rules such as coming to a complete stop at a stop sign. The simulation operation module 314 may provide playback of a training simulation after it is complete along with annotated critiques of the simulated driving. The simulation operation module 314 may include rules related to operating a boat, for example, providing a reaction when the virtual boat crosses another virtual boat's wake or when the virtual boat turns too fast at high speed. The simulation operation module 314 may include rules related to operating a plane, for example, providing a reaction when the virtual airplane goes below a stall speed or changing operation of the virtual airplane when flaps are extended. The simulation operation module 314 may include programming to simulate a virtual environment including other virtual vehicles, virtual pedestrians, virtual weather, virtual time of day, and other simulated details. The simulation operation module 314 may include programming to control vehicle systems, such as vibration, haptic, climate control, and audio outputs of the vehicle 10, based upon operation of the simulation. The simulation operation module 314 may provide for social interaction of the user 5 with other users through the wireless long-range communication device 12. The simulation operation module 314 may include programming to operate 360-degree sound, such that passing another virtual vehicle within the simulation results in directional sound from the sound system of vehicle 10 approximating the actual sounds and direction of passing another vehicle in the real world. The simulation operation module 314 may include a variety of rules, simulated environmental details, simulated vehicles, simulated driving or operating conditions, simulated race or game contexts, social applications, and other relevant programming to a simulation, and the disclosure is not intended to be limited to the examples provided herein.
The headset 60 of
The computerized simulation controller 20 is provided as an exemplary computerized device capable of executing programmed code to operate the disclosed method. In one exemplary embodiment, some of the functionality described in relation to the computerized simulation controller 20 may instead be operated by the headset 60. A number of different embodiments of the computerized simulation controller 20 and modules operable therein are envisioned, and the disclosure is not intended to be limited to examples provided herein.
While the best modes for carrying out the disclosure have been described in detail, those familiar with the art to which this disclosure relates will recognize various alternative designs and embodiments for practicing the disclosure within the scope of the appended claims.
