Patent Application
20230148927
The subject matter described herein relates, in general, to controlling a visual interface, and, more particularly, to controlling display attributes on the visual interface for producing stimuli and responses of a user.
Systems display visual interfaces for users to manage and control devices. For example, a user can touch or speak as inputs according to a displayed image on a device. In response, the device processes the input and responds with a visual change to the image or outputs audio. In various implementations, systems can use brainwaves from a user to receive an input or request a response. Using brainwaves may allow a user freedom to use hands for other tasks and reduce the distraction associated with controlling the device.
In one approach, a system uses visuals to stimulate a brainwave response from a user for control or alerts. For example, the system is a user interface displaying icons on a vehicle windshield that controls entertainment, safety, and temperature systems using brainwaves from an operator. The system varies an attribute of a displayed icon at a frequency associated with a task (e.g., navigation alert, radio control, etc.) for visual stimulation. However, these systems encounter difficulties enhancing or highlighting a task on a visual interface while maintaining operator awareness and engagement. As such, operator fatigue can increase particularly through prolonged exposure to visual stimulation for the displayed icon.
In one embodiment, example systems and methods relate to a manner of controlling display attributes within a visual interface for enhancing stimuli and responses of a user. In various implementations, systems using stimuli on a visual interface that evoke a user response encounter difficulties highlighting a task while maintaining user awareness. For example, a system varying an icon color between black and white as stimuli causes user fatigue and decreases response accuracy on a visual interface. Therefore, in one embodiment, a control system varies display attributes of a visual interface at values and frequencies for generating stimuli that prevents user fatigue. In particular, the control system in a brain-machine interface (BMI) oscillates a display attribute (e.g., a color scheme or a background pattern) for an image layer at a frequency. The control system also oscillates display attributes at a harmonic of the frequency to enhance stimuli, while mitigating user fatigue. In one approach, the image layer is an icon and the oscillation toggles the display attribute between a pair of states (e.g., two colors). In one approach, the oscillation may continue until the control system receives a response (e.g., icon selection) through the BMI or an event (e.g., alert) ends. In this way, the control system produces stimuli in the visual interface while preventing user fatigue by oscillating display attributes using harmonic frequencies.
In one embodiment, a control system for changing display attributes within a visual interface for enhancing stimuli and responses of a user is disclosed. The control system includes a processor and a memory communicably coupled to the processor and storing a visualization module including instructions that, when executed by the processor, cause the processor to select display attributes related to a first layer of an image and a second layer of the image, wherein the image is associated with the interface to control a component. The instructions also include instructions to oscillate, at a frequency, between first values of the display attributes for displaying the first layer. The instructions also include instructions to oscillate, at a harmonic of the frequency, between second values of the display attributes for displaying the second layer in parallel with the first layer until detection of a selection associated with the interface.
In one embodiment, a non-transitory computer-readable medium for controlling display attributes within a visual interface for enhancing stimuli and responses of a user and including instructions that when executed by a processor cause the processor to perform one or more functions is disclosed. The instructions include instructions to select display attributes related to a first layer of an image and a second layer of the image, wherein the image is associated with the interface to control a component. The instructions also include instructions to oscillate, at a frequency, between first values of the display attributes for displaying the first layer. The instructions also include instructions to oscillate, at a harmonic of the frequency, between second values of the display attributes for displaying the second layer in parallel with the first layer until detection of a selection associated with the interface.
In one embodiment, a method for controlling display attributes within a visual interface for enhancing stimuli and responses of a user is disclosed. In one embodiment, the method includes selecting display attributes related to a first layer of an image and a second layer of the image, wherein the image is associated with an interface to control a component. The method also includes oscillating, at a frequency, between first values of the display attributes for displaying the first layer. The method also includes oscillating, at a harmonic of the frequency, between second values of the display attributes for displaying the second layer in parallel with the first layer until detecting a selection associated with the interface.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various systems, methods, and other embodiments of the disclosure. It will be appreciated that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the figures represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component and vice versa. Furthermore, elements may not be drawn to scale.
Systems, methods, and other embodiments associated with controlling display attributes within a visual interface for enhancing stimuli and responses of a user are disclosed herein. In various implementations, systems using stimuli on a visual interface for a visually evoked potential (VEP) scheme encounter difficulties executing a task while maintaining user responsiveness. For example, a system varying colors within an image between black and white as stimuli causes user fatigue and decreases accuracy on a brain-machine interface (BMI). Therefore, in one embodiment, a control system selects and varies display attributes on a visual interface for optimizing a VEP scheme using harmonic frequencies. Here, display attributes may define values about colors or backgrounds (e.g., patterns) for areas within an image. Furthermore, the control system can control different portions of the image by layer, where each layer is associated with an icon or background. As such, the control system can use a VEP scheme that enhances or increases responsiveness to parts (e.g., layers) of a visual interface while mitigating fatigue by oscillating colors beyond black and white.
Moreover, the control system oscillates the display attributes at a frequency (e.g., a fundamental frequency or a first harmonic). In one approach, the control system selects the frequency that enhances or increases a VEP response using cycles of the display attributes between ON/OFF frames (e.g., display frames). For example, an OFF frame represents disengaging an attribute to reduce stimuli and an ON frame may achieve the opposite effect. In one example, the control system displays the ON frame of icon backgrounds as orange while the OFF frame is green. As such, the control system may oscillate the colors through the ON/OFF frames at the frequency. In this way, the different colors maximize contrast between the ON/OFF frames for triggering robust VEP responses in the visual cortex of a user. Here, the control system measures the VEP response using electroencephalography (EEG) sensors of the BMI and also adjusts the frequency for optimizing the VEP response.
As further enhancements, in one example the control system uses higher harmonics of the frequency that optimizes the VEP scheme and mitigates user fatigue. In particular, the control system overlays the display attributes within layers of the image that oscillate at different frequencies, thereby creating a unique and a robust VEP scheme. In other words, the control system may oscillate two display attributes between different frames. For example, the control system uses a third color as an ON frame for oscillating a symbol color within an icon at the second harmonic of the frequency (e.g., a fundamental frequency). Here, the control system may oscillate in parallel an icon background between two colors at the frequency. In other words, the control system may oscillate two colors for the visual interface at the fundamental frequency between frames while oscillating a third color at the second harmonic. This sequence for a composed image can continue until the control system detects a selection on the visual interface or the BMI measures a target response from the user. In this way, the control system improves a VEP response and input accuracy for the visual interface while mitigating fatigue by using colors and harmonics.
With reference to
Moreover, in one embodiment, the control system 100 includes a data store 140. In one embodiment, the data store 140 is a database. The database is, in one embodiment, an electronic data structure stored in the memory 120 or another data store and that is configured with routines that can be executed by the processor(s) 110 for analyzing stored data, providing stored data, organizing stored data, and so on. Thus, in one embodiment, the data store 140 stores data used by the visualization module 130 in executing various functions associated with varying (e.g., oscillate or toggle) or display attributes.
The data store 140 also includes the display attributes 150. In one approach, the display attributes 150 define values about colors, a color scheme, background colors, background patterns (e.g., checkered), frequency, and so on associated with an image. These values can be associated with an icon, a symbol, a background area, and so on within the image. Furthermore, the icon, the symbol, or the background may represent a layer of an image. In this way, the control system 100 can control different portions of the image by layer according to the display attributes 150.
As explained below, the visualization module 130, in one embodiment, is configured to perform tasks for varying the display attributes 150 within a visual interface as stimuli. For example, the visualization module 130 includes instructions that cause the processor(s) 110 to implement a visual scheme in a system (e.g., vehicle) using VEP. In systems using a VEP scheme, the control system 100 displays visual stimuli according to an oscillation frequency (e.g., a display refresh rate) for evoking a response. For instance, the visual cortex of a user responds to stimuli of display attributes according to the VEP scheme. Oscillating a display attribute in this manner is sometimes called a steady-state VEP (SSVEP) scheme. Furthermore, the control system 100 may use motion SSVEP (MSSVEP) when the oscillation includes animation or motion of a display attribute. In one approach, the user response is selecting or providing feedback by viewing a display attribute oscillating on a visual interface. For feedback or input, the control system 100 may measure responses to these visual stimuli using EEG sensors of a BMI. The EEG sensors are coupled to a body part (e.g., head) of a user and communicate measured stimuli to the control system 100 accordingly. In this way, the control system 100 measures effectiveness and accuracy to a VEP scheme.
In various implementations, the control system 100 is used for consumer electronics (CE), mobile devices, vehicles, and so on that benefit from the functionality discussed herein associated with controlling the display attributes 150 within a visual interface for enhancing stimuli and responses of a user. For instance, the control system 100 renders icons directly on a vehicle windshield, side-window, a heads-up display (HUD), and so on for driver control using a BMI according to the display attributes 150. The control system 100 uses measurements from EEG sensors on or near a body part of the operator to determine user feedback to stimuli. The control system 100 can vary display attributes (e.g., color contrast) of the icons at different frequencies for increasing focus, enhancing response time, or encouraging selection. For example, the visualization module 130 oscillates background display attributes of an icon between orange/green instead of black/white for producing effective while gentle stimuli. In this way, the control system 100 mitigates fatigue for the VEP scheme, particularly during extended exposure (e.g., extended road trips).
Turning now to
For the visual interface 200, the control system 100 may use a VEP scheme that enhances stimuli while mitigating fatigue through different color pairs instead of black and white. The enhanced stimuli may improve input accuracy and focus of users for systems using a BMI. For example, the visualization module 130 displays the ON frame 210 of four icon backgrounds as orange while the OFF frame 220 is green within the visual interface 200. The control system 100 may then oscillate the colors of the icon backgrounds through these ON/OFF frames at a frequency that enhances a VEP and a corresponding user response. In other words, the different colors maximize contrast between the ON/OFF frames for triggering robust VEP responses in the visual cortex of a user. The user response may be the control system 100 drawing the focus and the attention of the user to an alert or event associated with the image for the BMI. The user response may also be the control system 100 evoking a selection associated with the image.
Referring to
By overlaying the display attributes 150 within layers of an image and using higher harmonics, the control system 100 creates unique VEP schemes. Utilizing harmonic frequencies contributes to increased frequency band-power (e.g., micro volts{circumflex over ( )}2/Hz) on the fundamental, second harmonic, or higher harmonics. The control system 100 also increases the information transfer rate (ITR) as well as VEP detection in shorter time windows (e.g., 2 seconds) by using higher harmonics.
In one approach, portions of the visual interface 300 vary according to the display attributes 150 for optimizing the VEP scheme and BMI. As such, a pattern of one icon background may oscillate between ON/OFF frames at a fundamental frequency while other icon backgrounds remain ON/OFF. As explained above, the control system 100 may then also oscillate the color of the icon symbol at the second harmonic of the fundamental frequency. Such a VEP scheme can improve accuracy for a user selecting an icon visually through the BMI.
In various implementations, the control system 100 oscillates the display attributes 150 of a navigation icon within the visual interface 300 on a HUD due to changing traffic patterns. Operation is improved by the user selecting the navigation icon instead of other icons and receiving further information about the traffic patterns. For added enhancements, the control system 100 may use this approach to calibrate the BMI and the HUD according to selection times and measured stimuli.
In addition to selecting inputs and other tasks, the control system 100 may oscillate the display attributes 150 in the visual interface 300 using harmonic frequencies for authentication. A brainwave pattern represents a unique signature for a user of a BMI. As such, the control system 100 determines a brainwave pattern using measurements from EEG sensors of a user viewing an image. The control system 100 implements a VEP scheme for accurate authentication by oscillating the display attributes 150 at higher harmonics of the fundamental frequency between frames to increase unique responses. For example, the control system 100 oscillates a symbol within the image using an atypical color at the second harmonic. Meanwhile, the background of the symbol oscillates using a checkered pattern and an atypical color at the third harmonic. In this way, authentication accuracy is increased using the VEP scheme by the control system 100 acquiring a robust brainwave pattern from the BMI through different harmonic frequencies.
Turning now to
At 410, the control system 100 selects the display attributes and associated values to vary (e.g., oscillate or toggle) on a visual interface. In one approach, the display attributes define values about colors, a color scheme, background colors, or background patterns (e.g., checkered) associated with an image. These values can be for an icon, a symbol, a background area, and so on within the image. In various implementations, the icon, the symbol, or the background area may represent a layer of an image. In this way, the control system 100 can control different portions of the image by layer according to the display attributes.
Moreover, the control system 100 may select the display attributes to implement a VEP scheme specified by a manufacturer for a particular task. For example, a HUD displaying a temperature control system uses a VEP scheme that enhances or increases visual responses in a BMI to save energy by optimizing temperature adjustments. The control system 100 can also use a VEP scheme that increases focus on icon having an alert (e.g., badge) while mitigating fatigue by oscillating colors beyond black and white.
At 420, the control system 100 oscillates the display attributes and associated values selected on the visual interface at a frequency (e.g., a fundamental frequency or a first harmonic). In one approach, the control system 100 selects the frequency to attain a particular VEP response or meet the refresh rate of a display. As explained above, the frequency may also be related to target cycles of the display attributes between ON/OFF frames. In one approach, an OFF frame may represent disengaging an attribute to reduce stimuli and an ON frame may achieve the opposite effect. Furthermore, the control system 100 can use measurements from EEG sensors of a BMI to adjust the frequency or duty cycle and calibrate the BMI. In one approach, the control system 100 can capture the oscillation of visual stimuli and adjust the frequency from the EEG data by calibrating the BMI or through incremental learning that continuously finds the optimal settings for the visual stimuli. Accordingly, the control system 100 improves the responsiveness of a user through the adjustments for the VEP scheme.
In various implementations, the control system 100 uses a VEP scheme to enhance stimuli while mitigating fatigue through different color pairs. For example, the visualization module 130 displays the ON frame of icon backgrounds as orange while the OFF frame is green within a visual interface. The control system 100 may then oscillate the colors of the icon backgrounds through the ON/OFF frames at the frequency. In other words, the different colors maximize contrast between the ON/OFF frames for triggering robust VEP responses in the visual cortex of a user.
At 430, the control system 100 oscillates display attributes of the image at a harmonic (e.g., second harmonic) of the frequency. Here, using harmonics of the frequency optimizes responses to the VEP scheme while mitigating fatigue. In particular, the control system 100 overlays the display attributes within layers of an image that oscillate at different frequencies, thereby creating a unique and a robust VEP scheme. As such, the control system 100 may oscillate two display attributes between different frames. For example, the control system 100 uses a third color as an ON frame for oscillating a symbol color within an icon at the second harmonic of the frequency used to oscillate the icon background. Here, the OFF frame of the symbol color may be white. Accordingly, the control system 100 may oscillate two colors for the visual interface at the fundamental frequency between frames while oscillating the symbol color at the second harmonic. In this way, the control system 100 improves a VEP response and input accuracy for the visual interface while mitigating fatigue.
In various implementations, the pattern of one icon background may oscillate between ON/OFF frames at a fundamental frequency while other icon backgrounds remain in a current state. Meanwhile, the control system 100 oscillates the color of the icon symbol at the second harmonic of the fundamental frequency. This sequence for a composed image may continue until the control system 100 detects a selection on the visual interface or receives from the BMI a target response measured from the user. In other words, the control system 100 can use data from EEGs of the BMI to ensure that stimuli has increased beyond a threshold as a response and effect. Here, the effect may be viewing and physically processing a change on the visual interface. In another example, the control system 100 oscillates the display attributes 150 of a navigation icon within the visual interface on a HUD due to changing traffic patterns. This operation may continue until the user reacts by a driving maneuver, an input, or the control system 100 measures certain stimuli using the BMI. In this way, the control system 100 uses harmonic frequencies and a VEP scheme that improves accuracy and stimuli.
Detailed embodiments are disclosed herein. However, it is to be understood that the disclosed embodiments are intended as examples. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the aspects herein in virtually any appropriately detailed structure. Furthermore, the terms and phrases used herein are not intended to be limiting but rather to provide an understandable description of possible implementations. Various embodiments are shown in
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, a block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
The systems, components and/or processes described above can be realized in hardware or a combination of hardware and software and can be realized in a centralized fashion in one processing system or in a distributed fashion where different elements are spread across several interconnected processing systems. Any kind of processing system or another apparatus adapted for carrying out the methods described herein is suited. A combination of hardware and software can be a processing system with computer-usable program code that, when being loaded and executed, controls the processing system such that it carries out the methods described herein. The systems, components and/or processes also can be embedded in a computer-readable storage, such as a computer program product or other data programs storage device, readable by a machine, tangibly embodying a program of instructions executable by the machine to perform methods and processes described herein. These elements also can be embedded in an application product which comprises the features enabling the implementation of the methods described herein and, which when loaded in a processing system, is able to carry out these methods.
Furthermore, arrangements described herein may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied, e.g., stored, thereon. Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The phrase “computer-readable storage medium” means a non-transitory storage medium. A computer-readable medium may take forms, including, but not limited to, non-volatile media, and volatile media. Non-volatile media may include, for example, optical disks, magnetic disks, and so on. Volatile media may include, for example, semiconductor memories, dynamic memory, and so on. Examples of such a computer-readable medium may include, but are not limited to, a floppy disk, a flexible disk, a hard disk, a magnetic tape, another magnetic medium, an application-specific integrated circuit (ASIC), a CD, another optical medium, a RAM, a ROM, a memory chip or card, a memory stick, and other media from which a computer, a processor or other electronic device can read. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for various implementations. The examples are not intended to be limiting. Both singular and plural forms of terms may be within the definitions.
References to “one embodiment,” “an embodiment,” “one example,” “an example,” and so on, indicate that the embodiment(s) or example(s) so described may include a particular feature, structure, characteristic, property, element, or limitation, but that not every embodiment or example necessarily includes that particular feature, structure, characteristic, property, element, or limitation. Furthermore, repeated use of the phrase “in one embodiment” does not necessarily refer to the same embodiment, though it may.
“Module,” as used herein, includes a computer or electrical hardware component(s), firmware, a non-transitory computer-readable medium that stores instructions, and/or combinations of these components configured to perform a function(s) or an action(s), and/or to cause a function or action from another logic, method, and/or system. Module may include a microprocessor controlled by an algorithm, a discrete logic (e.g., an ASIC), an analog circuit, a digital circuit, a programmed logic device, a memory device including instructions that, when executed perform an algorithm, and so on. A module, in one or more embodiments, includes one or more CMOS gates, combinations of gates, or other circuit components. Where multiple modules are described, one or more embodiments include incorporating the multiple modules into one physical module component. Similarly, where a single module is described, one or more embodiments distribute the single module between multiple physical components.
Additionally, module, as used herein, includes routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular data types. In further aspects, a memory generally stores the noted modules. The memory associated with a module may be a buffer or cache embedded within a processor, a RAM, a ROM, a flash memory, or another suitable electronic storage medium. In still further aspects, a module as envisioned by the present disclosure is implemented as an ASIC, a hardware component of a system on a chip (SoC), as a programmable logic array (PLA), or as another suitable hardware component that is embedded with a defined configuration set (e.g., instructions) for performing the disclosed functions.
In one or more arrangements, one or more of the modules described herein can include artificial or computational intelligence elements, e.g., neural network, fuzzy logic, or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules can be distributed among a plurality of the modules described herein. In one or more arrangements, two or more of the modules described herein can be combined into a single module.
Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, radio frequency (RF), etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present arrangements may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java™, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
The terms “a” and “an,” as used herein, are defined as one or more than one. The term “plurality,” as used herein, is defined as two or more than two. The term “another,” as used herein, is defined as at least a second or more. The terms “including” and/or “having,” as used herein, are defined as comprising (i.e., open language). The phrase “at least one of . . . and . . . ” as used herein refers to and encompasses any and all combinations of one or more of the associated listed items. As an example, the phrase “at least one of A, B, and C” includes A, B, C, or any combination thereof (e.g., AB, AC, BC or ABC).
Aspects herein can be embodied in other forms without departing from the spirit or essential attributes thereof. Accordingly, reference should be made to the following claims, rather than to the foregoing specification, as indicating the scope hereof.
