Patent Grant
11429265
Many computing devices convey information through visual information displayed on a display and/or through audible information provided through a speaker. For example, a mobile device may display a weather chart through a weather application user interface. A tablet may play an audio book through an audio user interface. In this way, various types of computing devices can convey information in various ways.
Graphical representations of data allow users to quickly get a visual summary and key markers of the data, such as a domain of the data being represented, trends, outliers, minimum values, maximum values, starting values, ending values, and a range of the data. Such graphical representations provide users with more in-depth access to data points. Unfortunately, graphical representations of data and/or other visually represented data (e.g., images, videos, a text document, a website, etc.) provide little value to visually impaired users, those who don't have immediate access to the display, etc. Some computing devices provide visually impaired users with screen reader functionality that reads displayed content through audio output. However, audio descriptions of certain types of content may be inadequate for providing a user with a full understanding of the content. This leaves the user with the difficult task of constructing a mental picture of the content based upon an audio description that merely recites what is currently displayed (e.g. line graph trending up).
In accordance with the present disclosure, one or more computing devices and/or methods for controlling an output device are provided. In one embodiment, a computing device may store or provide access to an object. The object may represent information, such as a chart or graph comprising data points. It may be appreciated that any type of object comprising one or more datum points (e.g., values) may be processed. The data points of the object may be extracted, such as values extracted from data or metadata of the chart. Features are extracted from the data points, such as a minimum value, a maximum value, a trend, a change in slope such as from increasing values to decreasing values or vice versa, a first data point, a last data point, an outlier, and/or a variety of other features of data.
A set of data points are identified from the data points of the object based upon the set of data points having features corresponding to target features, such as those identified as interesting features (e.g., changes in slope, minimum values, maximum values, etc.). Marker elements, such as invisible user interface elements, are assigned to the set of data points. In one example, marker elements are assigned to every datum point of the object. In another example, marker elements are assigned to a subset of the data points of the object, such as the set of data points having the interesting features.
A mapping structure, mapping features to output indicators, is used to assign output indicators to marker elements based upon features of data points to which the marker elements are assigned. An output indicator can comprise a variety of different types of indicators, such as a vibration of a haptic sensor, an audio signal, and/or other types of output that can be provided by output devices. For example, an output indicator of an audio signal can have certain audio features corresponding to certain types of values of data points, such as different tones, melodies, instruments, volumes, pitches, frequencies, echoes, textures, etc. In this way, an audio signal can have a certain audio feature that invokes a feeling corresponding to the values of data points, such as a particular sound at a particular frequency and volume to convey a stock price dropping sharply (e.g., a sound of pots and pans dropping onto a floor).
Navigation among the marker elements is facilitated, such as through a touch screen of the computing device, voice commands spoken to the computing device, or any other type of user input. In one example, the navigation is provided through a user interface through which the object is rendered. In another example, the navigation is provided separate of the object, such as where the object is a chart file not displayed on the computing device but is stored within storage of the computing device. An output device, such as an audio device or a haptic device array, is controlled to provide output indicators assigned to marker elements as the marker elements become the current focus of the user input (e.g., the current focus may be a region of the display underneath a finger of the user). In this way, the user can gesture, swipe, or perform other input to navigate between marker elements in order to have corresponding output indicators provided to the user. Based upon the speed of the user's navigation, a macro view (e.g., zoom out to a larger granular view of the chart) or a micro view (e.g., zoom into a particular region of the chart for a finer granular view of the chart) of the object is provided.
While the techniques presented herein may be embodied in alternative forms, the particular embodiments illustrated in the drawings are only a few examples that are supplemental of the description provided herein. These embodiments are not to be interpreted in a limiting manner, such as limiting the claims appended hereto.
Subject matter will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific example embodiments. This description is not intended as an extensive or detailed discussion of known concepts. Details that are known generally to those of ordinary skill in the relevant art may have been omitted, or may be handled in summary fashion.
The following subject matter may be embodied in a variety of different forms, such as methods, devices, components, and/or systems. Accordingly, this subject matter is not intended to be construed as limited to any example embodiments set forth herein. Rather, example embodiments are provided merely to be illustrative. Such embodiments may, for example, take the form of hardware, software, firmware or any combination thereof.
1. Computing Scenario
The following provides a discussion of some types of computing scenarios in which the disclosed subject matter may be used and/or implemented.
1.1. Networking
The servers 104 of the service 102 may be internally connected via a local area network 106 (LAN), such as a wired network where network adapters on the respective servers 104 are interconnected via cables (e.g., coaxial and/or fiber optic cabling), and may be connected in various topologies (e.g., buses, token rings, meshes, and/or trees). The servers 104 may be interconnected directly, or through one or more other networking devices, such as routers, switches, and/or repeaters. The servers 104 may use a variety of physical networking protocols (e.g., Ethernet and/or Fiber Channel) and/or logical networking protocols (e.g., variants of an Internet Protocol (IP), a Transmission Control Protocol (TCP), and/or a User Datagram Protocol (UDP). The local area network 106 may include, e.g., analog telephone lines, such as a twisted wire pair, a coaxial cable, full or fractional digital lines including T1, T2, T3, or T4 type lines, Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communication links or channels, such as may be known to those skilled in the art. The local area network 106 may be organized according to one or more network architectures, such as server/client, peer-to-peer, and/or mesh architectures, and/or a variety of roles, such as administrative servers, authentication servers, security monitor servers, data stores for objects such as files and databases, business logic servers, time synchronization servers, and/or front-end servers providing a user-facing interface for the service 102.
Likewise, the local area network 106 may comprise one or more sub-networks, such as may employ differing architectures, may be compliant or compatible with differing protocols and/or may interoperate within the local area network 106. Additionally, a variety of local area networks 106 may be interconnected; e.g., a router may provide a link between otherwise separate and independent local area networks 106.
In the scenario 100 of
In the scenario 100 of
1.2. Server Configuration
The server 104 may comprise one or more processors 210 that process instructions. The one or more processors 210 may optionally include a plurality of cores; one or more coprocessors, such as a mathematics coprocessor or an integrated graphical processing unit (GPU); and/or one or more layers of local cache memory. The server 104 may comprise memory 202 storing various forms of applications, such as an operating system 204; one or more server applications 206, such as a hypertext transport protocol (HTTP) server, a file transfer protocol (FTP) server, or a simple mail transport protocol (SMTP) server; and/or various forms of data, such as a database 208 or a file system. The server 104 may comprise a variety of peripheral components, such as a wired and/or wireless network adapter 214 connectible to a local area network and/or wide area network; one or more storage components 216, such as a hard disk drive, a solid-state storage device (SSD), a flash memory device, and/or a magnetic and/or optical disk reader.
The server 104 may comprise a mainboard featuring one or more communication buses 212 that interconnect the processor 210, the memory 202, and various peripherals, using a variety of bus technologies, such as a variant of a serial or parallel AT Attachment (ATA) bus protocol; a Uniform Serial Bus (USB) protocol; and/or Small Computer System Interface (SCI) bus protocol. In a multibus scenario, a communication bus 212 may interconnect the server 104 with at least one other server. Other components that may optionally be included with the server 104 (though not shown in the schematic architecture diagram 200 of
The server 104 may operate in various physical enclosures, such as a desktop or tower, and/or may be integrated with a display as an “all-in-one” device. The server 104 may be mounted horizontally and/or in a cabinet or rack, and/or may simply comprise an interconnected set of components. The server 104 may comprise a dedicated and/or shared power supply 218 that supplies and/or regulates power for the other components. The server 104 may provide power to and/or receive power from another server and/or other devices. The server 104 may comprise a shared and/or dedicated climate control unit 220 that regulates climate properties, such as temperature, humidity, and/or airflow. Many such servers 104 may be configured and/or adapted to use at least a portion of the techniques presented herein.
1.3. Client Device Configuration
The client device 110 may comprise one or more processors 310 that process instructions. The one or more processors 310 may optionally include a plurality of cores; one or more coprocessors, such as a mathematics coprocessor or an integrated graphical processing unit (GPU); and/or one or more layers of local cache memory. The client device 110 may comprise memory 301 storing various forms of applications, such as an operating system 303; one or more user applications 302, such as document applications, media applications, file and/or data access applications, communication applications such as web browsers and/or email clients, utilities, and/or games; and/or drivers for various peripherals. The client device 110 may comprise a variety of peripheral components, such as a wired and/or wireless network adapter 306 connectible to a local area network and/or wide area network; one or more output components, such as a display 308 coupled with a display adapter (optionally including a graphical processing unit (GPU)), a haptic feedback array, a sound adapter coupled with a speaker, and/or a printer; input devices for receiving input from the user, such as a keyboard 311, a tactile feedback sensor, a mouse, a microphone, a camera, and/or a touch-sensitive component of the display 308; and/or environmental sensors, such as a global positioning system (GPS) receiver 319 that detects the location, velocity, and/or acceleration of the client device 110, a compass, accelerometer, and/or gyroscope that detects a physical orientation of the client device 110. Other components that may optionally be included with the client device 110 (though not shown in the schematic architecture diagram 300 of
The client device 110 may comprise a mainboard featuring one or more communication buses 312 that interconnect the processor 310, the memory 301, and various peripherals, using a variety of bus technologies, such as a variant of a serial or parallel AT Attachment (ATA) bus protocol; the Uniform Serial Bus (USB) protocol; and/or the Small Computer System Interface (SCI) bus protocol. The client device 110 may comprise a dedicated and/or shared power supply 318 that supplies and/or regulates power for other components, and/or a battery 304 that stores power for use while the client device 110 is not connected to a power source via the power supply 318. The client device 110 may provide power to and/or receive power from other client devices.
In some scenarios, as a user 112 interacts with a software application on a client device 110 (e.g., an instant messenger and/or electronic mail application), descriptive content in the form of signals or stored physical states within memory (e.g., an email address, instant messenger identifier, phone number, postal address, message content, date, and/or time) may be identified. Descriptive content may be stored, typically along with contextual content. For example, the source of a phone number (e.g., a communication received from another user via an instant messenger application) may be stored as contextual content associated with the phone number. Contextual content, therefore, may identify circumstances surrounding receipt of a phone number (e.g., the date or time that the phone number was received), and may be associated with descriptive content. Contextual content, may, for example, be used to subsequently search for associated descriptive content. For example, a search for phone numbers received from specific individuals, received via an instant messenger application or at a given date or time, may be initiated. The client device 110 may include one or more servers that may locally serve the client device 110 and/or other client devices of the user 112 and/or other individuals. For example, a locally installed webserver may provide web content in response to locally submitted web requests. Many such client devices 110 may be configured and/or adapted to use at least a portion of the techniques presented herein.
2. Presented Techniques
One or more computing devices and/or techniques for controlling an output device is provided. The output device, such as an audio device or haptic device, is controlled to provide output indicators used to convey information relating to data points of an object, such as a chart or graph. Such techniques improve the ability of visually impaired users and/or other users to interact with information conveyed through a computing device, such as an improved ability to understand dense information such as the ability to understand trends of data points within a graph or chart. Particular sounds, volume levels, pitches, instruments, songs, melodies, textures, tones, and/or other audio signal features can be used to convey emotional information associated with the data points, such as a sharp rise in a stock price.
An embodiment of controlling an output device is illustrated by an example method 400 of
At 404, features 518 are extracted from the plurality of data points 506. The features 518 may correspond to various trends, values, or range of values that may be interesting, such as a maximum value 508, a minimum value 512, a change 510 from increasing values to decreasing values, negative values, positive values, outliers, etc. At 406, a set of data points 522 are identified from the plurality of data points 506 based upon the set of data points 522 having features 518 corresponding to target features, as illustrated by
At 408, marker elements are assigned 524 to the set of data points 522 within the object 504, as illustrated by
At 410, a mapping structure 520 is used to assign 544 output indicators to marker elements based upon features 518 of data points to which the marker elements are assigned, as illustrated by
Features of data points, such as values of data points, trends, slopes, changes from increasing to decreasing values or vice versa, negative values, positive values, maximum values, minimum values, types of data being represented (e.g., stock values, sports scores, bank account values, etc.), and/or other features can be mapped to particular output indicators such as audio signals having particular audio signal output features. For example, a sharp rise in stock values may be assigned to an output indicator with a particular instrument, volume, and frequency, whereas a sharp decline in stock values may be assigned to an output indicator with a different instrument, volume, and frequency. For sports scores, different volumes, instruments, and/or other types of output features may be used compared to output features of output indicators used to represent stock values or other types of values. In another example, if output indicators are a different type of output such as haptic device vibrations, then different vibration intensities, vibration lengths, etc. can be used to represent different types of data points (e.g., sports scores compared to stock values) and/or data points having different types of features (e.g., a minimum value compared to an outlier).
At 412, user navigation is facilitated among the marker elements, as illustrated in
In one example, a display location of the marker element that is the current focus is determined (e.g., x/y coordinates). One or more haptic devices of a haptic device array that are proximate to the display location are identified. The haptic device array is controlled to provide the output indicator 552 by invoking the one or more haptic devices proximate to the display location to activate, such as to vibrate.
In one example, an indicator of a last accessed marker element is maintained. In response to determining a context change from a user interface associated with the object 504 to a second user interface (e.g., the user navigates from the user interface through which the object 504 is rendered to a different user interface such as a web browser interface, a game interface, an email app interface, etc.), the indicator may be persisted to storage, such as to memory, a storage device, etc. In response to determining a second context change to the user interface associated with the object 504, the indicator may be extracted from the storage. The current focus is set to the last accessed marker element (e.g., the last accessed marker element is brought into focus where the finger of the user is currently positioned, and a zoom level, such as a macro view or micro view, of the object 504 may be restored).
User navigation may be tracked to determine whether the user wants to zoom into a finer granularity of data points or zoom out to a more coarse granular view of the data points, as illustrated by the example system 600 of
In one example, user navigation is detected 630 within the graph object 604, as illustrated by
In another example, an input speed of the user navigation is determined. In response to the input speed exceeding a threshold speed (e.g., the user is scrolling quickly through the graph object 604), a macro scale object comprising a second set of data points from the plurality of data points is generated. The second set of data points may comprise a coarser macro view of the graph object 604. Macro marker elements are assigned to the second set of data points based upon features of data points within the second set of data points. The mapping structure 610 is used to assign output indicators to the macro marker elements based upon features of data points to which macro marker elements are assigned. The output device 608 is controlled to provide output indicators assigned to the macro marker elements as the macro marker elements become the current focus of the user navigation.
In another example, a current input speed of the user navigation is determined. In response to the current input speed falling below a threshold speed (e.g., the user starts to slowly scroll through the graph object 604), a micro scale object comprising a third set of data points from the plurality of data points is generated. The third set of data points may comprise a finer micro view of the graph object 604. Micro marker elements are assigned to the third set of data points based upon features of data points within the third set of data points. The mapping structure 610 is used to assign output indicators to the micro marker elements based upon features of data points to which micro marker elements are assigned. The output device 608 is controlled to provide output indicators assigned to the micro marker elements as the micro marker elements become the current focus of the user navigation.
3. Usage of Terms
As used in this application, “component,” “module,” “system”, “interface”, and/or the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
Unless specified otherwise, “first,” “second,” and/or the like are not intended to imply a temporal aspect, a spatial aspect, an ordering, etc. Rather, such terms are merely used as identifiers, names, etc. for features, elements, items, etc. For example, a first object and a second object generally correspond to object A and object B or two different or two identical objects or the same object.
Moreover, “example” is used herein to mean serving as an example, instance, illustration, etc., and not necessarily as advantageous. As used herein, “or” is intended to mean an inclusive “or” rather than an exclusive “or.” In addition, “a” and “an” as used in this application are generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. Also, at least one of A and B and/or the like generally means A or B or both A and B. Furthermore, to the extent that “includes”, “having”, “has”, “with”, and/or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising”.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing at least some of the claims.
Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. Of course, many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
Various operations of embodiments are provided herein. In an embodiment, one or more of the operations described may constitute computer readable instructions stored on one or more computer readable media, which if executed by a computing device, will cause the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated by one skilled in the art having the benefit of this description. Further, it will be understood that not all operations are necessarily present in each embodiment provided herein. Also, it will be understood that not all operations are necessary in some embodiments.
Also, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.
This application claims priority to and is a continuation of U.S. application Ser. No. 15/858,328, filed on Dec. 29, 2017, entitled “OUTPUT DEVICE CONTROL”, which is incorporated herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8643951 | Wheeler | Feb 2014 | B1 |
| 9007301 | Raffle | Apr 2015 | B1 |
| 10069771 | Lee | Sep 2018 | B2 |
| 20120109532 | George | May 2012 | A1 |
| 20150355794 | Spaulding | Dec 2015 | A1 |
| 20160246498 | Rav-Noy | Aug 2016 | A1 |
| 20170357317 | Chaudhri | Dec 2017 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20200117349 A1 | Apr 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15858328 | Dec 2017 | US |
| Child | 16715098 | US |
