Patent Application
20200402285
Digital cameras may be used to capture digital videos in a variety of settings. In view of the availability of expanded digital memory, users tend to capture more videos, and increasingly longer videos, as memory consumption becomes less of a concern. Increasing number and length of videos, however, results in inefficiencies in management of video libraries.
Manual review of videos to identify scenes the user finds exciting, entertaining, or otherwise interesting for view may be both time consuming, and resource intensive. For example, scrolling through multiple videos of various lengths to find such scenes burdens processors, memory, and battery of computing devices, such as mobile computing devices. Automated video processing technologies have been developed to identify scenes potentially interesting to the user. However, such video processing technologies may be resource-intensive, particularly with high-resolution, raw-format video data.
Further, after identifying an interesting scene, the user may wish to edit the scene using a digital editing environment. For example, the user may edit the scene to include effects (e.g., slow motion, zoom-in, zoom-out, color effects, and the like), which may enhance the scene, or otherwise affect user enjoyment of the scene. However, the user may undergo a trial-and-error process of adding/removing effects in different combinations, and/or with different effect attributes. Such editing processes may be time and resource intensive, thereby burdening processors, memory, and battery of computing devices, such as mobile computing devices.
This specification generally relates to digital media editing. More particularly, implementations of the present disclosure are directed to providing suggested edits to one or more moments captured in a digital video.
In some implementations, actions include receiving metadata associated with the digital video, the metadata including data representative of one or more of movement and an environment associated with recording of the digital video, processing the metadata to provide a suggested edit profile including at least one set of effects, the at least one set of effects including one or more effects configured to be applied to at least a portion of the digital video, providing a respective graphical representation of individual effect of the one or more effects within an effect interface, and receiving, through the effect interface, a user selection of a set of effects of the suggested edit profile, and in response, storing, in computer-readable memory, an edit profile comprising the set of effects for application to the digital video. Other embodiments of these aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.
These and other embodiments may each optionally include one or more of the following features: a processing the metadata to provide a suggested edit profile includes determining that at least one attribute value breaches a threshold, and in response, providing the at least one set of effects; processing the metadata to provide a suggested edit profile includes determining a type of effect to include in the at least one set of effects based on the metadata; determining a type of effect includes comparing at least one metadata value to a threshold value, and one of: including a first effect in the at least one set of effects if the metadata value breaches the threshold value, and including a second effect in the at least one set of effects if the metadata value does not breach the threshold value; each effect is associated with one or more attributes; actions further include receiving user input indicating a modification to at least one attribute; an attribute includes one of a degree of a respective effect, a beginning of a respective effect, an end of a respective effect, and a duration of a respective effect; the at least one set of effects is associated with an event that is identified as an event of interest based on the metadata; the at least one set of effects is associated with an event that is not identified as an event of interest based on the metadata; an effect includes slow motion, fast forward, zoom-in, zoom-out, a color effect, a sound effect, insertion of a music track, a volume effect, a pixilation effect; and the metadata is provided from one or more of a mobile processing device, a digital media device, and an ancillary device.
The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
Like reference numbers represent corresponding parts throughout.
Implementations of the present disclosure are generally directed to video editing applications. More particularly, implementations of the present disclosure are directed to providing one or more suggested edits to a digital video. In some examples, one or more suggested edits are provided for respective moments captured in the digital video. In some examples, a moment includes a scene, or multiple scenes recorded in the digital video. In some examples, each moment includes a plurality of frames of the digital video. In some implementations, a suggested edit may include one or more effects that may be applied to a respective moment, and are provided based on meta-data corresponding to the digital video. In some examples, a user may select the suggested edit to be applied to the moment. In some examples, the user may modify the suggested edit to provide a modified suggested edit, which may be applied to the moment.
As described in further detail herein, implementations of the present disclosure may include actions of receiving metadata associated with the digital video, the metadata including data representative of one or more of movement and an environment associated with recording of the digital video, processing the metadata to provide a suggested edit profile including at least one set of effects, the at least one set of effects including one or more effects configured to be applied to at least a portion of the digital video, providing a respective graphical representation of individual effect of the one or more effects within an effect interface, and receiving, through the effect interface, a user selection of a set of effects of the suggested edit profile, and in response, storing, in computer-readable memory, an edit profile comprising the set of effects for application to the digital video.
In some examples, the processor 102 may include a display manager 128 and an input manager 130. The display manager 128 may be operable to manage displayed on the display 112. In some implementations, an operating system running on the mobile device 102 may use the display manager 128 to arbitrate access to the display 112 for a number of applications 124 running on the mobile processing device 100. In some examples, a number of applications 124 may run concurrently on the mobile processing device 100, with each application displayed to the user in respective windows on the display 112. In such examples, the display manager 128 may control which portions of each application 124 are displayed on the display 112. In some implementations, the display manager 128 may control the orientation (e.g., portrait orientation, landscape orientation) of the display 112. In some examples, the input manager 130 may be operable control the handling of data received from a user via one or more input devices of the mobile processing device 100. For example, the input manager 130 may determine which application 124 should be invoked as a result of user received input. In some implementations, the UI module 114 may coordinate with the input devices to a interpret user input. The interpreted input may be evaluated by the input manager 130 to determine which application 124 should be invoked.
In some implementations, the sensors 120 may be responsive to physical movement and/or the physical environment of the mobile processing device 100, and generate data based thereon. Example sensors may include, without limitation, an inertial sensor, an accelerometer, a gyroscope, a compass, a barometer, and a temperature sensor. For example, responsive to movement of the mobile processing device 100, a sensor (e.g., an accelerometer) may provide data indicating attributes of the movement (e.g., force, speed, direction). As another example, responsive to a change in environment of the mobile processing device 100, a sensor (e.g., temperature sensor) may provide data indicating attributes of the change (e.g., increase/decrease in temperature). In accordance with implementations of the present disclosure, and as described in further detail herein, data profiles (e.g., a series of data values) from one or more sensors may be used to provide one or more suggested edits to one or more moments captured in a digital video.
In some implementations, the mobile processing device 100 may provide a digital media editing environment. For example, a digital media editing application may be downloaded to and installed on the mobile processing device 100, which may execute the digital media editing application to provide the digital media editing environment on the display 112 of the mobile processing device 100. Example digital media editing environments are disclosed in commonly assigned U.S. application Ser. No. 14/234,653, filed on Jan. 24, 2014, which issued as U.S. Pat. No. 9,557,885 on Jan. 31, 2017, and which claims priority to U.S. Prov. App. No. 61/521,624, filed on Aug. 9, 2011, and U.S. application Ser. No. 15/400, 660, filed Jan. 6, 2017, and which claims priority to U.S. Prov. App. No. 62/276,627, filed Jan. 8, 2016, and U.S. Prov. App. No. 62/291,336, filed Feb. 4, 2016. The disclosures of each of the above-referenced applications are expressly incorporated herein by reference in their entireties.
In some examples, the media device 204 includes any appropriate device for generating and storing, among other functionality, digital media objects (e.g., digital videos). An example media device may include a digital camera that may be used to capture digital images, video, and/or audio (as digital media objects). Example media devices include the GoPro Hero® series of digital video cameras provided by GoPro, Inc. In some examples, the media device 204 stores digital media objects in computer-readable memory. Example computer-readable memory may include removable memory, such as a non-volatile memory card (e.g., a secure digital (SD) storage care developed by the SD Card Association (SDA) for use in portable devices). In some examples, the media device 204 includes one or more components, such as components described above with reference to the mobile processing device 100 (e.g., one or more processors, memory, a transmitter, a receiver, a display, speakers, a microphone, sensors, a battery). In some examples, the media device 204 may provide a digital media editing environment. For example, a digital media editing application may be downloaded to and installed on the media device 204, which may execute the digital media editing application to provide the digital media editing environment.
In some examples, the server systems 206 includes one or more computing devices, and computer-readable memory, and may represent various forms of server systems including, but not limited to a web server, an application server, a proxy server, a network server, or a server farm.
In some examples, the mobile processing device 100, the media device 204, and/or the server system 206 may store one or more digital media units (also referred to herein as digital media objects, or objects) (e.g., a digital video file, a digital image file, a digital audio file). For example, the server system 206 may host one or more computer-implemented services, which may be used (e.g., by a user of the mobile processing device 100) to, for example, store, share, and/or edit digital media objects. Example computer-implemented services may include a computer-implemented social networking service, a computer-implemented micro-blogging service, and a computer-implemented content-sharing service. An example social networking service may include Facebook® provided by Facebook, Inc. An example micro-blogging service may include Twitter® provided by Twitter Inc. Example content-sharing services include image-sharing services and/or video-sharing services. An example content-sharing service includes Instagram® provided by Facebook, Inc. It is contemplated, however, that implementations of the present disclosure may be realized in any appropriate computer-implemented services (e.g., multiple social networking services, multiple micro-blogging services, multiple content-sharing services, and/or any other appropriate computer-implemented service).
In some examples, the mobile processing device 100 communicates with the media device 204, and/or the server system 206 over the network 202 to access objects stored thereby. In some examples, the network 202 may be a public communication network (e.g., the Internet, cellular data network, dialup modems over a telephone network), a wide area network (WAN), a local area network (LAN), a private communications network (e.g., private LAN, leased lines), a wireless network, a wireless connection between devices, or any appropriate combination thereof. In one example, the mobile processing device 100 may communicate with the media device 204 over the network 202 (e.g., as a wireless connection (WiFi, Bluetooth)) to access objects stored by the media device 204. In another example, the mobile processing device 100 may communicate with the server system 206 over the network 202 (e.g., the Internet) to access objects stored by the server system 206.
In the depicted example, the ancillary device 208 is provided as a wearable device (e.g., a fitness tracker, a smartwatch). In some implementations, the ancillary device 208 includes one or more components, such as components described above with reference to the mobile processing device 100 (e.g., one or more processors, memory, a transmitter, a receiver, a display, speakers, a microphone, sensors, a battery). Although depicted as a wearable device, the ancillary device 208 may include any appropriate device that is responsive to movement and/or environment, and that may provide respective data based thereon, as well as able to communicate with other devices over the network 202 (e.g., the Internet, wireless connection (WiFi, Bluetooth)). For example, the ancillary device 208 may be representative of a vehicle telemetry device, which provides vehicle telemetry data (e.g., altitude, speed, pitch, roll, acceleration, yaw) of a vehicle.
In accordance with implementations of the present disclosure, a digital video may be recorded by the media device 204 (e.g., digital video camera), and metadata that describes movement and/or environment associated with recordation of the digital video may be recorded. In some examples, the metadata may be recorded by one or more of the mobile processing device 100, the media device 204, and the ancillary device 208. For example, a user may include a skier, who uses the media device 204 to record digital video (e.g., mounted on a helmet of the skier), while the mobile processing device 100 is in a pocket, and the user is wearing the ancillary device 208. In some examples, in response to beginning recording of the digital video, a signal may be sent to each of the mobile processing device 100, and the ancillary device 208 (e.g., from the media device 204), instructing each to begin recording of metadata. In some examples, in response to ending recording of the digital video, a signal may be sent to each of the mobile processing device 100, and the ancillary device 208 (e.g., from the media device 204), instructing each to end recording of metadata. In some examples, the mobile processing device 100, and/or the ancillary device 208 each regularly record metadata, and cache metadata (e.g., regardless of activities of the media device 204). In some examples, in response to ending recording of the digital video, a request is sent to the mobile processing device 100, and/or the ancillary device 208 for historical metadata (e.g., going back to the time, at which recording of the digital video began). In some implementations, recorded metadata (e.g., recorded by the media device 204, the mobile processing device 100, and/or the ancillary device 208) is associated with the digital video (e.g., is appended to a file containing the digital video).
In accordance with implementations of the present disclosure, and as introduced above, the metadata includes data indicative of movement and/or an environment associated with the digital video. In some examples, metadata may also include information about the video digital itself, the device used to capture the video, the environment or setting in which a video is captured, or any other information associated with the capture of the digital video. Example metadata may include, without limitation, motion data, speed data, acceleration data, altitude data, GPS data, compass data, temperature data, and barometric data. For example, and continuing with the example of the user being a skier, metadata may include acceleration data representative of the acceleration of the media device 204 carried by the user as a digital video is recorded while skiing, as well as accelerations of the mobile processing device 100 and the ancillary device 208, compass data of the mobile processing device 100, a heart rate of the user, and any appropriate metadata captured during recording of the digital video.
In some implementations, the metadata may be processed to identify potential events of interest within the digital video (e.g., to identify moments within the digital video). Systems and methods for identifying interesting events in digital videos based on metadata are described in commonly assigned, U.S. application Ser. No. 14/513,149, filed on Oct. 13, 2014, the disclosure of which is expressly incorporated herein by reference in the entirety.
In accordance with implementations of the present disclosure, metadata may be processed to provide one or more suggested edits to a digital video. In some examples, one or more suggested edits are provided for respective moments captured in the digital video. In some examples, a moment includes a scene, or multiple scenes recorded in the digital video. In some implementations, a suggested edit may include one or more effects that may be applied to one or more frames of the digital video. Example effects may include, without limitation, slow motion, zoom-in, zoom-out, color effects, sound effects, music tracks, volume, pixilation, and the like.
In some implementations, a video editing module (e.g., of a video editing application executing on one of the mobile processing device 100, the media device 204, the server system 206) analyzes metadata associated with a digital video to identify scenes of interest (e.g., which include identified events of interest, and/or activities). Examples of events of interest may be indicated by abrupt changes or anomalies in the metadata, such as a peak or valley in metadata maximum or minimum values within the metadata, metadata exceeding or falling below particular thresholds, metadata within a threshold of predetermine values (for instance, within 20 meters of a particular location or within the particular location), and the like. The video editing module may identify events of interest based on any other type of metadata, such as a heart rate of a user, orientation information, and the like.
In some examples, the video editing module may identify any of the following as an event of interest within the metadata: a greater than threshold change in acceleration or velocity within a pre-determined period of time, a maximum or above-threshold velocity or acceleration, a maximum or local maximum altitude, a maximum or above-threshold heart rate or breathing rate of a user, a maximum or above-threshold audio magnitude, a user location within a pre-determined threshold distance from a pre-determined location, a threshold change in or pre-determined orientation of the camera or user, a proximity to another user or location, a time within a threshold of a pre-determined time, a pre-determined environmental condition (such as a particular weather event, a particular temperature, a sporting event, a human gathering, or any other suitable event), or any other event associated with particular metadata.
In some implementations, a user may manually indicate an event of interest during capture of the video, and the metadata may reflect such manual indication(s). For example, a user may press a button on the media device 204 (or a camera remote), or otherwise interact with the media device 204 during the capture of video to tag an instant in the video as including an event of interest. For example, if a user is capturing video while skiing, and presses a button of the media device 204 associated with manually tagging an event of interest, the media device 204 creates metadata associated with the captured video indicating that the video includes an event of interest, and indicating a time or portion within the captured video, at which the tagged event of interest occurs. In some examples, the manual tagging of an event of interest by a user while capturing video is stored as a flag within a resulting digital video file. The location of the flag within the digital video file corresponds to a time within the video, at which the user manually tags the event of interest.
In some implementations, a user may manually indicate an event of interest during capture of the video using a spoken command or audio signal. For example, a user may say “Tag,” or “Tag my moment,” during the capture of video to tag the video as including an event of interest. The audio-tagged event of interest may be indicated within metadata associated with the captured video. The spoken command may be pre-programmed, for example, by a manufacturer, programmer, or seller of the media device 204, or may be customized by a user of the media device 204. For example, a user may speak a command or other audio signal into the media device 204 during a training period (e.g., in response to configuring the media device 204 into a training mode, or in response to the selection of a button or interface option associated with training the media device 204 to receive a spoken command). An audio pattern is stored in the media device 204, and, during a video capture, the media device 204 may identify the audio pattern in a spoken command or audio signal received from the user, and may manually tag an event of interest during the capture of video in response to detecting the stored audio pattern within the received spoken command or audio signal.
In some implementations, the video editing module may identify events of interest based on audio captured in conjunction with the video. In some examples, the video editing module identifies events of interest based on one or more spoken words or phrases in captured audio. For example, if audio of an excited utterance of a user (e.g., “Holy Smokes!”) is captured, the video editing module may determine that an event of interest occurred contemporaneously with the excited utterance (e.g., within the previous 5 seconds or other threshold of time). As another example, if audio of a user saying an anticipatory excited utterance (e.g., “Oh no! Watch out!”) is captured, the video editing module may determine that an event of interest is about to occur (e.g., within the next 5 seconds, or other threshold of time). In addition to identifying events of interest based on captured dialogue, the video editing module may identify an event of interest based on captured sound effects, captured audio exceeding a magnitude or pitch threshold, or captured audio satisfying any other suitable criteria.
In some implementations, the video editing module may identify video that does not include events of interest. For example, and as describe further herein, the video editing module may identify video that is associated with metadata patterns determined to not be of interest to a user. Such patterns may include metadata associated with a below-threshold movement, a below-threshold luminosity, audio data that does not include dialogue or other notable sound effects, and the like. In some examples, and as described in further detail herein, video determined to not include events of interest may still have suggested edits associated therewith.
As described herein, the video editing module may identify events of interest based on activities and/or environment associated with recording of the video. For example, a jump while skiing, or a crash while skateboarding, may be considered events of interest. In some implementations, activities may be identified by an activity identifier module (e.g., of a video editing application executing on one of the mobile processing device 100, the media device 204, the server system 206) based on metadata associated with the video. Continuing with the previous example, metadata associated with a particular altitude and a parabolic upward and then downward velocity may be identified as a “jump” activity, and a sudden slowdown in velocity and accompanying negative acceleration (deceleration) may be identified as a “crash” activity.
In some implementations, the activity identifier module may receive a manual identification of an activity within videos from one or more users. In some examples, activities may be tagged during the capture of video. For example, if a user is about to capture video while performing a jump, the user may manually tag the video being captured or about to be captured as “jump.”
In accordance with implementations of the present disclosure, each effect, or a combination of effects that may be applied to the digital video may be suggested based on the metadata. For example, values of the metadata may be compared to one or more threshold values, and, if the values meet the thresholds, a corresponding effect, or multiple effects may be suggested (e.g., as described above with respect to identifying events of interest). For example, metadata may include acceleration values, which may be compared to one or more thresholds. In some examples, an acceleration value meeting a first threshold may be indicative of a beginning of an event (e.g., rapid acceleration/deceleration of the skier, a maneuver (jump, flip) performed by the skier). In response to the acceleration value exceeding the first threshold, a suggested effect (e.g., slow motion) may be provided. In some examples, the suggested effect may be associated with a corresponding timestamp of the digital video, and/or a particular frame (e.g., based on unique frame identifier) of the digital video.
In some implementations, one or more attributes may be provided for each suggested effect. Example attributes may include, without limitation, degree, and duration. For example, if the suggested effect includes slow motion, the degree of slow motion (e.g., three-quarter speed, half speed, quarter speed) may be provided based on the extent, to which the acceleration value exceeds the first threshold. For example, for a first acceleration, a first degree may be provided, and for a second acceleration, a second degree may be provided, where, for example, the first acceleration is greater than the second acceleration, and the first degree (e.g., three-quarter speed) is slower than the second degree (e.g., quarter speed). In other words, for more rapid accelerations, a slower motion effect may be suggested than for less rapid accelerations.
In some examples, an acceleration value meeting a second threshold may be indicative of an end of an event (e.g., rapid acceleration/deceleration of the skier, a maneuver (jump, flip) performed by the skier). In response to the acceleration value meeting the second threshold, an end to the suggested effect (e.g., slow motion) may be provided. In some examples, the end of the suggested effect may be associated with a corresponding timestamp of the digital video, and/or a particular frame (e.g., based on unique frame identifier) of the digital video. In some examples, the suggested effect may be associated with a duration (e.g., in seconds, minutes).
In some implementations, metadata may be indicative of an uninteresting event. For example, a period of time in the digital video, during which nothing of significant consequence occurs (e.g., a skier slowly skiing along a cat-track for a matter of minutes), may be determined to be an uninteresting event. Metadata (e.g., accelerometer data, GPS data, compass data) may be indicative of such an event (e.g., relatively constant, relatively slow speed, in a relatively constant direction). Consequently, a corresponding effect may be suggested (e.g., cropping of the digital video during the period of the event, fast-forward of the digital video during the period of the event).
In accordance with implementations of the present disclosure, a suggested edit profile may be provided, which includes one or more suggested effects for editing the digital video. In some implementations, for each of one or more events, a set of effects is provided. Continuing with the example above, a digital video may include a skier skiing slowly along a cat-track before rapidly accelerating and performing a jump. Accordingly, an example suggested edit profile may include a first set of effects, and a second set of effects, the first set of effects corresponding to the slow ski along the cat-track, and the second set of effects corresponding to the rapid acceleration and jump.
In some implementations, each set of effects includes one or more suggested effects that may be applied to the digital video, corresponding to a respective event. Continuing with the example above, a first set of effects may be provided for the slow ski along the cat-track, and a second set of effects may be provided for the rapid acceleration and jump. For example, the first set of effects may include cropping of the slow ski along the cat-track. As another example, the first set of effects may include fast forwarding of the slow ski along the cat-track, and a suggested music track (e.g., the theme music to the Benny Hill Show). As another example, the second set of effects may include slow motion of the rapid acceleration and jump, and a reduction in volume of audio.
In some examples, for each suggested effect, one or more attributes (e.g., a degree, and/or duration of the suggested effect) may be provided. For example, the first set of effects may include a degree of fast forwarding (e.g., double speed) for the duration of the slow ski along the cat-track. As another example, the second set of effects may include a degree of slow-motion (e.g., half speed) for the duration of the rapid acceleration and jump.
In some implementations, the suggested edit profile is provided as computer-readable file that stores a data set that includes data sub-sets. For example, the suggested edit profile (SEP) includes one or more sets of effects (SE1, . . . , SEn). In some examples, each set of effects includes one or more effects, each effect defined as a tuple of attributes (e.g., [T1, a1,1, . . . , a1,p]; [T2, a2,1, . . . , a2,q]). For purposes of illustration, an example suggested edit profile may include:
SEP→([T1, a1,1, a1,2A1,3]1; [T2a2,1, a2,2, a2,3]2)
In this example, the SEP includes a first set of effects ([. . . ]1), and a second set of effects ([. . . ]2), where T indicates a type of effect (e.g., slow motion, fast forward), and a indicates respective attributes of the effect (e.g., degree, duration). Continuing with the example above, the first set of effects may include an effect of fast forwarding (e.g., T1=FF) at double speed (e.g., a1,1=2.0) for the duration of the slow ski along the cat-track (e.g., a1,2=tstart, and a1,3=tstop; where t is a timestamp of the video, or a frame identifier). As another example, the second set of effects may include an effect slow-motion (e.g., T2=SM) at half speed (e.g., a2,1=0.5) for the duration of the rapid acceleration and jump (e.g., a2,2=tstart, and a2,3=tstop; where t is a timestamp of the video, or a frame identifier). Although each of the above-described example sets of effects includes a single effect, it is contemplated that a set of effects may include multiple effects (e.g., slow motion, volume reduction, color change to black-white).
In some implementations, a graphical representation of the suggested edit profile may be displayed to the user, and the user may accept one or more of the sets of effects, and/or modify one or more effects in the one or more sets of effects before accepting. For example, each set of effects may be visually displayed to the user in a digital media editing application, to enable the user to accept, delete, and/or modify one or more of the sets of effects in a suggested edit profile.
In some implementations, in response to user selection of a graphical representation, effects and respective attributes of the effects are displayed in the effect interface 306. In the example of
In some implementations, and as noted above, the edits may be provided in an edit profile provided as computer-readable file that is appended to, or otherwise associated with a file containing the digital video (e.g., a digital video file). In some examples, when the video is subsequently played (e.g., by a video player), the edit profile is read as a set of instructions for applying respective effects as the video is played. In this manner, an edited video is not stored as a separate digital video file, from the original video. In another example, an edited digital video is provided based on the edit profile, and is stored as a separate digital video file (e.g., separate from the digital video file of the original video).
Metadata associated with the digital video is received (402). For example, a video editing module (e.g., of a video editing application executing on one of the mobile processing device 100, the media device 204, the server system 206). In some examples, the metadata is received from one or more of the mobile processing device 100, the media device 204, and the ancillary device 208 of
A respective graphical representation of each effect of the one or more effects is displayed within an effect interface (406). For example, and as depicted in the examples of
Although described in the context of a mobile device application operating on a mobile processing device, it is contemplated that implementations of the present disclosure, and all of the functional operations provided herein, may be realized in any suitable type of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the present disclosure may also be realized as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
The processes and logic flows described in this disclosure may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. Elements of a computer may include a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, implementations of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input.
While this disclosure includes some specifics, these should not be construed as limitations on the scope of the disclosure or of what may be claimed, but rather as descriptions of features of example implementations of the disclosure. Certain features that are described in this disclosure in the context of separate implementations may also be provided in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be provided in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.
Thus, particular implementations of the present disclosure have been described. Other implementations are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Accordingly, other implementations are within the scope of the following claims.
| Number | Date | Country | |
|---|---|---|---|
| 62291336 | Feb 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16541671 | Aug 2019 | US |
| Child | 17009333 | US | |
| Parent | 15424281 | Feb 2017 | US |
| Child | 16541671 | US |
