Patent Application
20150243019
Embodiments herein relate to system architecture for aligning one or more components of a system.
Modern mechanical systems, for example bicycles, may use multiple gears which may require an indexing shifter. In general, the shifter may require very precise adjustment in order to shift a mechanical linkage such as a bicycle chain to each gear in a rear cassette. Generally, the shifting may be accomplished via a derailleur coupled with the shifter. When the shifter is activated, the derailleur may move slightly and thereby alter the position of the bicycle chain with respect to the cassette. This movement of the derailleur and bicycle chain may cause the bicycle chain to move to a different gear on the cassette.
Calibrating and adjusting the mechanical systems such as the shifter and derailleur may be required for a variety of reasons such as component wear, damage to the bicycle frame, shifter, or derailleur, or changing to a new cassette and/or wheel. However, in many cases, precisely calibrating and adjusting a shifter and derailleur may require extensive trial and error by a person with little calibration experience, or may require paying a person with more calibration experience to adjust the components. In either case, extensive time and/or financial resources may be spent in precisely adjusting the derailleur.
In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the spirit or scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of embodiments is defined by the appended claims and their equivalents.
Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding embodiments; however, the order of description should not be construed to imply that these operations are order dependent.
The description may use perspective-based descriptions such as up/down, back/front, and top/bottom. Such descriptions are merely used to facilitate the discussion and are not intended to restrict the application of disclosed embodiments.
The terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, “connected” may be used to indicate that two or more elements are in direct physical with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements are not in direct contact with each other, but yet still cooperate or interact with each other.
For the purposes of the description, a phrase in the form “NB” or in the form “A and/or B” means (A), (B), or (A and B). For the purposes of the description, a phrase in the form “at least one of A, B, and C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). For the purposes of the description, a phrase in the form “(A)B” means (B) or (AB) that is, A is an optional element.
The description may use the terms “embodiment” or “embodiments,” which may each refer to one or more of the same or different embodiments. Furthermore, the terms “comprising,” “including,” “having,” and the like, as used with respect to embodiments, are synonymous.
The tuning system described herein is built on a system architecture that may be readily applied to computer guidance applications in other industries. As such, it is described herein through the use of general terms, but also with reference to an embodiment specific to tuning a bicycle derailleur. The system may guide a user through a complex or adaptive procedure which may be driven mainly by computer vision input. In the tuning system, guidance may be given to the user in the form of spoken prompts as well as text and graphic instructions displayed on a computer screen, however, guidance may additionally or alternatively be given in any number of forms, including other visual, auditory, or haptic forms of feedback.
Although computer vision is the main form of input to the system, the tuning system may also solicit input from any number of computer sensors, including touch screen, auditory or voice recognition, position or orientation sensors (including but not limited to GPS, accelerometer, gyroscope, near field communications, or custom sensing accessories). Input to the system may also potentially be in the form of information sent to a computer device across a computer network.
Images such as raster images may be captured from one or more camera devices at 100. Images may be captured from a camera such as a camera coupled with or integrated into a computer, a personal digital assistant (PDA), a mobile phone, a smartphone, or some other type of camera. In the embodiment of a smartphone, for example, images may be captured from the smartphone front facing camera. Image capture may occur asynchronously, that is at a progression of different times, and each image may be processed through the stages of the process of
In order to perform the target detection 105, a plurality of physical targets may be attached to a mechanical system such as a bicycle in order to measure the position and orientation of different parts of the bicycle.
In embodiments, one or more targets may be attached to the frame 205, the cassette 210, and/or the derailleur 220. For example, a frame target 240 may be coupled with the frame 205, for example near the skewer that couples the cassette 210 and wheel 215 to the frame 205 as shown, though the frame target 240 may be coupled with the frame 205 at other locations in other embodiments. A cassette target 235 may be coupled with the cassette 210. A jockey target 250 may be coupled with the derailleur 220 at a position near the jockey pulley 225 or some other position. A derailleur target 245 may be coupled with the derailleur 220, for example near the idler pulley 230 or at some other position. In some embodiments, a calibration target (not shown) may be used. The calibration target may not attach to the system 200, but instead may be used for calibration of the intrinsic parameters of the camera taking the digital images. Additional or alternative targets may be used in other embodiments. For example, a plurality of frame, derailleur, jockey, and/or cassette targets may be used in other embodiments.
In general, a minimum of three fiducial markers may be used to locate the target's position and orientation (pose). However, in some embodiments targets such as the derailleur target 245 may contain additional fiducial markers such as fiducial marker 255d so that the target may be located from a wider range of viewing angles. In some embodiments, these additional fiducial markers may increase the accuracy of pose determination from single or multiple images.
The target detection process may be broken down into stages. Initially, fiducial markers such as fiducial markers 255a-d may be located in an image captured during the image capture 100. In one embodiment, the fiducial markers may be utilized to precisely locate 2-D points in the image. The fiducial markers may be detected in monochrome images, so a preliminary stage to convert a color image into monochrome intensity may occur prior to target detection 105. By averaging points on the perimeter of the outer circle of the fiducial markers, a very accurate (sub-pixel) 2-D coordinate of each fiducial marker may be calculated.
After the target detection 105, a full 3-D pose (position and orientation) relative to the camera may then be estimated at 110 for one or more of the targets in an image such as the cassette target 235, frame target 240, derailleur target 245, and jockey target 250. Pose may be calculated via a number of methods. For instance, software packages such as OpenCV may contain a number of pose solving algorithms (CV_ITERATIVE, CV_P3P, and CV_EPNP). A 3-D pose solver may be used to solve for pose with a closed form solution.
Using a camera calibration procedure, a number of intrinsic lens parameters may have been previously calculated such that the projection from 3-D space to a 2-D image coordinate may be precisely modeled. In one embodiment, an iterative solver may be used to minimize the reprojection error, which may be measured for example as root mean square (rms) error, to calculate the pose of a target. In some embodiments, these camera or lens parameters may be based on one or more targets that have, respectively, four or more fiducial markers.
Generally, a pose may consist of six degrees of freedom, for example X,Y,Z translation and X,Y,Z rotation. An initial pose for a target may be an estimate or simply be an arbitrary starting point. The initial pose for the target may be supplied to a minimization algorithm, and the minimization algorithm may adjust the six degrees of freedom of the pose to find the extrinsic parameters that yield a relatively low or lowest reprojection residual. A pose calculation may be potentially ambiguous. Specifically, there may be two minima of lowest reprojection error, one where the target is facing the camera and one where the target is facing away from the camera. Because the target may be known to be facing the camera, if the solver returns a pose facing away from the camera then the pose is rotated 180 degrees and then the iterative solver may then minimize from the new starting pose.
Once target poses have been identified, system state may be inferred at 115. For example, in the embodiment of a bicycle tuning system, the system state may be described through many measurable properties, including:
In order to infer these states, a number of calculations may be performed. Several of these measurable properties are described below. In some embodiments, these inferences may be based only on two targets. For example, the system state inference may be based on the cassette target 235, and the jockey target 250. In other embodiments, these inferences may be based on additional targets.
To determine which gear the bicycle is in, the alignment between the jockey pulley 225 and the different gears of the cassette 210 may be examined. However, the cassette target 235 and jockey target 250, which may respectively directly measure the position of the cassette 210 and jockey 225 pulley, may not be able to be attached to the bicycle while the bicycle is shifting because they may interfere with the shifting operation. In order to overcome this limitation, a calibration operation may be first performed on the bicycle to determine the spatial relationship (3D transform) between the cassette target 235 and the frame target 240 (‘C-F’ transform) and the jockey target 250 and the derailleur target 245 (‘J-D’ transform). This calibration may need to be performed only once so long as the mounts for the derailleur target 245 and frame 240 target are not moved.
Specifically, targets such as the cassette target 235, frame target 240, derailleur target 245, and jockey target 250 may be comprised of a rigid body containing multiple fiducial point markers such as fiducial markers 255a-d. Each marker may be assigned an X,Y,Z coordinate relative to the target's coordinate system. An interpose solver may be used to identify a spatial relationship (3-D “interpose” transform or full attitude pose) between two targets, or between multiple targets, based on observations gathered from multiple photographs of the set of targets.
This interpose solver may have many potential uses. For example a very accurate transform describing the spatial relationship between targets may be obtained by optimizing the pose to best match the results observed in multiple photographs taken from different viewing angles.
There may also be benefits from using multiple images captured from the same viewing angle, e.g. from the same camera in the same position. Specifically, given the inherent noise and inaccuracies of 2-D point location, a more accurate interpose transform may be obtained by calculating this relative transform as a best fit of a set of images. To use a further example that may be specific to the bicycle derailleur tuning process, it may be useful to find the average transform between the cassette target 235 and the frame target 240. Given that cassettes such as cassette 210 may not be perfectly concentric on the wheel hub and may exhibit a wobble when the hub they are mounted to rotates, a phenomenon that may be called the wheel 215 being “out of true,” it may be desirable to find a best-fit transform that averages the wobble. This may be accomplished by having the user spin the bicycle's rear wheel 215 while capturing a set of images of the cassette target 235 and the frame target 240.
In order to find the most accurate interpose transform, an iterative approach may be used that takes into account all images of the targets. In order to illustrate the algorithm, assume that there are two targets, for example, ‘A’ and ‘B’, and it is desired to find an accurate interpose between the two targets. It may be known that the spatial relationship between ‘A’ and ‘B’ is consistent in all images, therefore the problem may be restated as that of finding the pose of target ‘A’ for each image (“the ‘A’ pose), plus an ‘A’-to-′B′ pose that is shared between all images. This ‘A’-to-‘B’ pose may simply be the pose of target ‘B’ (“the ‘B’ pose”) relative to the pose of ‘A’. In embodiments, each pose may be represented by a 4×4 matrix, translation vector and Euler rotation angles, or a translation vector and quaternion.
It may be assumed that each pose is rigid and consists of only six degrees of freedom as described above. Therefore, if there are 10 images showing ‘A’ and ‘B’ targets, then there may be a total of 66 degrees of freedom to optimize—6×10 degrees of freedom for ‘A’, and an additional 6 degrees of freedom for the ‘A’-to-′B′ pose that is common to all images. This ‘A’-to-‘B’ pose transform may be the desirable interpose result. In embodiments the additional degrees of freedom, e.g. the 60 degrees of freedom of ‘A’ may be useful to facilitate solution of the problem.
Initially, approximations may be made for all degrees of freedom. The poses for ‘A’ in each image may be calculated through pose estimation techniques discussed earlier in this text. The initial guess at the ‘A’-to-′B′ pose transform may be estimated by solving for the ‘B’ pose and then multiplying the 4×4 matrix of ‘B’ pose by the inverse of the 4×4 matrix for ‘A’ pose in any single image.
The interpose solver may then minimize the total reprojection error observed across all images. Reprojection error may be calculated by transforming all fiducial points on target ‘A’ into 3-D coordinates relative to the camera for each image. Next, the 3-D coordinates may be projected (using intrinsic lens parameters) into image coordinates. Each projected image coordinate may be calculated and the observed location of the correlated fiducial marker in the image may be identified. A reprojection error may then be calculated by summing the squares of the distances between the projected and observed locations. The same procedure may be performed on the ‘B’ target. The ‘B’ fiducial points may then be transformed into 3-D camera coordinates by using the concatenation of the ‘A’ and ‘A’-to-′B′ pose transforms. These calculated camera coordinates may then be projected by the intrinsic lens parameters into image coordinates, and the squared distance between the projected image coordinates and observed image coordinates are added to the running total of reprojection error.
A nonlinear optimizer may then be used to adjust all degrees of freedom in order to minimize this reprojection error. Any number of solver algorithms may be used for this task, for example, the Nelder-Mead algorithm (downhill simplex method) the Levenberg-Marquardt algorithm, or other algorithms may be applied.
In embodiments, the poses between any number of rigid targets may be found with the interpose process described above. For example, if an additional ‘C’ target was used, an additional six degrees of freedom for the ‘A’-to-′C′ transform may be used. If there were 10 images containing targets ‘A’, ‘B’ and ‘C’, then 72 degrees of freedom (6×10 for ‘A’, 6 for ‘A’-to-′B′, and 6 for ‘A’-to-′C′) may be optimized. Reprojection error may then be the total squared distance between observed image coordinates of fiducial markers and the projected image coordinates of all three targets in one or more, or all, of the 10 images.
In some embodiments, it may not be necessary that all targets (or fiducial markers) be present in all images. In these embodiments, the total reprojection error may be constituted only from the targets (and fiducial markers) that are present in the captured images.
Once calibrated, the cassette target 235 and the jockey target 250 may not be necessary to determine the positions of the cassette 210 and jockey pulley 225, respectively. The derailleur target 245 and frame target 240 may be located, and then the poses of these targets may be concatenated with the pre-calculated ‘C-F’ and ‘J-D’ poses described above in order to determine the poses of the largest sprocket of the cassette 210 (e.g. the lowest gear or first gear) and the jockey pulley 225 relative to the camera.
The position where the chain 207 detaches from the jockey pulley 225, referred to as the “jockey-point”, may then be determined, and the normal distance between that point and the plane of the largest sprocket of the cassette 210, referred to as the “jockey-distance,” may be determined. If the jockey-distance is zero or approximately zero, the jockey pulley may be directly aligned with the largest sprocket of the cassette 210. In embodiments, the spacing between sprockets of the cassette 210 may be known, for example it may be 0.1555 inches in common cassettes, and there it may be possible to find the current bicycle gear by determining which sprocket of the cassette 210 the jockey-point is closest to. The value of 0.1555 inches is merely one example and in other embodiments the spacing between sprockets of the cassette 210 may be larger or smaller.
The derailleur target 245 may also indicate the angle at which the derailleur cage 232 is hanging. Given that it may be known what gear the rear derailleur 220 is in, this angle of the derailleur cage 232 may indicate the status of the front derailleur (not shown) of the bicycle. For example, when the front derailleur is on the large sprocket of the front cassette (not shown), typically referred to as being in a high gear, the 207 may pull the derailleur cage to a more forward position than when the front derailleur is on a smaller sprocket, typically referred to as being in a low gear. Through a calibration procedure, these angles may be measured and then later the front derailleur gear may readily be recognized.
System state may also include prior settings of any measurements so that additional information may be extracted. For example, by seeing that the jockey-distance is consistent over a number of captured images, it may be reasonable to infer that the user has stopped adjusting the bicycle gearshift. By seeing that the gear number has increased, it may be reasonable to infer that the user has shifted up.
Based on the changes in system state, and the related inferences made at 115, one or more events may be generated at 120. Specifically, the changes in system state may be described by events such as:
The events discussed above are merely examples and additional or alternative events may be used. For example, additional events may describe what information is available from the current state:
By examining the current system state and noting changes from a previous system state such as the last system state, a list of events may be generated at 120. These events may then be passed to the current controller which may choose to respond or not respond to any event type.
The generated events may then be processed at 125. Specifically, an application may be described by a state map that organizes the stages necessary to guide the user through the tuning process. Each stage of the state map may have a specific goal, for example, getting the user to shift to a specific gear, recording a measurement, or asking the user to turn an adjusting screw. An example of a state map is depicted in
The state map may contain ‘nodes’, which may be designated by rectangular boxes in
Some examples of classes may include:
Nodes may also contain configuration information that configure the class. For example, the ShiftToGear class may be configured with the desired gear, the direction the user must shift into the gear, and how the instructions will be given. These configuration properties may be contained within a dictionary that may be passed into the class on its construction.
In embodiments, nodes may also contain transitions to other nodes. For example, in
As described above, there are several variations of processes that may be used in different embodiments. Additionally, the processes discussed above are described with reference to a bicycle, but may be easily applied to other mechanical systems used in industries such as:
In some embodiments, the processes described above may be well suited to industrial automation or market areas where orientation and/or location of bodies may be desired. For example, with regard to industrial automation, the above described processes may be useful when performed with respect to:
In some embodiments, the processes described above may be well suited to the following market and/or product areas:
In response, the user 405 may make an adjustment 420 to the mechanical system 410 in accordance with the instructions. For example, the user may shift the directed gear or adjust the specified screw or knob.
The mechanical system 410 may be changed, and so the adjustment system 400 may be able to identify an inferred state 425 of the system as described above at
It will be recognized that the above description of the interaction flow is merely one example. Specifically, in this embodiment the interaction flow is based on the premise that the adjustment system 400 may not directly identify an action of the user 405, but rather is reliant upon observing a change in the mechanical system 410 which can be used to identify an inferred state 425 and generate new instructions 415 based on that inferred state 425. In other embodiments, the adjustment system 400 may be able to directly identify an action or movement of the user 405, or the interaction flow may include additional or alternative elements.
Embodiments of the present disclosure may be implemented into a system using any suitable hardware and/or software to configure as desired.
In some embodiments, the system 500 may include one or more computer-readable media (e.g., system memory or NVM/storage 520) having instructions and one or more processors (e.g., processor(s) 505) coupled with the one or more computer-readable media and configured to execute the instructions to implement a module to perform actions described herein.
System control module 510 for one embodiment may include any suitable interface controllers to provide for any suitable interface to at least one of the processor(s) 505 and/or to any suitable device or component in communication with system control module 510.
System control module 510 may include memory controller module 530 to provide an interface to system memory 515. The memory controller module 530 may be a hardware module, a software module, and/or a firmware module.
System memory 515 may be used to load and store data and/or instructions, for example, for system 500. System memory 515 for one embodiment may include any suitable volatile memory, such as suitable DRAM, for example. In some embodiments, the system memory 515 may include double data rate type four synchronous dynamic random-access memory (DDR4 SDRAM).
System control module 510 for one embodiment may include one or more input/output (I/O) controller(s) to provide an interface to NVM/storage 520 and communications interface(s) 525.
The NVM/storage 520 may be used to store data and/or instructions, for example. NVM/storage 520 may include any suitable non-volatile memory, such as flash memory, for example, and/or may include any suitable non-volatile storage device(s), such as one or more hard disk drive(s) (HDD(s)), one or more compact disc (CD) drive(s), and/or one or more digital versatile disc (DVD) drive(s), for example. In some embodiments,
The NVM/storage 520 may include a storage resource physically part of a device on which the system 500 may be installed or it may be accessible by, but not necessarily a part of, the device. For example, the NVM/storage 520 may be accessed over a network via the communications interface(s) 525.
Communications interface(s) 525 may provide an interface for system 500 to communicate with a user, for example by providing visual cues via the display 540 and/or audio cues via the speaker 545. In some embodiments, the communications interface(s) 525 may include the input device 550 to receive commands from the user, for example in response to prompts provided by the display 540 and/or speaker 545.
For one embodiment, at least one of the processor(s) 505 may be packaged together with logic for one or more controller(s) of system control module 510, e.g., memory controller module 530. For one embodiment, at least one of the processor(s) 505 may be packaged together with logic for one or more controllers of system control module 510 to form a System in Package (SiP). For one embodiment, at least one of the processor(s) 505 may be integrated on the same die with logic for one or more controller(s) of system control module 510. For one embodiment, at least one of the processor(s) 505 may be integrated on the same die with logic for one or more controller(s) of system control module 510 to form a System on Chip (SoC).
In some embodiments the processor(s) 505 may include or otherwise be coupled with one or more of a graphics processor (GPU) (not shown), a digital signal processor (DSP) (not shown), wireless modem (not shown), multimedia circuitry (not shown), sensor circuitry (not shown), and/or global positioning satellite (GPS) circuitry (not shown).
In various embodiments, the system 500 may be or include, but is not limited to, a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, a smartphone, a gaming console, etc.). In various embodiments, the system 500 may have more or fewer components, and/or different architectures. For example, in some embodiments, the system 500 may include one or more of a non-volatile memory port, multiple antennas, graphics chip, application-specific integrated circuit (ASIC), and speakers.
This application claims priority to U.S. Provisional Patent Application No. 61/944,705, filed Feb. 26, 2014 and titled “ALIGNMENT SYSTEM ARCHITECTURE,” the entire contents of which are incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 61944705 | Feb 2014 | US |
