Patent Application
20210368748
This specification generally describes enhanced mechanical fish sorting equipment.
A population of farmed fish may include normal, healthy fish that are fit for human consumption, as well as fish that are not suited for human consumption, such as juvenile fish, unhealthy or diseased fish, fish that are heavily infested with parasites, or runts. In the aquaculture context, when a population of fish are harvested, a worker may visually evaluate each harvested fish based on a quick observation, and may manually separate the healthy fish from the unhealthy fish so that only healthy fish are sent to market.
Oftentimes these quick visual observations can result in inaccurate classifications of fish, which may result in some fish being improperly sent to market. At the same time, when classifying thousands or tens of thousands of fish using manual processes, the sorting process can last a long period of time, during which time harvested fish may be kept in suboptimal or stressed conditions, resulting in additional fish losses.
In some implementations, electronic devices, including computers, imaging devices, and actuating mechanical devices, are used to analyze and sort fish. An imaging device obtains one or more images of a fish that is in water. The one or more images are analyzed by electronic devices such as computers or software algorithms, including machine learning models, to detect the fish. Physical characteristics of the fish are determined and used to compute related values. Physical characteristics can include key points on the body of a fish, the positioning of the fish in three dimensions (3D), truss lengths related to distances between key points on the body of the fish, weight or biomass of the fish, body length among others. The physical characteristics and related values are used to classify and sort the fish into a predefined area. Corresponding data, including data elements corresponding to the fish, is stored in a database and information related to the data element, analysis, or sorting is presented to a user. The presentation of information can be an interactive graphic displaying data related to a population of fish.
Advantageous implementations can include one or more of the following features. For example, a user interface (UI) providing relevant information about a fish population can be generated and sent to a user device. The user interface can be a population graph that uses a physical characteristic and a condition factor as coordinate points for plotting the location of one or more fish within the fish population. In some cases, the physical characteristic can be the weight of the fish and the condition factor can be defined as a function of weight and length of the fish. Subpopulations, based on one or more clusters within the population graph, can then be determined such as runts within a population of fish.
In some implementations, population information is recorded for separate pens of fish. For example, a first pen can be composed of 10 percent outliers. A second pen can be composed of 2 percent outliers. The user can access images of the outliers as well as images of members of the entire population. Statistics on the population's size, weight, geometry, feeding behavior, and movement within the pen can be recorded as data related to the pen, the population, or one or more specific fish associated with the pen or the population. Historical data of a specific fish can also be provided by using fish identification. The historical data may include any information captured during one or more detections of the specific fish including previous locations, physical characteristics, physical changes, growth, or behavior over time among others.
In some implementations, visual data is captured and used to determine characteristics, traits, or other detection data in an automated process. For example, instead of an employee evaluating each fish based on a visual observation performed manually, a computer, or group of computers, can obtain images automatically while a fish is in water. The nature of automatically obtaining images allows this type of system to collect more data within a given period of time. The data can be obtained by using algorithms that can learn and improve. In this way, the data can be more accurate than manual observation approaches. The data can be used to double check previous data or be combined to create larger databases of records or individual data elements or sets related to fish. The larger databases can be used to determine trends, risk factors, or other output associated with the population. This output can be used to automatically manage the population through automated sorting or be sent to a user.
In some implementations, one or more images of a particular fish within a population of fish are obtained. Based on the one or more images of the fish, a data element can be determined that includes a first value that reflects a physical characteristic of the particular fish, and a second value that reflects a condition factor of the particular fish. Based on the data element, the fish can be classified as a member of a particular subpopulation of the population of fish. Based on classifying the particular fish as a member of the particular subpopulation of the population of fish, an actuator of an automated fish sorter can be controlled.
In some implementations, the one or more images of the fish include at least a first image representing a first view of the fish and a second image representing a different, second view of the fish.
In some implementations, at least the first image and the second image are used to determine a three dimensional (3D) pose of the fish.
In some implementations, the condition factor is calculated as a function of one or more weight values associated with the fish and one or more length values associated with the fish.
In some implementations, the data element is determined, in part, by estimating a set of truss lengths corresponding to distances between locations on a body of the fish.
In some implementations, one or more other data elements corresponding to other fish within the population of fish are generated.
In some implementations, classifying the fish as a member of the particular subpopulation includes generating one or more clusters based on the data element of the particular fish and the one or more other data elements corresponding to the other fish within the population of fish.
In some implementations, the actuator controls a flap that selectively opens a particular passage among multiple passages that are associated with the automated fish sorter.
In some implementations, the flap includes holes to reduce force required to actuate the actuator.
In some implementations, a visual representation is generated using the first value of the data element and the second value of the data element as a coordinate pair to represent a data point within a graph associated with the particular fish.
The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will become apparent from the description, the drawings, and the claims.
Like reference numbers and designations in the various drawings indicate like elements.
The system 100 further includes a control unit 110 connected to an imaging device 108 and a sorting actuator 116. The sorting actuator 116 controls access to the second fish pen 118 and the third fish pen 124 from the first fish pen 102. The control unit 110 is used to generate population graphs 112 and 120 that show relevant information of a fish population. The fish population shown in population graphs 112 and 120 include the runt fish 104 and the healthy fish 106 among others.
The population graphs 112 and 120 include weight and K-Factor data for fish within the fish population as well as related information including historical data and images of particular fish. Individual points within the population graphs 112 and 120 correspond to individual fish. Additional information related to a fish can be accessed by clicking on a point within the graphic within a given user interface rendered on a computer or other electronic device.
A flow rate of water is adjusted to allow the runt fish 104 into the channel and to progress forwards through the channel in the direction of the second fish pen 118 and the third fish pen 124. In some implementations, other processes are used to control a fishes' movement from one element to another element. For example, an incentive can be placed in the channel connecting the first fish pen 102 to the second fish pen 118. The incentive can be used to control movement of the fish from the first fish pen 102 to the second fish pen 118. In another example, because fish like to swim into a current, a pump, or other current generating sources, may create a current in the channel to entice the fish to swim into the fish channel.
Stage B shows the imaging device 108 obtaining one or more images of the runt fish 104 when the runt fish 104 is in view of the imaging device 108. A motion sensor is used to detect motion of a fish in the channel connecting the first fish pen 102 to the second fish pen 118 and the third fish pen 124. The motion sensor triggers the capturing of one or more images by the imaging device 108. In this case, the motion sensor detects the motion of the runt fish 104 and the imaging device 108 subsequently obtains one or more images of the runt fish 104. The images are sent to the control unit 110.
Stage C shows the control unit 110 receives the one or more images of the runt fish 104. The detailed process of analysis performed by the control unit 110 is described in
The population graph 112 shows data related to the fish population including the runt fish 104 and the healthy fish 106. Data related to a detection of the runt fish 104 updates a database used to create the population graph 112. The population graph 112 is described in more detail in reference to
Stage D shows an example of a graphical element, the population graph 112, generated by the control unit 110. The x axis of the population graph 112 is weight and the y axis is a K-Factor that is related to physical characteristics of the fish. Points on the graph near the top right represent larger fish while points near the bottom left represent smaller fish. The value of the condition factor may be determined based on any mathematical formula that quantifies the condition of fish. One example of such a condition factor is a K-Factor, which is one typical measure that is used by industry to objectively and quantitatively estimate the condition of an individual fish within a population of fish. While a condition factor alone provides one powerful measure of a fish's health, the condition factor plotted against other values provides an even more powerful tool, e.g., for use in identifying different subpopulations of fish by using clustering techniques. In other implementations, other computed parameters based on determined physical characteristics can be used.
The industry standard condition factor is traditionally computed by physically weighing and measuring the fish, a time-consuming process. One example expression used to compute the condition factor is shown in Equation (1), below:
In Equation (1), k is the condition factor, weight is a weight of a fish e.g., in grams, and length is a length of the fish, e.g., in centimeters, cubed. By automating the process and using machine learning techniques to determine significant parameters, the system 100 shown in
An image 114 of the runt fish 104 shows example statistics and parameters related to the identification of the runt fish 104. The image 114 can be stored within a graphical element such as the population graph 112 and shown to a user after a selection is made on the particular data entry related to the runt fish 104. As shown in the image 114, the runt fish 104 is classified as “Runt” by the control unit 110 based on the population graph 112 and physical characteristics related to the runt fish 104. The K-Factor is calculated as 1.1 and the weight is determined to be 2.1 kilograms (kg).
A data element corresponding to the runt fish 104 is generated based on the weight of the runt fish 104 (e.g., 2.1 kg) and the condition factor or K-Factor (e.g., 1.1). In this case, the data element is used as a coordinate pair to plot a representation of the runt fish 104 within the population graph 112. One or more previously recorded data elements for other fish can be used to generate the other points within the population graph 112. In some cases, groups of fish can be plotted together within one point.
In stage E, the control unit 110 determines where to sort the runt fish 104 based on a location of the runt fish 104 in the population graph 112, the physical characteristics of the runt fish 104, and predefined sorting criteria. The sorting criteria for the system 100 include sorting fish classified as “Runt” into the second fish pen 118 and sorting fish classified as “Healthy” into the third fish pen 124. The control unit 110 activates the sorting actuator 116 to allow the runt fish 104 to progress into the second fish pen 118. In the example of
The sorting actuator 116 is connected to an element with holes. The element with holes is used, in stage E, to block the passage from the first fish pen 102 to the third fish pen 124. The holes can be included to reduce drag when moving the element that blocks passage of the fish. The element attached to the sorting actuator 116 is made of plastic. Similar constructions can be made out of other materials including aluminum, wood, glass, other metals or substances. The holes may be circular or another shape that reduces resistance of motion while not allowing the fish or other animal, to swim through.
The sorting actuator 116 receives a signal from the control unit 110 to close the passage to the third fish pen 124 or to open the passage to the second fish pen 118, depending on implementation. The runt fish 104, unable to swim through to the third fish pen 124, swims through to the second fish pen 118 and is sorted. In stage F, the runt fish 104 joins other similar size fish, classified as “Runt” in the second fish pen 118.
In stage G, the healthy fish 106 moves from the first fish pen 102 to the channel connecting the first fish pen 102 to the second fish pen 118 and the third fish pen 124. The imaging device 108 obtains one or more images of the healthy fish 106. The motion detector used to detect the motion of the runt fish 104 can be used again to detect the motion of the healthy fish 106 as it moves into the view of the imaging device 108.
Stage H shows the control unit 110 receiving one or more images obtained by the imaging device 108 of the healthy fish 106. As mentioned above, the detailed process of the analysis performed by the control unit 110 is shown in
In Stage I, the population graph 120 is generated by the control unit 110 based on the analysis of one or more images obtained by the imaging device 108. The population graph represents population data of the given population with the addition of the recent detection of the healthy fish 106. A database storing information related to members of a population, such as the healthy fish 106, can be updated based on a recent detection or detections. In this case, the detection of the healthy fish 106 is added to the database as an additional data point within an identification profile associated with the healthy fish 106. The identification profile and corresponding data related to the healthy fish 106 can be accessed by a user or automatically by a member of the system 100.
The updated database can be presented to a user or referenced to make automated decisions, such as sorting decisions. The graphical element shown in the population graph 112 and 120 represent two versions of a data visualization tool corresponding to a database that stores information for one or more fish with identification profiles associated with specific identified fish. The data related to the identification profiles can be updated, changed, or referenced by a detection, user input, or other events within the system 100. The data can be presented to the user or used for automated decisions.
Images associated with identified fish can be accessed using a graphical element such as the population graph 120. For example, the image 122 shows an image of the recently detected healthy fish 106. The image 122 also shows information related to the healthy fish. The specific information shown can be any form of gathered or computed information and is not limited in the current specification. In the example of
A data element corresponding to the healthy fish 106 is generated based on the weight of the healthy fish 106 (e.g., 4.3 kg) and the condition factor or K-Factor (e.g., 1.4). In this case, the data element is used as a coordinate pair to plot a representation of the healthy fish 106 within the population graph 120. One or more previously recorded data elements for other fish can be used to generate the other points within the population graph 120. In some cases, groups of fish can be plotted together within one point.
In some implementations, the data element is generated based on one or more variables within a system. For example, the data element can be a multi-dimensional vector. The multi-dimensional vector can be used within a multi-dimensional plot. Within the multi-dimensional plot, a multi-dimensional clustering algorithm could be used to group subpopulations together. In some cases, instead of weight and condition factor, weight, condition factor, and length are used. Any other measured or perceived value related to a member of the population can be added to the data element. Any value related to the member of the population can be represented as an element within a multi-dimensional vector.
In stage J, the control unit 110 sends a signal to the sorting actuator 116. The data in the signal is based on a location of the healthy fish 106 in the population graph 120, the physical characteristics of the healthy fish 106, and instruction as to where to sort members of the population. In this case, the signal is an instruction for the sorting actuator 116 to close the passage to the second fish pen 118 and allow the healthy fish 106 to progress into the third fish pen 124. The sorting actuator 116 prompts the movement of the flap over the channel connecting the first fish pen 102 to the second fish pen 118 and thus allows the healthy fish 106 to swim through to the third fish pen 124.
As described above, the sorting actuator 116, in this example, is a motorized element activated by the control unit 110 to swing one way or the other depending upon the sorting criteria. The sorting criteria for the system 100 include sorting fish classified as “Runt” into the second fish pen 118 and sorting fish classified as “Healthy” into the third fish pen 124. Stage K shows the healthy fish 106 joining other fish classified as “Healthy” in the third fish pen 124.
The imaging device 204 is a stereo camera that captures one image from one viewpoint of the fish 202, and another image from another viewpoint of the fish 202. Stereo images can be sent to the control unit 110 as part of the one or more images obtained by the imaging device 204. In some implementations, multiple imaging devices are used to obtain one or more images of the fish 202. The multiple imaging devices can be situated at different viewpoints around the fish 202 and can capture different views of the fish 202.
A flow chart shown in item 206 illustrates a number of processing steps and actions performed by the control unit 110. The control unit 110 receives the one or more images of the fish 202 from the imaging device 204 and performs fish detection 208. The fish detection 208 includes the use of a supervised neural network trained on labeled bounding boxes to detect fish using visual data. Boxes labeled as fish are used as a ground truth database to train the supervised neural network to detect objects sharing characteristics with the objects in the labeled bounding boxes, as fish. The supervised neural network, after being trained, is able to take one or more images from the imaging device 204 and generate fish detections and bounding boxes around the fish within an image of one or more fish.
In some implementations, the supervised neural network is a form of convolutional neural network. Various image processing techniques can be used. For example, imaging processing techniques that involve a form of supervised neural network, such as a convolutional neural network, can receive the one or more images of the fish 202 as input. In the case of a convolutional network, various learned filters can be applied to the one or more images. The process of applying a number of filters can be used to detect specific elements, including features on the body of the fish 202, within the one or more images of the fish 202. In some cases, padding on an image can be used to fully capture elements within an image of the one or more images using the given filters within the neural network. Alternative image processing techniques known in the art can also be used to process one or more images.
In the one or more images processed by the fish detection 208, each fish present in the one or more images is a separate detection of a fish. In the example of
In some implementations, the neural network used to detect the fish is separate from the neural network used to detect key points. For example, the two neural networks can be trained on separate data and a new version of one may be used with an older version of another allowing separate updates to the control unit 110. In other implementations, the detections take place in a single neural network trained to detect fish and key points associated with the fish. For example, a neural network trained using images of labeled fish and key points of the labeled fish can perform the operations of both the fish detection 208 and the key point detection 210. Other consolidations are possible in the control unit 110 process flow. For example, pose estimation may be included in the neural network detecting fish and key points.
Key points on a fish include body parts such as eyes, upper lip, and dorsal fin among others. Specific key points can vary for different species of fish or different animals. Training data containing sections of images labeled as specific key points are used to train the neural network. The trained neural network is used to label unlabeled data as including one or more key points. In some cases, a key point is not detected in a region. In this case, a label indicated no key point is included in the region, or the absence of any label can be used to indicate that no key point is within the region. The unlabeled data is a section of an image including a detection of the fish 202. The key points are determined based on the section of the one or more images containing a detection of the fish 202.
A process of pose estimation 212 receives the detected key points from the key point detection 210. The pose estimation 212 receives key point detections from images including images captured from multiple viewpoints of the fish 202. The multiple viewpoints, and corresponding key points detected in images from the multiple viewpoints, enable the pose estimation 212 to estimate a pose in three dimensions (3D). Key points identified in an image from one viewpoint and an image from another viewpoint are matched and projected into 3D space. The projections of the key points are used to estimate the pose in 3D. In some cases, the projections of the key points are an estimate of the pose in 3D.
A process of computing truss lengths 214 receives information related to the 3D pose estimation and the key points detected on the body of the detected fish. The result of the computed truss lengths is shown in item 215. The fish 202 is shown with truss lengths along its body. A more detailed image of the fish 202 shown in item 215 is discussed in reference to
The pose estimation 212 is used to interpret distances between key points and thus determine truss lengths. For example, key points projected into 3D space can be used to determine various distances associated with the key points. The key points can represent specific locations on the body of a fish. If the fish 202 is at a 15 degree angle from an imaginary line extending from the imaging device 204, locations on the body of the fish 202 may appear close in two dimensions (2D). Projected key points in 3D corresponding to the locations on the body of the fish 202 are used to determine a true distance value by accounting for effects such as the angle of the fish 202 within 3D space. In some cases, other effects including certain camera parameters including lens shape or focal length may affect the apparent distance between the locations on the body of the fish 202. By using pose estimation 212, the 3D pose of the fish is accounted for and accurate measurements of the fish 202 are made.
In some implementations, effects from a visual device alters the obtained one or more images of a fish. For example, the shape of the lens on a camera may distort an image. The distortion of the image may affect the visual appearance of the fish. The true value of one or more truss lengths can be determined by the 3D pose and subsequent truss measurements in 3D. Visual distortions may be alleviated through standard calibration based on known effects on known locations within an image. Similarly, one or more images taken in sequence can be used to determine more accurate values that are affected less by visual artifacts such as lens distortion.
A process of modeling and analysis 216 receives information related to one or more of the previous processing steps including detected fish, detected key points of the detected fish, pose estimations of the detected fish, or computed truss lengths. The modeling and analysis 216 is a linear regression model trained using truss length measurements and corresponding ground truth weight. In some implementations, other algorithms, or processes performed on a computing device can be used to determine weight based on the information of the fish 202. In this way, the modeling and analysis 216 determines a weight of the fish 202 or any other fish detected within the one or more images obtained by the imaging device 204 and processed by the control unit 110.
The modeling and analysis 216 includes computing a K-Factor as one or more values representing weights divided by one or more values representing lengths cubed. The weight, as discussed, has been calculated based on truss lengths obtained from one or more images of the imaging device 204. The traditional approach of manually weighing the fish is significantly slower and can be more prone to mistakes than the current method. In addition, weighing a sufficient quantity of fish in order to generate predictions and sorting is impractical within a traditional manual approach. In the current method, multiple measurements can be made automatically to improve accuracy or check results. Both the lengths and weights can be determined based on the key points and subsequent truss lengths determined based on a 3D pose estimation. The control unit 110 performs the calculation as previously stated and obtains the industry standard K-Factor. Other values can be computed based on the determined physical information corresponding to the fish 202 or additional information can be used to compute additional values.
The modeling and analysis 216 sends information to a database that tracks information corresponding to members of the population. In this case, the detection of the fish 202 can be added to an existing entry corresponding to the identity of the fish 202 or a new entry can be created to hold information related to the fish 202 and the detection. The data, including the computed weight, length, and at least one image of the one or more images, is added to the database.
A process of aggregated clustering 218 uses data within the database that stores information related to members of a population to generate clusters within the population. A graph 219 similar to the population graphs 112 and 120 of
The fish 202 may be classified according to either the graph 219 or the graph 220. Corresponding data is sent to action mapping 221. The action mapping 221 uses predetermined criteria to sort the fish 202 based on the physical characteristics and the clustering of the fish 202 with respect to the population. In this case, like
Based on the action mapping 221, a corresponding actuator activation 222 is performed. In the example of
Other truss lengths not shown can be used by the system 200. For example, a truss length from the upper lip 302 to the tail 313 can be used as the length of the fish 202 within the calculation of the K-Factor used in corresponding graphical elements such as the population graph. In addition, specific truss lengths can be used to recognize specific deformities. Deformities such as shortened operculum can be detected using truss lengths such as a truss length from the upper lip 302 to the gill. Processing one or more images of a fish can include determining any of the following health conditions: shortened abdomen, shortened tail, scoliosis, lordosis, kyphosis, deformed upper jaw, deformed lower jaw, shortened operculum, runting or cardiomyopathy syndrome (CMS).
Cluster 402 corresponds to smaller fish or outliers within the population represented by the graph 400. Cluster 404 corresponds to larger fish or normal, healthy fish within the population. An element 406 represents a selection within the population represented by the graph 400. In some implementations, the selection is one fish. In other implementations, the selection is a group of similar fish or similar hatches of fish. Images similar to images 114 or 122 in
In some implementations, feeding behavior or movement within the pen for members within a population can be recorded and used for analysis. For example, if a number of fish are not feeding in normal areas, not feeding at regular intervals, or not feeding to a sufficient degree, related tracked information can be recorded and attached to the specific fish or multiple fish.
In some implementations, other parameters can be used to generate specific clusters or graphs. For example, sea lice can be detected on a number of fish within a fish population. The graph can include the number of sea lice a fish is infected with related to another health metric such as K-Factor. Other combinations are possible and can be tailored by users or generated in response to one or more received detections. For example, multiple detections of a specific disease can result in a generation of a graph showing the effect of the disease within the population. Dynamically generated graphs can help orient users to issues within a population before other outward signs within a population become visible.
In some implementations, yield predictions can be generated based on observing one or more fish within a population. For example, within a first pen, ten fish may be detected. Corresponding data related to the detections of the ten fish can be used to compare the ten fish with other similar pens. Similar groups of fish can be used to generate predictions. For example, historical data can be used to compare the ten fish within the first pen to a similar group of fish detected previously. A given group of fish can be matched with a similar group of fish. A yield associated with the historical data of the similar group of fish detected previously can be used to predict the yield of the ten or more fish in the first pen. In other implementations, any number of fish can be used to compare with historical data in order to generate a yield prediction for members of a population. The yield predictions can be used for business purposes or decision making including sorting members of a population, preparing one or more fish for market, predicting revenue, among others.
The process 500 includes obtaining one or more images of a fish (502). For example, the imaging device 108 can obtain one or more images of the runt fish 104 as the runt fish moves from the first fish pen 102 to the channel connecting the first fish pen 102 to the second fish pen 118 and the third fish pen 124. The imaging device 108 can be a form of digital camera or a set of multiple digital cameras.
The process 500 includes detecting the fish within the one or more images (504). For example, the control unit 110 can detect the runt fish 104 using a neural network that processes one or more images. The one or more images can be obtained from the imaging device 108 after a motion sensor detects movement corresponding to the runt fish 104.
In some implementations, the control unit 110 performs analysis as described in reference to
The process 500 includes determining physical characteristics of the fish and computing related values (506). For example, in
The process 500 includes storing data related to the fish within a database (508). For example, the control unit 110 can be communicably connected to one or more electronic storage devices. The electronic storage devices can store data related to one or more members of the population within a database. The database can be used to generate population graphs as shown in item 219 of
The process 500 includes based on the physical characteristics of the fish, sorting the fish into a predefined location (510). For example, the control unit 110 classifies the runt fish 104 as “Runt” and according to the predefined sorting criteria for the system 100, the control unit 110 sends a signal to the sorting actuator 116 to allow passage for the runt fish 104 to the second fish pen 118. The control unit 110 can use data from the population, as shown in the population graph 112, to determine where to sort the runt fish 104.
The process 500 includes outputting information to a user (512). For example, the control unit 110 can output data corresponding to a graphical element similar to 112 or 120, such as the population graph 400 in
In some implementations, multiple sightings of a fish are aggregated together in a database. For example, multiple sightings of a fish in one pen and the same fish in another pen, in cases where fish move from pen to pen, can be stored. Information, including images of the fish, corresponding locations of the images, current location of the fish, or last known location, can be recorded. Other data, such as condition factor, disease, skeletal deformity including spinal deformity, among others, can also be recorded.
In some implementations, a subset of obtained images corresponding to a fish is recorded. For example, images of a fish that feature the fish unobstructed or in other ways are clear, and without visual defects in the region of a detection of the fish, can be aggregated together. The aggregated images of the fish can be used to form video from which behavior clues can be devised. The behavioral cues can be used to further record a current mental state or infer other non-visual traits of the fish including non-visual disease symptoms. In some cases, behaviors associated with a fish can be determined based on the aggregated images of the fish. The behaviors can be used to aid in the determination of diseases or other automated processes including predictions or sorting.
In some implementations, clusters or groups within a population are tracked to infer information related to the cluster or group. For example, smaller sized fish in a population can be recorded or tracked. Data related to the smaller sized fish can be used to inform understandings of causes or enable automatic preventative measures such as moving fish away from a certain pen or location that is correlated with stunted growth possibly because of a chemical leak, environmental factors including water temperature or water quality among others, or other external influence. By gathering data on a population, understandings can be generated and used to better manage the population or generate predictions including yield or profit.
In some implementations, sorting the fish can be accompanied by other processes. For example, a fish with sea lice can be sorted into a location with an automated delouse machine that can delouse the fish. Similar sorters in other pens or locations can similarly sort the fish back into specific pens after the lice have been removed. Similar actions can be performed for other diseases or ailments.
In some implementations, sites can be flagged based on recorded data of a population. For example, boats moving between fish pens can spread disease between pens. If a number of fish are ill within a specific pen, the site can be flagged and boats and other spreaders of disease, like humans, other animals, machines, or instruments, can be prevented from going to the flagged site or, if there, can be prevented from leaving without first disinfecting or taking other preventative measures. In one example, the control unit 110 can send a signal to a fleet of boats with a location of the flagged site as a “flagged site” alerting boats not to move in that region.
In some implementations, other sorting criteria are used. For example, instead of sorting fish classified as “Runt” from fish classified as “Healthy”, a system can sort fish with specific deformities from fish that do not have physical deformities. Physical deformities can be detected based on certain truss lengths determined by the control unit 110. Clustering based on fish sharing the characteristics of specific deformities or one or more deformities including scoliosis or lordosis among others, can be used to sort particular fish with deformities from fish without deformities.
In some implementations, fish with diseases are detected. For example, the control unit 110 can detect sea lice on a fish. The detection of sea lice on a fish can be an additional data element stored within a database. Fish with the disease of sea lice can be detected and sorted from fish that do not have the disease of sea lice. In general, any disease can be used as criteria for the control unit 110 to sort a given population.
In some implementations, the control unit 110 interprets elements within an image that are not visible. For example, if another fish swims between the imaging device 204 and the fish 202 while the imaging device 204 is capturing one or more images of the fish 202, the other fish may obscure or cover up portions of the fish 202. For example, the other fish may be obscuring the tail of the fish 202. In this example, the control unit 110 can, based on knowledge of a shape of a fish, species, or animal, infer that the tail should be at a specific location at the opposite end of the head and equidistant from the dorsal fin. In some cases, proportions of other elements of the animal or fish can be used to determine likely locations or proportions of other elements that may not be visible.
In some implementations, images determined to be not useful of a fish can be disregarded or discarded. For example, if in the example given above, the other fish covers the entire body of the fish 202 except for the tail, the image of the fish 202 can be discarded or disregarded as lacking sufficient data to interpret information of the fish 202. Similarly, if the fish 202 is at an extreme angle to an imaging device, a resulting image with multiple key points stacked, or too close to one another can be used to determine that a certain image should be discarded or disregarded. Obscurities from optical elements, such as smudges or lens cracks, or digital data corruption can similarly be sufficient to disregard or discard.
In some implementations, a score representing an indication of the quality of an image is used to determine whether an image is used for analysis or sorting. For example, in the examples discussed above, if the other fish covers the entire body of the fish 202 except for the tail, the quality score can be a low value indicating poor quality. Various scoring systems and ranges can be used. On the other hand, if an image is captured with the fish and exactly a 90 degree angle as shown in item 215 of
In some implementations, the imaging device 204 includes a depth sensor to enable 3D pose estimation. For example, images obtained by the imaging device 204 include depth information from the depth sensor on the imaging device 204. The pose estimation 212 determines a 3D pose of the fish 202 based on the depth sensor information obtained by the imaging device 204. In this way, stereo images of the fish 202, images from two or more viewpoints of the fish 202 are not used for 3D pose estimation. In some cases, the depth sensor can be a form of sonar or structured light.
In some implementations, the control unit 110 performs a subset of the processes depicted in
In some implementations, image analysis performed may be composed of multiple steps. For example, the image analysis performed by the control unit 110 in the fish detection 208 step, may include a rough object identifier used to detect the fish 202 within the one or more images. A second object identifier may be the key point detection 210 process and may include receiving the output of the first object identifier to locate objects on the fish 202 including key points of the fish. The multiple steps can be performed by various computational methods including, but not limited to, algorithms, neural networks, or linear regressions.
In some implementations, one or more images obtained by an imaging device contain two or more fish. For example, the imaging device 108 can capture images of both the runt fish 104 and the healthy fish 106. Image processing performed by the control unit 110 can be used to identify the runt fish 104 and the healthy fish 106 within one or more images containing both the runt fish 104 and the healthy fish 106. Sections of the one or more images that contain the runt fish 104 can be used as images of the runt fish 104. Similarly, sections of the one or more images that contain the healthy fish 106 can be used as images of the healthy fish 106. In general, this approach can be applied to any number of fish within a view of an imaging device or multiple imaging devices.
In some implementations, the imaging device 108 obtains one or more images of the healthy fish 106 without a signal from the motion detector. For example, the imaging device 108 can capture one or more images and send the one or more images to the control unit 110. By performing image detection on the one or more images, the control unit 110 can determine images of the healthy fish 106.
In some implementations, the motion detector or other element is used to send a control signal to the imaging device. For example, if the motion detector detects movement, the motion detector can send a signal to the imaging device and, responsive to receiving the signal from the motion detector, the imaging device 108 can increase the frequency of images obtained.
In some implementations, other sorting criteria is used. For example, instead of separating members classified as “Runt” into one area and members classified as “Healthy” into another area, the system 100 can be used to separate those determined to be sick or otherwise unwell from members of the population that are not sick or are well. Determinations based on one or more images collected by the system 100, the imaging device 108, and processed by the control unit 110 can be used to classify fish as well or unwell. Based on the determination, the control unit 110 can either automatically sort the unwell fish from the well fish or flag certain fish for sorting by an external system. The identification abilities of the control unit 110 can be used to identify and therefore aid in any form of population segmentation.
In some implementations, the sorting actuator 116 is another form or element. For example, instead of the flap shown in
In some implementations, the sorting actuator 116 includes a closing or opening mechanism. For example, the closing mechanism can include an element similar to an iris of a human eye. The element can be used to restrict the size of a passage to only allow certain members of a population through. Other shapes may be used or generated to accomplish similar goals. A benefit of using an actuator to create the shapes for members of a population to move through is the ability to change the kind of shape dynamically through the use of a computer or other controller such as the control unit 110.
In some implementations, the sorting actuator 116 includes a linear actuator. For example, instead of swinging to the left or right depending on a sorting decision, the sorting actuator 116, or element attached to the sorting actuator 116, can move linearly left or right, up or down, to alternately block or open passage from one area to another area.
In some implementations, the control unit 110 is used to establish sorting decisions based on a sorting criteria. For example, the sorting decisions can be generated based on determined characteristics of a fish and corresponding sorting criteria for a system. The sorting decisions can be sent to a user or owner of a system for external involvement not carried out directly by the control unit 110. For example, instructions based on the sorting decision can be sent to a worker within an aquaculture environment. The worker can manually sort an identified fish into a given location based on the sorting decision determined by the control unit 110. In some cases, the control unit 110 can automatically sort animals, such as the runt fish 104 in
In some implementations, one or more other images are captured by the imaging device 108. For example, instead of using a motion sensor, the imaging device 108 can capture one or more images and send the one or more images to the control unit 110. The control unit 110 analyzes the one or more images and, based on the analysis, determines whether or not a fish is in view and thus whether or not a fish is present in the viewing area of the imaging device 108. In this way, image recognition of fish within images obtained by the imaging device 108 can be used to replace or supplement any existing motion sensor.
In some implementations, the imaging device 108 includes multiple sub elements. For example, the imaging device 108 can be a form of an imaging assembly comprised of one or more cameras, storage devices, or other electronic components. The imaging device 108 can be capable of capturing images of an object from two or more viewpoints to aid in 3D analysis including pose estimation.
In some implementations, other storage devices are used to store fish. For example, instead of the first fish pen 102, the runt fish 104 can be stored in an open water environment. The open water environment can similarly be connected to one or more other areas used for fish storage. A sorting actuator can be used to control access to the one or more other areas used for fish storage. In general, any element able to store aquatic animals, including fish, can be used in place of the first fish pen 102 of
In some implementations, more than two elements can be used to sort animals into. For example, instead of two elements in the second fish pen 118 and the third fish pen 124, three elements can be used. A sorting actuator can be used to sort fish, or other items into one of the three elements.
In some implementations, instead of fish, other animals can be sorted. For example, other forms of aquaculture or forms of non-aquaculture may be similarly sorted using a system similar to the system 100. Cattle raised in pastures may be similarly sorted into separate fields or tagged for further processes based on visual processing by a control unit similar to the control unit 110 of
For example, at early ages, the lengths of runt fish may appear similar to the lengths of normal fish. This may be in part due to the tendency for bone growth of the fish to be comparable between runt and non-runt fish. However, runt fish may be skinnier than non-runt counterparts. The R-Factor analysis may use components of the determined truss network (e.g., truss network 300 of
In some implementations, R-Factor values may be computed according to Equations (2) and (3):
For example, the length from the upper lip 302 to the trailing upper caudal peduncle 314, the length from the upper lip 302 to the trailing low caudal peduncle 312, and the length from the leading edge dorsal fin 306 to the pelvic fin 307, as shown in the truss network 300 of
A weight may be determined according to the modeling and analysis 216 described in
Similar to the system 100, by automating the process and using machine learning techniques to determine significant parameters, the system 600 shown in
Similar to the system 100 as described in reference to
The system 600 further includes a control unit 610 connected to an imaging device 608 and a sorting actuator 616. The sorting actuator 616 controls access to the second fish pen 618 and the third fish pen 624 from the first fish pen 602. The control unit 610 is used to generate population graphs 612 and 620 that show relevant information of a fish population. The fish population shown in population graphs 612 and 620 include the runt fish 604 and the healthy fish 606 among others.
The population graphs 612 and 620 include weight and R-Factor data for fish within the fish population as well as related information including historical data and images of particular fish. Individual points within the population graphs 612 and 620 correspond to individual fish. Additional information related to a fish can be accessed by clicking on a point within the graphic within a given user interface rendered on a computer or other electronic device.
A flow rate of water is adjusted to allow the runt fish 604 into the channel and to progress forwards through the channel in the direction of the second fish pen 618 and the third fish pen 624. In some implementations, other processes are used to control a fishes' movement from one element to another element. For example, an incentive can be placed in the channel connecting the first fish pen 602 to the second fish pen 618. The incentive can be used to control movement of the fish from the first fish pen 602 to the second fish pen 618. In another example, because fish like to swim into a current, a pump, or other current generating sources, may create a current in the channel to entice the fish to swim into the fish channel.
Stage B shows the imaging device 608 obtaining one or more images of the runt fish 604 when the runt fish 604 is in view of the imaging device 608. A motion sensor is used to detect motion of a fish in the channel connecting the first fish pen 602 to the second fish pen 618 and the third fish pen 624. The motion sensor triggers the capturing of one or more images by the imaging device 608. In this case, the motion sensor detects the motion of the runt fish 604 and the imaging device 608 subsequently obtains one or more images of the runt fish 604. The images are sent to the control unit 610.
Stage C shows the control unit 610 receives the one or more images of the runt fish 604. Based on analysis performed, the control unit 610 determines physical characteristics corresponding to the runt fish 604. For example, the control unit 610 determines that the runt fish 604 has a weight of 1.8 kilograms (kg). Information related to the physical characteristics of each fish within a population of fish is used to generate the population graph 612.
In some implementations, the process of analysis performed by the control unit 610 may include actions depicted and described in reference to
The population graph 612 shows data related to the fish population including the runt fish 604 and the healthy fish 606. Data related to a detection of the runt fish 604 updates a database used to create the population graph 612.
Data related to the runt fish 604 is added to an identification profile corresponding to the runt fish 604. Data related to the runt fish 604 can be accessed by a user or other element of the system 600.
Stage D shows an example of a graphical element, the population graph 612, generated by the control unit 610. The x axis of the population graph 612 is weight and the y axis is an R-Factor that is related to physical characteristics of the fish. Points on the graph near the bottom right represent fish less likely to be runts while points near the top left represent fish more likely to be runts. The value of the runt factor may be determined based on any mathematical formula that quantifies the condition of fish in terms of one or more truss lengths and weight. One example of such a factor, similar to the K-Factor, is the R-Factor as discussed herein. A runt factor value may be plotted against other runt factor values to identify different subpopulations of fish by using clustering techniques or thresholds. In other implementations, other computed parameters based on determined physical characteristics can be used.
An image 614 of the runt fish 604 shows example statistics and parameters related to the identification of the runt fish 604. The image 614 can be stored within a graphical element such as the population graph 612 and shown to a user after a selection is made on the particular data entry related to the runt fish 604. As shown in the image 614, the runt fish 604 is classified as “Runt” by the control unit 610 based on the population graph 612 and physical characteristics related to the runt fish 604. The R-Factor is calculated as 0.04 and the weight is determined to be 1.8 kilograms (kg).
A data element corresponding to the runt fish 604 is generated based on the weight of the runt fish 604 (e.g., 1.8 kg) and the runt factor (e.g., 0.04). In this case, the data element is used as a coordinate pair to plot a representation of the runt fish 604 within the population graph 612. One or more previously recorded data elements for other fish can be used to generate the other points within the population graph 612. In some cases, groups of fish can be plotted together within one point.
In stage E, the control unit 610 determines where to sort the runt fish 604 based on a location of the runt fish 604 in the population graph 612, the physical characteristics of the runt fish 604, and predefined sorting criteria. The sorting criteria for the system 600 include sorting fish classified as “Runt” into the second fish pen 618 and sorting fish classified as “Healthy” into the third fish pen 624. The control unit 610 activates the sorting actuator 616 to allow the runt fish 604 to progress into the second fish pen 618. In the example of
The system 600 may obtain or determine a threshold 612a. The threshold 612a may be determined based on one or more size determinations of fish within a population. For example, the threshold 612a may include a slope indicating which region of the population graph 612 is considered runt fish and which region is considered healthy. In the example of
In the example of
The sorting actuator 616 is connected to an element with holes. The element with holes is used, in stage E, to block the passage from the first fish pen 602 to the third fish pen 624. The holes can be included to reduce drag when moving the element that blocks passage of the fish. The element attached to the sorting actuator 616 is made of plastic. Similar constructions can be made out of other materials including aluminum, wood, glass, other metals or substances. The holes may be circular or another shape that reduces resistance of motion while not allowing the fish or other animal, to swim through.
The sorting actuator 616 receives a signal from the control unit 610 to close the passage to the third fish pen 624 or to open the passage to the second fish pen 618, depending on implementation. The runt fish 604, unable to swim through to the third fish pen 624, swims through to the second fish pen 618 and is sorted. In stage F, the runt fish 604 joins other similar size fish, classified as “Runt” in the second fish pen 618.
In stage G, the healthy fish 606 moves from the first fish pen 602 to the channel connecting the first fish pen 602 to the second fish pen 618 and the third fish pen 624. The imaging device 608 obtains one or more images of the healthy fish 606. The motion detector used to detect the motion of the runt fish 604 can be used again to detect the motion of the healthy fish 606 as it moves into the view of the imaging device 608.
Stage H shows the control unit 610 receiving one or more images obtained by the imaging device 608 of the healthy fish 606. As mentioned above, the detailed process of the analysis performed by the control unit 610 is shown in
In Stage I, the population graph 620 is generated by the control unit 610 based on the analysis of one or more images obtained by the imaging device 608. The population graph represents population data of the given population with the addition of the recent detection of the healthy fish 606. A database storing information related to members of a population, such as the healthy fish 606, can be updated based on a recent detection or detections. In this case, the detection of the healthy fish 606 is added to the database as an additional data point within an identification profile associated with the healthy fish 606. The identification profile and corresponding data related to the healthy fish 606 can be accessed by a user or automatically by a member of the system 600.
The updated database can be presented to a user or referenced to make automated decisions, such as sorting decisions. The graphical element shown in the population graph 612 and 620 represent two versions of a data visualization tool corresponding to a database that stores information for one or more fish with identification profiles associated with specific identified fish. The data related to the identification profiles can be updated, changed, or referenced by a detection, user input, or other events within the system 600. The data can be presented to the user or used for automated decisions.
Images associated with identified fish can be accessed using a graphical element such as the population graph 620. For example, the image 622 shows an image of the recently detected healthy fish 606. The image 622 also shows information related to the healthy fish. The specific information shown can be any form of gathered or computed information and is not limited in the current specification. In the example of
A data element corresponding to the healthy fish 606 is generated based on the weight of the healthy fish 606 (e.g., 2.5 kg) and the runt factor (e.g., 0.03). In this case, the data element is used as a coordinate pair to plot a representation of the healthy fish 606 within the population graph 620. One or more previously recorded data elements for other fish can be used to generate the other points within the population graph 620. In some cases, groups of fish can be plotted together within one point.
In some implementations, the data element is generated based on one or more variables within a system. For example, the data element can be a multi-dimensional vector. The multi-dimensional vector can be used within a multi-dimensional plot. Within the multi-dimensional plot, a multi-dimensional clustering algorithm could be used to group subpopulations together. In some cases, instead of weight and condition factor, weight, condition factor, and length are used. Any other measured or perceived value related to a member of the population can be added to the data element. Any value related to the member of the population can be represented as an element within a multi-dimensional vector.
In stage J, the control unit 610 sends a signal to the sorting actuator 616. The data in the signal is based on a location of the healthy fish 606 in the population graph 620, the physical characteristics of the healthy fish 606, and instruction as to where to sort members of the population. In this case, the signal is an instruction for the sorting actuator 616 to close the passage to the second fish pen 618 and allow the healthy fish 606 to progress into the third fish pen 624. The sorting actuator 616 prompts the movement of the flap over the channel connecting the first fish pen 602 to the second fish pen 618 and thus allows the healthy fish 606 to swim through to the third fish pen 624.
In the example of
As described above, the sorting actuator 616, in this example, is a motorized element activated by the control unit 610 to swing one way or the other depending upon the sorting criteria. The sorting criteria for the system 600 include sorting fish classified as “Runt” into the second fish pen 618 and sorting fish classified as “Healthy” into the third fish pen 624. Stage K shows the healthy fish 606 joining other fish classified as “Healthy” in the third fish pen 624.
In some implementations, one or more components of the system 600 may be used for runt filtering. For example, because runts are a subpopulation in the pen, they may not appear uniformly in the pen. If the runts appear intermittently then they may cause swings in the mean estimate of the weight (especially when they are much smaller than the mean weight). Filtering out the runts while tracking the population weight provides far more consistent results than the same weight estimates without filtering out the runts.
In some implementations, runt classification may be used to estimate ratios corresponding to the number of runts in a population. For example, estimation of the true ratio of runt to non-runt fish may traditionally require sampling fish over a longer period of time (e.g. 10-20 days) to determine the ratio. With runt filtering using an appropriate value, such as the R-Factor, the ratio may be determined more accurately and more quickly than simply extending the window of detection or resorting to manual techniques. Ratio determination may be an important consideration in yield estimation as the ratio tends to be relatively stable over time and may indicate a corresponding future yield for a population where greater numbers of runt fish may reduce the yield, thus allowing operators to plan or adjust the aquaculture operation accordingly.
A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed.
Embodiments of the invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, embodiments of the invention can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Embodiments of the invention can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the invention, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
While this specification contains many specifics, these should not be construed as limitations on the scope of the invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of the invention. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
In each instance where an HTML file is mentioned, other file types or formats may be substituted. For instance, an HTML file may be replaced by an XML, JSON, plain text, or other types of files. Moreover, where a table or hash table is mentioned, other data structures (such as spreadsheets, relational databases, or structured files) may be used.
Particular embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the steps recited in the claims can be performed in a different order and still achieve desirable results.
This application is a continuation-in-part of U.S. patent application Ser. No. 16/885,646, filed May 28, 2020, which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16885646 | May 2020 | US |
| Child | 17381666 | US |
