Patent Grant
12154405
The present invention relates to pharmacy packaging systems and, more particularly, to verification systems for pharmacy packaging systems.
One embodiment provides a pharmacy packaging system including a feed roll for forming a pharmaceutical pouch, a dispenser configured to dispense pharmaceuticals into the pharmaceutical pouch formed by the feed roll, and a verification system. The verification system includes a camera system configured to capture images in a visible spectrum and an infrared spectrum, a first light source configured to output infrared spectrum light, a second light source configured to output visible spectrum light, and a processor communicatively coupled to the camera system, the first light source, and the second light source. The processor is operable to activate the first light source to illuminate the pharmaceutical pouch and capture a first image, using the camera system, of the pharmaceutical pouch while illuminated by the first light source. The processor is also operable to activate the second light source to illuminate the pharmaceutical pouch and capture a second image, using the camera system, of the pharmaceutical pouch while illuminated by the second light source. The processor is further operable to generate a third image based on the first image and the second image and generate a dashboard to simultaneously display first images, second images, and third images from a plurality of pharmaceutical pouches, wherein the first images, the second images, and the third images are arrayed in a grid on the dashboard. The processor is also operable to provide a number of pills indication of a number of pills detected in the pouch against a number of pills expected in the pouch.
Another embodiment provides a method of verifying pharmaceuticals packaged within a pharmaceutical pouch using a pharmacy packaging system. The pharmacy packaging system includes a feed roll for forming the pharmaceutical pouch, a dispenser configured to dispense pharmaceuticals into the pharmaceutical pouch formed by the feed roll, a camera system configured to capture images in a visible spectrum and an infrared spectrum, a first light source configured to output an infrared spectrum light, a second light source configured to output a visible spectrum light, and a processor communicatively coupled to the camera system, the first light source, and the second light source. The method includes activating, by the processor, the first light source to illuminate the pharmaceutical pouch and capturing a first image, by the processor using the camera system, of the pharmaceutical pouch while illuminated by the first light source. The method also includes activating, by the processor, the second light source to illuminate the pharmaceutical pouch and capturing a second image, by the processor using the camera system, of the pharmaceutical pouch while illuminated by the second light source. The method further includes generating, by the processor, a third image based on the first image and the second image and generating, by the processor, a dashboard to simultaneously display first images, second images, and third images from a plurality of pharmaceutical pouches, wherein the first images, the second images, and the third images are arrayed in a grid on the dashboard. The method also includes providing a number of pills indication of a number of pills detected in the pouch against a number of pills expected in the pouch and displaying the dashboard.
Other aspects of the invention will become apparent by consideration of the detailed description and accompanying drawings.
Before any embodiments of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. The invention is capable of other embodiments and of being practiced or of being carried out in various ways.
Packaging System
The manifold 320, or chute, includes a plurality of discrete tracks 370 corresponding to each of the canisters 130 mounted on the motor base 310. The illustrated tracks 370 are independent channels that together form the manifold 320. The tracks 370 direct pharmaceuticals from the motor base 310 toward packaging equipment of the packaging unit 120. The motor base 310 is positioned generally above the packaging equipment such that pharmaceuticals slide down the manifold 320 toward the packaging equipment. The tracks 370 isolate the pharmaceuticals from each other as the pharmaceuticals slide down the manifold 320 to the receptacle 330.
The receptacle 330 collects the pharmaceuticals from the manifold 320. After the pharmaceuticals pass through to the receptacle 330, the pharmaceuticals are sandwiched between two strips of material (e.g., plastic, paper, etc.) from the feed stock rolls 340, 350. The first feed roll 340 may be made of a first material that is white in color (e.g., a translucent material). The second feed roll 350 may be made of a second material that is clear (e.g., a transparent material) to allow a pharmacist to look through the pouch to see the pharmaceuticals inside the pouch. The two strips of material are then heat sealed together to form a pouch for the pharmaceuticals. In some embodiments, the packaging unit 120 may include a single feed stock roll having a single strip of material that is folded and/or heat sealed to form pouches. Once filled and sealed, the pouches are wrapped around the take-up roll 360 to create a single spool of pouches. The spool may correspond to pharmaceuticals requested by a particular patient or a particular facility. In other embodiments, the pouches may be cut and separated as they are filled, rather than spooled onto the take-up roll 360 continuously. The pouches are dispensed through, for example, a dispenser or dispensing port connected to the pharmacy packaging system 100.
In some embodiments, each packaging unit 120 may include a printer 380 to print a patient's name, the date, the amount and type of pharmaceuticals contained within, a bar code, and/or other indicia on the pouches as the pouches are formed. The printer 380 may be, for example, a thermal printer. In other embodiments, the printer 380 may include an ink ribbon or an ink jet. In addition, each packaging unit 120 may include a bar code scanner or vision system 390 to monitor and check the pouches as they are spooled onto the take-up roll 360 or dispensed.
In the example illustrated, the control system 400 includes a processor 410, a memory 420, a transceiver 430, and an input/output interface 440. The processor 410, the memory 420, the transceiver 430, and the input/output interface 440 communicate over one or more control and/or data buses (e.g., a communication bus 450).
In some embodiments, the processor 410 is implemented as a microprocessor with separate memory, such as the memory 420. In other embodiments, the processor 410 may be implemented as a microcontroller (with memory 420 on the same chip). In other embodiments, the processor 410 may be implemented using multiple processors. In addition, the processor 410 may be implemented partially or entirely as, for example, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), and the like, and the memory 420 may not be needed or be modified accordingly. In the example illustrated, the memory 420 includes non-transitory, computer-readable memory that stores instructions that are received and executed by the processor 410 to carry out functionality of the control system 400 described herein. The memory 420 may include, for example, a program storage area and a data storage area. The program storage area and the data storage area may include combinations of different types of memory, such as read-only memory and random-access memory.
The transceiver 430 enables wireless communication from the control system 400 to, for example, a remote electronic device such as a server or a smart telephone or a tablet computer of a remote pharmacist. In other embodiments, rather than the transceiver 430, the control system 400 may include separate transmitting and receiving components, for example, a transmitter and a receiver. In yet other embodiments, the control system 400 may not include a transceiver 430 and may communicate with a remote device via a network interface and a wired connection to a communication network such as the Internet.
As noted above, the control system 400 may include the input/output interface 440 (or more commonly referred to as a user interface). The input/output interface 440 may include one or more input mechanisms (e.g., a touch screen, a keypad, a button, a knob, and the like), one or more output mechanisms (e.g., a display, a printer, a speaker, and the like), or a combination thereof. The input/output interface 440 receives input from the input devices actuated by a user, and provides output to the output devices with which a user interacts. In some embodiments, as an alternative or in addition to managing inputs and outputs through the input/output interface 440, the control system 400 may receive user inputs, provide user outputs, or both by communicating with an external device, such as a console computer, over a wired or wireless connection.
A user can interact with the packaging unit 120 through the control system 400 to input patient information, facility information, and/or the pharmaceuticals needed. The control system 400 can control movement of the gantry assembly 220 to move the canisters 130 from the shelves of the storage unit 110 to one of the motor bases 310. In addition, the control system 400 can control operation of the motor bases 310 to selectively open and close the canisters 130. Furthermore, the control system 400 may optimize orders by minimizing movement of the gantry assembly 220 and the canisters 130 or by filling all the orders for a particular patient or facility consecutively.
Verification System
As the pouches enter the verification system 610, the agitator 710 engages (e.g., agitates, vibrates, etc.) the pouches to help the pharmaceuticals within each pouch settle and separate or spread out. In some embodiments, the agitator 710 may include fingers, brushes, or the like that engage and agitate the pouches. Additionally, the fingers or brushes may be driven by, for example, a vibration motor to help agitate the pouches and move the pharmaceuticals around within each pouch. In other embodiments, the fingers or brushes may be passive elements that help comb the pharmaceuticals apart. In further embodiments, the agitator 710 may be omitted.
After the pouches are agitated, the pouches pass a sensor 720. In some embodiments, the sensor 720 may be, for example, a barcode scanner, a QR code reader, or an image sensor. In other embodiments, the sensor 720 may be a label camera that captures images. The sensor 720 is positioned on one side of the pouches (e.g., a label side of the pouches) and reads a code or captures an image of the label on each pouch. A light source 725 (e.g., a visible light) is also positioned on the same side of the pouches as the sensor 720 to selectively illuminate the pouches while the sensor 720 reads codes or captures images. The sensor 720 can communicate with the processor 410 and memory 420 of the control system 400 to store the image or data related to the read/scanned code. The processor 410 can also control operation of the light source 725 (e.g., when the light source 725 is turned on and off).
The pouches then pass the camera system 730. In the example illustrated, the camera system 730 includes a single camera with an infrared filter removed to capture both infrared images and the visible light images. The camera system 730 is positioned on an opposite side of the pouches (e.g., on the side of the pouches facing the transparent material) from the sensor 720 so the labels on the pouches do not interfere with the camera system 730. A first light source 760 (e.g., an infrared (IR) or near infrared (NIR) spectrum light) is positioned on a side of the pouches opposite the camera system 730 (e.g., the side of the pouches facing the translucent material and label). The first light source 760 shines light through the pouches toward the camera system 730 to illuminate the pouches. Light from the first light source 760 passes through the labels so that the pharmaceuticals within each pouch cast shadows against, for example, the opaque or translucent white paper of the labels. As described above, in some embodiments, the label information on the pharmaceutical pouch is printed by a thermal printer. In these embodiments, the label information lets infrared light through the printed indicia without causing a reflection on the white paper or plastic that makes up one-half of the pharmaceutical pouch. The camera system 730 can then capture an image of the shadows cast by the pharmaceuticals within each pouch without any additional shadows or noise created by the label information. Similar to the sensor 720, the camera system 730 can communicate with the processor 410 and memory 420 to store the image.
As the pouches are positioned under the camera system 730, a second light source 770 also illuminates the pouches. The second light source 770 (e.g., a visible spectrum light) is positioned on the same side of the pouches as the camera system 730 (e.g., on the side of the pouches facing the transparent material) to shine light on the pouch and illuminate the contents of the pouch. When the second light source 770 illuminates the pouches, the camera system 730 captures a visible light image of each pouch, showing the color, shape, contour, surface finish, etc. of each pharmaceutical. In the illustrated embodiment, the pouches are stopped under the camera system 730 to capture the IR image and the visible light image. The camera system 730 captures both images without moving so the images are lined-up for computer manipulation.
In the illustrated embodiment, the first light source 760 and the second light source 770 operate in relatively rapid succession. For example, the first light source 760 can be turned on to briefly illuminate a pouch with infrared light, and the camera system 730 can capture an image of the pouch while the pouch is illuminated with the infrared light. Then, the first light source 760 can be turned off, the second light source 770 can be turned on to briefly illuminate the pouch with visible light, and the camera system 730 can capture an image of the pouch while the pouch is illuminated with the visible light (and before the pouch is disturbed/moved). In some embodiments, the order in which the light sources 760, 770 are turned on and the images are captured may be reversed (e.g., the camera system 730 may first capture an image of the pouch while the pouch is illuminated with visible light, and then capture an image of the pouch while the pouch is illuminated with infrared light). In further embodiments, the camera system 730 may capture another image while both light sources 760, 770 are on.
Although the camera system 730 is described as having a single camera with the infrared filter removed, in other embodiments, the camera system 730 may include two separate cameras. For example, one camera may be an infrared (IR) or near infrared (NIR) camera that operates in conjunction with the first light source 760 to capture images in the infrared spectrum, and the other camera may be a color camera or visible spectrum camera that operates in conjunction with the second light source 770 to capture images in the visible spectrum.
In some embodiments, the verification system 610 may provide real-time monitoring of the pharmaceutical pouches as they pass through the system. For example, after the camera system 730 captures an infrared image of a pouch, the processor 410 analyzes the image to determine the number and approximate size of each pharmaceutical within the pouch. Based on the label (which is associated with an expected prescription order), the processor 410 determines whether the correct number of pharmaceuticals were loaded into the pouch. In addition, after the camera system 730 captures a visible light image of the pouch, the processor 410 analyzes the image (in conjunction with the infrared image and the data from the label) to verify that each pouch is properly filled with the correct type of pharmaceuticals. Once the pouches are verified, the pouches are directed to the take-up roll 360, a bin, or other location for distribution.
When the verification system 610 determines that a pouch is improperly filled, the verification system 610 can provide immediate feedback to discard the pouch and/or fill a new pouch. For example, the verification system 610 can include a printer or other device to mark an improperly filled pouch with a large X, a colorful dot, a spray, or the like to identify that the pouch has an error. A pharmacist or other technician can then discard this pouch before the pouch is distributed. In addition, the verification system 610 can communicate with the packaging unit 120 to notify the packaging unit 120 to fill another pouch with the same prescription order, without having to wait for further input from the pharmacist or other technician. As such, disruptions to the packaging system are reduced.
Chain of Custody
First, a canister 130 is placed on a filling station (e.g., a platform, a countertop, a table, etc.) with the ability to read an RFID tag, or other information, of the canister 130. The filling station then prompts an operator (e.g., pharmacist, technician, or other authorized personnel) which National Drug Code (NDC) number is needed. The user is instructed to obtain a bulk storage bottle with this NDC number, and scans a barcode on the bulk storage bottle to verify it is correct. If the barcode does not match the expected NDC number, the canister 130 will not later be accepted by the packaging system 100 (e.g., a security door on the packaging system 100 will not open to allow the filled canister to be loaded into the system 100).
At step 810, the canister 130 (e.g., one of the cassettes/container) is filled with pharmaceuticals by the operator. The canister 130 is filled by opening the canister 130 and inserting the desired pharmaceuticals into the canister 130. After filling, the canister 130 is closed. In some embodiments, the canister 130 may be locked and tamper-proof such that only authorized personnel know how or are able to reopen the canister 130. During this filling step, a camera may be positioned (e.g., located over the top of the canister 130) to record that the canister 130 is filled. The camera can take pictures or record a video as the pharmaceuticals are poured into the canister 130. In some embodiments, the canister 130 may include an RFID tag or other identifier to help track the canister 130, as noted above.
At step 820, a pharmacist verifies the canister 130 was properly filled by either filling the canister 130 him or herself, or by watching a technician or other authorized personnel fill the canister 130 via a video or still images. The video may be a live stream of the filling or a replay of the filling. The pharmacist then logs that the canister 130 was properly filled into the electronic chain of custody file. Among other things, the pharmacist may enter his/her name, the date of filling, the type of pharmaceuticals, and/or the amount of pharmaceuticals into the electronic file. By recording the filling process, tagging the canister 130, and identifying the operator (e.g., through biometric, a login, etc.), the filled canister 130 can be certified as correctly filled by the remote pharmacist.
At step 830, the filled canister 130 is loaded into the pharmacy packaging system 100. For example, the canister 130 is inserted by operator into an input port (when the security door is open, if applicable) of the storage unit 110. The storage unit 110 scans a barcode, RFID tag, or other indicia on the canister 130 to identify the canister 130. Once verified as the correct canister 130 filled from the correct bulk storage bottle, the security door is closed. While the door is closed and no tampering is possible, the canister 130 is automatically weighed by a scale positioned in the input port to determine the number of pharmaceuticals that were added. This data can be stored by the system 100 to help accurately track on-hand inventory and avoid interruptions in production. The gantry assembly 220 of the storage unit 110 then moves the filled canister 130 from the input port to a suitable storage location. In some embodiments, the storage location may be randomized (i.e., the canister 130 is not always stored the same location) based on size, frequency of use, etc. As the gantry assembly 220 moves the filled canister 130, the packaging system automatically logs where the canister 130 is moved to in the electronic chain of custody file.
In some embodiments, the filled canister 130 may be loaded into the pharmacy packaging system 100 before the pharmacist verifies that the canister 130 was filled properly. In such embodiments, the pharmacist can verify proper filling of the canister 130 after the canister 130 is stored in the storage unit 110, or can flag that the canister 130 was improperly filled and needs to be corrected. The pharmacy packaging system 100 inhibits using the canister 130 to fill prescription orders until the canister 130 is verified by the pharmacist. In other embodiments, the canister 130 may be used to fill pouches, but the final pouches may not be released until the canisters 130 are verified by the pharmacist.
When a prescription order is to be filled, a technician or other personnel logs into the system 100 via biometrics, a passcode, or the like. The technician then verifies that the manifold 320 of the pharmacy packaging system is clear, at step 840. If necessary, the technician may remove any pharmaceuticals or other debris from the manifold 320. Once the manifold 320 is clear, the technician notifies the pharmacy packaging system 100 that the manifold 320 was inspected and to start filling the prescription order. In some embodiments, the technician may confirm that the manifold 320 is clear by pushing a button on or entering a code into the pharmacy packaging system 100.
After the manifold 320 is cleared, at step 850, the pharmacy packaging system 100 begins filling the prescription order. In particular, the gantry assembly 220 of the storage unit 110 retrieves the desired canister 130 and moves the canister 130 to the motor base 310 of the packaging unit 120. As discussed above, the motor base 310 can operate the canister 130 to release pharmaceuticals from the canister 130. The pharmaceuticals travel down the manifold 320 to be packaged into a pouch.
At step 860, the pharmacy packaging system 100 monitors the pharmaceuticals as the pharmaceuticals are released from the canister 130. For example, the motor base 310 may include a sensor (e.g., a light sensor, a laser beam or array, etc.) adjacent each outlet to verify whether a pharmaceutical was properly released from the canister 130. The sensor can also verify that only the expected number of pharmaceuticals (e.g., one) is being released from the canister 130. This information is logged in the electronic chain of custody file to record that the proper number of pharmaceuticals was released.
After the pharmaceuticals drop down the manifold 320 and are inserted into a pouch, at step 870, the pharmacy packaging system 100 verifies that the correct number of pharmaceuticals reached the pharmacy packaging system and was loaded into the pouch. For example, a sensor (e.g., an image sensor or camera) on the packaging unit 120 captures an image of the filled pouch. The image acquired by the sensor can be analyzed by the processor 410 of the pharmacy packaging system 100 or a remote computer for object recognition. That is, the processor 410 simply needs to detect that the correct number of objects reached the pouch, since the pharmaceuticals have been monitored and verified from the time of loading into the canister 130. When the expected number of objects is detected, the processor 410 logs this verification in the electronic chain of custody file. The pouch can then be released and delivered to a patient. On the other hand, if the processor 410 does not detect the expected number of objects, the pharmacy packaging system 100 can flag the pouch for further review by a pharmacist. The pharmacist then can manually verify whether the pouch was properly filled, discard the pouch, and/or refill the prescription order in a new pouch.
Method of Verifying Pharmaceuticals
The method 800 depicted in
After the pouches are filled, the pouches are directed into the verification system 610. In some embodiments, the agitator 710 (
Once the pouches enter the verification system 610, the sensor 720 (e.g., label camera) can capture an image of the label on each pouch. The processor 410 sends a signal or command to the sensor 720 to capture the image. The processor 410 also directs the sensor 720 to store the captured image in the memory 420 or other suitable storage device.
As illustrated in
At step 920, the camera system 730 captures a first image of the pouch while the pouch is illuminated by the first light source 760. The processor 410 sends a signal or command to the camera system 730 to capture the image while the first light source 760 is on. The processor 410 also directs the camera system 730 to store the captured image in the memory 420 or other suitable storage device.
As noted above, the label may be printed on the pouch by a thermal printer. In such embodiments, the infrared light from the first light source 760 passes through the indicia without the indicia appearing as “noise” on the image captured by the camera system 730. In embodiments where the label is applied to the pouch using other techniques, the indicia may need to be removed from the captured image. In such embodiments, the processor 410 can remove the indicia from the image using, for example, a software module in the control system 400. For example, the processor 410 can analyze the label image captured by the sensor 720 to determine the location of the indicia “noise.” The processor 410 can then flip the label image (due to the orientation of the sensor 720 relative to the pouch versus the orientation of the camera system 730 relative to the pouch), and subtract the flipped label image from the infrared image captured by the camera system 730.
At step 930, the processor 410 activates the second light source 770 to illuminate the pharmaceutical pouch when the pharmaceutical pouch is under the camera system 730. In particular, the processor 410 sends a signal or command to the second light source 770 to turn the second light source 770 on. The second light source 770 illuminates the pouch such that the pharmaceuticals within the pouch are visible to the camera system 730 through the transparent side of the pouch.
At step 940, the camera system 730 captures a second image of the pouch. In the illustrated embodiment, the processor 410 sends a signal or command to the camera system 730 to capture the image while the second light source 770 is on. The processor 410 also directs the camera system 730 to store the captured image in the memory 420 or other suitable storage device.
At step 950, the processor 410 generates a third image based on the first (i.e., infrared) image and the second (i.e., visible light) image. Several techniques are contemplated by this invention to generate the third image. One example technique of generating the third image includes super imposing the first image on the second image, and removing all other information from the second image that is not within the borders of the shadows created by the pharmaceuticals in the first image. Removing all other information may include, for example, filling regions in the third image that are not within the borders of the shadows created by the pharmaceuticals in the first image with a solid color (e.g., black). Another example technique includes filling in the colors from the second image corresponding to the shadow areas of the first image, and filling the remained of the image with a solid color (e.g., black). Once the third image is created, the processor 410 stores the image in the memory 420 or other suitable storage device.
In some embodiments, the pill images shown in the third image 1030 can be rearranged by the processor 410 on the solid background. For example, the processor 410 can reorganize the pill images in a row or column to align the pill images in the same order from pouch to pouch. Such an arrangement enhances the ability of a pharmacist to observe and inspect the individual pills in each pouch. Additionally, processor 410 can order the individual pill images in the same order as images of expected pills 1050 (
The above-described methods 800 and 900 provide, among other things, verification that a pouch is filled with the proper type of pharmaceuticals because the pharmaceuticals are monitored from the time they are loaded into the canister 130 until they are dispensed to the customer. After the pharmaceuticals are loaded into a canister 130, the pharmaceuticals are maintained in a closed system that is generally not accessible to outside users. In addition, the methods 800 and 900 verify that the pouch is filled with the proper number of pharmaceuticals using relatively low-cost sensors (e.g., light sensors on the motor base 310, and an image sensor on the packaging unit 120). Further, the methods 800 and 900 create an electronic chain of custody file that can be referenced later, if necessary. The method 900 may also create a database of filled pouch images should it be necessary to review an allegedly faulty pouch at a later time.
Dashboard
In the example illustrated, the dashboard 1000 includes an infrared first image 1010, a visible light second image 1020, a third image 1030 based on the first and second images 1010, 1020, label information 1040 (e.g., a label fourth image), and images of expected pills 1050. The illustrated dashboard 1000 also includes an interface for navigating the dashboard 1000, including patient information 1060, a “Previous Pouches” button 1070, an “Approve Pouches” button 1080, and pouch information 1090. As shown, the first image 1010, the second image 1020, the third image 1030, and the label information 1040 may be arrayed in a grid on the dashboard 1000, with each column of the grid corresponding to the images associated with one of the pouches. The first image 1010 is the infrared image captured by the camera system 730. The second image 1020 is the visible light image captured by the camera system 730. The third image 1030 is generated by the processor 410, as explained above. The label information 1040 may be the label information printed on the pharmaceutical pouch captured by the sensor 720, or may be a computer-generated list of data associated with the barcode of the pouch. The images of expected pills 1050 are stock images (e.g., “gold images” from a database) of the medication that a pharmacist should be expecting to see in the pouch based on the information input to pack the pouch.
The patient information 1060 provides information about a patient to whom the pharmaceutical pouch is being dispensed. The patient information 1060 may include, for example, patient name, administration time, facility, and the like. The “Previous Pouches” button 1070 allows a user (for example, a pharmacist) to navigate to screens or columns of previous pouches stored in the system. The “Approve Pouches” button 1080 allows the user to approve the current pouches displayed on the dashboard 1000 and dispense the pouches to the patient. Once the displayed pouches are approved by actuating the button 1080, the dashboard 1000 can automatically navigate to the next screen or columns of pouches. The pouch information 1090 displays information about the pouches being shown on the dashboard 1000. The pouch information 1090 may include, for example, a pouch number, an administration time, a prescription number, and the like.
In some embodiments, the processor 410 may also automatically determine whether the pouches are correctly filled or incorrectly filled. This automatic verification can be used to assist a pharmacist in making his or her determination on whether the pouches were properly filled. In one example technique, the processor 410 determines whether the number of pharmaceuticals or the type of pharmaceuticals (e.g., a first characteristic of pharmaceuticals) included in the pouch matches the number of pharmaceuticals or the type of pharmaceuticals (e.g., a second characteristic of expected pharmaceuticals) of expected pharmaceuticals, based on the label information 1040 or information that is in the prescription order input. The processor 410 may analyze the first image 1010, the second image 1020, the third image 1030, or a combination of the three images to determine the number of pharmaceuticals included in the pouch. For example, the processor 410 may determine the number of pharmaceuticals included in the pouch based on the distinct shadows created in the first image 1010. In other embodiments, other known techniques may be used to determine the number of pharmaceuticals included in the pharmaceutical pouch based on analyzing an image of the pharmaceutical pouch.
The processor 410 may also analyze the third image 1030 of the pharmaceutical pouch to determine the type of pharmaceuticals included in the pharmaceutical pouch. The processor 410 may employ known color analysis techniques on the third image 1030 in addition to the known information from the expected pills 1050 to determine whether the correct type of pharmaceuticals are included in the pharmaceutical pouch. Additionally or alternatively, the processor 410 may use images captured by cameras on the motor base 310 to help identify the pharmaceuticals and to verify that the correct pharmaceuticals were dropped.
Based on these comparisons, the processor 410 provides indications on the dashboard 1000 to identify whether the processor 410 thinks the pouches were correctly or incorrectly filled. For example, the dashboard 1000 may display a first indication on one or more of the images 1010, 1020, 1030, 1040, identifying that the processor 410 thinks the pouch was properly filled. In some embodiments, the dashboard 1000 may display a border 1015 having a first color (e.g., green) around the images 1010, 1020, 1030, 1040 to indicate the pharmaceutical pouch is correctly filled. Similarly, the dashboard 1000 may display a second indication on one or more of the images 1010, 1020, 1030, 1040, identifying that the processor 410 thinks the pouch was not properly filled. In some embodiments, the dashboard 1000 displays a border 1015 having a second color (e.g., red) around the images 1010, 1020, 1030, 1040 to indicate the pharmaceutical pouch might be incorrectly filled. Other indicia (e.g., check marks, X's, etc.) may also be used to show that a pharmaceutical pouch is correctly or incorrectly filled.
In some embodiments, the fourth image 1130 may be generated based on the third image 1030. As described above, the first image 1010 is used to define borders or boundaries of the pharmaceuticals within the pouch. Once the third image 1030 is generated, individual pharmaceuticals may similarly be identified based on the borders or boundary information from first image 1010. The individual pharmaceuticals may then be arranged in rows or columns to generate the fourth image 1130. The fourth image 1130 may be displayed such that the individual pharmaceuticals in the fourth image 1130 align with individual pharmaceuticals in the images of expected pills 1050, as shown.
In addition, the processor 410 may determine whether the individual pharmaceuticals in the fourth image 1130 correspond to the individual pharmaceuticals in the images of expected pharmaceuticals 1050. When the processor 410 determines that an individual pharmaceutical in the fourth image 1130 corresponds with the images of expected pharmaceuticals, the dashboard 1100 may display an indication, for example, a green border around the individual pill to indicate that the pouch includes the expected pharmaceutical (shown in
In some embodiments, the pharmacist may select or click the “Approve Pouches” button 1080 after reviewing all the pouches and each individual suspected pouch to verify that the pouch corresponding to the first column of images is correctly filled. When the pharmacist selects or clicks the “Approve Pouches” button 1080, the dashboard moves the grid of images to the left by one set of images to display the contents of the next pouch. Scrolling through the pouch images in this manner allows each set of images to be aligned next to the “gold images” of the expected pills 1050. As such, the pharmacist may quickly and efficiently verify the contents of the pouch when the fourth image 1130 is next to the images of expected pills 1050.
In addition, the processor 410 may determine whether the individual pharmaceuticals in the fourth image 1130 correspond to the individual pharmaceuticals in the images of expected pharmaceuticals 1050. When the processor 410 determines that an individual pharmaceutical in the fourth image 1130 corresponds with the images of expected pharmaceuticals, the dashboard 1400 may display a first pill indication around the individual pill to indicate that the pouch includes the expected pharmaceutical. When the processor 410 determines that an individual pharmaceutical in the fourth image 1130 does not correspond with the images of expected pharmaceutical (i.e., a wrong kind of pharmaceutical is included in the pouch), the dashboard 1400 may display a second pill indication around the individual pill to indicate that the pouch includes a pharmaceutical that should not be included in the pouch. In addition, the dashboard 1400 may also display a third pill indication around an empty space aligned with an image of an expected pill 1050 when the expected pill is not found in the fourth image 1130 (e.g., when the pill is missing from the pouch). Similarly, when the processor 410 determines that an unidentified pharmaceutical or debris in the pharmaceutical pouch, the unidentified pill or debris in the fourth image 1130 is displayed in a separate box below the images of the identified pharmaceuticals in the fourth image 1130 (as shown in
In addition, the dashboard 1400 may also display a number of pills indication 1410 providing the number of pills detected in a pouch against a number of pills expected in the pouch. The number of pills indication 1410 may be provided on one of the first mage 1010, the second image 1020, and/or the label image 1040. In one embodiment, the number of pills indication 1410 is provided on the second image 1020 (i.e., the infrared image). In other embodiments, the numbers of pills indication 1410 may be provided in a row or column separate from the images 1010, 1020, 1040. Additionally or alternatively, the number of pills indication 1410 may only display the number of pills detected (and not the number of pills expected).
In some embodiments, the processor 410 may also provide indication around individual pills within the first image 1010 and the second image 1020. For example, as shown in
After inspection of the dashboard, the pharmacist may determine whether the errors are correctly identified on the dashboard 1400. When the errors are correctly identified, the pharmacist may click on or select one of the first image 1010, the second image 1020, and the label image 1040 of the corresponding pouch to mark the pouch for further work or further inspection. The processor 410 may print an indication on the pouch such that a packaging technician or pharmacist can identify the incorrectly packaged pouches or pouches that need further work. When the errors are incorrectly identified, the pharmacist may ignore the indications for the misidentified pills in the corresponding pouch. In some embodiments, as described with respect to
The method 1700 includes receiving, at the user interface 440, a user input to add the image of the individual pharmaceutical to a correct location (at block 1720). With reference to
The method 1700 includes receiving, at the user interface 440, a user input to add the image of the individual pharmaceutical to the database storing images of expected pharmaceuticals (at block 1730). By approving the image, the image of the pill may also be added to the database storing images of expected pharmaceuticals. When the second pill is selected, the processor generates a graphical user interface providing an option for the pharmacist to select adding the image of the pill to the database storing images of expected pharmaceuticals. For example, a “Train” option 1420 is provided on the user interface 1400 for selection by the pharmacist. The pharmacist may select the “Train” option to add the image of the individual pharmaceutical to the database storing images of expected pharmaceuticals. In some embodiments, the pharmacist may also specify the orientation of the pill. For example, the pharmacist may specify whether the image of the pill is captured in a first orientation with the camera facing the length, width, or diameter of the pill or in a second orientation with the camera facing the height of the pill.
The method 1700 includes determining, using the processor 410, whether the image of individual pharmaceutical is within a tolerance level of the expected pharmaceutical (at block 1740). When the images of expected pharmaceuticals are loaded into the database, the processor 410 extracts characteristics of the images to determine the features of the expected pharmaceuticals. For example, the processor 410 uses pixel data to determine the length, width, height, diameter, color, shape, and the like of the expected pharmaceutical based on the images of the expected pharmaceuticals. The database may store more than one image of an expected pharmaceutical. The processor 410, based on the features extracted from the one or more images of the expected pharmaceutical, determines the tolerance level for one or more features of the expected pharmaceutical. For example, based on the one or more images of the expected pharmaceutical, the processor 410 may determine that the length of the expected pharmaceutical is 2 centimeters (cm)+/−10% tolerance. That is, a packaged pharmaceutical will be accepted as the expected pharmaceutical when, based on the image captured of the packaged pharmaceutical, the processor 410 determines that the length of the package is between 1.8 to 2.2 cm. When the image of individual pharmaceutical from the dashboard 1400 is being trained to the database, the processor 410 compares the features of the individual pharmaceutical based on the image of the individual pharmaceutical to the tolerance levels of the expected pharmaceutical determined based on the images of the expected pharmaceuticals or based on information received from the NDC database. The processor 410 may determine the orientation of the image of the individual pharmaceutical to determine which features (for example, length, width, or diameter vs. height) to compare to the tolerance levels.
When the image of individual pharmaceutical is not within a tolerance level of the images of expected pharmaceutical, the method 1700 includes generating, at the user interface, an alert indicating that the image of the individual pharmaceutical is not within the tolerance level of the expected pharmaceutical (at block 1750). The processor 410 may generate the alert as a second graphical user interface 1800 as shown in
When the image of individual pharmaceutical is within the tolerance level of the images of expected pharmaceutical and/or when the pharmacist approves the image of the individual pharmaceutical to be added to the database, the method 1700 includes adding, using the processor 410, the image of the individual pharmaceutical to the database storing images of expected pharmaceuticals (at block 1760). The processor 410 adds the selected image of the individual pharmaceutical (for example, the second pill of first column in
The method 1700 also includes re-analyzing, using the processor 410, the images of the current dashboard 1400 based on the updated database (at block 1770). The processor 410 re-analyzes the images of the current dashboard, that is, the pouches illustrated on the screen when the database is updated in response to user input. The processor 410 regenerates the dashboard based on the analyzing the images of the current dashboard using the updated database and displays the regenerated dashboard. In some embodiments, the processor 410 may only re-analyze the images of the current dashboard that were previously identified as incorrectly packaged. In other embodiments, the processor 410 may re-analyze all the images of the current dashboard. The processor 410 may also analyze any future dashboards using the updated database. The method 1700 may be repeated for the next misidentified pharmaceutical in the dashboard.
Various features and advantages of the invention are set forth in the following claims.
This application is a continuation of U.S. patent application Ser. No. 16/999,327, filed on Aug. 21, 2020, which is a continuation-in-part of U.S. patent application Ser. No. 16/821,823, filed on Mar. 17, 2020, which is a continuation of U.S. patent application Ser. No. 16/216,184, filed on Dec. 11, 2018, which is a continuation of U.S. patent application Ser. No. 15/277,493, filed Sep. 27, 2016, the entire contents of all of which are incorporated by reference herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3709598 | Vandenberg et al. | Jan 1973 | A |
| 4546901 | Buttarazzi | Oct 1985 | A |
| 4679068 | Lillquist et al. | Jul 1987 | A |
| 4751571 | Lillquist | Jun 1988 | A |
| 5522512 | Archer et al. | Jun 1996 | A |
| 5568715 | Ebel et al. | Oct 1996 | A |
| 6079284 | Yamamoto et al. | Jun 2000 | A |
| 6108030 | Yamamoto et al. | Aug 2000 | A |
| 6286421 | Ackley | Sep 2001 | B1 |
| 6324253 | Yuyama et al. | Nov 2001 | B1 |
| 6330351 | Yasunaga | Dec 2001 | B1 |
| 6345487 | Luciano et al. | Feb 2002 | B1 |
| 6434911 | Yamamoto et al. | Aug 2002 | B1 |
| 6450089 | Ackley | Sep 2002 | B2 |
| 6481347 | Ackley | Nov 2002 | B2 |
| 6505461 | Yasunaga | Jan 2003 | B1 |
| 6535637 | Wootton et al. | Mar 2003 | B1 |
| 6739455 | Yamamoto et al. | May 2004 | B2 |
| 6741731 | Yamamoto et al. | May 2004 | B1 |
| 6805245 | Kenneway | Oct 2004 | B2 |
| 7079236 | Yamamoto et al. | Jul 2006 | B2 |
| 7102741 | Ackley, Jr. et al. | Sep 2006 | B2 |
| 7218395 | Kaye | May 2007 | B2 |
| 7701568 | Ackley, Jr. et al. | Apr 2010 | B2 |
| 7706596 | Garvey | Apr 2010 | B2 |
| 7769221 | Shakes et al. | Aug 2010 | B1 |
| 7782365 | Levien et al. | Aug 2010 | B2 |
| 7794394 | Frangioni | Sep 2010 | B2 |
| 7994480 | Johnson et al. | Aug 2011 | B2 |
| 7995831 | Eller et al. | Aug 2011 | B2 |
| 8037992 | Ackley, Jr. et al. | Oct 2011 | B2 |
| 8060249 | Bear et al. | Nov 2011 | B2 |
| 8072590 | Ackley, Jr. et al. | Dec 2011 | B2 |
| 8146331 | Soloman | Apr 2012 | B2 |
| 8269958 | Ackley, Jr. et al. | Sep 2012 | B2 |
| 8271128 | Schultz | Sep 2012 | B1 |
| 8295582 | Eller et al. | Oct 2012 | B2 |
| 8300922 | Garvey, III | Oct 2012 | B1 |
| 8340392 | Kim | Dec 2012 | B2 |
| 8457361 | Julius et al. | Jun 2013 | B2 |
| 8467897 | Holmes et al. | Jun 2013 | B2 |
| 8538581 | Kuehnrich et al. | Sep 2013 | B2 |
| 8571297 | Eller et al. | Oct 2013 | B2 |
| 8582863 | Van Den Brink | Nov 2013 | B2 |
| 8583281 | Bear et al. | Nov 2013 | B2 |
| 8584715 | Stoeckel et al. | Nov 2013 | B2 |
| 8646591 | De Ruijter et al. | Feb 2014 | B2 |
| 8712163 | Osheroff | Apr 2014 | B1 |
| 8727208 | Poisner | May 2014 | B2 |
| 8729528 | Klem et al. | May 2014 | B2 |
| 8742398 | Klem et al. | Jun 2014 | B2 |
| 8749635 | Högasten et al. | Jun 2014 | B2 |
| 8755597 | Tantalo et al. | Jun 2014 | B1 |
| 8791993 | Kim | Jul 2014 | B2 |
| 8943779 | Amano et al. | Feb 2015 | B2 |
| 8959878 | Okuma | Feb 2015 | B2 |
| 8964054 | Jung et al. | Feb 2015 | B2 |
| 8966863 | Amano et al. | Mar 2015 | B2 |
| 9008408 | Sinbar et al. | Apr 2015 | B2 |
| 9041826 | Jung et al. | May 2015 | B2 |
| 9098900 | Helgason et al. | Aug 2015 | B2 |
| 9135701 | Guissin et al. | Sep 2015 | B2 |
| 9184204 | Hu | Nov 2015 | B2 |
| 9202194 | Mistry | Dec 2015 | B1 |
| 9233767 | Amano et al. | Jan 2016 | B2 |
| 9235690 | Bear et al. | Jan 2016 | B2 |
| 9259766 | Ackley et al. | Feb 2016 | B2 |
| 9272795 | Chudy | Mar 2016 | B2 |
| 9311451 | Bae et al. | Apr 2016 | B2 |
| 9334096 | Luciano et al. | May 2016 | B2 |
| 9387155 | Morioka et al. | Jul 2016 | B2 |
| 9406127 | Helgason et al. | Aug 2016 | B2 |
| 9407838 | Butte et al. | Aug 2016 | B2 |
| 9413988 | Heinke et al. | Aug 2016 | B2 |
| 9468948 | Ackley et al. | Oct 2016 | B2 |
| 9517855 | Murokh | Dec 2016 | B2 |
| 9599461 | Gerlach et al. | Mar 2017 | B2 |
| 9621749 | Jung et al. | Apr 2017 | B2 |
| 9637323 | Kim | May 2017 | B2 |
| 9716843 | Fox et al. | Jul 2017 | B2 |
| 9730649 | Jepsen | Aug 2017 | B1 |
| 9731909 | Kim | Aug 2017 | B2 |
| 9741761 | Jiang et al. | Aug 2017 | B2 |
| 9757772 | Ackley et al. | Sep 2017 | B2 |
| 9766114 | Ademe | Sep 2017 | B2 |
| 9776755 | Kondo et al. | Oct 2017 | B2 |
| 9835445 | Strandemar et al. | Dec 2017 | B2 |
| 9836583 | Garcia et al. | Dec 2017 | B2 |
| 9843743 | Lewis et al. | Dec 2017 | B2 |
| 9886751 | Jacobs et al. | Feb 2018 | B2 |
| 9906739 | Sugano et al. | Feb 2018 | B2 |
| 9912913 | Johnson et al. | Mar 2018 | B2 |
| 9994347 | Yuyama | Jun 2018 | B2 |
| 10015474 | Rhead et al. | Jul 2018 | B2 |
| 10187593 | Holmes | Jan 2019 | B2 |
| 10713540 | Zhang | Jul 2020 | B2 |
| 10839501 | Futase | Nov 2020 | B2 |
| 10881581 | Koike et al. | Jan 2021 | B2 |
| 10926907 | Maruyama et al. | Feb 2021 | B2 |
| 11195270 | Hellenbrand | Dec 2021 | B2 |
| 11289179 | Brown | Mar 2022 | B1 |
| 11753193 | Holmes | Sep 2023 | B2 |
| 20020066867 | Denwood, III et al. | Jun 2002 | A1 |
| 20040119722 | Hilbert et al. | Jun 2004 | A1 |
| 20060015536 | Buchanan | Jan 2006 | A1 |
| 20060271237 | Kim | Nov 2006 | A1 |
| 20070008326 | Levien et al. | Jan 2007 | A1 |
| 20070189597 | Limer | Aug 2007 | A1 |
| 20080056556 | Eller et al. | Mar 2008 | A1 |
| 20080307757 | Heim et al. | Dec 2008 | A1 |
| 20090055116 | Chen et al. | Feb 2009 | A1 |
| 20100071320 | Ali et al. | Mar 2010 | A1 |
| 20110137177 | Toma et al. | Jun 2011 | A1 |
| 20110184751 | Holmes | Jul 2011 | A1 |
| 20110188707 | Shutt | Aug 2011 | A1 |
| 20120002042 | Okuma | Jan 2012 | A1 |
| 20120200596 | Gotou et al. | Aug 2012 | A1 |
| 20120304596 | Koike et al. | Dec 2012 | A1 |
| 20130058550 | Tanimoto et al. | Mar 2013 | A1 |
| 20130142406 | Lang et al. | Jun 2013 | A1 |
| 20130170732 | Gotou et al. | Jul 2013 | A1 |
| 20130176424 | Weil | Jul 2013 | A1 |
| 20130188038 | Tanimoto et al. | Jul 2013 | A1 |
| 20130218330 | Chudy et al. | Aug 2013 | A1 |
| 20130279774 | Helgason | Oct 2013 | A1 |
| 20130307992 | Erlandsson et al. | Nov 2013 | A1 |
| 20130318931 | Holmes | Dec 2013 | A1 |
| 20130342676 | Amano et al. | Dec 2013 | A1 |
| 20140033644 | Amano et al. | Feb 2014 | A1 |
| 20140094960 | Holmes | Apr 2014 | A1 |
| 20140231881 | Parks | Aug 2014 | A1 |
| 20140236349 | Bae et al. | Aug 2014 | A1 |
| 20140261881 | Chudy | Sep 2014 | A1 |
| 20150077534 | Derenne et al. | Mar 2015 | A1 |
| 20150170373 | Yonaha | Jun 2015 | A1 |
| 20150192525 | Sinbar et al. | Jul 2015 | A1 |
| 20150266604 | Amano et al. | Sep 2015 | A1 |
| 20150269742 | Bergstrom et al. | Sep 2015 | A1 |
| 20150347832 | Shutt | Dec 2015 | A1 |
| 20150379712 | Guissin et al. | Dec 2015 | A1 |
| 20160065909 | Derenne et al. | Mar 2016 | A1 |
| 20160199260 | Andersen, Sr. | Jul 2016 | A1 |
| 20160323522 | Heinke et al. | Nov 2016 | A1 |
| 20160364858 | Butte et al. | Dec 2016 | A1 |
| 20170015445 | Holmes | Jan 2017 | A1 |
| 20170020785 | McCullough | Jan 2017 | A1 |
| 20170061663 | Johnson et al. | Mar 2017 | A1 |
| 20170081069 | Murokh | Mar 2017 | A1 |
| 20170163972 | Köhler et al. | Jun 2017 | A1 |
| 20170227942 | Thomson et al. | Aug 2017 | A1 |
| 20170254762 | Yoo | Sep 2017 | A1 |
| 20170301087 | Yuyama et al. | Oct 2017 | A1 |
| 20170305589 | Yuyama et al. | Oct 2017 | A1 |
| 20170312182 | Ma | Nov 2017 | A1 |
| 20170354574 | Feng et al. | Dec 2017 | A1 |
| 20170355476 | Kim | Dec 2017 | A1 |
| 20170357775 | Ekin | Dec 2017 | A1 |
| 20170358621 | Jiang et al. | Dec 2017 | A1 |
| 20180039564 | Zimmerman et al. | Feb 2018 | A1 |
| 20180068087 | Garcia et al. | Mar 2018 | A1 |
| 20180170591 | Koike et al. | Jun 2018 | A1 |
| 20180177682 | Tanaka | Jun 2018 | A1 |
| 20180225818 | Jacobs et al. | Aug 2018 | A1 |
| 20190050660 | Iwami | Feb 2019 | A1 |
| 20190112080 | Holmes | Apr 2019 | A1 |
| 20190178815 | Tojo | Jun 2019 | A1 |
| 20190259143 | Futase | Aug 2019 | A1 |
| 20200143545 | Weng et al. | May 2020 | A1 |
| 20200342972 | Chudy | Oct 2020 | A1 |
| 20210015711 | Iwami | Jan 2021 | A1 |
| 20220287915 | LeBrun | Sep 2022 | A1 |
| 20230101967 | Holmes | Mar 2023 | A1 |
| Number | Date | Country |
|---|---|---|
| 2591761 | May 2013 | EP |
| 2006069618 | Mar 2006 | JP |
| 1999017218 | Apr 1999 | WO |
| 2002025568 | Mar 2002 | WO |
| WO-0225568 | Mar 2002 | WO |
| 2013129099 | Sep 2013 | WO |
| 2013181416 | Dec 2013 | WO |
| 2014119994 | Aug 2014 | WO |
| 2016047569 | Mar 2016 | WO |
| WO-2016047295 | Mar 2016 | WO |
| Entry |
|---|
| J. J. Caban, et al., “Automatic identification of prescription drugs using shape distribution models”, 2012 19th IEEE International Conference on Image Processing, pp. 1005-1008. |
| R. Palenychka, et al., “Verification of Medication Dispensing Using the Attentive Computer Vision Approach”, 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-4. |
| Number | Date | Country | |
|---|---|---|---|
| 20230206714 A1 | Jun 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16999327 | Aug 2020 | US |
| Child | 18173599 | US | |
| Parent | 16216184 | Dec 2018 | US |
| Child | 16821823 | US | |
| Parent | 15277493 | Sep 2016 | US |
| Child | 16216184 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16821823 | Mar 2020 | US |
| Child | 16999327 | US |
