None.
The present disclosure generally relates a method of ascertaining identity, verification, and authentication of a product, and in particular to a method of watermarking with digital printing using inkjet or laser printers utilizing edible bioprinting and of extracting an embedded watermark from a watermarked image after error corrections.
This section introduces aspects that may help facilitate a better understanding of the disclosure. Accordingly, these statements are to be read in this light and are not to be understood as admissions about what is or is not prior art.
The prevalence of fake drugs has emerged as a continually growing problem worldwide, regardless of the economic status of countries. In general, fake drugs can be categorized as substandard, falsified, counterfeit, and diverted drugs, and the World Health Organization (WHO) broadly defines a counterfeit medicine as “one which is deliberately and fraudulently mislabeled with respect to identity and/or source.” Pharmaceutical products are the most vulnerable goods that counterfeiters and illicit drug manufacturers deliberately produce for illegal monetary gain worldwide. Notably, counterfeit medicines for malaria and pneumonia are estimated to cause 250,000 child deaths per year. In sub-Saharan Africa, such counterfeit antimalaria drugs are estimated to cost tens of millions of dollars. According to WHO and the United Nations Office on Drugs and Crime, counterfeit drugs account for approximately 10% of the global pharmaceutical trade and 20-30% of all medicines in Africa, Asia, and the Middle East. Furthermore, counterfeit medicines pertaining to both lifestyle and lifesaving drugs appear increasingly common in the U.S. Counterfeit opioids have caused deaths in almost all states of the U.S.
The bolstered distribution of counterfeit medicines and pharmaceutical products can be attributed to the increased use of online pharmacies in recent years and the ongoing COVID-19 pandemic scenario. Approximately 40,000 online pharmacies are currently active worldwide, with an increase of 600 every month. The operation of nearly 96-97% of these online pharmacies is illegal. In the U.S., the opioid epidemic can also be attributed to the illegal online sales of controlled substances without a prescription. The COVID-19 pandemic has promoted the emergence and use of online pharmacies, resulting in the recent surge in counterfeit pharmaceutical products (e.g. fake chloroquine and hydroxychloroquine) and counterfeit medical supplies (e.g. substandard surgical mask and respirator). This trend is expected to continue as the demand exceeds the supply in part because of lockdowns in many drug-producing countries. Several governments have strengthened legislation and enforcement to deter counterfeit medicines. For example, in the U.S., unit-level traceability will be mandated by 2023, following the Drug Supply Chain Security Act set by the United States Food and Drug Administration (FDA). In Europe, the Falsified Medicines Directive requires the implementation of safety features. Although public resources to avoid the unintentional use of illicit online pharmacies are available, patients and healthcare providers (e.g. physician, nurse, and pharmacist) are not well aware of such information.
Traditional methods to identify counterfeit medicines rely on analytical chemistry and spectroscopy. Fluorescence, Raman, infrared, and near-infrared spectrometers represent the first screening tools that can detect counterfeit medicines by analyzing the exact chemical compositions. However, these instrument-dependent methods require specialized devices, complicated readers, and trained personnel. Recently, advanced anticounterfeit technologies, including fluorescent ink printing, microprinting, fingerprint-like encoding strategies, organic electronic devices, nanopillar array paper, 3D microstructure films, and invisible photonic printings, have been developed. However, patients and healthcare professionals have limited access to such analytical chemistry methods and sophisticated anticounterfeit technologies to avoid the unintentional use of counterfeit medicines.
Therefore, there is an unmet need for a novel approach to provide dosage-level (on-dose) anticounterfeit measures and authentication features that can provide protection against counterfeited pharmaceutical and food-related products.
A method of ascertaining identity, verification, or authentication of a product is disclosed. The method includes embedding a predetermined watermark on a host image, thereby generating a watermarked image, printing the watermarked image using edible ink on an attack-resistant edible substrate, affixing the printed substrate onto the product, providing a color reference chart along with the watermarked image, obtaining an image of the watermarked image from the product, establishing correspondence between the watermarked image and the color reference chart, correcting the obtained image based on the established correspondence, extracting the embedded predetermined watermark from the corrected watermarked image, and determining identity, verification, and authentication of the product by analyzing the extracted predetermined watermark of the watermarked image.
In the above method, the product is one or more of a pharmaceutical product, a nutritional supplement product, and a food product.
In the above method, the watermarked image is printed using an edible ink.
In the above method, the edible ink is a Food and Drug Administration approved food coloring dye.
In the above method, the watermarked image is printed with an attack-resistant edible substrate material.
In the above method, the attack-resistant edible substrate material includes fluorescent materials.
In the above method, the fluorescent material for the substrate includes material selected from the group consisting of naturally occurring fluorescent proteins, animal-based proteins, plant-based proteins and pigments, lipids, genetically hybridized fluorescent proteins, edible polymers, and fluorescent food coloring dyes.
In the above method, the watermarked image is printed by an inkjet printer or laser printer.
In the above method, the correction of the watermarked image is further based on correcting geometrical distortions which have occurred during the printing and the obtaining steps.
In the above method, the color correction of the watermarked image based on the established correspondence includes establishing a relationship between International Commission on Illumination (CIE) RGB color values of colors in the color reference chart as originally printed and the obtained CIE RGB color values.
In the above method, the established relationship is based on:
where T and M are 3×m matrices of the original CIE RGB color values of m different reference colors in the color reference chart and the obtained RGB color values of m different reference colors in the color reference chart, and C is a conversion matrix that is used to correct color values of the obtained watermarked image back to the originally printed CIE RGB color values of the watermarked image.
In the above method, the step of extracting the embedded predetermined watermark from the corrected watermarked image is carried out at a first location associated with a priori knowledge of the predetermined watermark.
The above method, further including communicating identity, verification, and authentication of the product from the first location to a second location.
In the above method, the first location is related to where the watermarked image was first printed and the second location is related to wherein the watermarked image was obtained.
In the above method, the first location is related to where the watermarked image was obtained and the second location is related to where the watermarked image was printed.
In the above method, the communicated identity, verification, an authentication include information relevant to the product.
In the above method, the product is one of a pharmaceutical or a food product.
In the above method, the information relevant to the pharmaceutical includes one or more of name of the pharmaceutical, dosage, date of manufacture, date of expiration, location of manufacture, warning of interactions with other drugs, or base ingredients.
In the above method, the information relevant to the pharmaceutical along with information related to where the watermarked image was obtained are communicated to a third location.
In the above method, the third location includes one or more of a physician's office, a medical care facility, a central medical record facility, or the first location.
In the above method, the color reference chart includes metadata associated with the CIE RGB color values of the color reference chart.
In the above method, the metadata is used to establish degradation of the obtained watermarked image since it was first printed on the product.
In the above method, the degradation is result of exposure to one or more of light, moisture, or chemical contaminants.
In the above method, the metadata is used to correct color distortion of the obtained watermarked image since it was first printed on the product.
In the above method, the degradation is result of color distortions during printing and photo acquisition.
In the above method, determining identity of the product includes one or more of i) validation associated with ensuring that the identity represents actual product issued by an actual manufacturer, ii) verification associated with ensuring the identity is corresponding to a particular product; or iii) authentication associated with ensuring the determined product is the product for which the predetermined watermark was embedded in the printed watermarked image.
In the above method, the genetically hybridized fluorescent silk proteins include silk fibroin with one or more of enhanced green fluorescent protein (eGFP), far-red fluorescent protein (mKate2), enhanced cyan fluorescent protein (eCFP), and enhanced yellow fluorescent protein (eYFP).
In the above method, the edible polymers include one or more of hydrocolloids: Starch, cellulose derivatives, chitosan, pectin, alginates, gums, and carrageenans.
In the above method, the animal-based proteins include one or more of silk, gelatin, collagen, albumin, and milk protein.
In the above method, the plant-based proteins include one or more of zein, soy, wheat gluten, and lectins.
In the above method, the lipids include one or more of fatty acids, triglycerides, and phospholipids.
In the above method, the fluorescent food dyes and plant pigments include one or more of Citrus Red (FD&C Red #2), Erythrosine B (FD&C Red #3), Allura Red (FD&C Red #40), Brilliant Blue (FD&C blue #1), Indigo carmine (FD&C blue #2), Fast Green (FD&C Green #3), Orange B, Tartrazine (FD&C Yellow #6), Sunset Yellow (FD&C Yellow #6), Tartrazine (FD&C Yellow #5), Betaxanthins, Chlorophyll, Carotenoids, Anthocyanin, Betalains, Fisetin, Quercetin, Quinine, Curcumin, Anthocyanin, Riboflavin, Vanillin, and Benzaldehyde.
In the above method, the established relationship is further based on incorporating color distortion between the originally printed watermarked image and the obtained CIE RGB color values of the watermarked image.
In the above method, the M3×m matrix is expanded into an Mp×m matrix, and thereby the established relationship is based on:
T3×m=C3×p Mp×m, wherein the p rows in Mp×m include polynomial or root-polynomial expansion terms of CIE RGB values to thereby incorporate a nonlinearity in addition to the CIE RGB color values of the obtained watermark image.
In the above method, an inverse of an expanded conversion matrix C is solved using a least-squares method.
In the above method, the least-squares method includes one of QR decomposition or Moore-Penrose pseudo inverse.
In the above method, the edible ink includes water, glycerin (polyalcohol), ethanol (monoalcohol), FDA-approved color dyes, and a preservative (polysorbate 80, propylene glycol).
In the above method, the FDA-approved color dyes include one or more of FD&C Red #2/Red #3/Red #40, Blue #1/Blue #2, Green #3, Orange B, and Yellow #5/Yellow #6.
In the above method, the preservative includes methylparaben.
For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of this disclosure is thereby intended.
In the present disclosure, the term “about” can allow for a degree of variability in a value or range, for example, within 10%, within 5%, or within 1% of a stated value or of a stated limit of a range.
In the present disclosure, the term “substantially” can allow for a degree of variability in a value or range, for example, within 90%, within 95%, or within 99% of a stated value or of a stated limit of a range.
A novel approach is presented herein to provide dosage-level or item-level (collectively, on-dose) anticounterfeit measures and authentication features that can provide protection against counterfeited pharmaceutical and food-related products. Towards this end, each product can be verified and authenticated even if it is separated from its package. Patients and consumers can identify their medicines and food product in real time with important dose information and food product history such as location of origin, date of manufacture or harvest, intermediate handling, etc. at the point of administration or consumption. Furthermore, such on-dose authentication for individual medicines or food products allows patients or consumers to serve as the last line of defense and actively participate in combating the prevalence of illicit pharmaceutical and food-related products. In addition, companies can implement serialization directly to individual medicines and food products, thereby enabling the tracking and tracing of medicines and foods to ensure safety, quality, and brand protection.
To implement the above-referenced on-dose features, it is necessary to ensure the nontoxicity and edibility (or digestibility) of constituent materials without compromising the safety or introducing interactions with main ingredients. Another requirement of on-dose features is a physical form because the security measure is directly applied to the solid oral dosage form (e.g. pill, tablet, or capsule) or affixed to the individual medicine, or food product. Among encryption and steganography methods, digital watermarking provides an immediately available security solution for copyright protection, identification, and authentication. The use of digital watermarking to protect medical images and diagnostic results has been widely recommended. However, as a purely digital process, digital watermarking does not involve any physical form. Physical watermarking, which has been extensively used for high-value objects (e.g. currency, document, and artwork) to discourage and identify counterfeiting, lacks digital and cyber aspects.
In the present disclosure, we introduce edible print-camera watermarking by combining the digital and physical properties of watermarking to establish anticounterfeit measures and authentication features at the dosage level (i.e., individual pharmaceutical or food product level) instead of the secondary package level. First, we refine an inkjet printing method to use FDA-approved food coloring as edible ink. Examples of edible ink are many, however several are provided in patent references including WO1998029514A1, WO2005122784A1, US20050061184A1, WO2004003089A1, and JP2001011355A, each of which is incorporated by reference in its entirety into the present disclosure. Inkjet printing is scalable for printing edible materials and can be implemented for additive manufacturing. Second, as an edible and digestible natural biopolymer, we use fluorescent protein, e.g., silk fibroin genetically hybridized with enhanced green fluorescent protein (eGFP) to fabricate an attack-resistant print sheet. Fluorescent substrate is a robust attack-resistant material since it shows unique optical properties that have absorption and emission in visible light (wavelength range of 400-700 nm). As another candidate, a colored material also can protect against a copy or scan attack because it has light absorption at visible wavelengths of 400-700 nm. In other words, such materials with light absorption and/or emission work as an attack-resistant factor during scanning based on white light for a copy of the original watermarked image. Third, we enhance the robustness of digital watermark extraction against significant color distortions that occur during the print-camera process. In particular, an integrated color correction using reference colors printed on a watermarked taggant is developed to ensure the reliability of edible watermarked taggants under scenarios involving different smartphone models and light conditions. Finally, for the edible watermarked taggants affixed to individual solid oral dosage forms, we investigate the readout repeatability of the watermark extractions and printing reproducibility of statistically identical watermarked taggants.
Thus, an inkjet printer using safe food coloring is adapted to print a watermarked image on a recombinant luminescent taggant to enhance attack resistance. Machine learning of color accuracy recovers unavoidable color distortions during printing and acquisition, allowing robust smartphone readability. An edible watermarked taggant affixed to each individual medicine or food product can offer anticounterfeit and authentication features at the dosage level, empowering every patient and consumer to aid in abating illicit medicines and food products.
Referring to
Thus,
As alluded to above, to ensure resistance to a copy (scan-print) attack, a fluorescent material, e.g., a biopolymer, is used to make a print sheet on which the edible watermarked image (i.e., the watermark image and the host image) is printed. There are many such edible fluorescent materials. One such example, is silk proteins genetically fused with eGFP which provide several advantages. From an edibility standpoint, silk fibroin is not only biocompatible with low immunogenicity and minimal inflammatory responses, but is also generally recognized as safe as designated by FDA (i.e., Generally Recognized as Safe (GRAS)). The commonly applied silk fibroin extraction method does not introduce heavy metals and toxic trace elements. Moreover, silk fibroin is easily digestible in the presence of proteolytic enzymes (e.g. pepsin or trypsin) produced in the gastrointestinal tract. Fluorescent proteins are often consumed as genetically modified dietary products. In addition, eGFP does not have common allergen epitopes. From a manufacturing standpoint, the genetic fusion (i.e. eGFP silk fibroin) of silk fibroin and eGFP can be readily realized via the piggyBac transposon method or clustered regularly interspaced short palindromic repeats (CRISPR) methods, both known to a person having ordinary skill in the art, which allow mass, scalable, and sustainable production.
To produce eGFP silk, we fuse the eGFP gene with the N-terminal and C-terminal domains of the silk fibroin heavy (H)-chain promoter (pFibH) using the piggyBac transposon method, creating a p3×P3-DsRed2-pFibH-eGFP transformation vector, as shown in
Specifically, to examine the digestibility of edible watermarked taggants, the enzymatic degradation and denaturation of eGFP silk proteins are investigated using two major proteolytic enzymes (i.e. pepsin and trypsin) produced in the gastrointestinal tract under physiologically relevant conditions. Notably, eGFP fluorescence can serve as a biomarker for quantifying protein denaturation and degradation, because the protein unfolding of the eGFP chromophore from proteolytic enzyme exposure or any damage in the tertiary structure results in a loss of fluorescence. After immersing the eGFP silk fibroin print sheets into each enzyme solution, we monitor the fluorescence emission intensity of eGFP at 525 nm upon 470-nm excitation. Compared to that associated with the pH 2.2 and pH 7.2 buffer solutions, the fluorescence of the eGFP silk fibroin print sheets immersed in 0.25% trypsin (pH 7.2) and 0.1% pepsin (pH 2.2) enzymes is considerably decreased after 90 min. In other words, the protein-based print sheet can be easily digested in the gastrointestinal tract after oral intake.
Referring to
The fabrication process of the eGFP silk fibroin, according to one embodiment, is shown in
To fabricate safe edible watermarked taggants for on-dose applications, we use commercially available FDA-approved food coloring dyes formulated in a food-grade laboratory. These edible dyes have the appropriate physical properties (e.g. viscosity <16 mPa·s) to be compatible to inkjet printer ink. Inkjet printing is an attractive option to ensure the scalable printing of edible taggants and can potentially be integrated into an additive pharmaceutical manufacturing process. However, it is challenging to print images or patterns using these edible coloring dyes on an eGFP silk fibroin print sheet. Edible coloring dyes were originally developed for frosting sheet printing. The inkjet printing ability is affected by the surface properties of print sheets. It should be noted that the surface of eGFP silk fibroin films is water repellent. The main ingredients of each color ink solution and the corresponding Food, Drugs and Cosmetics (FD&C) color information are provided in Table 1.
Referring to
The optimal density of droplets in the proposed inkjet printing method (also known as halftoning in inkjet printing) was determined. A simple square pattern with a green color is printed using the food coloring dyes on eGFP silk fibroin sheets at opacity levels ranging from 20 to 100% in the CIE RGB space (see
To extract the embedded watermark image from image acquired by the smartphone, it is critical to correct color artifacts and geometrical distortions that occur during printing and image acquisition. Both geometrical distortion realignment and color correction are needed (steps 118, 120, and 122 shown in
Photographs obtained using a smartphone camera or any digital camera exhibit different colors and brightness depending on the models and ambient light conditions during such image acquisition. In particular, each digital camera and smartphone model (i.e. three-color image sensors or trichromatic cameras) has unique spectral response functions (also known as spectral sensitivity) in the red (R), green (G), and blue (B) channels. A white balance is often used to adjust colors to represent a natural appearance; however, this aspect is not sufficient to compensate for color distortions. Notably, print-camera watermarking involves not only imaging but also printing, both of which are intrinsically lossy processes for color integrity. This color correction is shown in
In particular,
To recover the true RGB color values significantly distorted during printing and image acquisition, we incorporate machine learning of fixed-design regression in which polynomial features capture a nonlinear relationship between the original CIE and acquired RGB color values resulting from diverse spectral responses (or sensitivity) functions of different smartphone models. Specifically, we implement a color correction methodology by adding 32 primary colors into the periphery of a watermarked image (see
where T and M are 3×m matrices of the original CIE and measured RGB color values for m (e.g., 32) primary colors, respectively. By solving Equation (1) for the unknown matrix C, C can be used to convert the measured RGB color values of a watermarked image to the CIE RGB color space. First, we use the CIE RGB color space as a reference space because the CIE RGB color space defines physiologically perceived colors in the human visual system on the basis of the electromagnetic spectrum, and other color spaces are often derived from the CIE RGB (or XYZ) color space. Second, to incorporate nonlinearity between the original CIE and measured RGB color values, M3×m can be expanded to Mp×m based on:
where the p rows in Mp×m include the polynomial terms and cross-terms in addition to the RGB color values (see the Experimental Section). Third, an inverse of the expanded conversion matrix C in Equation (2) can be solved using a least-squares method (i.e. I2-norm minimization), such as QR decomposition or Moore-Penrose pseudo inverse.
Next, we evaluate the color correction ability of representative print-camera cases through leave-one-out cross-validation (see
The watermark image is next extracted from the acquired image after the realignment and the color correction, discussed above. To evaluate the accuracy of watermark image extraction, we use a structural similarity index that can comprehensively quantify the image luminance, contrast, and structural pattern in comparison with the original watermark. Referring to
Referring to
To demonstrate the robustness of resistance to a scan-copy counterfeiting attack. an experiment was conducted assuming that a counterfeiter has full access to all of the technologies used in the present disclosure, including the FDA-approved edible dyes, the inkjet printer, and the eGFP silk fibroin print sheets. Specifically, we presume that a counterfeiter uses a high-resolution digital scanner (600 dots per inch, DPI) to obtain a digital watermarked image from the original watermarked taggant and then reprints the scanned digital watermarked image on an eGFP silk fibroin print sheet. In this simulated case, the average structural similarity indices between the original and extracted watermark images duplicated by the scan-print processes are drastically low, with structural similarity index values of 0.20±0.04. 0.21±0.03, 0.21±0.04. 0.20±0.02, and 0.20±0.03 (mean±standard deviation) for the five smartphone models (see
Silkworm transgenesis for eGFP silk fibroin is provided herein. We obtained eGFP silk from transgenic silkworms expressing eGFP by constructing a transformation vector pBac-3×P3-DsRed2-pFibH-eGFP. First, to form the fibroin promoter, a DNA fragment containing the promoter domain (1,124 base pairs (bp)) and the N-terminal region (1,430 bp) with intron (972 bp) of the fibroin heavy (H) gene [GenBank Accession, nucleotides (nt) 61,312 to 63,870 of No. AF226688] was amplified by polymerase chain reaction (PCR) using the genomic DNA from Bombyx mori and primers (pFibHN-F: 5′-GGCGCGCCGTGCGTGATCAGGAAAAAT-3′ and pFibHN-R: 5′-TGCACCGACTGCAGCACTA GTGCTGAA-3′). This DNA fragment was cloned into the pGEM-T Easy Vector System (Promega Co., Madison, WI, USA). The resulting plasmid was designated as pGEMT-pFibH-NTR. The DsRed2 cDNA used as a marker was amplified by PCR using specific primers with Nhel/AfII sites from pDsRed2-C1 (Nhel-DsRed2-F: 5′-GCTAGCATGGCCTCCTCCGAGAAC-3′ and DsRed2-AflII-R: 5′-CTTAAGCTACAGGAACAGGTGGTGGCG-3′; Clontech, Mountain View, CA, USA). The resultant DNA was cloned into the pGEM-T Easy Vector System, which was named as pGEMT-DsRed2. The DsRed2 gene was excised from pGEMT-DsRed2 digested with restriction enzymes of Nhel/AflII and replaced with the cGFP gene from pBac-3×P3-eGFP to form pBac-3×P3-DsRed2. The DNA fragment included the 180 bp of 3′ terminal sequence of the H-chain gene open reading frame and the 300 bp of 3′ region of the fibroin H gene (GenBank Accession, nt 79,021 to 80,009 of No. AF226688). This DNA fragment was amplified by PCR using genomic DNA isolated from Bombyx mori silkworm and primers (pFibHC-F: 5′-AGCGTCAGTTACG GAGCTGGCAGGGGA-3′ and pFibHC-R: 5′-TATAGTATTCTTAGTTGAGAAGGCATA-3′). The produced DNA fragment was cloned into pGEM-T Easy Vector System, resulting in pGEMT-CTR. The fragments were prepared by restriction enzyme treatment for pGEMT-pFibH-NTR with Ascl/BamHI and for pGEMT-CTR with Sall/Fsel, respectively. These fragments were cloned together in a pBluescriptIl SK (-) vector (Stratagene, CA, USA) treated with restriction enzymes with Apal/NotI, named as pFibHNC-null. The N- and C-terminals had the NotI and Sbfl restriction sites, respectively. The eGFP gene fragment without a termination codon was amplified from peGFP-1 using primers (eGFP-F: 5′-CGGCCGCATGGTGAGCAAGGGCGAGGAG-3′ and eGFP-R: 5′-GCTGAGGCTTTGTACAGC TCGTCCAT-3′), cloned into pGEM-T Easy Vector. This fragment was treated with NotI/Bbvcl, and then cloned into pFibHNC-null vector digested with NotI/Bbvcl, producing pFibHNC-eGFP. Finally, pFibHNC-eGFP was restriction enzyme-treated with Ascl/Fsel and was subcloned into pBac-3×P3-DsRed2 digested with Ascl/Fsel, obtaining the transformation vector pBac-3×P3-DsRed2-pFibH-eGFP.
Those having ordinary skill in the art will recognize that numerous modifications can be made to the specific implementations described above. The implementations should not be limited to the particular limitations described. Other implementations may be possible.
The present non-provisional patent application is related to and claims the priority benefit of U.S. Provisional Patent Application Ser. No. 63/301,426, entitled CYBER-PHYSICAL WATERMARKING WITH INKJET EDIBLE BIOPRINTING which was filed Jan. 20, 2022, the contents of which are hereby incorporated by reference in its entirety into the present disclosure
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US2022/053918 | 12/23/2022 | WO |
Number | Date | Country | |
---|---|---|---|
63301426 | Jan 2022 | US |