BACKGROUND
Field
The present disclosure relates to systems and methods for identifying particular types of data from larger sets of data. More specifically, the present disclosure relates to systems and methods for automatic Uniform Resource Locator (URL) identification from data.
Related Art
A URL is a specific type of data which identifies a web resource, such as a web page, a File Transfer Protocol (FTP) site, an e-mail, a database, or other web resource. Typically, URLs are used by web browsers to access one or more web pages. URLs are in heavy use in today's web-based and cloud-based computing environments, and follow a specific syntax in order to address a specific web resource.
In various fields, it is very useful to be able to rapidly and accurately identify URLs from large volumes of data. For example, in the retail field, it would be beneficial to provide a computer-based system which can rapidly process large volumes of merchant data, with high precision, to identify one or more merchant URLs. Such merchant URLs, once identified, can be utilized for a variety of purposes, such as monitoring a merchant's online portfolio (accessed via the identified URL) for one or more violations by the merchant, enforcement of agreement terms, etc. Often, URLs are not provided by merchant and other parties to agreements. As a result, the task of identifying merchant URLs (or URLs of other parties) from large volumes of data is performed manually. This results in undue cost as well as lost time.
Accordingly, what would be desirable are systems and methods for automatic URL identification from data, which address the foregoing, and other needs.
SUMMARY
The present disclosure relates to systems and methods for automatic URL identification from data. The system receives and processes one or more sources of data, such as merchant data, and processes the input data to identify one or more URLs present in the data. The identified URLs are automatically validated by the system using one or more fuzzy and/or exact matching algorithms. The validation could be performed by matching one or more non-URL data items, such as a business name, address, e-mail, country, zip code, or any other suitable non-URL data item, to ensure that only valid URLs are identified. Once the URLs are validated, a report is generated by the system and delivered to a recipient. The report could be electronically transmitted to a recipient data processing system, such as a merchant e-commerce platform.
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing features of the invention will be apparent from the following Detailed Description, taken in connection with the accompanying drawings, in which:
FIG. 1 is a flowchart illustrating processing steps carried out by the systems and methods of the present disclosure;
FIG. 2 is a flowchart illustrating the processing steps of FIG. 1, in greater detail;
FIGS. 3-4 are diagrams illustrating sample work flows carried out by the systems and methods of the present disclosure;
FIG. 5 is a diagram illustrating processing phases carried out by the systems and methods of the present disclosure;
FIG. 6 is a table illustrating various validation rules that can be utilized by the systems and methods of the present disclosure to validate identified URLs; and
FIG. 7 is a diagram illustrating sample hardware and software components which can be utilized to implement the systems and methods of the present disclosure.
DETAILED DESCRIPTION
The present disclosure relates to systems and methods for automatic URL identification from data, as discussed in detail below in connection with FIGS. 1-7.
FIG. 1 is a flowchart illustrating processing steps carried out by the systems and methods of the present disclosure, indicated generally at 10. In step 12, the system receives input data to be processed. Such data can be stored in one or more databases (e.g., relational database, etc.) and accessed by and/or transmitted to the system in either CSV or JSON format. Such data can include, but is not limited to, merchant data such as merchant address, zip code, phone number, email, business description, etc. In step 14, the system allows a user to conduct one or more searches through the merchant data using one or more search combinations. For example, the user can search using one or more (or combinations of) the following queries: business name (“Doing Business As” (DBA)), telephone number, postal code, address, city, and any other suitable queries. In step 16, the system searches for (“scrapes” for) one or more matching URLs, using the search combinations identified in step 14. In step 18, the system validates any returned matching URLs using one or more non-URL data items, such as a phone number, an address, a business name, postal code, or other information corresponding to the returned record. If the matching URLs are validated, transmits the matching URLs in step 20. Importantly steps 12-20 can be performed in parallel (e.g., in a multiprocessing environment, using a plurality of processors, processing cores, processing threads, etc.) in order to speed up searching for URLs by the system.
FIG. 2 is a flowchart 30 illustrating the processing steps of FIG. 1 in greater detail. In step 32, the system receives input data to be processed. As noted above, such data can be stored in one or more databases (e.g., relational database, etc.) and accessed by and/or transmitted to the system. In step 34, the system processes the input data using a column mapping process which includes standardization and mapping of incoming columnar metadata to existing column metadata in the system, and also performs any required data preprocessing such as cleaning and standardization of metadata. In step 36, the system allows the user to build one or more search combinations as discussed above in connection with FIG. 1. In step 38, the system accesses a search engine using a suitable web browser and, optionally, an application programming interface (API) call. For example, the system can access the Microsoft Bing search engine using a secure web browser, such as the Tor secure web browser and a suitable API call to the search engine. In step 46, the system fetches a URL list from the search engine via the web browser. Optionally, the URL list could be limited to all URL hits for a particular region, such as in a country. In step 44, the system removes prohibited (“blacklisted”) URLs from the list, if such prohibited URLs exist. In step 42, the system selects the top 8 URLs in the list and preserves the order of the list. Of course, any other number of URLs could be selected.
In step 40, the system “scrapes” each URL content including the URL's main page content and the “Contact US” page content in the list (one by one). In step 48, the system validates the incoming metadata (which could be standardized) against URL content using one or more matching algorithms, which could apply fuzzy (approximate) or exact matching processes to the URLs. The matching process follows the logic of FIG. 6 and scores URL content to determine how relevant the scraped URL content is to the merchant metadata. If validation of the URLs is unsuccessful based on the score assigned in the matching process, step 50 occurs, wherein the system scrapes other web pages of the URL. Then, in step 52, the system validates the URL content again using a fuzzy or exact matching process.
If the URLs are successfully validated, step 54 occurs, wherein the system appends the URLs to the input data obtained in step 32. This could be performed by appending the URLs to columns of data in the input data. Finally, in step 56, the system generates and transmits an output file which includes the URLs and the input data.
FIGS. 3-4 are diagrams illustrating sample workflows carried out by the systems and methods of the present disclosure. As shown in the workflow 60 of FIG. 3, the system executes a full-cycle work flow 62, which includes a pre-processing step 64, an automation cycle 68, a review cycle 78, and a delivery and integration step 84. Specifically, in pre-processing step 64, the system carries out one or more pre-processing steps on the input data, including, but not limited to, client-level sorting, standardization of columns, and preparation of files. In step 66, the system determines whether a master file exists. If so, step 74 of the automation cycle occurs, wherein the master file is retrieved and processed by the system. Then, in step 76, a python tool is executed, which is a predefined process that looks up incoming merchant metadata and validates the metadata against IP addresses stored in a database. This process also involves strict look-up and cross-checking of URL addresses based on one or more of a business name and an e-mail. Then, step 72 occurs, wherein further processing of the master file occurs as discussed below in connection with FIG. 4.
The review cycle 78 includes a quality assurance (QA) review process 80, wherein the system reviews and confirms the accuracy of the URLs returned by the system. In step 82, the system also optionally allows for a manual review process, wherein one or more users of the system can manually review and confirm the accuracy of URLs returned by the system. In step 84, once the QA review process 80 is complete, the system generates and delivers a report that includes and summarizes all of the URLs returned by the system. Optionally, in step 80, the system can determine whether an e-commerce platform is in communication with the system of the present disclosure.
As shown in the workflow 72 of FIG. 4, the system performs the URL identification processes discussed above in connection with FIGS. 1-2. Specifically, in step 90, the system obtains a standard input file, which could be stored in any suitable format such as an Excel spreadsheet, a comma-separated value (CSV) file, etc. In step 92, the system performs the match processes discussed above in connection with FIGS. 1-2 (e.g., in connection with a search engine such as the Microsoft Bing search engine or any suitable search engine). Then, in step 94, the system performs the validation processes discussed above in connection with FIGS. 1-2. Finally, in step 96, the system generates an output file in a suitable format (which includes the identified URLs), including, but not limited to, a .JSON file, a .CSV file, or any other suitable format. Additionally, it is noted that the system can flag one or more of the URL results, and/or generate a reason as to why the URL match was identified.
FIG. 5 is a diagram 100 illustrating processing phases carried out by the systems and methods of the present disclosure. The processes include the input processes 102 discussed above in connection with FIGS. 1-4, one or more parallel processes 104, which could include the manual review process, the system processes discussed in connection with FIGS. 1-4, and one or more master files. Additionally, the parallel processes 104 could also include the python tool 76 discussed above. Advantageously, since the processes 104 could be carried out in parallel, system processing speed is greatly increased. The process 106 allows for quality assurance (QA) processes discussed above in connection with FIG. 3. Finally, in process 108, the results are delivered by the system (e.g., electronically in a file/report).
FIG. 6 is a table 110 illustrating various validation rules that can be utilized by the systems and methods of the present disclosure to validate identified URLs. Examples of various criteria that can be used to validate the URLs include, but are not limited to, business or merchant name or DBA name or “transacting business as” (T/A) name, street address, city, postal code, state or province, telephone number, e-mail, name, business description, or country. Any desired combinations can be used, and can be toggled as desired in order to “tweak” the validation processes and/or accuracy thereof.
FIG. 7 is a diagram 120 illustrating sample hardware and software components which can be utilized to implement the systems and methods of the present disclosure. The system can scrape URL information from one or more websites 122a-122n (n being any desired number), which can communicate with a URL identification processor 126 via a network connection 124 (e.g., the Internet, a wide area network (WAN), a local area network (LAN), a wireless network, an optical network, etc.). The URL identification processor 126 could include a hardware processor such as a computer system, computer server, cloud processing service, mobile device, etc., which executes system code 128 programmed in accordance with the processes discussed herein in connection with FIGS. 1-6. The system code 128 could comprise non-transitory, computer-readable code stored on one or more computer-readable media capable of being accessed by the processor 126, including, but not limited to, random-access memory (RAM), read-only memory (ROM), electrically-erasable programmable ROM (EEPROM), non-volatile (NV) memory, flash memory, disk storage, tape storage, or any other suitable memory capable of being accessed by the processor 126. Additionally and/or alternatively, the systems and methods discussed herein could be implemented as one or more customized hardware components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other suitable customized hardware component.
Having thus described the system and method in detail, it Is to be understood that the foregoing description is not intended to limit the spirit or scope thereof. It will be understood that the embodiments of the present disclosure described herein are merely exemplary and that a person skilled in the art may make any variations and modification without departing from the spirit and scope of the disclosure. All such variations and modifications, including those discussed above, are intended to be included within the scope of the disclosure. What is desired to be protected by Letters Patent is set forth in the following claims.
Patent Application
20230161831
