The invention relates to systems, methods and interfaces for managing information technology systems, and is more particularly, but not by way of limitation, directed to technology for schedule migration and validation for distributed systems and mainframe systems.
Application workflow orchestration platforms allow businesses to run hundreds of thousands of batch jobs and use data to optimize complex business operations. Workload management software may define, monitor and manage scheduled and event-based workloads across enterprise applications in different operating systems, hybrid and multi-cloud environments. Tools and systems may provide access to enterprise batch jobs through graphical interfaces. Some workflow management software may integrate with distributed storage systems. Distributed systems and mainframe systems require different schedules, but a same workflow management system may be required to schedule jobs across these systems. Workflow automation systems (e.g., BMC Control-M (CTM), Broadcom's ESP) provide different capabilities and cost advantages.
Accordingly, there is a need for tools, systems, methods and interfaces for migrating and/or validating schedules generated by different workflow automation systems for mainframe and/or distributed systems. Some embodiments ensure that job linkages between different schedulers are appropriately configured with proper predecessor and successor relationships. To facilitate efficient verification of migrated schedules, a schedule validator may be provided to application owners.
One or more embodiments of the invention are directed to an improved method and system for migrating and/or validating schedules for mainframe systems and/or distributed systems. The method may be performed at a server having one or more processors, memory, one or more displays, and one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for performing the steps described herein. The method may include interfacing with a plurality of schedulers including a source scheduler and a target scheduler. Each scheduler may be configured to schedule jobs across one or more distributed systems and one or more mainframe systems. The method may include obtaining a source schedule generated by the source scheduler and a target schedule generated by the target scheduler for a same system of the one or more distributed systems and one or more mainframe systems. The method may include determining validity of migration of jobs between the source schedule and the target schedule. The method may include, in accordance with a determination that one or more jobs in the target schedule are not valid, transmitting an indication to the target scheduler causing the target scheduler to correct the target schedule.
In some embodiments, the method may further include, prior to determining validity of migration of jobs, cleansing and pre-processing the source schedule and the target schedule.
In some embodiments, cleansing and pre-processing the source schedule and the target schedule includes eliminating predetermined schedules or invalid schedules.
In some embodiments, determining validity of migration of jobs may include identifying correct predecessors for each job based on whether a specific predecessor job is listed as a successor in any other job's definition.
In some embodiments, determining validity of migration of jobs between the source schedule and the target schedule may include using a first set of rules for schedules for the one or more distributed systems and a second set of rules for schedules for the one or more mainframe systems.
In some embodiments, either (i) the source scheduler generates schedules that have job names with at least a predetermined number of characters and the target scheduler generates schedules that uses application identifiers having a predetermined size, or (ii) the target scheduler generates schedules that have job names with at least a predetermined number of characters and the source scheduler generates schedules that uses application identifiers having a predetermined size.
In some embodiments, determining validity of migration of jobs may include in accordance with a determination that the source schedule and the target schedule corresponds to the one or more distributed systems, using a distributed system schedule validation pattern. In some embodiments, determining validity of migration of jobs may include in accordance with a determination that the source schedule and the target schedule corresponds to the one or more mainframe systems, using a mainframe system schedule validation pattern.
In some embodiments, determining validity of migration of jobs may include identifying one or more mismatches between the source schedule and the target schedule using one or more pattern matching techniques.
In some embodiments, determining validity of migration of jobs may include identifying one or more source schedule matching parameters and one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying matching schedules from the one or more source schedule matching parameters and the one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying non-matching schedules from the one or more source schedule matching parameters and the one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying non-matching schedules in the one or more source schedule matching parameters from the one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying non-matching schedules in the one or more target schedule matching parameters from the one or more source schedule matching parameters.
In some embodiments, determining validity of migration of jobs may include extracting a first list of predecessors for the source schedule and a second list of predecessors for the target schedule, and comparing the first list and the second list to generate a comparison report.
In some embodiments, the method may further include generating a report based on predecessor jobs of the source schedule and/or the target schedule with broken linkages.
In some embodiments, the method may further include providing an interface to query job names to retrieve schedule details for the source schedule and the target schedule, and/or in response to receiving a job name, returning schedule details for the source schedule or the target schedule corresponding to the job name.
In some embodiments, a computer system has one or more processors, memory, and a display. The one or more programs include instructions for performing any of the methods described herein.
In some embodiments, a non-transitory computer readable storage medium stores one or more programs configured for execution by a computer system having one or more processors, memory, and a display. The one or more programs include instructions for performing any of the methods described herein.
The following descriptions of embodiments of the invention are exemplary, rather than limiting, and many variations and modifications are within the scope and spirit of the invention. Although numerous specific details are set forth in order to provide a thorough understanding of the present invention, it will be apparent to one of ordinary skill in the art, that embodiments of the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail in order to avoid unnecessarily obscuring the present invention.
One or more embodiments of the invention are directed to an improved method and system for migrating and/or validating schedules for distributed systems and/or mainframe systems. Some embodiments verify that job linkages between different schedulers are accurately configured with proper predecessor and successor relationships. Some embodiments facilitate efficient verification of migrated schedules by providing a schedule validator for application owners. Some embodiments determine whether a specific predecessor job is listed as a successor in any other job's definition. Some embodiments use multiple validation iterations rather than text parsing, which can be error-prone in identifying predecessors of a job.
Some embodiments facilitate migration and/or validation for mainframe and distributed systems. The pre-requisites for these systems may be different. For example, for distributed systems, some schedulers may use a job name with a predetermined number of characters (e.g., more than 55 characters), while other schedulers may use an application identifier (e.g., an 8-byte application identifier). Some embodiments compare source and target schedules having different formats. Some embodiments validate all jobs scheduled by a scheduler to identify discrepancies. Some embodiments identify broken linkages on converted (or migrated) schedules. Some embodiments use knowledge of job linkages defined in different schedulers. In some embodiments, a platform features a user-friendly interface that enables end-users to provide necessary inputs to identify differences at the job level, minimizing the need for validation intervention. Some embodiments allow users to provide highly refined job dependencies, improving accuracy and efficiency in validation and migration.
A mainframe schedule validator 106 and/or a distributed system validator 108 may be provided. The schedule validators may have different requirements, capabilities and/or output, as described below. After obtaining the source schedule (e.g., a schedule generated by a current system or scheduler) and the target schedule (e.g., a schedule generated by a target system or scheduler where the scheduling is migrated to), the schedulers perform schedule preprocessing (112). The preprocessing may include cleansing (114) of input schedules, data pre-processing (116) of the schedules, performing pattern matching (118) on the schedules, and generating (120) schedules (e.g., using SQLite). The data pre-processing may include formatting source and/or target dump files (e.g., files in XML format) to prepare files for merging and/or validation. For each job, the system may extract a list of predecessors for both source and target schedulers. Some embodiments reconcile differences in formats (e.g., CTM jobs use more than 55 bytes and ESP jobs use 8 bytes) by converting jobs to their corresponding job names and/or by performing a matching process.
For illustration, for cleansing/pre-processing that may require normalization across systems, suppose the migration is from CTM to ESP scheduler for both mainframe and distributed systems. In the case of mainframe systems, a source (e.g., CTM) schedule may include job names that are 8 bytes each, a target (e.g., ESP) schedule may include job names that are 8 bytes each. In the case of distributed systems, however, a source (e.g., CTM) schedule may include job names that are between 8 and 55 bytes each, and a target (e.g., ESP) schedule may include job names that are 8 bytes each. Since the source and target are not in same format, the target (ESP) schedule may be converted from the 55 byte job names to 8 byte job names to normalize the job names across the schedulers.
A schedule validator 124 may use scheduling information extracted from the schedules 120 to compare predecessor lists and/or generate a comparison report 126. The schedule validator 124 may compare source schedule matching parameter and target schedule matching parameter, to generate matching schedules 128 from the source schedule and the target schedule, non-matching schedules 130 from the source schedule and the target schedule, non-matching schedules 132 in the source schedule from the target schedule, and/or non-matching schedules 134 in the target schedule from the source schedule. Matching Parameters may include source predecessors (e.g., CTM predecessors), target predecessors (e.g., ESP predecessors), and/or job start times and end times for source and target. Comparison parameters may include CTM predecessors versus ESP predecessors, CTM job start time, end time versus ESP job start time, end time, and so on. Resulting set may include predecessor or time matching between the schedulers. For example, this may include predecessor time for CTM mismatch (e.g., available in CTM but absent in ESP), predecessor time for ESP Mismatch (e.g., available in ESP but absent in CTM).
Some embodiments generate comparison reports for schedule validation and/or migration, as described below in reference to
Some embodiments generate the comparison report between the different schedules (e.g., CTM and ESP) and/or reports for predecessor jobs with broken linkages from ESP after conversion. Some embodiments may identify mismatches between the schedulers. Some embodiments reduce the risk of scheduling inconsistencies across systems, mitigates the adverse effects on downstream processes. Each schedule validation may take several seconds (e.g., 40 seconds). Multiple executions of the schedule validator may improve accuracy. For example, the validation may be performed in stages, whereby a first stage provides an initial and/or quick validation of the schedules, followed by subsequent stages where the validation is performed more exhaustively. In some situations, an exhaustive validation process may be necessary so a complete migration may be performed. The validation may be performed in stages as applied to different portions of a schedule and/or different portions of different schedules. Some embodiments facilitate the retrieval of job schedule dependencies, allowing application owners and/or automatic schedule correction to validate and/or migrate job execution.
The method may include interfacing (802) with a plurality of schedulers including a source scheduler and a target scheduler (e.g., via the users 102). Each scheduler may be configured to schedule jobs across one or more distributed systems and one or more mainframe systems.
The method may include obtaining (804) a source schedule generated by the source scheduler and a target schedule generated by the target scheduler for a same system of the one or more distributed systems and one or more mainframe systems. For example, in
The method may include determining (806) validity of migration of jobs between the source schedule and the target schedule. In some embodiments, the method may further include, prior to determining validity of migration of jobs, cleansing and pre-processing the source schedule and the target schedule. In some embodiments, cleansing and pre-processing the source schedule and the target schedule includes eliminating predetermined schedules or invalid schedules. In some embodiments, determining validity of migration of jobs may include identifying correct predecessors for each job based on whether a specific predecessor job is listed as a successor in any other job's definition. In some embodiments, determining validity of migration of jobs between the source schedule and the target schedule may include using a first set of rules for schedules for the one or more distributed systems and a second set of rules for schedules for the one or more mainframe systems.
In some embodiments, determining validity of migration of jobs may include in accordance with a determination that the source schedule and the target schedule corresponds to the one or more distributed systems, using a distributed system schedule validation pattern. In some embodiments, determining validity of migration of jobs may include in accordance with a determination that the source schedule and the target schedule corresponds to the one or more mainframe systems, using a mainframe system schedule validation pattern.
In some embodiments, determining validity of migration of jobs may include identifying one or more mismatches between the source schedule and the target schedule using one or more pattern matching techniques. The pattern matching may include regular expression.
In some embodiments, determining validity of migration of jobs may include identifying one or more source schedule matching parameters and one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying matching schedules from the one or more source schedule matching parameters and the one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying non-matching schedules from the one or more source schedule matching parameters and the one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying non-matching schedules in the one or more source schedule matching parameters from the one or more target schedule matching parameters. In some embodiments, determining validity of migration of jobs may include identifying non-matching schedules in the one or more target schedule matching parameters from the one or more source schedule matching parameters.
In some embodiments, determining validity of migration of jobs may include extracting a first list of predecessors for the source schedule and a second list of predecessors for the target schedule, and comparing the first list and the second list to generate a comparison report.
Some embodiments compare and/or validate dataset triggers between the source schedule and the target schedule. For example, validation may be performed on execution of the application job which runs when the file is created/received from different systems/sources. Some embodiments compare and/or validate cyclic Jobs between the source schedule and the target schedule. For example, validation may be performed on execution of the application job which runs multiple times in a day (e.g., every 15 or 30 min depending on the definitions).
To validate accuracy, in some embodiments, the source scheduler and the target scheduler may be executed in parallel in separate test environments. Job logs from each of these environments may be captured and compared to identify discrepancy between pre- and post-migration processing. After the comparison results are consistently successful, target scheduler may be installed in the production environment. The source scheduler may be decommissioned post install.
The method may include, in accordance with a determination that one or more jobs in the target schedule are not valid, transmitting (808) an indication to the target scheduler causing the target scheduler to correct the target schedule. In some embodiments, the comparison reports described earlier may be used by a target scheduling team to fix broken linkages to align with a source schedule definition. For example, suppose there are 10 broken linkages issues between CTM (source) and ESP (target). A comparison report may be shared with the ESP to fix the issues in ESP to align the schedule with the CTM schedule.
In some embodiments, the method may further include generating a report based on predecessor jobs of the source schedule and/or the target schedule with broken linkages. In some embodiments, the method may further include generating a report identifying mismatches between the source schedule and the target schedule.
In some embodiments, the method may further include providing an interface to query job names to retrieve schedule details for the source schedule and/or the target schedule, and/or in response to receiving a job name, returning schedule details for the source schedule or the target schedule corresponding to the job name. The query interface may allow developers to directly access the scheduling definition without accessing mainframe systems. The developers may view a defined set of IN conditions and OUT conditions for end-to-end application analysis purposes.
Some embodiments handle different application groups in multiple servers. Applications may be hosted on different servers that may add an additional layer of complexity. Dependencies and configuration of each application group may need to be preserved and required to be compatible with the target environment. The performance and functionality of each application group may require focused validation to prevent any impact.
In some embodiments, job status information may be used to monitor the progress and outcome of the migration. This information may be used to troubleshoot any run-time issues that may arise. Job status information may also be used to validate that the migrated jobs are running as expected and producing the desired results.
Some embodiments ensure that events that trigger the jobs (for event-based workloads) are consistent and reliable across different environments.
Some embodiments perform validation across enterprise applications, across different operating systems, and/or hybrid and/or multi-cloud environments. The techniques described herein are flexible and may be applied across different environments, and may be applied simultaneously across such platforms or systems. For mainframe systems, the schedulers (e.g., CTM, ESP) may operate within the mainframe systems, whereas for distributed systems, schedulers (e.g., CTM) may operate outside the mainframe and the schedulers (e.g., ESP) may operate within the mainframe systems.
While embodiments and alternatives have been disclosed and discussed, the invention herein is not limited to the particular disclosed embodiments or alternatives but encompasses the full breadth and scope of the invention including equivalents, and the invention is not limited except as set forth in and encompassed by the full breadth and scope of the claims herein.
This application claims priority to U.S. Provisional Application No. 63/614,777, titled “Systems, Methods And Interfaces for Schedule Migration And Validation for Distributed Systems And Mainframe Systems,” filed Dec. 26, 2023, which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63614777 | Dec 2023 | US |