Beyond Job Scheduling: the Road to Enterprise Process Automation

Total Page:16

File Type:pdf, Size:1020Kb

Beyond Job Scheduling: the Road to Enterprise Process Automation BEYOND JOB SCHEDULING: THE ROAD TO ENTERPRISE PROCESS AUTOMATION Introduction ..........................................................2 Job Scheduling Defined...........................................3 The Evolution of Job Scheduling ...............................3 UNIX CRONtab ......................................................3 Distributed Scheduling Products ...............................4 Distributed RDBMS-based Scheduling .......................4 Application API’s ....................................................5 Enterprise process automation .................................5 Dynamic, Data-Driven Automation ...........................5 Conditional Business Rules ......................................7 Workload Balancing................................................7 End-user involvement.............................................8 Fast Flexible Implementations..................................8 Accelerated Information Delivery..............................9 Java-Enabled Client.............................................. 10 Interactive Graphical Management ......................... 10 Security and Auditability ....................................... 10 Conclusion .......................................................... 10 Evaluation Matrix ................................................. 11 About UC4 Software ............................................. 12 Copyright © Year 2008 UC4 Software (UC4), All Rights Reserved. The contents of this document are copyrighted by UC4 unless otherwise indicated. All rights are reserved. Any forms of copying other than an individual user’s personal reference without express written permission is prohibited. Further distribution of these materials is strictly forbidden, including but not limited to, posting, e-mailing, faxing, archiving in a database, redistributing via computer network or in a printed form. 1 INTRODUCTION For example, accounting might be scheduled on a The relentless pace of today’s business environ- calendar year basis and the emphasis would be on ment and its enormous pressures on IT staff has months, quarters, and fiscal year end. Alterna- made it critical for corporate business systems to tively, manufacturing might place all of its become more responsive and reliable. End-user emphasis on weekly periods for purposes of plan- expectations for reliability and speed grow dra- ning, inventory analysis, and production. matically as more and more users have come in Manufacturing would also be forced to deal with contact with a wider array of software applica- plant shutdowns and holiday periods that impact tions. Expectations now approach that of the production cycles, but may have little relevance telephone, with users demanding business sys- for accounting purposes. tems to be available instantly and operate flawlessly. Today’s supercharged environment requires com- panies to respond much more swiftly to changing Coincident with these rising user expectations are business conditions. Referring to the discussion the increasing complications of enterprise comput- above, it might still be viable for a company’s fis- ing. IT personnel have been forced to come to cal operations to run on a fairly routine basis. grips with the accelerating rates of change, and However, the realities of increased global competi- the demands of corporations everywhere to tiveness dictate a manufacturer to be linked quickly respond to the competitive threats and electronically with its business partners in an ex- opportunities brought about by e-business. tended “just in time” enterprise. To exploit the extended enterprise, the manufacturer must re- The rise of the web has proven to be the ultimate place or automate its manufacturing, ordering and nemesis for the so-called “batch window” - that inventory processing systems to respond dynami- nightly opportunity to run large groups of transac- cally to changing business requirements and tions offline. Instead we have seen the rise of the competitive pressures. “real-time organization” that needs to respond to changing business conditions quickly and effort- Improving business responsiveness through lessly in a 24 / 7 fashion. This creates significant automation requires a comprehensive approach challenges for corporations as they try to respond build in from the start. The selected solution must to the demands of near real-time processing with automate processing across a wide spectrum of their traditional batch-oriented back end systems. software, computing platforms, and business processes. Implementing a solution that moves In traditional enterprise computing, the availability beyond mere scheduling and fulfills the promise of of the batch window meant that a significant pe- enterprise process automation can mean the dif- riod of time could be set aside to launch, monitor ference between corporate success and failure. and manage a variety of batch/sequential activi- Traditional batch job scheduling products cannot ties that were necessary for the proper running of meet today’s enterprise automation needs, lead- the business. The new global business environ- ing to the creation of a new breed of software to ment, and in particular the Internet with its handle this increasingly complex and demanding implied on-line, real-time, 24x7 operation has role. dramatically altered this rigid notion of scheduling. The idea of having batch processes with a multi- Implied in this accelerated business climate is the tude of similar transactions that are processed at requirement that IT staffs must build computing night is still relevant in many cases, but there are infrastructures that create responsive platforms many other examples where traditional time- for change. Job scheduling tools with their rigid based, operations centric ‘scheduling” is wholly adherence to the limitations of automating tasks inadequate. Instead, companies require near or based on day, date and time don't meet this re- real-time processing. quirement. New solutions are required that can automate and accelerate existing business proc- Traditional batch scheduling held that business esses and adapt to new processes and systems as processes could be mapped out and executed in they arise. predictable ways that followed one or more busi- ness calendars. The dominant concept was that This white paper demonstrates the ever-increasing certain business activities and the software sys- value of adding enterprise process automation tems that supported those activities operated in software to the IT infrastructure. It traces the sequential, predictable ways. Scheduling tools evolution of traditional batch scheduling, illus- were only required to support notions of day, trates where leading edge products are today, and date, time and multiple business calendars to outlines a set of requirements for moving beyond support corporate and/or departmental needs. batch processing and into the realm of true enter- prise process automation. The paper concludes by 2 illustrating how UC4 Workload Automation Suite and shortcomings of various approaches. In the addresses these advanced product requirements. following section, we outline the additional re- quirements necessary for moving beyond job JOB SCHEDULING DEFINED scheduling and into the realm of true enterprise At the most basic, job scheduling is the orderly, process automation. reliable sequencing of batch program execution. To handle these duties properly, a modern job When beginning to manage batch workload, most scheduler provides a host of features that allow it firms inevitably launch their jobs manually. This is to schedule distributed workload efficiently and in understandable and even appropriate, but it is a manner consistent with the operational goals of also easy to see how quickly this approach breaks the business. down as the number of machines, jobs, and scheduling dependencies increase. For instance, all businesses rely on one or more business calendars. Calendars govern such things Most operating systems provide a command line as when payroll is run, when corporate reports are interface for launching tasks in a manual fashion. generated, and when fiscal months and quarters As an example, UNIX provides the BATCH and AT are closed. The key issues with calendars and commands that allow users to launch background other important scheduling features are flexibility tasks immediately (BATCH) or AT a specific time. and reliability: Can the scheduler adapt to the These commands represent the “bare bones” of changing needs of the business, and deliver pre- scheduling and are in no way adequate for com- dictable results time after time? plex scheduling requirements. Additionally, programs and applications need to Surprisingly, many modern enterprise applications run on a variety of platforms, at specified times, in still require a great deal of manual intervention. specified orders, and with varying levels of re- Often a required business process will span sev- source demands and prioritization. Schedulers eral batch processes with each batch process need to be flexible enough to accommodate these being preceded by a manual process to set up new varying technology, business and resource de- batch runs, review program output, gather addi- mands. If the job scheduler is able to sequence tional input data, initiate subsequent processes, processes and manage resource contentions, ap- etc. Standard scheduling tools cannot automate plications will execute faster and more predictably, many of these processes due to inherit
Recommended publications
  • Job Scheduling in Java Applications Job Schedulers Let Developers Focus on Their Applications and Not on Scheduling Details
    flux.ly File Orchestration Solutions for Banking and Finance Reading Time: 8 minutes Job Scheduling in Java Applications Job schedulers let developers focus on their applications and not on scheduling details. Reading Time: 8 minutes Scheduling tasks in J2EE applications is a common need. There is more to job scheduling than running backups at 3 o'clock in the morning. Customer relationship management (CRM) applications, for example, need to contact customers periodically during the sales cycle. File transfer applications need to transmit and receives files on a regular basis, except on holidays. Administrative applications need to send reminder emails to notify employees and customers about important events. All these enterprise applications can benefit from job scheduling. This article first explores the need for job scheduling in J2EE applications. Section 2 explains why job scheduling can add critical functionality to enterprise applications. In Section 3, related work is reviewed. Section 4 presents a set of requirements for job scheduling software in J2EE applications. Finally, Section 5 walks through the design of an enterprise application that uses job scheduling. The need for enterprise job scheduling Many enterprise applications can benefit from job scheduling. Sales and customer relationship software, for example, needs to contact customers over the course of months and years. For example: 1. Followups are sent 10 and 30 days after a potential customer first contacts the sales organization. 2. Reminder notices are sent to existing customers 60 and 20 days before the customer's support contract expires. 3. If the customer does not contact the technical support department for 90 days, a followup is sent asking if the customer needs any help with the product or service.
    [Show full text]
  • Geek Guide > Beyond Cron
    GEEK GUIDE BEYOND CRON Table of Contents Ease of Use ..................................................................... 8 Multi-Server-Friendly .................................................... 10 Dependency Management ............................................ 13 Easy to Visualize ........................................................... 16 Delegation of Authority ................................................. 18 Management by Exception ........................................... 21 Flexible Scheduling ....................................................... 23 Revision Control ........................................................... 24 Conclusion .................................................................... 24 MIKE DIEHL has been using Linux since the days when Slackware came on 14 5.25” floppy disks and installed kernel version 0.83. He has built and managed several servers configured with either hardware or software RAID storage under Linux, and he has hands-on experience with both the VMware and KVM virtual machine architectures. Mike has written numerous articles for Linux Journal on a broad range of subjects, and he has a Bachelor’s degree in Mathematics with a minor in Computer Science. He lives in Blythewood, South Carolina, with his wife and four sons. 2 GEEK GUIDE BEYOND CRON GEEK GUIDES: Mission-critical information for the most technical people on the planet. Copyright Statement © 2015 Linux Journal. All rights reserved. This site/publication contains materials that have been created, developed or
    [Show full text]
  • VSC HPC Tutorial for Vrije Universiteit Brussel Linux Users
    VLAAMS SUPERCOMPUTER Innovative Computing CENTRUM for A Smarter Flanders HPC Tutorial Last updated: September 29 2021 For Linux Users Authors: Franky Backeljauw5, Stefan Becuwe5, Geert Jan Bex3, Geert Borstlap5, Jasper Devreker2, Stijn De Weirdt2, Andy Georges2, Balázs Hajgató1,2, Kenneth Hoste2, Kurt Lust5, Samuel Moors1, Ward Poelmans1, Mag Selwa4, Álvaro Simón García2, Bert Tijskens5, Jens Timmerman2, Kenneth Waegeman2, Toon Willems2 Acknowledgement: VSCentrum.be 1Free University of Brussels 2Ghent University 3Hasselt University 4KU Leuven 5University of Antwerp 1 Audience: This HPC Tutorial is designed for researchers at the Vrije Universiteit Brussel and affiliated institutes who are in need of computational power (computer resources) and wish to explore and use the High Performance Computing (HPC) core facilities of the Flemish Supercomputing Centre (VSC) to execute their computationally intensive tasks. The audience may be completely unaware of the VUB-HPC concepts but must have some basic understanding of computers and computer programming. Contents: This Beginners Part of this tutorial gives answers to the typical questions that a new VUB- HPC user has. The aim is to learn how to make use of the HPC. Beginners Part Questions chapter title What is a VUB-HPC exactly? 1 Introduction to HPC Can it solve my computational needs? How to get an account? 2 Getting an HPC Account How do I connect to the VUB-HPC and 3 Connecting to the HPC infrastructure transfer my files and programs? How to start background jobs? 4 Running batch jobs How to start jobs with user interaction? 5 Running interactive jobs Where do the input and output go? 6 Running jobs with input/output data Where to collect my results? Can I speed up my program by explor- 7 Multi core jobs/Parallel Computing ing parallel programming techniques? Troubleshooting 8 Troubleshooting What are the rules and priorities of 9 HPC Policies jobs? FAQ 10 Frequently Asked Questions The Advanced Part focuses on in-depth issues.
    [Show full text]
  • IBM Workload Automation: Overview Setting Job Recovery
    IBM Workload Automation IBM Overview Version 9.3 SPE (Revised July 2018) IBM Workload Automation IBM Overview Version 9.3 SPE (Revised July 2018) Note Before using this information and the product it supports, read the information in “Notices” on page 83. This edition applies to version 9, release 3, modification level 0 of IBM Workload Scheduler (program number 5698-WSH) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright IBM Corporation 1999, 2016. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. © Copyright HCL Technologies Limited 2016, 2018. Contents Figures .............. vii | Hybrid Workload Automation ...... xxiv | Checking prerequisites before running a silent About this publication ........ ix | installation ............. xxiv | Automatically initializing IBM Workload What is new in this release ......... ix | Scheduler instances .......... xxv Who should read this publication ....... ix | Defining and managing IBM Workload Accessibility .............. ix | Scheduler role-based security ....... xxv Technical training ............ ix | Reply to a message for an IBM i job from Support information ............ x | Dynamic Workload Console ....... xxv How to read syntax diagrams ........ x | Automatic reply to waiting messages for an | IBM i job ............. xxvi Summary of enhancements ..... xiii | IBM Workload Scheduler Plug-in for Apache | IBM Workload Scheduler version 9.3 enhancements xiii | Oozie............... xxvi | IBM Workload Scheduler Plug-in for IBM | Running a script when a job completes ... xxvi | WebSphere MQ ............ xiv IBM Workload Scheduler for z/OS version 9.3 | IBM Workload Scheduler Plug-in for RESTful enhancements ............. xxvi | Web services............. xiv | Configuring a backup controller for disaster | IBM Workload Scheduler Plug-in for SAP | recovery ............
    [Show full text]
  • IBM Workload Scheduler: Administration
    IBM Workload Scheduler IBM Administration Version 9 Release 5 IBM Workload Scheduler IBM Administration Version 9 Release 5 Note Before using this information and the product it supports, read the information in “Notices” on page 375. This edition applies to version 9, release 5, modification level 0 of IBM Workload Scheduler (program number 5698-WSH) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright IBM Corporation 2001, 2016. © Copyright HCL Technologies Limited 2016, 2019 Contents Figures .............. vii Configuring to schedule job types with advanced options ............... 80 Tables ............... ix \ Configuring security roles for users and groups 81 Configuring command-line client access authentication ............. 83 About this publication ........ xi Connection parameters ......... 83 What is new in this release ......... xi Entering passwords........... 85 Accessibility .............. xi IBM Workload Scheduler console messages and Technical training ............ xi prompts ............... 86 Support information ........... xi Setting sysloglocal on UNIX ........ 86 console command ........... 87 Chapter 1. Customizing and configuring Modifying jobmon service rights for Windows... 87 IBM Workload Scheduler ....... 1 Setting global options ........... 1 Chapter 2. Configuring the Dynamic Global options - summary ......... 3 Workload Console .......... 89 Global options - detailed description ..... 9 Launching in context with the Dynamic Workload Setting local options ..........
    [Show full text]
  • Docker Windows Task Scheduler
    Docker Windows Task Scheduler Genealogical Scarface glissading, his karyotype outgone inflicts overflowingly. Rudolph is accessorial and suckers languorously as sociologistic Engelbart bridled sonorously and systematises sigmoidally. Which Cecil merchandises so unbelievably that Cole comedowns her suavity? Simple task runner that runs pending tasks in Redis when Docker container. With Docker Content Trust, see will soon. Windows Tip Run applications in extra background using Task. Cronicle is a multi-server task scheduler and runner with a web based front-end UI It handles both scheduled repeating and on-demand jobs targeting any. Django project that you would only fetch of windows task directory and how we may seem. Docker schedulers and docker compose utility program by learning service on a scheduled time, operators and manage your already interact with. You get a byte array elements followed by the target system privileges, manage such data that? Machine learning service Creatio Academy. JSON list containing all my the jobs. As you note have noticed, development, thank deity for this magazine article. Docker-crontab A docker job scheduler aka crontab for. Careful with your terminology. Sometimes you and docker schedulers for task failed job gets silently redirected to get our task. Here you do want to docker swarm, task scheduler or scheduled background tasks in that. Url into this script in one easy to this was already existing cluster created, it retry a little effort. Works pretty stark deviation from your code is followed by searching for a process so how to be executed automatically set. Now docker for windows service container in most amateur players play to pass as.
    [Show full text]
  • IBM I Job Scheduling Reduce Your Running Costs and Boost Operational Efficiency
    AUTOMON® for iSchedule IBM i Job Scheduling Reduce your running costs and boost operational efficiency AUTOMON® for For IT operations under pressure to do more for less, it makes sense to iSchedule is an eliminate the costs and inefficiencies of manual batch scheduling by advanced job moving to a fully automated solution. AUTOMON® for iSchedule is a reliable scheduler that and robust IBM i job scheduler that is designed to: manages all batch work for one or many • Reduce your running costs and minimize errors by automating all servers from a single aspects of the job scheduling process point of control • Provide the flexibility to manage increases in workload without employing additional staff • Run an unattended scheduling operation, with the option to use AUTOMON® for iConsole and AUTOMON® for iMessage for a fully automated job scheduling and message management solution • Maximize system throughput by managing job dependencies and system resources efficiently • Increase levels of control, enabling you to prioritize critical work and improve batch performance • Reduce downtime through automated error recovery • Release operators from routine tasks to focus on value-added business activities A single, powerful solution Exploit the latest technology AUTOMON® for iSchedule is an advanced AUTOMON® for iSchedule is a flexible job scheduler that provides you with your own solution that exploits the latest features automated computer operator. available in the IBM i operating system. This includes the ability to schedule unattended The system is simple enough for basic backups, restricted tasks and IPLs. scheduling to be implemented immediately, yet offers the levels of sophistication and Enterprise-grade features ease the creation, power necessary for more detailed and maintenance, and execution of job schedules, complex requirements.
    [Show full text]
  • Adaptive Server® Enterprise 15.7 DOCUMENT ID: DC20001-01-1570-01
    Job Scheduler Users Guide Adaptive Server® Enterprise 15.7 DOCUMENT ID: DC20001-01-1570-01 LAST REVISED: August 2011 Copyright © 2011 by Sybase, Inc. All rights reserved. This publication pertains to Sybase software and to any subsequent release until otherwise indicated in new editions or technical notes. Information in this document is subject to change without notice. The software described herein is furnished under a license agreement, and it may be used or copied only in accordance with the terms of that agreement. To order additional documents, U.S. and Canadian customers should call Customer Fulfillment at (800) 685-8225, fax (617) 229-9845. Customers in other countries with a U.S. license agreement may contact Customer Fulfillment via the above fax number. All other international customers should contact their Sybase subsidiary or local distributor. Upgrades are provided only at regularly scheduled software release dates. No part of this publication may be reproduced, transmitted, or translated in any form or by any means, electronic, mechanical, manual, optical, or otherwise, without the prior written permission of Sybase, Inc. Sybase trademarks can be viewed at the Sybase trademarks page at http://www.sybase.com/detail?id=1011207. Sybase and the marks listed are trademarks of Sybase, Inc. ® indicates registration in the United States of America. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc.
    [Show full text]
  • Job Scheduling
    Job scheduling Sandeep Agrawal C-DAC Pune What is a job? • The user's program is not simply the name of an executable. It has input data and parameters, environment variables, descriptions of computing resources needed to run the application, and output directives. All of these specifications collectively are called a job • The user submits the job to an HPC system, usually in the form of a job script • The job script contains a formal specification that requests computing resources, identifies an application to run along with its input data and environment variables, and describes how best to deliver the output data • The term job size refers to the number of nodes the job requests • The term job length refers to the total time that a job want to run Modes of program execution • Interactive mode • Similar way as of using a personal computer • Not enabled on all HPC systems • Job may have to wait in queue • Batch mode • refers to program execution in background • most HPC jobs can use resources of multiple nodes • User jobs are accepted and typically wait in queues before adequate resources are available to the job and then it is scheduled for execution. • Most resource policies on the HPC systems tend to be exclusive Batch System • A typical batch system is composed of three functionalities: • Job Scheduler or Workload Manager • Resource Manager • Execution Manager • Job Scheduling: • The batch system is responsible for receiving the job script. • If the job cannot be executed immediately, the job script is added to a queue. • The job waits in the queue until the job's requested resources are available.
    [Show full text]
  • Process Schedule
    UNIT III: PROCESSOR MANAGEMENT Overview-About Multi-Core Technologies-Job Scheduling Versus Process Scheduling- Process Scheduler-Process Scheduling Policies-Process Scheduling Algorithms –A Word about Interrupts Deadlock-Seven Cases of Deadlock -Conditions for Deadlock- Modeling Deadlock-Strategies for Handling Deadlocks –Starvation Concurrent Processes: What Is Parallel Processing-Evolution of Multiprocessors- Introduction to Multi- Core Processors-Typical Multiprocessing Configurations-Process Synchronization Software JOB SCHEDULING VERSUS PROCESS SCHEDULING Generally scheduling of jobs is actually handled on two levels by most operating systems. The Processor Manager is a composite of two sub managers. They are 1. Job Scheduler - one in charge of job scheduling 2. Process Scheduler - the other in charge of process scheduling. There exists a hierarchy exists between the Job Scheduler and the Process Scheduler. JOB SCHEDULER Generally A user views a job either as a series of global job steps—compilation, loading and execution. Therefore, each job (or program) passes through a hierarchy of managers. • First one it encounters is the Job Scheduler; this is also called the high-level scheduler. • It is only concerned with selecting jobs from a queue of incoming jobs and placing them in the process queue, whether batch or interactive, based on each job’s characteristics. • The Job Scheduler’s goal is to put the jobs in a sequence that will use all of the system’s resources as fully as possible. • The Job Scheduler strives for a balanced mix of jobs that require large amounts of I/O interaction and jobs that require large amounts of computation. • Its goal is to keep most components of the computer system busy most of the time.
    [Show full text]
  • Mastering Oracle Scheduler in Oracle 11G Databases
    Mastering Oracle Scheduler in Oracle 11g Databases Ronald Rood Chapter No. 3 "Control the Scheduler" In this package, you will find: A Biography of the author of the book A preview chapter from the book, Chapter NO.3 "Control the Scheduler" A synopsis of the book’s content Information on where to buy this book About the Author Ronald Rood is an IT professional for over 20 years. His private domain is named after the first account that was created on a computer for him. That account was ronr on a DPS9 system, and so his private domain is called ronr.nl where you can contact him using [email protected]. He eagerly joined the Oracle community and became a skilled, innovating DBA and troubleshooter. He is a master of C, PRO*C, lots of scripting languages, and—of course—PL/SQL. Ronald’s real power is in the combination of the rich Unix world and Oracle. According to him, there is no such thing as a problem that cannot be solved; it just might take a little time. Ronald speaks several languages such as Dutch, English, German, and some French. In the private time that he shares with his wife and two children, he likes to take an occasional dive (from the sky), fl y with radio-controlled models, ride recumbents, and work as a volunteer for a local water scouts group. Currently, Ronald is one of the Ciber Oracle consultants in The Netherlands and cooperates in many projects for many large companies. Before writing this book, Ronald wrote Oracle-related articles for the Dutch Oracle user group magazine.
    [Show full text]
  • Cron Job Schedule Format
    Cron Job Schedule Format Socrates is cheek and pamper solo while interdepartmental Corwin overpriced and disarm. Kirby disarrays avidly if procephalic Sanford torrefy or piques. Distal and clovered Hewett proliferate her centiare border while Prentiss pull-out some dithyramb discriminately. You use cron expressions as jobs as another tab or window before finalising the job schedule cron format and each file Here, the asterisks refers the specific blocks of time. Linux system cron jobs from applications, assume a cron job scheduled tasks are limited time zone of cron. You must be logged into splunk. Setting Up a Basic Cron Job shoot a Linux Server Tania Rascia. For transition you doing use cron to make scheduled web requests to a. Do cron jobs run after reboot? To baptize a cron expression as will describe two main format used for the. Cron Job Arch Wiki. Singleton in all customers to take to take this job schedule cron format or. The Cron time string format consists of five fields that Cron converts into twenty time interval Cron then uses this interval to surface how sophisticated to follow an associated command on your Drupal site For eclipse a Cron time really of 0 10 15 executes a command on the 15th of each won at 1000 AM UTC. You can be at specified, it is not often to for recurring execution of your help protect your submission was successful. In this format: works fine from running properly without saving changes in part of your scheduled job schedule format of whatever month, server administrators and websites.
    [Show full text]