AutoSys

Overview

AutoSys integration with Acceldata Data Observability Cloud (ADOC) allows you to track and monitor scheduled jobs and pipelines in one place.

AutoSys is typically used to run and manage jobs. By connecting it to ADOC, you can:

See how jobs are running in real time
Understand how different jobs are connected
Identify failures or delays quickly
Track how jobs impact your data

This helps you get a clear picture of how your data pipelines are working without needing to switch between tools.

Key Capabilities:

Job Monitoring: View real-time statuses of AutoSys jobs, including job dependencies, execution times, and failure alerts.
Pipeline Views: Visualize nested box jobs and job hierarchies to better understand complex workflows.
Data Impact Visibility: Understand which datasets are used as input and which are produced as output.
Custom Alerts: Set up alerts for job failures, delays, or other critical events to proactively manage your data pipelines.

Prerequisite

AutoSys Environment: An active AutoSys environment with access credentials. Ensure that AutoSys server and agents are properly installed and configured.
ADOC Account: An Acceldata Data Observability Cloud account with Read level API access privileges.( administrative privileges.)
Network Access:
- Ensure network connectivity between your AutoSys server and the ADOC platform.
- Update firewall rules to allow communication between AutoSys and ADOC.
Firewall Configuration: Ensure that the necessary ports are open for communication (e.g., port 9443 for AutoSys API access).

Setting up and Managing AutoSys Integration in ADOC

1. Accessing AutoSys Data in ADOC

1.1. Firewall Configuration

To enable communication between your AutoSys server and ADOC:

Identify External IP Addresses:
- Determine the external IP addresses of your ADOC data plane nodes.
- For example, if your data plane is hosted on AWS or GCP, obtain the public IP addresses of the nodes.
Update Firewall Rules:
- Access your firewall management console (e.g., GCP Firewall Policies).
- Edit the relevant firewall rule .
- Add the external IP addresses of your ADOC data plane nodes to the Source IPv4 ranges.
- Enable the necessary ports (e.g., port 9443 for AutoSys API).
Enable Enforcement: Ensure that the enforcement option is set to Enabled.
Save Changes: Click Save to apply the updated firewall settings.

1.2. Access Credentials: Gather the necessary credentials for both UI and CLI access to your AutoSys environment.

UI Access:
- URL: Example: https://<your-autosys-url>:9443/AEWS
- Username: Provided by your administrator.
- Password: Provided by your administrator.

Note Depending on your infrastructure, you need to grant access to the data plane nodes that will interact with the remote AutoSys server.

2. Configuring the AutoSys Integration

Access your ADOC account and log-in with your credentials.
Go to the Register menu.
Click on Add Data Source and choose AutoSys from the list.
Enter Connection Details: Fill in the required fields:
- Username: Your AutoSys username.
- Password: Your AutoSys password.
- Use Secret Manager: (Optional) If using secret manager for credentials, configure accordingly.
Click on Test Connection to verify connectivity. After the connection is successful, save the integration settings.

3. Define Pipelines Using Meta Configuration

To organize jobs in ADOC, you need to define how they should appear as pipelines.

This is done using a Meta JSON configuration, which helps:

Group related jobs into a pipeline
Show how jobs are connected
Link jobs to the data they use and produce

Meta Configuration Template

JSON
    
 
{  "pipelines": [    {      "name": "<<Parent Pipeline Name to be observed>>",      "meta": {        "assets": {          "upstream": ["<<Asset UID as upstream>>"],          "downstream": ["<<Asset UID as downstream>>"]        }      },      "child_jobs_meta": [        {          "name": "<<Child Job Name for which meta needs to be added>>",          "meta": {            "assets": {              "upstream": ["<<Asset UID as upstream>>"],              "downstream": ["<<Asset UID as downstream>>"]            }          }        }      ]    }  ]}
Copy

Asset Format

Assets must follow this format: <DataSource Name>.<Schema/Database>.<Table/View Name>

Example: bigquery.sales.orders

You can find these values on the Assets Details page in ADOC.

Field Descriptions

Pipeline Fields

Field	Type	Description	Example
`name`	String	The name of the main job in the pipeline.	box_job
`meta`	JSON (Optional)	Contains metadata about the main job, including assets information as upstream and downstream.	See Meta Fields
`child_jobs_meta`	Array of JSON (Optional)	Contains metadata for sub-jobs associated with the main job, each defined as an object in this array.	See Child Jobs Meta Fields

Meta Fields

Field	Sub Field	Type	Description	Example
`assets`	upstream	Array of String (Optional)	List of assets involved as input (upstream).	"meta": { "assets": { "upstream": [ "bigqueryds.sales.order", "bigqueryds.sales.transaction" ] } } Note: "bigqueryds.sales.order" is the asset uid per acceldata which we can get from the Asset page for a table. Syntax used as:<Datasource Name>.<Schema / Database>.<Table / View name>
`assets`	downstream	Array of String(Optional)	List of assets which are the outcome of part of the job treated as downstream.	"meta": { "assets": { "downstream": [ "Bigqueryds.sales.order-output", "bigqueryds.sales.order-output", ] } Note: "bigqueryds.sales.order-output" is the asset uid per acceldata which we can get from the Asset page for a table. Syntax used as: <Datasource Name>.<Schema / Database>.<Table / View name>

Child Jobs Meta Fields

Field	Type	Description	Example
`name`	String	The name of the child job for which metadata is to be added.	`"child_job_1"`
`meta`	JSON (Optional)	Contains metadata about the child job, including	See Meta Fields

Examples:

Simple Basic Job Meta

JSON
    
 
{  "pipelines": [    {      "name": "data_processing_job"    }  ]}
Copy

Basic Job with Upstream/Downstream Assets Meta

JSON
    
 
{  "pipelines": [    {      "name": "data_processing_job",      "meta": {        "assets": {          "upstream": ["source_dataset_1", "source_dataset_2"],          "downstream": ["target_dataset_2"]        }      }    }  ]}
Copy

Jobs with Child Jobs Metadata

JSON
    
 
{  "pipelines": [    {      "name": "parent_job",      "meta": {        "assets": {          "upstream": ["source_dataset_4"],          "downstream": ["target_dataset_3"]        }      },      "child_jobs_meta": [        {          "name": "child_job_1",          "meta": {            "assets": {              "upstream": ["source_dataset_4"],              "downstream": ["target_dataset_4"]            }          }        },        {          "name": "child_job_2",          "meta": {            "assets": {              "upstream": ["target_dataset_4"],              "downstream": ["target_dataset_5"]            }          }        }      ]    }  ]}
Copy

Multiple Jobs with One Having Meta and Another Without

JSON
    
 
{  "pipelines": [    {      "name": "data_processing_job"    },    {      "name": "data_processing_job_with_meta",      "meta": {        "assets": {          "upstream": ["source_dataset_1", "source_dataset_2"],          "downstream": ["target_dataset_2"]        }      }    }  ]}
Copy

4. Add Meta Configuration in ADOC

Prepare the Meta JSON: Use the JSON template and examples provided to create your Meta JSON configuration. Ensure that the names in the Meta JSON match the actual job names in AutoSys.
Input the Meta JSON: During the onboarding process, you will be prompted to enter the Meta JSON configuration. Paste your prepared Meta JSON into the configuration field.
Submit the Configuration: Click Submit to add the AutoSys data source with your Meta JSON configuration.

5. Monitoring Jobs and Pipelines in ADOC

Navigate to the Pipelines section in ADOC.
Filter by Data Source: Use the filter options to select AutoSys as the data source.
View Pipelines: The pipelines configured via the Meta JSON will appear. Click on a pipeline to view its details, including job hierarchies and execution timelines.
Pipeline Visualization: Visualize job dependencies, including nested box jobs and conditional jobs. Navigate between parent and child jobs within pipelines.

Troubleshooting

If you encounter issues, consider the following steps:

Connection Issues

Firewall Settings: Revisit your firewall configurations to ensure ADOC can access your AutoSys server.
Network Connectivity: Verify network connections and DNS settings.
Credentials Verification: Double-check your AutoSys credentials entered in ADOC.
Permissions: Ensure the user account has the necessary permissions to access AutoSys data.

Data Not Appearing

Job Configuration: Confirm that your AutoSys job names match the Meta JSON and the jobs are active.
Integration Status: Check the integration status in ADOC for any error messages.

Last updated on

Was this page helpful?