Data Sources APIs

This document outlines the process of managing and crawling data sources within Acceldata. ADOC facilitates crawling across more than fifteen varied data sources. The following list consists of the data source APIs that seamlessly integrate with the SDK. It elaborates on acquiring data sources by name or ID, retrieving data sources of a specific type, initiating crawlers for designated data sources or assembly names, checking the crawling status, and deleting a data source process using the Java SDK. The accompanying code snippets illustrate these functionalities through both DataSource objects and the ADOC Client.

Obtaining a DataSource by its ID or Name

To get data sources by their name or IDs use the following code examples for each method:

// Either assemblyId or assemblyName is needed as an assemblyIdenfier.

​

// properties should be passed as True to get assembly property

String dataSourceName = "java_sdk_snowflake_ds";

boolean fetchAssemblyPropertiesFlag = true;

//1. Get datasource by name

DataSource dataSourceByName = adocClient.getDataSource(dataSourceName, Optional.of(fetchAssemblyPropertiesFlag));

System.out.println(dataSourceByName);

​

//2. Get data source by id

Long dataSourceId = dataSourceByName.getId();

DataSource dataSourceById = adocClient.getDataSource(String.valueOf(dataSourceId), Optional.of(fetchAssemblyPropertiesFlag));

System.out.println(dataSourceById);

Retrieve All Data Sources

The following sections explains how to retrieve all available data sources. It provides a code snippet to obtain a list of all data sources without type filtering.

AssetSourceType assetSourceType = AssetSourceType.AWS_S3;

List<DataSource> dataSources = adocClient.getDataSources(Optional.of(assetSourceType));

Initiate Crawler for a Specified DataSource or Assembly Name

To initiate a crawler for a designated data source or assembly name, use the provided code snippets as it demonstrates the process using both the data source object and the ADOC client.

//Using DataSource object

dataSource.startCrawler();

​

//or

​

//Using AdocClient

adocClient.startCrawler(assemblyName);

Retrieve Crawler Status for a Specified DataSource or Assembly Name

To check the status of the crawling process for a specific data source or assembly name for both the data source object and ADOC client, use the following code snippet:

//Using DataSource object

CrawlerStatus crawlerStatus = dataSource.getCrawlerStatus();

//or

​

//Using AdocClient

adocClient.getCrawlerStatus(assemblyName);

Deleting a DataSource

Use the following code snippet to delete a particular data source within the Acceldata environment:

adocClient.deleteDataSource(dataSourceId);

By following this guide, you can efficiently manage data sources and optimize their data operations in the Acceldata platform.