Amazon | S3

Amazon S3 is AWS’s object storage service. Use ADOC to monitor and profile S3 buckets and files—providing data quality, freshness, usage, schema drift, and reliability insights.

Prerequisites

Before you connect Amazon S3 as a data source in ADOC, ensure the following:

• You have access to an AWS S3 bucket containing the data you want to monitor.
• The ADOC Data Plane is deployed and has network access to the S3 bucket.
• Authentication to AWS is configured using one of the following methods:
• EC2 Instance Profile with the appropriate IAM role attached,
• Kubernetes IAM Roles for Service Accounts (IRSA), or
• AWS Access Key and Secret Key credentials.
• Your IAM policy includes the necessary permissions to access S3, Secrets Manager (if applicable), and SQS (if you're using event-based or incremental monitoring).

Add S3 as a Data Source

Follow these steps to set up S3 in ADOC:

Step 1: Start Setup

1. Select Register from the left main menu.

2. Select Add Data Source.

3. Select AWS S3 from the list of data sources.

4. On the Data Source Details page:

   1. Enter a name for this data source that is unique within your tenant.
   2. Optionally, add a brief description to clarify its purpose.
   3. Enable the Data Reliability toggle and select your data plane from the drop-down list.

5. Select Next to proceed.

Step 2: Add Connection Details

Enter your AWS S3 connection information. Required fields vary depending on the selected Authentication Type.

Common Fields (Displayed for All Authentication Types)

Authentication Types

1. Select Test Connection. If successful, you’ll see “Connected.” If the test fails, troubleshoot credentials, region, network issues, and ensure that the ADOC Data Plane service (ad-analysis-standalone) is running.
2. Select Next to proceed.

Step 3: Setup Observability

Configure how ADOC will monitor your S3 bucket:

1. Asset Name: Enter a logical name or label.
2. Path Expression: Enter the complete path expression using the syntax s3://bucket name/file. For example, s3://acceldatabucket/observability.csv
3. File Type: Select a file type and provide file-type specific parameters. ADOC supports the following file types: .bz2, .deflate, .gz, .z4, .snappy, JSON, CSV, ORC, PARQUET. ADOC also supports profiling of zipped and KMS encrypted files. For details, refer to the below table.

Configure Observability Options:

4. Enable Schema Drift Monitoring to detect changes in file schemas (e.g., added, removed, or renamed columns) over time.
5. Enable Crawler Execution Schedule to set up scheduled scans of your S3 bucket:

• Choose how often the crawler runs (e.g., daily)
• Set execution time and time zone
• Add multiple execution times if needed

6. Set Notifications
7. Notify on Crawler Failure: Choose one or more channels to receive failure alerts.
8. Notify on Success: Toggle this if you'd like to receive success notifications.
9. Click Submit to save your configuration to register and begin monitoring the AWS S3 data source.

You have successfully added AWS S3 as a data source. A new card for AWS S3 will appear on the Data Sources page, displaying crawler status and basic connection details.

What’s Next

Once you've successfully connected your Amazon S3 bucket as a data source in ADOC, you can:

• Profile your S3 data: Run profiling jobs to gather metrics such as row count, null percentage, file size, and freshness across supported file formats.
• Monitor data quality in real time: Enable schema drift detection and track changes to file structure, format, or volume using scheduled crawlers or SQS-based triggers.
• Apply data reliability rules and policies: Set up and enforce data quality rules—such as column-level validations, null checks, or file arrival thresholds—directly on your S3 data.

Additional References

IRSA-Based Authentication

IAM Roles for Service Accounts (IRSA) is a secure way to manage AWS access in Kubernetes without hard-coded credentials. IRSA allows each ADOC service in the Data Plane to assume a role with only the permissions it needs, following the principle of least privilege.

How IRSA Works in ADOC? The ADOC Data Plane connects to AWS services in your environment using service accounts mapped to IAM roles. These roles allow access to:

• S3 (for data ingestion)
• AWS Secrets Manager (to retrieve credentials)
• Amazon Athena (for metadata crawling)
• SQS (for event-driven ingestion)

Kubernetes Service Account Mapping

Annotating Service Accounts for IRSA

To map a Kubernetes service account to an IAM role:

eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>

Replace placeholders with your AWS values.

Required IAM Policies

1. S3 Read Access

{

  "Effect": "Allow",

  "Action": ["s3:GetObject", "s3:ListBucket"],

  "Resource": [

    "arn:aws:s3:::<bucket-name>",

    "arn:aws:s3:::<bucket-name>/*"

  ]

}

2. Secret Manager Access

{

  "Effect": "Allow",

  "Action": "secretsmanager:GetSecretValue",

  "Resource": "<secret-arn>"

}

3. SQS Access (for Incremental Processing)

{

  "Effect": "Allow",

  "Action": ["sqs:ReceiveMessage"],

  "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>"

}

Secret Manager Configuration (for IRSA or EKS Pod Identity)

To retrieve secrets via ADOC’s Data Plane:

[

  {

    "name": "<config-name>",

    "type": "AWS_SECRETS_MANAGER",

    "details": {

      "secretName": "<aws-secret-name>",

      "accessKey": "",

      "secretKey": "",

      "region": "<region>",

      "authType": "IAM_ROLES_FOR_SA_BASED"

    }

  }

]

Other authType values (for alternate methods):

• KEY_BASED
• INSTANCE_PROFILE_BASED

EKS Pod Identity (Alternative to IRSA)

EKS Pod Identity enables IAM role assignment directly to pods, removing the need for annotations. It provides the same security benefits as IRSA but with more granularity.

IAM policies for EKS Pod Identity are the same as those listed above under IRSA.