Deploying the Connector Component on a Standalone Node

This documentation provides the steps needed to deploy ad-sa-connector as a standalone add-on.

Prerequisites

Ensure the following criteria are met:

• You are using “Pulse 3.5.0 or higher” or “Pulse 3.4.3 or higher”.
• You have not deployed Core Connectors. If it’s already deployed, uninstall it using accelo deploy add-ons and unselect Core Connectors.
• You have already configured the standalone node.

Deploying ad-sa-connector as a Standalone add-on

Follow the steps listed below to successfully deploy ad-sa-connector as a standalone add-on.

1. On the core node, run the following command to generate ad-core.yml if it doesn’t exist already.

accelo admin makeconfig ad-core

Open the file in a text editor and navigate to the following section.

ad-graphql:

    image: ad-graphql

    container_name: ""

    environment:

....................

In case of Non-Multi KDC:

Add the following environment variables under the section environment:

• AD_CONNECTOR_HOST: Set its value to the hostname of the node where you deploy StandAlone Connector.
• AD_CONNECTOR_PORT: Set its value to 19025.

Save your changes. If done correctly, the following changes appear on the screen.

• AD_SPARK_PROXY_HOST: Set its value to the hostname of the node where you deploy StandAlone Connector.
• AD_SPARK_PROXY_PORT: Set its value to 19025.

In case of Multi-KDC:

Add the following environment variable under the section environment:

• IS_MULTIKDC_MULTINODE: Set its value to true only in case of a multi-node multi-KDC environment.

[root@plat01:docker (ad-default)]$ diff -y --suppress-common-lines ad-core.yml.orig ad-core.yml

                    >     - AD_CONNECTOR_HOST=plat03.acceldata.dvl

                    >     - AD_CONNECTOR_PORT=19025

                    >     - AD_SPARK_PROXY_HOST=plat03.acceldata.dvl

                    >     - AD_SPARK_PROXY_PORT=19025

                                  >     - IS_MULTIKDC_MULTINODE=true

2. On the core node, run the following command and provide the following string when prompted.

accelo admin encrypt

• mongodb://accel:ACCELUSER_01082018@<HOSTNAME_OF_CORE_NODE>:27017

Replace <HOSTNAME_OF_CORE_NODE> with the hostname of node where the core services are deployed. Note down the output as you need the encrypted string in the next step.

3. On the core node, backup $AcceloHome/config/acceldata_<cluster_name>.conf.Open the file in a text editor and navigate to the following section.

# General dependencies are expressed as accessible service url's. Spark, Yarn,

# ATS, Hive Metastore, Spark and Webhdfs urls are required.

# At this point in time, AccelData does not support compression by default.

group {

  name = "odp_zoro"

  displayName = "odp_zoro"

  originalName = "ODP_Zoro"

  distroType = "HWX"

  onprem = true

  active = true

  priority = 1000

  connections = {

    mongo = [

    ........

Make changes to the value of the following fields using a text editor:

• uri: Change its value to the output of the command that you ran in the previous step.
• host: Change its value from ad-elastic to the host/hosts where Logsearch is deployed.

• port: Change its value to 19013.
• nats.server.host.list: Change the value of all occurrences of this field to <HOSTNAME_OF_CORE_NODE>:19009, replacing <HOSTNAME_OF_CORE_NODE> with the hostname of node where the core services are deployed.
• spark.events.url: Change the value of all occurrences of this field to <HOSTNAME_OF_STANDALONE_NODE>:<PORT>, replacing <HOSTNAME_OF_STANDALONE_NODE> with the hostname of node where you’ll be deploying the Standalone Connectors and <PORT> with 19004.

[root@plat02:config (ad-default)]$ diff -y --suppress-common-lines acceldata_odp_zoro.conf.cpy acceldata_odp_zoro.conf

        uri = "ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8 |         uri = "ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXobon

        uri = "ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8 |         uri = "ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXobon

        host = "ad-elastic"             |         host = "plat03.acceldata.dvl"

        port = 9200                   |         port = 19013

        host = "ad-elastic"             |         host = "plat03.acceldata.dvl"

        port = 9200                   |         port = 19013

        host = "ad-elastic"             |         host = "plat03.acceldata.dvl"

        port = 9200                   |         port = 19013

    nats.server.host.list="ad-events:4222"                  |     nats.server.host.list="plat02.acceldata.dvl:19009"

    spark.events.url =  "http://ad-sparkstats:19004/events"   |     spark.events.url =  "http://plat03.acceldata.dvl:19004/ev

    nats.server.host.list="ad-events:4222"                  |     nats.server.host.list="plat02.acceldata.dvl:19009"

    nats.server.host.list="ad-events:4222"                  |     nats.server.host.list="plat02.acceldata.dvl:19009"

4. Run the following command to push the configuration.

accelo admin database push-config -a

5. On the core node, navigate to $AcceloHome and run the following command.

tar -cvf work_dir3.tar.gz  $(ls -d config/krb/security work/*/hadoop)

It generates a tar file in $AcceloHome named work_dir.tar.gz which needs to be copied over to $AcceloHome on the standalone node. One way to copy would be using scp or an NFS export.

6. On the standalone node, navigate to $AcceloHome and run the following command.

tar -xvf work_dir3.tar.gz

7. On the standalone node, create a copy of /etc/hosts and save it under $AcceloHome/config/docker. Open the file in a text editor and find the entry corresponding to the node where core services are deployed. Add the aliases ad-db ad-events ad-streaming . If done correctly, it must look something like this.

<IP_OF_CORE_NODE>           <HOST_NAME_OF_CORE_NODE> ad-db ad-events ad-streaming

Post that, run the following command to generate ad-sa-connectors.yml if it doesn’t exist already.

accelo admin makeconfig ad-sa-connectors

Open the file in a text editor and navigate to the following section.

ad-sa-connectors:

    image: ad-connectors

    container_name: ""

    ....................

    volumes:

....................

Make the following changes.

• Replace/etc/hosts:/etc/hosts:ro with $AcceloHome/config/docker/hosts:/etc/hosts:rosubstituting the environment variable $AcceloHome with its value.
• Replace the value of MONGO_URI with the output of the command you ran in step 2.

Add the following section under services.

ad-sparkstats:

    image: ad-sparkstats

    environment:

    - AD_STREAMS_URL=http://ad-streaming:19005/events

    - MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXqyScAm+LrS8Y9EWT8A8/30

    - MONGO_ENCRYPTED=true

    - KAFKA_ENABLED=false

    - MONGO_SECRET=Ah+MqxeIjflxE8u+/wcqWA==

    - NATS_SERVER_HOST_LIST=ad-events:19009

    - KRB5_CONF_LOCATION=/krb/security/krb5.conf

    - JAAS_LOGIN_CONF_LOCATION=/krb/security/krb5JAASLogin.conf

    - KERBEROS_ENABLED=true

    - ES_USERNAME=pulse

    - ES_PASSWORD=pPBrVKaoB0QsmCJZNZyYAw==

    volumes:

    - /etc/localtime:/etc/localtime:ro

    - $AcceloHome/config/docker/hosts:/etc/hosts:ro

    - ./config/users/group:/etc/group:ro

    - ./config/users/passwd:/etc/passwd:ro

    - ./work:/etc/acceldata:ro

    - ./config/krb/security:/krb/security

    opts:

      memory: "30064771072"

    ports:

    - 19004:19004

    depends_on:

    - ad-streaming

Add the following port mapping to the service ad-sa-connectors.

ad-sa-connectors:

    image: ad-connectors

    ...................

    ports:

    - 19025:19025

• Replace the environment variable $AcceloHome with its value.
• Replace the value of MONGO_URI with the output of the command you ran in step 2.

Save your changes. If done correctly, the following changes appear on the screen.

- MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXqyS |     - MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXqcd

    ports: []                                     |     ports:

                                                 >      - 19025:19025

    - MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXqyS |     - MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXqcd

8. Perform this step only if Multi KDC is enabled. On the Standalone Node:
   1. Update the ports so that ad-sa-connectors and ad-sparkstats start listening on different ports.

ports:

- 19029:19025

For details about enabling Multi KDC, see Enable MultiKDC.

b. Save your changes.

c. Switch to the other cluster by running the following command.

accelo set

9. Perform this step only if Multi-KDC is enabled.
   1. On the core node, backup $AcceloHome/config/accelo.yml.
   2. Open the file in a text editor and add the following entry to the list KerberosAddonsad-sa-connectors.``
   3. Update MultiKDCContainersPortMap.

You need to specify the host and the host port corresponding to ad-sa-connectors for each of the clusters that Pulse is observing. The host and host port are the hostname of the standalone node and the host port that you have specified in the previous step for each of the clusters respectively.

Note that the host port differs for each of the clusters.

ad-connectors:

        odp_ace:

            hostname: ""

            port: ""

        odp_ace2:

            hostname: ""

            port: ""

        odp_zoro:

            hostname: ""

            port: ""

On the core node, make sure that the port number for the field spark.events.url in the acceldata_<cluster_name>.conf file is correct and match with the host ports you’ve specified in this step for ad-sparkstats for each of the clusters.

You need to repeat this step for each of the of the acceldata_<cluster_name>.conf files.

Save your changes and run the following command.

accelo admin database push-config -a

10. On the standalone node, run the following command and select StandAlone Connector.

accelo deploy addons