Search
After installing the Knox service via the Ambari UI and activating the Knox plugin in Ranger (if not done already), execute the following steps to assess Knox functionality.
Demo LDAP Server
Start Demo Knox LDAP server from the actions menu in Knox.
Proxy Users in HDFS
Update the following configs in custom core-site in HDFS:
hadoop.proxyuser.knox.groups=*
hadoop.proxyuser.knox.hosts=*
Attempt the given curl command:
curl -iku admin:admin-password "https://$(hostname -f):8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"
This command is expected to fail unless you have disabled Audit to SOLR in the Knox configuration.
Disable Audit to SOLR
On disabling Audit to SOLR in the Knox configuration, the API calls must function as expected.
root@newreleaseub1:~# curl -iku admin:admin-password "https://$(hostname -f):8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"
HTTP/1.1 200 OK
Date: Mon, 11 Dec 2023 19:32:31 GMT
Set-Cookie: KNOXSESSIONID=node01mitiwazsjo0gskicoox7x3v933.node0; Path=/gateway/default; Secure; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Sun, 10-Dec-2023 19:32:31 GMT; SameSite=lax
Date: Mon, 11 Dec 2023 19:32:31 GMT
Cache-Control: no-cache
Expires: Mon, 11 Dec 2023 19:32:31 GMT
Date: Mon, 11 Dec 2023 19:32:31 GMT
Pragma: no-cache
X-Content-Type-Options: nosniff
X-FRAME-OPTIONS: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16395,"group":"hadoop","length":0,"modificationTime":1701708150259,"owner":"yarn","pathSuffix":"app-logs","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17307,"group":"hdfs","length":0,"modificationTime":1701680533763,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16392,"group":"hadoop","length":0,"modificationTime":1701680011525,"owner":"yarn","pathSuffix":"ats","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16415,"group":"hdfs","length":0,"modificationTime":1701680420023,"owner":"hdfs","pathSuffix":"atsv2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16398,"group":"hdfs","length":0,"modificationTime":1701680020018,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16400,"group":"hadoop","length":0,"modificationTime":1701680083247,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16404,"group":"hdfs","length":0,"modificationTime":1701680022069,"owner":"hdfs","pathSuffix":"odp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16389,"group":"hdfs","length":0,"modificationTime":1701680004677,"owner":"hdfs","pathSuffix":"ranger","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":127,"fileId":17548,"group":"hadoop","length":0,"modificationTime":1701719034844,"owner":"spark","pathSuffix":"spark2-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":21891,"group":"hdfs","length":0,"modificationTime":1701706404538,"owner":"hdfs","pathSuffix":"system","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":12,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1701718929865,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":15,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1702303725817,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":17315,"group":"hdfs","length":0,"modificationTime":1701705629173,"owner":"hdfs","pathSuffix":"warehouse","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}
Update Advanced Topology
In Knox configurations, update or add the URL's based on the hostname at your end for the services as shown below:
<topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://{{knox_host_name}}:33389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>{{namenode_address}}</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
</service>
<service>
<role>WEBHDFS</role>
{{webhdfs_service_urls}}
</service>
<service>
<role>WEBHCAT</role>
<url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
</service>
<service>
<role>OOZIE</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
</service>
<service>
<role>OOZIEUI</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie/</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
</service>
<service>
<role>HIVE</role>
<url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://{{rm_host}}:{{rm_port}}/ws</url>
</service>
<service>
<role>DRUID-COORDINATOR-UI</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-COORDINATOR</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-OVERLORD-UI</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-OVERLORD</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-ROUTER</role>
{{druid_router_urls}}
</service>
<service>
<role>DRUID-BROKER</role>
{{druid_broker_urls}}
</service>
<service>
<role>ZEPPELINUI</role>
{{zeppelin_ui_urls}}
</service>
<service>
<role>ZEPPELINWS</role>
{{zeppelin_ws_urls}}
</service>
<service>
<role>YARNUI</role>
<url>http://newreleaseub2.acceldata.ce:8088</url>
</service>
<service>
<role>AMBARIUI</role>
<url>http://newreleaseub1.acceldata.ce:8080</url>
</service>
<service>
<role>AMBARIWS</role>
<url>ws://newreleaseub1.acceldata.ce:8080</url>
</service>
<service>
<role>RANGER</role>
<url>http://newreleaseub1.acceldata.ce:6080</url>
</service>
<service>
<role>RANGERUI</role>
<url>http://newreleaseub1.acceldata.ce:6080</url>
</service>
<service>
<role>HDFSUI</role>
<version>2.7.0</version>
<url>http://newreleaseub1.acceldata.ce:50070</url>
</service>
<service>
<role>SOLR</role>
<version>6.0.0</version>
<url>http://newreleaseub1.acceldata.ce:8886/solr</url>
</service>
</topology>
Note Change the hostnames according to your requirement.
Info
Additionally, modify the gateway.dispatch.whitelist in the Advanced gateway-site section of the Knox configuration to use regular expressions as illustrated below:
For hostnames resembling newreleaseub1.acceldata.ce:
^https?:\/\/(.+\.acceldata\.ce):[0-9]+\/?.*$
For hostnames resembling odp01.ubuntu.ce:
^https?:\/\/(.+\.ubuntu\.ce):[0-9]+\/?.*$
Setup Ambari SSO
Generate the Certificate
root@newreleaseub1:/usr/odp/3.2.3.3-2/knox/bin# ./knoxcli.sh export-cert --type PEM
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/odp/3.2.3.3-2/knox/bin/../dep/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/odp/3.2.3.3-2/knox/bin/../dep/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Certificate gateway-identity has been successfully exported to: /usr/odp/3.2.3.3-2/knox/data/security/keystores/gateway-identity.pem
Display the contents using cat as you'll require them in the next step.
root@newreleaseub1:/usr/odp/3.2.3.3-2/knox/bin # cat /usr/odp/3.2.3.3-2/knox/data/security/keystores/gateway-identity.pem
-----BEGIN CERTIFICATE-----
MIIDjDCCAnSgAwIBAgIJAI3Nc0sM1HuuMA0GCSqGSIb3DQEBBQUAMHAxCzAJBgNVBAYTAlVTMQ0w
CwYDVQQIEwRUZXN0MQ0wCwYDVQQHEwRUZXN0MQ8wDQYDVQQKEwZIYWRvb3AxDTALBgNVBAsTBFRl
.
.
.
Chd3qTNvU8y0Sbb7qEF8mxvqB9m/6MjzvWW5OMiJZ8YFGccKLGihG6ziJn/oir+i1fiB7j8LU5o/
P++SwUaEbq9/TX1ankw5CGvOUiPGzqk7aOwvwxMOwAh++PgVPW/nnqeCBNZYXdXeVwVrKMhGpoMP
-----END CERTIFICATE-----
Ambari Server Setup SSO
root@newreleaseub1:/var/log/knox# ambari-server setup-sso
Using python /usr/bin/python
Setting up SSO authentication properties...
Enter Ambari Admin login: admin
Enter Ambari Admin password:
SSO is currently not configured
Do you want to configure SSO authentication [y/n] (y)? y
Provider URL (https://knox.example.com:8443/gateway/knoxsso/api/v1/websso): https://newreleaseub1.acceldata.ce:8443/gateway/knoxsso/api/v1/websso
Public Certificate PEM (empty line to finish input):
-----BEGIN CERTIFICATE-----
MIIDjDCCAnSgAwIBAgIJAI3Nc0sM1HuuMA0GCSqGSIb3DQEBBQUAMHAxCzAJBgNVBAYTAlVTMQ0w
CwYDVQQIEwRUZXN0MQ0wCwYDVQQHEwRUZXN0MQ8wDQYDVQQKEwZIYWRvb3AxDTALBgNVBAsTBFRl
.
.
.
Chd3qTNvU8y0Sbb7qEF8mxvqB9m/6MjzvWW5OMiJZ8YFGccKLGihG6ziJn/oir+i1fiB7j8LU5o/
P++SwUaEbq9/TX1ankw5CGvOUiPGzqk7aOwvwxMOwAh++PgVPW/nnqeCBNZYXdXeVwVrKMhGpoMP
-----END CERTIFICATE-----
Use SSO for Ambari [y/n] (n)? y
Manage SSO configurations for eligible services [y/n] (n)? y
Use SSO for all services [y/n] (n)? y
JWT Cookie name (hadoop-jwt): hadoop-jwt
JWT audiences list (comma-separated), empty for any ():
Ambari Server 'setup-sso' completed successfully.
Restart the Ambari server.
LDAP Setup and Sync
To setup LDAP, perform the following:
root@newreleaseub1:/var/log/knox# ambari-server setup-ldap --ldap-url=newreleaseub1.acceldata.ce:33389
Using python /usr/bin/python
Currently 'no auth method' is configured, do you wish to use LDAP instead [y/n] (y)?
Enter Ambari Admin login: admin
Enter Ambari Admin password:
Fetching LDAP configuration from DB. No configuration.
Please select the type of LDAP you want to use [AD/IPA/Generic](Generic):
Primary LDAP Host (ldap.ambari.apache.org):
Primary LDAP Port (389):
Secondary LDAP Host <Optional>:
Secondary LDAP Port <Optional>:
Use SSL [true/false] (false):
User object class (posixUser): person
User ID attribute (uid): uid
Group object class (posixGroup): groupofnames
Group name attribute (cn): cn
Group member attribute (memberUid): member
Distinguished name attribute (dn):
Search Base (dc=ambari,dc=apache,dc=org): dc=hadoop,dc=apache,dc=org
Referral method [follow/ignore] (follow):
Bind anonymously [true/false] (false):
Bind DN (uid=ldapbind,cn=users,dc=ambari,dc=apache,dc=org): uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
Enter Bind DN Password:
Confirm Bind DN Password:
Handling behavior for username collisions [convert/skip] for LDAP sync (skip): convert
Force lower-case user names [true/false]:true
Results from LDAP are paginated when requested [true/false]:true
====================
Review Settings
====================
Primary LDAP Host (ldap.ambari.apache.org): newreleaseub1.acceldata.ce
Primary LDAP Port (389): 33389
Use SSL [true/false] (false): false
User object class (posixUser): person
User ID attribute (uid): uid
Group object class (posixGroup): groupofnames
Group name attribute (cn): cn
Group member attribute (memberUid): member
Distinguished name attribute (dn): dn
Search Base (dc=ambari,dc=apache,dc=org): dc=hadoop,dc=apache,dc=org
Referral method [follow/ignore] (follow): follow
Bind anonymously [true/false] (false): false
Handling behavior for username collisions [convert/skip] for LDAP sync (skip): convert
Force lower-case user names [true/false]: true
Results from LDAP are paginated when requested [true/false]: true
ambari.ldap.connectivity.bind_dn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
ambari.ldap.connectivity.bind_password: *****
Save settings [y/n] (y)? y
Saving LDAP properties...
Saving LDAP properties finished
Ambari Server 'setup-ldap' completed successfully.
Sync LDAP as shown below:
root@newreleaseub1:/var/log/knox# ambari-server sync-ldap --all
Using python /usr/bin/python
Syncing with LDAP...
Enter Ambari Admin login: admin
Enter Ambari Admin password:
Fetching LDAP configuration from DB.
Syncing all...
Completed LDAP Sync.
Summary:
memberships:
removed = 0
created = 4
users:
skipped = 0
removed = 0
updated = 1
created = 3
groups:
updated = 0
removed = 0
created = 3
Ambari Server 'sync-ldap' completed successfully.
Restart the Ambari server and clear all cookies after setting up LDAP or SSO in the browser.
Perform UI checks for the following services
For details about configuring Okta SSO for Knox, see Configure Okta SSO for Apache Knox.
UI Checks
Oozie
Added the following configs in Oozie configuration to move forward:
oozie.service.ProxyUserService.proxyuser.knox.groups=*
oozie.service.ProxyUserService.proxyuser.knox.hosts=*
Oozie UI URL
https://newreleaseub1.acceldata.ce:8443/gateway/default/oozie/
http://newreleaseub1.acceldata.ce:11000/oozie/?user.name=admin
HDFS
HDFS UI URL
http://newreleaseub1.acceldata.ce:50070/dfshealth.html #tab-overview
https://newreleaseub1.acceldata.ce:8443/gateway/default/hdfs/dfshealth.html #tab-overview
Yarn
Yarn UI URL
http://newreleaseub1.acceldata.ce:8088/ui2/#/cluster-overview
https://newreleaseub1.acceldata.ce:8443/gateway/default/yarn
Troubleshooting
If the service UI, when accessed in incognito mode (similar to attempting to log in without cookies) in Chrome, redirects to the Knox login page again, include the following configuration (add the last parameter section) in the Advanced knoxsso-topology under the service attribute:
<param>
<name>knoxsso.cookie.samesite</name>
<value>Lax</value>
</param>
This will set the cookie property as mentioned above. After restarting the service and clearing old cookies in the browser, the UIs should also function correctly in Chrome.
Knox Ranger Test Connection Failure
To fix the Knox Ranger test connection failure, perform the following steps on the Ranger Admin host.
## Get knox cert and store to tmp location
echo -n | openssl s_client -connect ce04.acceldata.dvl:8443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/knoxcert.crt
## Locate your java cacerts
ll /etc/java/java-11-openjdk/java-11-openjdk-11.0.24.0.8-3.el8.x86_64/lib/security/cacerts
## Import knox cert to cacerts
keytool -import -file /tmp/knoxcert.crt -keystore /etc/java/java-11-openjdk/java-11-openjdk-11.0.24.0.8-3.el8.x86_64/lib/security/cacerts -alias "knox-trust"
## default password : changeit
Add properties as shown below.
Verify the Knox policy test connection.
Knox Uninstallation
To uninstall Knox, perform the following:
Disable SSO setup on Ambari by doing the following:
root@odp03:~# ambari-server setup-sso
Using python /usr/bin/python
Setting up SSO authentication properties...
Enter Ambari Admin login: admin
Enter Ambari Admin password:
SSO is currently enabled
Do you want to disable SSO authentication [y/n] (n)? y
Ambari Server 'setup-sso' completed successfully.
Restart your Ambari server.
Stop and delete the Knox service from the Ambari UI:
To revert to the older quicklinks and other configurations added by Knox, the simplest approach is to roll back the version in the configurations of all services associated with Knox to an earlier version used just before Knox was introduced. Alternatively, if you have incorporated other required configurations into the service, you can remove these configurations by searching for the specified keywords in each of the services.
For HDFS, YARN, and MapReduce, navigate to Advanced Ambari configs and remove the following configurations:
hadoop.http.authentication.public.key.pem
hadoop.http.authentication.authentication.provider.url
hadoop.http.authentication.type=org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler
proxyuser.knox.groups
proxyuser.knox.hosts
For Ranger and Oozie, go to Advanced Ambari configs and remove the following configurations:
proxyuser.knox.groups
proxyuser.knox.hosts
proxyuser.knox.ip
proxyuser.knox.users
SSO public key
SSO provider URL
Enable Ranger SSO - Set to False
oozie.authentication.type
oozie.authentication.expected.jwt.audiences
oozie.authentication.jwt.cookie
Additionally, check other services where additional Knox configurations may have been added.
Was this page helpful?