Configuring YarnSpawner and HDFSCM

Follow the steps below to configure YarnSpawner and HDFSCM for JupyterHub.

1. Update the JupyterHub Configuration: To enable YarnSpawner and HDFSCM just click the checkbox for both on ambari ui.

2. Save and Restart JupyterHub

   1. After making the above changes, save the configuration file.
   2. Restart the JupyterHub service to apply the changes.

3. Launch the JupyterHub Server

   1. Click on Launch Server to start the JupyterHub server.
   2. Refresh the JupyterHub web interface.

4. Resource Allocation via YARN
   1. Upon launching, a job will be submitted to the YARN queue.
   2. YARN allocates the required resources for the JupyterHub server.

5. Using the JupyterHub Server
   1. Access the JupyterHub server from the web interface.
   2. Create a Python notebook or any other notebook of your choice.

6. File Storage on HDFS
   1. Notebooks will be automatically saved to the HDFS path, organized under the logged-in user's directory.
   2. This setup ensures that all notebook files are retained in the HDFS path associated with the logged-in user.

By following these steps, you can seamlessly integrate YarnSpawner and HDFSCM with JupyterHub, enabling efficient resource management and persistent notebook storage.