Cloudera HIVE BDR schedule via Python script

Worked on new user case or may  be to reduce my workload i force to write this script. replicating all the hive databases from production to DR cluster is very slow taking one day to complete.

thought of splitting  100 databases into multiple batches like 10 on each hive bdr configuration.  i felt creating 10 batches and updating configuration of more of painful work. so wrote below script to do that,

Python Script to Monitor and send email Cloudera Services and Roles health status

I know cloudera has in built feature to send the cluster status via email. but in our environment mail function is not working  through cloudera for some reason. tried all the configuration but no luck.

We could not contact Mail server support team to resolve issue. fortunately mail command is working from Unix server. so thought of writing own monitoring script using cloudera cm_client python module and send the alert to support group.

Try below my script

Cloudera cluster creation on google compute instance

Wanted to quickly launch my cloudera cluster in google cloud(since i had some free credit wanted to try effectively)  similar to on prem cluster like single sign  on  all the linux nodes.

initially i though of integrate all linux servers with Active directory + SSSD client but later i moved to MIT kerberos + Open LDAP client + SASL passthrough.

This script has 2 part

PART 1: will create No. of gcp instances, create hadoop users/groups, install SASL/openldap/MIT kerberos/Cloudera agent and Manager

PART 2: Will add the hosts into cloudera manager,create cluster/add hdfs and zookeeper services.(still working on adding more services)

Part 2 of this script can be easily  scale up with any cloud providers(AWS,AZURE) as long as cloudera manager url  is exposed to internet.

cloudera manager TLS via python api

import socket
from cm_api.api_client import ApiResource
from cm_api.api_client import ApiException
from cm_api.endpoints.cms import ClouderaManager
import ssl


#CM_HOST = "cm.tanu.com"
CM_HOST = "cm.tanu.com"
#api = ApiResource(CM_HOST,version=13, username="admin", password="admin")
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
cxt = ssl.create_default_context(cafile="/app/ca/ca.pem")

api = ApiResource(CM_HOST,version=12, username="admin", password="admin",use_tls=True,ssl_context=cxt)

clu=api.get_cluster('Cluster 1')
hdfs=clu.get_service('hdfs')

#hdfs_ssl_enable = { 'hdfs_hadoop_ssl_enabled' : 'true','ssl_server_keystore_location' : '/var/tmp/cm.jks','ssl_server_keystore_password':'test123','ssl_server_keystore_keypassword':'test123' }
cm_ssl_conf = {'WEB_TLS':'true','KEYSTORE_PATH':'/opt/cloudera-manager/ssl/jks/javakeystore.jks','KEYSTORE_PASSWORD':'iCpjC"7]','TRUSTSTORE_PATH':'/opt/cloudera-manager/ssl/jks/ca_combined.jks','TRUSTSTORE_PASSWORD':'test123'}


#hdfs.update_config(svc_config=hdfs_ssl_enable)
#for name,config in hdfs.get_config(view="full")[0].items():
#       print "%s - %s - %s" %(name,config.relatedName,config.description)
#       print "%s --> %s" %(name,config.relatedName)
x=ClouderaManager(api)
for name,config in x.get_config(view="full").items():
        print "%s  --> %s" %(name,config)
#print(x.get_config(view="full"))
#x.update_config(cm_ssl_conf)
print(hdfs)
print(clu)

for h in api.get_all_hosts():
        print(h.hostname)
        print(h.get_config())

print(api)