Python script to automate Yarn pool utilization report mail

i was working on simple requirement to send the each Yarn pool utilization hourly to management team.

Although cloudera offer cluster utilization report which will provide detailed report about Yarn utilization, there are some limitation. so i used my scripting knowledge to implement on my own.

please find below python and script and the email output.

It will generate report like below.

From: ClouderaAdministrator-
Sent: Thursday, April 2, 2020 8:00:01 AM (UTC+00:00) Monrovia, Reykjavik
To: admins; admins@tanu.com
Subject: Prod Cluster Yarn Utilization report

Report generation time:	2020-04-02 04:00:01.043615
Total Resource availabe:	1440 Vcores,	7.03TB
Total Resource Utilized(value):	916 Vcores,	3.95TB
Total Resource Utilized(%):	63.61 %,	56.24%
Queue Name	Max Allocation(cpu)	Max Allocation(memory)	Cpu Utilizied(%)	Memory Utilized(%)
root.default	1.0	1.0	0.0	0.0
root.dev1	1.0	1.0	0.0	0.0
root.hive	1.0	1.0	0.0	0.0
root.qa1	60.0	60.0	23.0	23.0
root.admin1	1.0	1.0	0.0	0.0
root.dev2	1.0	1.0	0.0	0.0
root.bigdadmin	1.0	1.0	0.0	0.0
root.qa2	1.0	1.0	0.0	0.0
root.cloudera	1.0	1.0	0.0	0.0
root.bigdata	1.0	1.0	0.0	0.0
root.reporting	10.0	10.0	6.0	10.0
root.testing	1.0	1.0	1.0	0.0
root.informatica	1.0	1.0	0.0	0.0
root.admin2	1.0	1.0	0.0	0.0
root.banking	5.0	5.0	5.0	4.0
root.informatica	1.0	1.0	0.0	0.0
root.java	1.0	1.0	0.0	1.0
root.users	1.0	1.0	0.0	0.0

Cloudera HIVE BDR schedule via Python script

Worked on new user case or may  be to reduce my workload i force to write this script. replicating all the hive databases from production to DR cluster is very slow taking one day to complete.

thought of splitting  100 databases into multiple batches like 10 on each hive bdr configuration.  i felt creating 10 batches and updating configuration of more of painful work. so wrote below script to do that,

Python Script to Monitor and send email Cloudera Services and Roles health status

I know cloudera has in built feature to send the cluster status via email. but in our environment mail function is not working  through cloudera for some reason. tried all the configuration but no luck.

We could not contact Mail server support team to resolve issue. fortunately mail command is working from Unix server. so thought of writing own monitoring script using cloudera cm_client python module and send the alert to support group.

Try below my script

Sorting Yarn running jobs using python

Although hadoop Yarn resource manager provide nice UI interface to sorting  yarn jobs, sometimes it is difficult to filter the ideal session with custom sorting.

Below my script will sort the ideal session running for more than hours (3600000) and print with the application ID.

we can manually kill the session using  yarn command or we could also automate in the same script.



App Name: dev-claim_report1
Application id: application_1580724162250_1127
Total epsed time: 3.0 hours
queue Name: root.devqueue-1
Allocated memory: 1651 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1127/')
No long running jobs!

App Name: Spark shell
Application id: application_1580724162250_1151
Total epsed time: 1.0 hours
queue Name: root.devqueue-2
Allocated memory: 55 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1151/')

App Name: Spark shell
Application id: application_1580724162250_1152
Total epsed time: 1.0 hours
queue Name: root.devqueue-3
Allocated memory: 55 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1152/')

App Name: dev-claim_report2
Application id: application_1580724162250_1141
Total epsed time: 1.0 hours
queue Name: root.devqueue-4
Allocated memory: 1 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1141/')

Kudu tablet servers Metric check using python

Recently got chance to work on Kudu issue. developers started getting below error. after going through lot of documents i come to know that, cluster did not plan according to the kudu recommendation.

dropped due to backpressure. The service queue is full; it has 50 items

Below is kudu recommendation.

Scale

• Recommended maximum number of tablet servers is 100.
• Recommended maximum number of masters is 3.
• Recommended maximum amount of stored data, post-replication and post-compression, per tablet server is 8TB.
• Recommended maximum number of tablets per tablet server is 2000, post-replication.
• Maximum number of tablets per table for each tablet server is 60, post-replication, at table-creation time.

I need to find the numbers of tablets per server to developer, so that they can cleanup the tables or reduce the partition to meet the recommendation,

Below is my python script which will connect tablet server metrics and print the details

+------------------------------------------+-----------------+---------------+
|                  Server                  | Running tablets | Total_tablets |
+------------------------------------------+-----------------+---------------+
| devkn-01.tanu.com:8050 |      1790       |     11235     |
+------------------------------------------+-----------------+---------------+
| devkn-02.tanu.com:8050 |      1787       |     10970     |
+------------------------------------------+-----------------+---------------+
| devkn-03.tanu.com:8050 |      1924       |     11349     |
+------------------------------------------+-----------------+---------------+
| devkn-04.tanu.com:8050 |      1923       |     11325     |
+------------------------------------------+-----------------+---------------+
| devkn-05.tanu.com:8050 |      1838       |     11297     |
+------------------------------------------+-----------------+---------------+
| devkn-06.tanu.com:8050 |      1924       |     11299     |
+------------------------------------------+-----------------+---------------+
| devkn-07.tanu.com:8050 |      1788       |     11050     |
+------------------------------------------+-----------------+---------------+
| devkn-08.tanu.com:8050 |      1790       |     10564     |
+------------------------------------------+-----------------+---------------+
| devkn-09.tanu.com:8050 |      1921       |     10758     |
+------------------------------------------+-----------------+---------------+
| devkn-10.tanu.com:8050 |      1923       |     10899     |
+------------------------------------------+-----------------+---------------+
| devkn-11.tanu.com:8050 |      1868       |     9254      |
+------------------------------------------+-----------------+---------------+
| devkn-12.tanu.com:8050 |      2269       |     8101      |
+------------------------------------------+-----------------+---------------+
| devkn-13.tanu.com:8050 |      1802       |     10467     |
+------------------------------------------+-----------------+---------------+
| devkn-14.tanu.com:8050 |      1927       |     10875     |
+------------------------------------------+-----------------+---------------+
| devkn-15.tanu.com:8050 |      1601       |     10867     |
+------------------------------------------+-----------------+---------------+
| devkn-16.tanu.com:8050 |      2017       |     10088     |
+------------------------------------------+-----------------+---------------+
| devkn-17.tanu.com:8050 |      1793       |     10391     |
+------------------------------------------+-----------------+---------------+
| devkn-18.tanu.com:8050 |      1683       |     11631     |
+------------------------------------------+-----------------+---------------+
| devkn-19.tanu.com:8050 |      1946       |     9793      |
+------------------------------------------+-----------------+---------------+
| devkn-20.tanu.com:8050 |      1719       |     10488     |
+------------------------------------------+-----------------+---------------+
| devkn-21.tanu.com:8050 |      1703       |     9213      |
+------------------------------------------+-----------------+---------------+
| devkn-22.tanu.com:8050 |      1740       |     9920      |
+------------------------------------------+-----------------+---------------+
| devkn-23.tanu.com:8050 |      1827       |     9953      |
+------------------------------------------+-----------------+---------------+
| devkn-24.tanu.com:8050 |      1929       |     10094     |
+------------------------------------------+-----------------+---------------+

Cloudera cluster creation on google compute instance

Wanted to quickly launch my cloudera cluster in google cloud(since i had some free credit wanted to try effectively)  similar to on prem cluster like single sign  on  all the linux nodes.

initially i though of integrate all linux servers with Active directory + SSSD client but later i moved to MIT kerberos + Open LDAP client + SASL passthrough.

This script has 2 part

PART 1: will create No. of gcp instances, create hadoop users/groups, install SASL/openldap/MIT kerberos/Cloudera agent and Manager

PART 2: Will add the hosts into cloudera manager,create cluster/add hdfs and zookeeper services.(still working on adding more services)

Part 2 of this script can be easily  scale up with any cloud providers(AWS,AZURE) as long as cloudera manager url  is exposed to internet.

cloudera manager TLS via python api

import socket
from cm_api.api_client import ApiResource
from cm_api.api_client import ApiException
from cm_api.endpoints.cms import ClouderaManager
import ssl


#CM_HOST = "cm.tanu.com"
CM_HOST = "cm.tanu.com"
#api = ApiResource(CM_HOST,version=13, username="admin", password="admin")
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
cxt = ssl.create_default_context(cafile="/app/ca/ca.pem")

api = ApiResource(CM_HOST,version=12, username="admin", password="admin",use_tls=True,ssl_context=cxt)

clu=api.get_cluster('Cluster 1')
hdfs=clu.get_service('hdfs')

#hdfs_ssl_enable = { 'hdfs_hadoop_ssl_enabled' : 'true','ssl_server_keystore_location' : '/var/tmp/cm.jks','ssl_server_keystore_password':'test123','ssl_server_keystore_keypassword':'test123' }
cm_ssl_conf = {'WEB_TLS':'true','KEYSTORE_PATH':'/opt/cloudera-manager/ssl/jks/javakeystore.jks','KEYSTORE_PASSWORD':'iCpjC"7]','TRUSTSTORE_PATH':'/opt/cloudera-manager/ssl/jks/ca_combined.jks','TRUSTSTORE_PASSWORD':'test123'}


#hdfs.update_config(svc_config=hdfs_ssl_enable)
#for name,config in hdfs.get_config(view="full")[0].items():
#       print "%s - %s - %s" %(name,config.relatedName,config.description)
#       print "%s --> %s" %(name,config.relatedName)
x=ClouderaManager(api)
for name,config in x.get_config(view="full").items():
        print "%s  --> %s" %(name,config)
#print(x.get_config(view="full"))
#x.update_config(cm_ssl_conf)
print(hdfs)
print(clu)

for h in api.get_all_hosts():
        print(h.hostname)
        print(h.get_config())

print(api)

Python CGI Webserver with SSL

This is sample code to run python CGI Web server with SSL. Note that i used CA signed certificate  due to that i need to provide CA certificate as well in my python code which may not be necessary if you use self signed certificate.

       
#!/usr/bin/python

import BaseHTTPServer, SimpleHTTPServer,CGIHTTPServer

import ssl

import cgitb; cgitb.enable()



#httpd = BaseHTTPServer.HTTPServer(('web.tanu.com', 8000), SimpleHTTPServer.SimpleHTTPRequestHandler)

handler = CGIHTTPServer.CGIHTTPRequestHandler

httpd = BaseHTTPServer.HTTPServer(('web.tanu.com', 8000), handler)

handler.cgi_directories = ["/cgi-bin"]



httpd.socket = ssl.wrap_socket (httpd.socket,keyfile='./certs/key1.pem', certfile='./certs/cert.pem',ca_certs='./ca_cert.pem,server_side=True,do_handshake_on_connect=True)

handler.have_fork=False

httpd.serve_forever()



       
 

User defined exception class in python

Today i learned something about user defined exception class in python and posting  in my blog so that i can remember in future :)


This is my check_age.py file

from own_Exeption import teenageException,kidException,childException

def age(x):
        try:
                if( x >= 14 ) and (x <= 19):
                        raise teenageException
                elif(x >= 8 ) and (x <= 13) :
                        raise kidException
                elif(x >= 2 ) and (x <= 7) :
                        raise childException
                #else:
                #       print("Ags is : "+str(x))
        except teenageException:
                print("WARNING : You are teenage not allowed here")
        except kidException:
                print("WARNING: you are kid not allowed here")
        except childException:
                print("WARNING: you are child not allowed here")
        else:
                print("INFO: You are the right age {} adult person".format(x))
        finally:
                print("INFO: End of the try/except block")

age(3)
age(17)
age(21)

Here is my user defined exception class

cat own_Exeption.py

class teenageException(Exception):
        pass
class kidException(Exception):
        pass
class childException(Exception):
        pass  

Initially i tried without try/except block. it seems like execution getting halt as soon as raise block invoked. So i tried with try/except block to catch the respective and print the warning message. here is my output.

       

WARNING: you are child not allowed here
INFO: End of the try/except block
WARNING : You are teenage not allowed here
INFO: End of the try/except block
INFO: You are the right age 21 adult person
INFO: End of the try/except block


       
 


i could also use raise statement inside except block like below to halt the code from further execution

       

from own_Exeption import teenageException,kidException,childException

def age(x):
        try:
                if( x >= 14 ) and (x <= 19):
                        raise teenageException
                elif(x >= 8 ) and (x <= 13) :
                        raise kidException
                elif(x >= 2 ) and (x <= 7) :
                        raise childException
                #else:
                #       print("Ags is : "+str(x))
        except teenageException:
                print("WARNING : You are teenage not allowed here")
                raise teenageException
        except kidException:
                print("WARNING: you are kid not allowed here")
                raise kidException
        except childException:
                print("WARNING: you are child not allowed here")
                raise childException
        else:
                print("INFO: You are the right age {} adult person".format(x))
        finally:
                print("INFO: End of the try/except block")

age(3)
age(17)
age(21)


       
 

This  will stop execution  withing first error.

       

WARNING: you are child not allowed here
INFO: End of the try/except block
Traceback (most recent call last):
  File "sat1.py", line 27, in 
    age(3)
  File "sat1.py", line 21, in age
    raise childException
own_Exeption.childException

       
 

Python script to Upload files into AWS S3 bucket

Run this script with below arguments

./aws-s3upload.py    S3_BUCKET_NAME SOURCE_FILE  S3_TARGET_DIR

Example ./aws-s3upload.py  test_s3_bucket /app/file/tanu.jpg  Images/data

above command will upload the files into test_s3_bucket/Images/data/tanu.jpg



#!/usr/bin/env python
import boto.ec2
import sys
import os
import ntpath

#### Configuration section ####
IAM_ID = 'PLACE IAM ID HERE'
IAM_SECRET ='PLACE IAM SECRET HERE'
REGION = 'us-east-1'


conn = boto.s3.connect_to_region(REGION, aws_access_key_id=IAM_ID, aws_secret_access_key=IAM_SECRET)

if (len(sys.argv) != 4 ):
        print "USAGE ./aws-s3upload.py S3_BUCKET_NAME FileName TARGETDIR"
        sys.exit(1)

s3_bucket = sys.argv[1]
filename = sys.argv[2]
targetpath = sys.argv[3]

def percent_cb(complete, total):
    sys.stdout.write('.')
    sys.stdout.flush()


try:
        print 'Uploading %s to Amazon S3 bucket %s' % (filename, s3_bucket)
        bucket = conn.get_bucket(s3_bucket)
        file=ntpath.basename(filename)
        full_key_name = os.path.join(targetpath, file)
        print("Target Upload Location " + full_key_name)
        k = bucket.new_key(full_key_name)
        k.set_contents_from_filename(filename,cb=percent_cb, num_cb=10)
except Exception,e:
    print str(e)
    print "error"