Python script to automate Yarn pool utilization report mail

i was working on simple requirement to send the each Yarn pool utilization hourly to management team.

Although cloudera offer cluster utilization report which will provide detailed report about Yarn utilization, there are some limitation. so i used my scripting knowledge to implement on my own.

please find below python and script and the email output.

It will generate report like below.

From: ClouderaAdministrator-
Sent: Thursday, April 2, 2020 8:00:01 AM (UTC+00:00) Monrovia, Reykjavik
To: admins; admins@tanu.com
Subject: Prod Cluster Yarn Utilization report

Report generation time:	2020-04-02 04:00:01.043615
Total Resource availabe:	1440 Vcores,	7.03TB
Total Resource Utilized(value):	916 Vcores,	3.95TB
Total Resource Utilized(%):	63.61 %,	56.24%
Queue Name	Max Allocation(cpu)	Max Allocation(memory)	Cpu Utilizied(%)	Memory Utilized(%)
root.default	1.0	1.0	0.0	0.0
root.dev1	1.0	1.0	0.0	0.0
root.hive	1.0	1.0	0.0	0.0
root.qa1	60.0	60.0	23.0	23.0
root.admin1	1.0	1.0	0.0	0.0
root.dev2	1.0	1.0	0.0	0.0
root.bigdadmin	1.0	1.0	0.0	0.0
root.qa2	1.0	1.0	0.0	0.0
root.cloudera	1.0	1.0	0.0	0.0
root.bigdata	1.0	1.0	0.0	0.0
root.reporting	10.0	10.0	6.0	10.0
root.testing	1.0	1.0	1.0	0.0
root.informatica	1.0	1.0	0.0	0.0
root.admin2	1.0	1.0	0.0	0.0
root.banking	5.0	5.0	5.0	4.0
root.informatica	1.0	1.0	0.0	0.0
root.java	1.0	1.0	0.0	1.0
root.users	1.0	1.0	0.0	0.0

Sorting Yarn running jobs using python

Although hadoop Yarn resource manager provide nice UI interface to sorting  yarn jobs, sometimes it is difficult to filter the ideal session with custom sorting.

Below my script will sort the ideal session running for more than hours (3600000) and print with the application ID.

we can manually kill the session using  yarn command or we could also automate in the same script.



App Name: dev-claim_report1
Application id: application_1580724162250_1127
Total epsed time: 3.0 hours
queue Name: root.devqueue-1
Allocated memory: 1651 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1127/')
No long running jobs!

App Name: Spark shell
Application id: application_1580724162250_1151
Total epsed time: 1.0 hours
queue Name: root.devqueue-2
Allocated memory: 55 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1151/')

App Name: Spark shell
Application id: application_1580724162250_1152
Total epsed time: 1.0 hours
queue Name: root.devqueue-3
Allocated memory: 55 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1152/')

App Name: dev-claim_report2
Application id: application_1580724162250_1141
Total epsed time: 1.0 hours
queue Name: root.devqueue-4
Allocated memory: 1 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1141/')