Thursday, April 2, 2020

Python script to automate Yarn pool utilization report mail

i was working on simple requirement to send the each Yarn pool utilization hourly to management team.

Although cloudera offer cluster utilization report which will provide detailed report about Yarn utilization, there are some limitation. so i used my scripting knowledge to implement on my own.

please find below python and script and the email output.

It will generate report like below.

From: ClouderaAdministrator-
Sent: Thursday, April 2, 2020 8:00:01 AM (UTC+00:00) Monrovia, Reykjavik
To: admins; admins@tanu.com
Subject: Prod Cluster Yarn Utilization report

Report generation time:

2020-04-02 04:00:01.043615

Total Resource availabe:

1440 Vcores,

7.03TB

Total Resource Utilized(value):

916 Vcores,

3.95TB

Total Resource Utilized(%):

63.61 %,

56.24%

Queue Name

Max Allocation(cpu)

Max Allocation(memory)

Cpu Utilizied(%)

Memory Utilized(%)

root.default

1.0

1.0

0.0

0.0

root.dev1

1.0

1.0

0.0

0.0

root.hive

1.0

1.0

0.0

0.0

root.qa1

60.0

60.0

23.0

23.0

root.admin1

1.0

1.0

0.0

0.0

root.dev2

1.0

1.0

0.0

0.0

root.bigdadmin

1.0

1.0

0.0

0.0

root.qa2

1.0

1.0

0.0

0.0

root.cloudera

1.0

1.0

0.0

0.0

root.bigdata

1.0

1.0

0.0

0.0

root.reporting

10.0

10.0

6.0

10.0

root.testing

1.0

1.0

1.0

0.0

root.informatica

1.0

1.0

0.0

0.0

root.admin2

1.0

1.0

0.0

0.0

root.banking

5.0

5.0

5.0

4.0

root.informatica

1.0

1.0

0.0

0.0

root.java

1.0

1.0

0.0

1.0

root.users

1.0

1.0

0.0

0.0