Cloudera HIVE BDR schedule via Python script

Worked on new user case or may  be to reduce my workload i force to write this script. replicating all the hive databases from production to DR cluster is very slow taking one day to complete.

thought of splitting  100 databases into multiple batches like 10 on each hive bdr configuration.  i felt creating 10 batches and updating configuration of more of painful work. so wrote below script to do that,

Python Script to Monitor and send email Cloudera Services and Roles health status

I know cloudera has in built feature to send the cluster status via email. but in our environment mail function is not working  through cloudera for some reason. tried all the configuration but no luck.

We could not contact Mail server support team to resolve issue. fortunately mail command is working from Unix server. so thought of writing own monitoring script using cloudera cm_client python module and send the alert to support group.

Try below my script

Sorting Yarn running jobs using python

Although hadoop Yarn resource manager provide nice UI interface to sorting  yarn jobs, sometimes it is difficult to filter the ideal session with custom sorting.

Below my script will sort the ideal session running for more than hours (3600000) and print with the application ID.

we can manually kill the session using  yarn command or we could also automate in the same script.



App Name: dev-claim_report1
Application id: application_1580724162250_1127
Total epsed time: 3.0 hours
queue Name: root.devqueue-1
Allocated memory: 1651 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1127/')
No long running jobs!

App Name: Spark shell
Application id: application_1580724162250_1151
Total epsed time: 1.0 hours
queue Name: root.devqueue-2
Allocated memory: 55 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1151/')

App Name: Spark shell
Application id: application_1580724162250_1152
Total epsed time: 1.0 hours
queue Name: root.devqueue-3
Allocated memory: 55 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1152/')

App Name: dev-claim_report2
Application id: application_1580724162250_1141
Total epsed time: 1.0 hours
queue Name: root.devqueue-4
Allocated memory: 1 gb
('Tracking Url: ', u'http://devmn-02.tanu.com:8088/proxy/application_1580724162250_1141/')

Kudu tablet servers Metric check using python

Recently got chance to work on Kudu issue. developers started getting below error. after going through lot of documents i come to know that, cluster did not plan according to the kudu recommendation.

dropped due to backpressure. The service queue is full; it has 50 items

Below is kudu recommendation.

Scale

• Recommended maximum number of tablet servers is 100.
• Recommended maximum number of masters is 3.
• Recommended maximum amount of stored data, post-replication and post-compression, per tablet server is 8TB.
• Recommended maximum number of tablets per tablet server is 2000, post-replication.
• Maximum number of tablets per table for each tablet server is 60, post-replication, at table-creation time.

I need to find the numbers of tablets per server to developer, so that they can cleanup the tables or reduce the partition to meet the recommendation,

Below is my python script which will connect tablet server metrics and print the details

+------------------------------------------+-----------------+---------------+
|                  Server                  | Running tablets | Total_tablets |
+------------------------------------------+-----------------+---------------+
| devkn-01.tanu.com:8050 |      1790       |     11235     |
+------------------------------------------+-----------------+---------------+
| devkn-02.tanu.com:8050 |      1787       |     10970     |
+------------------------------------------+-----------------+---------------+
| devkn-03.tanu.com:8050 |      1924       |     11349     |
+------------------------------------------+-----------------+---------------+
| devkn-04.tanu.com:8050 |      1923       |     11325     |
+------------------------------------------+-----------------+---------------+
| devkn-05.tanu.com:8050 |      1838       |     11297     |
+------------------------------------------+-----------------+---------------+
| devkn-06.tanu.com:8050 |      1924       |     11299     |
+------------------------------------------+-----------------+---------------+
| devkn-07.tanu.com:8050 |      1788       |     11050     |
+------------------------------------------+-----------------+---------------+
| devkn-08.tanu.com:8050 |      1790       |     10564     |
+------------------------------------------+-----------------+---------------+
| devkn-09.tanu.com:8050 |      1921       |     10758     |
+------------------------------------------+-----------------+---------------+
| devkn-10.tanu.com:8050 |      1923       |     10899     |
+------------------------------------------+-----------------+---------------+
| devkn-11.tanu.com:8050 |      1868       |     9254      |
+------------------------------------------+-----------------+---------------+
| devkn-12.tanu.com:8050 |      2269       |     8101      |
+------------------------------------------+-----------------+---------------+
| devkn-13.tanu.com:8050 |      1802       |     10467     |
+------------------------------------------+-----------------+---------------+
| devkn-14.tanu.com:8050 |      1927       |     10875     |
+------------------------------------------+-----------------+---------------+
| devkn-15.tanu.com:8050 |      1601       |     10867     |
+------------------------------------------+-----------------+---------------+
| devkn-16.tanu.com:8050 |      2017       |     10088     |
+------------------------------------------+-----------------+---------------+
| devkn-17.tanu.com:8050 |      1793       |     10391     |
+------------------------------------------+-----------------+---------------+
| devkn-18.tanu.com:8050 |      1683       |     11631     |
+------------------------------------------+-----------------+---------------+
| devkn-19.tanu.com:8050 |      1946       |     9793      |
+------------------------------------------+-----------------+---------------+
| devkn-20.tanu.com:8050 |      1719       |     10488     |
+------------------------------------------+-----------------+---------------+
| devkn-21.tanu.com:8050 |      1703       |     9213      |
+------------------------------------------+-----------------+---------------+
| devkn-22.tanu.com:8050 |      1740       |     9920      |
+------------------------------------------+-----------------+---------------+
| devkn-23.tanu.com:8050 |      1827       |     9953      |
+------------------------------------------+-----------------+---------------+
| devkn-24.tanu.com:8050 |      1929       |     10094     |
+------------------------------------------+-----------------+---------------+