Kudu tablet servers Metric check using python

Recently got chance to work on Kudu issue. developers started getting below error. after going through lot of documents i come to know that, cluster did not plan according to the kudu recommendation.

dropped due to backpressure. The service queue is full; it has 50 items

Below is kudu recommendation.

Scale

• Recommended maximum number of tablet servers is 100.
• Recommended maximum number of masters is 3.
• Recommended maximum amount of stored data, post-replication and post-compression, per tablet server is 8TB.
• Recommended maximum number of tablets per tablet server is 2000, post-replication.
• Maximum number of tablets per table for each tablet server is 60, post-replication, at table-creation time.

I need to find the numbers of tablets per server to developer, so that they can cleanup the tables or reduce the partition to meet the recommendation,

Below is my python script which will connect tablet server metrics and print the details

+------------------------------------------+-----------------+---------------+
|                  Server                  | Running tablets | Total_tablets |
+------------------------------------------+-----------------+---------------+
| devkn-01.tanu.com:8050 |      1790       |     11235     |
+------------------------------------------+-----------------+---------------+
| devkn-02.tanu.com:8050 |      1787       |     10970     |
+------------------------------------------+-----------------+---------------+
| devkn-03.tanu.com:8050 |      1924       |     11349     |
+------------------------------------------+-----------------+---------------+
| devkn-04.tanu.com:8050 |      1923       |     11325     |
+------------------------------------------+-----------------+---------------+
| devkn-05.tanu.com:8050 |      1838       |     11297     |
+------------------------------------------+-----------------+---------------+
| devkn-06.tanu.com:8050 |      1924       |     11299     |
+------------------------------------------+-----------------+---------------+
| devkn-07.tanu.com:8050 |      1788       |     11050     |
+------------------------------------------+-----------------+---------------+
| devkn-08.tanu.com:8050 |      1790       |     10564     |
+------------------------------------------+-----------------+---------------+
| devkn-09.tanu.com:8050 |      1921       |     10758     |
+------------------------------------------+-----------------+---------------+
| devkn-10.tanu.com:8050 |      1923       |     10899     |
+------------------------------------------+-----------------+---------------+
| devkn-11.tanu.com:8050 |      1868       |     9254      |
+------------------------------------------+-----------------+---------------+
| devkn-12.tanu.com:8050 |      2269       |     8101      |
+------------------------------------------+-----------------+---------------+
| devkn-13.tanu.com:8050 |      1802       |     10467     |
+------------------------------------------+-----------------+---------------+
| devkn-14.tanu.com:8050 |      1927       |     10875     |
+------------------------------------------+-----------------+---------------+
| devkn-15.tanu.com:8050 |      1601       |     10867     |
+------------------------------------------+-----------------+---------------+
| devkn-16.tanu.com:8050 |      2017       |     10088     |
+------------------------------------------+-----------------+---------------+
| devkn-17.tanu.com:8050 |      1793       |     10391     |
+------------------------------------------+-----------------+---------------+
| devkn-18.tanu.com:8050 |      1683       |     11631     |
+------------------------------------------+-----------------+---------------+
| devkn-19.tanu.com:8050 |      1946       |     9793      |
+------------------------------------------+-----------------+---------------+
| devkn-20.tanu.com:8050 |      1719       |     10488     |
+------------------------------------------+-----------------+---------------+
| devkn-21.tanu.com:8050 |      1703       |     9213      |
+------------------------------------------+-----------------+---------------+
| devkn-22.tanu.com:8050 |      1740       |     9920      |
+------------------------------------------+-----------------+---------------+
| devkn-23.tanu.com:8050 |      1827       |     9953      |
+------------------------------------------+-----------------+---------------+
| devkn-24.tanu.com:8050 |      1929       |     10094     |
+------------------------------------------+-----------------+---------------+

Cloudera cluster creation on google compute instance

Wanted to quickly launch my cloudera cluster in google cloud(since i had some free credit wanted to try effectively)  similar to on prem cluster like single sign  on  all the linux nodes.

initially i though of integrate all linux servers with Active directory + SSSD client but later i moved to MIT kerberos + Open LDAP client + SASL passthrough.

This script has 2 part

PART 1: will create No. of gcp instances, create hadoop users/groups, install SASL/openldap/MIT kerberos/Cloudera agent and Manager

PART 2: Will add the hosts into cloudera manager,create cluster/add hdfs and zookeeper services.(still working on adding more services)

Part 2 of this script can be easily  scale up with any cloud providers(AWS,AZURE) as long as cloudera manager url  is exposed to internet.

google cloud compute instance creation using python script

First steps to create dedicated service account for our python script with name libcloud( since we are going to use apache libcloud python framework )

And then map the necessary roles to the service account to create compute instances

After mapping above roles, i was not able to create instances in my python scripts, it was keep on throwing below exception


    response = responseCls(**kwargs)
  File "/home/sathish/miniconda3/lib/python3.7/site-packages/libcloud/common/base.py", line 154, in __init__
    self.object = self.parse_body()
  File "/home/sathish/miniconda3/lib/python3.7/site-packages/libcloud/common/google.py", line 267, in parse_body
    raise GoogleBaseError(message, self.status, code)
libcloud.common.google.GoogleBaseError: "The user does not have access to service account '123333333333-compute@developer.gserviceaccount.com'.  User: 'libcloud@xxxxxxxxx.iam.gserviceaccount.com'.  Ask a project owner to grant you the iam.serviceAccountUser role on the service account"

Then i granted additional below roles.

then try below python script to create instance

       
from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

ComputeEngine = get_driver(Provider.GCE)
# Note that the 'PEM file' argument can either be the JSON format or
# the P12 format.
driver = ComputeEngine('libcloud@xxxxx.iam.gserviceaccount.com','/home/sathish/gcp_pem.json',
                       project='ferrous-weaver-xxxxx')

#(driver.list_images())

### Function to findout the gcp image name to provide arg in create instance function ###

def list_all_gcp_images(driver):
        images = driver.list_images()
        for image in images:
                print(image)

### use below function to create compute instance ##

def create_instance(driver):
        s = 'n1-standard-1'
        i = 'centos-7-v20191121'
        z = 'us-central1-a'

        sa_scopes = [{'email': 'default','scopes': ['storage-ro']}]
        node_1 = driver.create_node("n2", s, i, z, ex_service_accounts=sa_scopes)

create_instance(driver)
list_all_gcp_images(driver)
       
 

cloudera manager TLS via python api

import socket
from cm_api.api_client import ApiResource
from cm_api.api_client import ApiException
from cm_api.endpoints.cms import ClouderaManager
import ssl


#CM_HOST = "cm.tanu.com"
CM_HOST = "cm.tanu.com"
#api = ApiResource(CM_HOST,version=13, username="admin", password="admin")
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
cxt = ssl.create_default_context(cafile="/app/ca/ca.pem")

api = ApiResource(CM_HOST,version=12, username="admin", password="admin",use_tls=True,ssl_context=cxt)

clu=api.get_cluster('Cluster 1')
hdfs=clu.get_service('hdfs')

#hdfs_ssl_enable = { 'hdfs_hadoop_ssl_enabled' : 'true','ssl_server_keystore_location' : '/var/tmp/cm.jks','ssl_server_keystore_password':'test123','ssl_server_keystore_keypassword':'test123' }
cm_ssl_conf = {'WEB_TLS':'true','KEYSTORE_PATH':'/opt/cloudera-manager/ssl/jks/javakeystore.jks','KEYSTORE_PASSWORD':'iCpjC"7]','TRUSTSTORE_PATH':'/opt/cloudera-manager/ssl/jks/ca_combined.jks','TRUSTSTORE_PASSWORD':'test123'}


#hdfs.update_config(svc_config=hdfs_ssl_enable)
#for name,config in hdfs.get_config(view="full")[0].items():
#       print "%s - %s - %s" %(name,config.relatedName,config.description)
#       print "%s --> %s" %(name,config.relatedName)
x=ClouderaManager(api)
for name,config in x.get_config(view="full").items():
        print "%s  --> %s" %(name,config)
#print(x.get_config(view="full"))
#x.update_config(cm_ssl_conf)
print(hdfs)
print(clu)

for h in api.get_all_hosts():
        print(h.hostname)
        print(h.get_config())

print(api)

Python CGI Webserver with SSL

This is sample code to run python CGI Web server with SSL. Note that i used CA signed certificate  due to that i need to provide CA certificate as well in my python code which may not be necessary if you use self signed certificate.

       
#!/usr/bin/python

import BaseHTTPServer, SimpleHTTPServer,CGIHTTPServer

import ssl

import cgitb; cgitb.enable()



#httpd = BaseHTTPServer.HTTPServer(('web.tanu.com', 8000), SimpleHTTPServer.SimpleHTTPRequestHandler)

handler = CGIHTTPServer.CGIHTTPRequestHandler

httpd = BaseHTTPServer.HTTPServer(('web.tanu.com', 8000), handler)

handler.cgi_directories = ["/cgi-bin"]



httpd.socket = ssl.wrap_socket (httpd.socket,keyfile='./certs/key1.pem', certfile='./certs/cert.pem',ca_certs='./ca_cert.pem,server_side=True,do_handshake_on_connect=True)

handler.have_fork=False

httpd.serve_forever()



       
 

How to implement SPNEGO authentication in kerberos with python

Let me explain my best  how spnego authentication work and how can we implement SPNEGO authentication in a few lines in python.

In this test, I used MIT kerberos server (Active directory installation and configuration is more painful for me) refer my blog on how to setup and configure MIT kerberos in centos. Linux Ldap authentication with kerberos backend and openldap SASL Passthrough  and kerberos  (python module to implement the spengo authentication (pip install kerberos))


We need to create service principle in Kerberos server for the domain we are going to use www.sathish.com. since webserver use HTTP protocol, kerberos service principle would be HTTP.

So we need to create HTTP service principle with below name for www.sathish.com. ( TANU.COM is my kerberos domain.)

HTTP/www.sathish.com@TANU.COM

Run following commands in kerberos server to create the keytab.

kadmin.local -q "addprinc -randkey HTTP/www.sathish.com@TANU.COM"
kadmin.local -q "ktadd -k /www/http/secure/www.sathish.com.keytab HTTP/www.sathish.com

set below environment variable to point the keytab

export KRB5_KTNAME=/www/http/secure/www.sathish.com.keytab

(if we dont set above variable python code with throw below exception

    rc, state = kerberos.authGSSServerInit('HTTP')
GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('', 100002))

)

Now we are good to run our own code to implement the spnego kerberos authentication.


Webserver (simple python http server with custom code) running on CENTOS  with url http:/www.sathish.com



Client machine(one Linux  and  one windows 10 desktop)

How SPNEGO work:

1) client request url using Internet explore   ----> http://www.sathish.com

2) webserver get request and check  "Authorization" header. if header not present in the request , webserver send the response with 401 and WWW-Authenticate','Negotiate' header  back to client

                       header=s.headers.get('Authorization')
                        if not header:
                        s.send_response(401)
                        s.send_header('WWW-Authenticate','Negotiate')

3) Internet explore get the response from webserver and request URL with Authorization header with valid token.

4)  Then webserver read the token from Authorization header and decrypt  token using service key stored in the keytab for this HTTP Service principle HTTP/www.sathish.com@TANU.COM.

If the decryption is successful then user token is valid and we can mark user authentication is completed.

               rc, state = kerberos.authGSSServerInit('HTTP')
                if rc != kerberos.AUTH_GSS_COMPLETE:
                        return None
                rc = kerberos.authGSSServerStep(state, token)
                if rc == kerberos.AUTH_GSS_COMPLETE:
                        user = kerberos.authGSSServerUserName(state)

  if token is valid, webserver can  retrieve the username.

5) Post Authentication webserver response back to client.

Here is the full code for testing

       
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
import sys
import kerberos
import re

class AuthHandler(SimpleHTTPRequestHandler):
    ''' Main class to present webpages and authentication. '''
    def do_HEAD(self):
        print "send header"
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_AUTHHEAD(self):
        print "send header"
        self.send_response(401)
        #self.send_header('WWW-Authenticate', 'Basic realm=\"Test\"')
        self.send_header('WWW-Authenticate', 'Negotiate')
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        ''' Present frontpage with user authentication. '''
        if self.headers.getheader('Authorization') == None:
            self.do_AUTHHEAD()
            self.wfile.write('no auth header received')
            pass
        #elif self.headers.getheader('Authorization') == 'Negotiate':
        elif re.match(r'^Negotiate', self.headers.getheader('Authorization')):
            #print self.headers.getheader('Authorization')
            header=self.headers.getheader('Authorization')
            token = ''.join(header.split()[1:])
            print token
            rc, state = kerberos.authGSSServerInit('HTTP')
            if rc != kerberos.AUTH_GSS_COMPLETE:
                return None
            rc = kerberos.authGSSServerStep(state, token)
            if rc == kerberos.AUTH_GSS_COMPLETE:
                user = kerberos.authGSSServerUserName(state)
                print user
                SimpleHTTPRequestHandler.do_GET(self)
                pass
        else:
            self.do_AUTHHEAD()
            self.wfile.write(self.headers.getheader('Authorization'))
            self.wfile.write('not authenticated')
            pass

def test(HandlerClass = AuthHandler,
         ServerClass = BaseHTTPServer.HTTPServer):
    BaseHTTPServer.test(HandlerClass, ServerClass)


if __name__ == '__main__':
    if len(sys.argv)<2: code="" port="" print="" simpleauthserver.py="" sys.exit="" test="" usage="">
 

Here is the curl with negotiate request output



[root@krbserver ~]# curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt  http://www.sathish.com/ -v
* About to connect() to www.sathish.com port 80 (#0)
*   Trying 192.168.100.31...
* Connected to www.sathish.com (192.168.100.31) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.sathish.com
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 401 Unauthorized
< Server: BaseHTTP/0.3 Python/2.7.5
< Date: Sun, 13 May 2018 17:09:17 GMT
< WWW-Authenticate: Negotiate
* Closing connection 0
* Issue another request to this URL: 'http://www.sathish.com/'
* About to connect() to www.sathish.com port 80 (#1)
*   Trying 192.168.100.31...
* Connected to www.sathish.com (192.168.100.31) port 80 (#1)
* Server auth using GSS-Negotiate with user ''
> GET / HTTP/1.0
> Authorization: Negotiate YIICWwYJKoZIhvcSAQICAQBuggJKMIICRqADAgEFoQMCAQ6iBwMFACAAAACjggFiYYIBXjCCAVqgAwIBBaEKGwhUQU5VLkNPTaIZMBegAwIBA6EQMA4bBEhUVFAbBmtyYmNuMaOCASowggEmoAMCARChAwIBA6KCARgEggEUoewKZoNfchhCOjox2e4y465Wnz2E94henxLTzQq70A270SbVPef5oIkqXXupDK9/JPhT2QPRUXEhWYoH41KwxfK4Q28H2OLpzkvCCk0wVLjNFrJZjhEm1mxV4eKUl4AAi/nSwYUCZGAEFrEUVwWQwP8apFCsGolvZXP9Skc0XViAZTM5JKL6mVgQcwwuvs+J6Cf6elzZPR+7BLrJRFHlUTvMS3C8H+WClAh9KDeSvSbxYvG4ZRhNgXqRrD259O5iGKySf126ToG00Zece4Fz0rHjSRJUnrV4H2Npku2sMSo7M9DnwBiDW8kQ84yQnpXbc1hOqiFIOWRc31ZuQ2O0OUormHm3mUZY+ABC2zA8BdIPLZvspIHKMIHHoAMCARCigb8EgbxC+6wp6oKhdTtVS6nFomRylJ1p8t1hn8nqs/rsNTHoia6csinnPP2ZR2oqydnaVq9cGMqA6q7U5Yz1iXlf0dGOWmOewucN8URLkYDC6sne2acqNkWCpOVyJJ9/AaC5QTFvVyR+Nj7aRW3LpSgJdrwrMybrfgYawthj0q44z7+FQi1T0K42sLmTvelkqvXFdVBn2/aqcaXCkkA9OI5IrZH0fqwt25nMm/mTkhsgH+ntuzjtm4VDViLV2DuZcA==
> User-Agent: curl/7.29.0
> Host: www.sathish.com
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BaseHTTP/0.3 Python/2.7.5
< Date: Sun, 13 May 2018 17:09:19 GMT
< Content-type: text/html
<
* Closing connection 1
<html><head><title>Title goes here.</title></head><body><p>This is a test.</p></body></html>[root@krbserver ~]#

User defined exception class in python

Today i learned something about user defined exception class in python and posting  in my blog so that i can remember in future :)


This is my check_age.py file

from own_Exeption import teenageException,kidException,childException

def age(x):
        try:
                if( x >= 14 ) and (x <= 19):
                        raise teenageException
                elif(x >= 8 ) and (x <= 13) :
                        raise kidException
                elif(x >= 2 ) and (x <= 7) :
                        raise childException
                #else:
                #       print("Ags is : "+str(x))
        except teenageException:
                print("WARNING : You are teenage not allowed here")
        except kidException:
                print("WARNING: you are kid not allowed here")
        except childException:
                print("WARNING: you are child not allowed here")
        else:
                print("INFO: You are the right age {} adult person".format(x))
        finally:
                print("INFO: End of the try/except block")

age(3)
age(17)
age(21)

Here is my user defined exception class

cat own_Exeption.py

class teenageException(Exception):
        pass
class kidException(Exception):
        pass
class childException(Exception):
        pass  

Initially i tried without try/except block. it seems like execution getting halt as soon as raise block invoked. So i tried with try/except block to catch the respective and print the warning message. here is my output.

       

WARNING: you are child not allowed here
INFO: End of the try/except block
WARNING : You are teenage not allowed here
INFO: End of the try/except block
INFO: You are the right age 21 adult person
INFO: End of the try/except block


       
 


i could also use raise statement inside except block like below to halt the code from further execution

       

from own_Exeption import teenageException,kidException,childException

def age(x):
        try:
                if( x >= 14 ) and (x <= 19):
                        raise teenageException
                elif(x >= 8 ) and (x <= 13) :
                        raise kidException
                elif(x >= 2 ) and (x <= 7) :
                        raise childException
                #else:
                #       print("Ags is : "+str(x))
        except teenageException:
                print("WARNING : You are teenage not allowed here")
                raise teenageException
        except kidException:
                print("WARNING: you are kid not allowed here")
                raise kidException
        except childException:
                print("WARNING: you are child not allowed here")
                raise childException
        else:
                print("INFO: You are the right age {} adult person".format(x))
        finally:
                print("INFO: End of the try/except block")

age(3)
age(17)
age(21)


       
 

This  will stop execution  withing first error.

       

WARNING: you are child not allowed here
INFO: End of the try/except block
Traceback (most recent call last):
  File "sat1.py", line 27, in 
    age(3)
  File "sat1.py", line 21, in age
    raise childException
own_Exeption.childException