Server-Server mutual authentication - Apache ZooKeeper (2024)

During leader activation(leader-election) phase, all the servers in an ensemble will participate to form quorum. By default, this communication is not authenticated. This guide describes how to enable secure communication between the quorum peer servers using SASL mechanism. ZooKeeper supports Kerberosor DIGEST-MD5 as your authentication scheme.

Presently, this feature is supported only in ZooKeeper 3.4.10+ version. Thisis implemented using JIRA issueZOOKEEPER-1045,and thepatch is availablelinked from that JIRA.The feature code will be forward ported to ZooKeeper 3.5.x and 3.6.x versions viaseparate JIRA task ZOOKEEPER-2639.

Following configurations should be added in $conf/zoo.cfg configuration file.

quorum.auth.enableSasl=truequorum.auth.learnerRequireSasl=truequorum.auth.serverRequireSasl=truequorum.auth.learner.saslLoginContext=QuorumLearnerquorum.auth.server.saslLoginContext=QuorumServerquorum.auth.kerberos.servicePrincipal=servicename/_HOSTquorum.cnxn.threads.size=20

1. Sets to enable quorum authentication using SASL.
   

   zoo.cfg

   # Defaulting to falsequorum.auth.enableSasl=true

2. Sets to connect using quorum authentication. If this is true, quorum peer learner will send authentication packet to quorum peer server then proceeds with LE on successful authentication. If false, then proceeds with LE without any authentication. This can be used while upgrading ZooKeeper server.
   

   zoo.cfg

   # Defaulting to falsequorum.auth.learnerRequireSasl=true

3. Sets to connect using quorum authentication. If this is true, then all unauthenticated quorum peer learner connection requests will be rejected. If false, then quorum peer server will accept quorum peer learner connection request and then proceeds with Leader Election even if the authentication did not succeed. This can be used while upgrading ZK server.

   zoo.cfg

   # Defaulting to falsequorum.auth.serverRequireSasl=true

4. (Optional) If you want to use different login context for learner/server other than the default values, then configure the following.

   zoo.cfg

   # Defaulting to QuorumLearnerquorum.auth.learner.saslLoginContext=QuorumLearner# Defaulting to QuorumServerquorum.auth.server.saslLoginContext=QuorumServer

   See Also

   ZooKeeper Getting Started GuideChapter 1. ZooKeeperSetting Up an External ZooKeeper EnsembleClient-Server mutual authentication - Apache ZooKeeper

5. The maximum number of threads to allow in the “connectionExecutors” thread pool, which will be used to process quorum server connection requests during Leader Election. This has to be tuned depending on the cluster size. For example, consider a 3-node cluster, during quorum formation at least 3 outgoing connection requests and 3 incoming connection requests will occur. So total 6 threads will be used. It is recommended to configure 2x number of threads for smooth execution, where 'x' represents the cluster size.

   zoo.cfg

   # Defaulting to 20quorum.cnxn.threads.size=20

conf/java.env

Add the following settings to thejava.envfile located in the ZooKeeper configuration directory. (Create the file if it does not already exist.)

SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/server/jaas/file.conf"

ZooKeeper servers will talk to each other using the credentials configured in “jaas/file.conf” file. They will act like learner-server when creating connections during quorum formation. Set up the Java Authentication and Authorization Service (JAAS) by creating a “jaas/file.conf” file in the ZooKeeper configuration directory and add configuration entries into this file specific to your selected authentication schemes.

Following section describes the details of supported authentication schemes, Kerberos or DIGEST-MD5.

ZooKeeper uses Kerberos principals and Keytabs to support quorum peermutual authentication. Kerberos assigns tickets to Kerberos principals to enable them todo the Kerberos-secured communication.

You can use principal in the form of servicename/fully.qualified.domain.name@EXAMPLE.COM,here "servicename" has to be substituted with the proper service name. For example, for ZooKeeper quorum service you can substitute "servicename" with "zkquorum". Assume, we have two hosts (myhost1.foo.com and myhost2.foo.com) we would have Kerberos principals like, zkquorum/myhost1.foo.com@EXAMPLE.COM and zkquorum/myhost2.foo.com@EXAMPLE.COM

Configure service name and host details servicename/fully.qualified.domain.name of quorum peer server to the following configuration property in zoo.cfg file. This service principal will be used by the quorum learner to send authentication packet to the peer quorum server.

quorum.auth.kerberos.servicePrincipal

The principal name should be in either of the following formats:

servicename/localhost@EXAMPLE.COM

Admin has to be configure the below property with the principal name in zoo.cfg file,

quorum.auth.kerberos.servicePrincipal=servicename/localhost

Important Note: Authorization is not supported in this format.

2) Host based Kerberos principal name with _HOST wildcard

servicename/fully.qualified.domain.name@EXAMPLE.COM

ZooKeeper simplifies the deployment of configuration files by allowing the fully qualified domain name component of the service principal to be specified as the _HOST wildcard. Internally each quorum learner will substitute _HOST with the respective FQDN from zoo.cfg at runtime and then send authentication packet to that server. This allows administrators to avoid the overhead of configuring all other's principal names on all nodes. However, the keytab files will be different. A keytab file for a ZooKeeper server is unique to each host if the principal name contains the hostname. This file is used to authenticate a principal on a host to Kerberos without human interaction or storing a password in a plain text file. Access to the keytab files should be tightly secured because having access to the keytab file for a principal allows one to act as that principal.

ZooKeeper ensemble server details will be configured in zoo.cfg of the form

server.id=host:port:port

Make sure that you use FQDN in zoo.cfg like below, FQDN value will be used to replace the special string pattern “_HOST”

server.id=FQDN:port:port

QuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="zkquorum/fully.qualified.domain.name@EXAMPLE.COM";};QuorumLearner { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/keytab" storeKey=true useTicketCache=false debug=false principal="learner/fully.qualified.domain.name@EXAMPLE.COM";};

Supports authorization:

ZooKeeper server will support authorization if the principal name is in the format servicename/fully.qualified.domain.name@EXAMPLE.COM and configure “_HOST” wildcard in quorum server principal in zoo.cfg file as below,

quorum.auth.kerberos.servicePrincipal= servicename/_HOST

Now, QuorumServer will do the authorization checks against configured authorized hosts. This authorized host list will be prepared using the ensemble server details in zoo.cfg file. During LE, QuorumLearner will send an authentication packet to QuorumServer. Now, QuorumServer will check that the connecting QuorumLearner’s hostname exists in the authorized hosts. If not exists then connecting peer is not authorized to join this ensemble and the request will be rejected immediately.

For example, zoo.cfg contains the below server details. Now, only FQDN1, FQDN2, FQDN3 hosts are allowed to join ZooKeeper cluster. All others will be rejected as unauthorized connections.

server.1=FQDN1:port:portserver.2=FQDN2:port:portserver.3=FQDN3:port:port

QuorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_test="test";};QuorumLearner { org.apache.zookeeper.server.auth.DigestLoginModule required username="test" password="test";};

Important Note: Authorization is not supported in this format

This section describes the possible rolling upgrade to use ZOOKEEPER-1045 feature.The rolling upgrade in ZOOKEEPER-1045 is designed based on a key observation that a Quorum Peer can be configured such that a ZooKeeper ensemble can be composed of a mixed servers, both old and new. Rolling upgrade is achieved by a combination of configuration flags in zoo.cfg configuration file.

quorum.auth.enableSasl: If false, no authentication at all. If true, could be either in rolling upgrade, or finished rolling upgrade.

quorum.auth.learnerRequireSasl: Initially false. Sets to true in second step of rolling upgrade, this is to prepare each server ready to send authentication packet to other servers. This flag can’t be set to false if quorum.auth.serverRequireSasl isset to true.

quorum.auth.serverRequireSasl: Initially false. Sets to true in third step of rolling upgrade (quorum.auth.learnerRequireSaslshould be true) to enable server-to-­server SASL authentication strictly.

How to perform rolling upgrade

Prior to enabling ZooKeeper to work with security on your cluster, make sure ZooKeeper cluster works well in no-authentication setup. Rolling upgrade should be completed in three steps. After every step, admin has to ensure that all the servers have completed this step before moving on to the next step.

Step-1) Loop through each server, update zoo.cfg with quorum.auth.enableSasl=true, then restart the server with the new ZooKeeper 3.4.10+ binaries. Verify everything works well after restarting the server.

Step-2) Loop through each server, update zoo.cfg with quorum.auth.learnerRequireSasl=true, then restart the server. Each server is now ready to talk with other servers in an auth­ enabled way but on the other end the receiving server is not auth enabled yet. Verify everything works well after restarting the server.

Step-3) Loop through each server, update zoo.cfg with quorum.auth.serverRequireSasl=true, then restart server. Now for each server, both its sending part (modeled as QuorumLearner) and its receiving part (modeled as QuorumServer) are auth enabled. Verify everything works correctly after restarting the server.

For verification, you can use ZooKeeper client operations like, create/delete znode.