This guide describes how to enable secure communication between client and server using SASL mechanism. ZooKeeper supportsKerberosorDIGEST-MD5as your authentication scheme.
This feature was added in ZooKeeper3.4.0+version and is available in all higher versions.ZOOKEEPER-938is the JIRA issue, and thepatch is availablelinked from that JIRA.
This proposed implementation builds on the existing ZooKeeper authentication and authorization design in a straightforward way. To briefly review, ZooKeeper supports pluggable authentication schemes. A node may have any number of <scheme:expression,perms> pairs. The left member of the pair specifies authentication as the authentication scheme and the principal. The right member indicates what permissions are given to this principal. For example, one ACL pair on a given node might be:
<ip:19.22.0.0/16 , READ>
The left side,ip:19.22.0.0/16, means that the authentication scheme is by Internet address, and that any client whose IPv4 address begins with "19.22" has whatever permissions are indicated on the right side. The right side indicates that the user the permissions "READ" on the given node.
The designated name of the SASL authentication scheme is simply "sasl", so if you are using Kerberos, you may set a ZooKeeper's node to be:
<sasl:myclient@EXAMPLE.COM , READ>
meaning that the client whose Kerberos principal is myclient@EXAMPLE.COM may read the given node.
create
In non-SASL ZooKeeper, you may add authentication credentials when you create a node, for example, usingorg.apache.zookeeper.server.auth.DigestAuthenticationProvider, you would do:
password
# create a digest form of the password "password":$ java -cp build/classes:build/lib/log4j-1.2.15.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider user:passworduser:password->user:tpUq/4Pn5A64fVZyQ0gOJ8ZWqkY=
Then, after connecting to ZooKeeper, you would do the following to grant all permissions to the user "user" using password "password":
create
create /mynode content digest:user:tpUq/4Pn5A64fVZyQ0gOJ8ZWqkY=:cdrwa
With SASL ZooKeeper, the password generation depends on the mechanism (currently DIGEST-MD5 or Kerberos). How to set passwords for both mechanisms is described below in the Configuration section. Unlike withDigestAuthenticationProvideras shown above, with SASL, thecreatecommand does not include password information. Instead, (assuming your Kerberos domain isEXAMPLE.COM):
create
create /mynode content sasl:user@EXAMPLE.COM:cdrwa
addauth
The SASL authentication scheme differs from certain other schemes in that the "addauth <scheme> <auth>" command has no effect if scheme is "sasl". This is because authentication is performed using SASL-enabled token exchange immediately after connection, rather than occuring any time after connection, as addauth is.
addAcl
As withcreate, you do not include credential information. So whereas with theDigestAuthenticationProvideryou would do:
addAcl
addAcl /mynode digest:user:tpUq/4Pn5A64fVZyQ0gOJ8ZWqkY=:cdrwa
withSaslAuthenticationProvider, you instead do:
addAcl
addAcl /mynode sasl:user@EXAMPLE.COM:cdrwa
You may continue to use existing ZooKeeper authentication providers, such asDigestAuthenticationProvidertogether withSaslAuthenticationProvider, if you wish. Existing unit tests that test existing authentication providers still pass and code that uses these authentication providers should also work.
LoginThread is a new class that starts a new thread that periodically refreshes thejavax.security.auth.Subjectcredentials, and is used for this purpose on both the ZooKeeper client and server. If ZooKeeper is configured to use Kerberos (see "Server Configuration" below for how to do this), both client and server should be configured to use a keytab or credential cache that the LoginThread will use to refresh the Subject's credentials.
org.apache.zookeeper.ZooKeeper
If the System Propertyjava.security.auth.login.configis defined, theZooKeeperconstructor initializes its member variableorg.apache.zookeeper.LoginThread loginThread:
ZooKeeper.java
LoginThread loginThread = null; if (System.getProperty("java.security.auth.login.config") != null) { // zookeeper.client.ticket.renewal defaults to 19 hours (about 80% of 24 hours, which is a typical ticket expiry interval). loginThread = new LoginThread("Client",new ClientCallbackHandler(null),Integer.getInteger("zookeeper.client.ticket.renewal",19*60*60*1000)); } cnxn = new ClientCnxn(connectStringParser.getChrootPath(), hostProvider, sessionTimeout, this, watchManager, getClientCnxnSocket(), canBeReadOnly, loginThread); cnxn.start(); As shown above, theloginThreadis then passed to theClientCnxnconstructor, whose class is discussed in the next section.
org.apache.zookeeper.ClientCnxn
ClientCnxn's constructor has one new parameter:LoginThread loginThread. The above code fragment shows how theZooKeeperobject initializesClientCnxnusing this new parameter.
ClientCnxnuses the suppliedloginThreadobject to initialize itssaslClientmember variable in thestartConnect()method, which is called duringClientCnxn'srun()loop when the client attempts to connect to a ZooKeeper Quorum server.
TheloginThreadobject is also used to generate SASL tokens to send to the ZooKeeper server, as will be shown below in the code fragment showing the definition ofprepareSaslResponseToServer().
When the ZooKeeper client connects to a ZooKeeper Quorum member, it creates aClientCnxnas shown above, which in turn starts an EventThread to communicate with the quorum member. If SASL is enabled, then the client goes fromCONNECTINGtoSASL_INITIAL. At this state, the client checks whether itssaslClientshould send an initial response (which is a SASL-internal detail that depends on the mechanism). If it should send an initial response, it creates the initial token and sends it to the ZooKeeper server and goes to stateSASL. If not, it simply goes to stateSASL.
If, on the other hand, SASL is not configured on the client, then the client simply goes fromSASL_INITIALtoCONNECTEDstate. This allows non-SASL authenticated ZooKeeper clients to interact without modification with a SASL-configured ZooKeeper Quorum.
While the client is inSASLstate, it exchanges tokens with the ZooKeeper server until authentication has succeeded or failed. If the former, it goes toCONNECTEDstate; if the latter, it goes toAUTH_FAILEDstate. The token-exchange process on the client side is done using packets of typeSaslServerResponseCallback(the definition of this class is shown below). We modify the ClientCnxn's event thread to support processing packets of typeSaslServerResponseCallback:
ClientCnxn.java
class EventThread { . . run() { . . processEvent(event); . . } private void processEvent(Object event) { . . Packet p = (Packet) event; . . if (p.cb instanceof ServerSaslResponseCallback) { ServerSaslResponseCallback cb = (ServerSaslResponseCallback) p.cb; SetSASLResponse rsp = (SetSASLResponse) p.response; cb.processResult(rc,null,p.ctx,rsp.getToken(),null); } . . }}TheprocessResult()called in the above has the following implementation:
ClientCnxn.java
static class ServerSaslResponseCallback implements DataCallback { public void processResult(int rc, String path, Object ctx, byte data[], Stat stat) { // data[] contains the ZooKeeper Server's SASL token. // ctx is the ClientCnxn object. We use this object's prepareSaslResponseToServer() method // to reply to the ZooKeeper Server's SASL token ClientCnxn cnxn = (ClientCnxn)ctx; byte[] usedata = data; if (data != null) { LOG.debug("ServerSaslResponseCallback(): saslToken server response: (length="+usedata.length+")"); } else { usedata = new byte[0]; LOG.debug("ServerSaslResponseCallback(): using empty data[] as server response (length="+usedata.length+")"); } cnxn.prepareSaslResponseToServer(usedata); }}Thecnxn.prepareSaslResponseToServer()called in the above is implemented as:
ClientCnxn.java
private byte[] saslToken = new byte[0];public void prepareSaslResponseToServer(byte[] serverToken) { saslToken = serverToken; LOG.debug("saslToken (server) length: " + saslToken.length); if (!(saslClient.isComplete() == true)) { try { saslToken = createSaslToken(saslToken, saslClient); if (saslToken != null) { LOG.debug("saslToken (client) length: " + saslToken.length); queueSaslPacket(saslToken); } if (saslClient.isComplete() == true) { LOG.info("SASL authentication with ZooKeeper server is successful."); } } catch (SaslException e) { LOG.error("SASL authentication failed."); } }}Finally, createSaslToken is defined as follows (with some exception-handling code not shown):
ClientCnxn.java
Subject subject = this.loginThread.getLogin().getSubject(); if (subject != null) { synchronized(this.loginThread) { try { final byte[] retval = Subject.doAs(subject, new PrivilegedExceptionAction<byte[]>() { public byte[] run() throws SaslException { try { LOG.debug("ClientCnxn:createSaslToken(): ->saslClient.evaluateChallenge(len="+saslToken.length+")"); return saslClient.evaluateChallenge(saslToken); } . . } } } }Note the use of thejavax.security.auth.Subject subjectin the above: this allows use of a Kerberos-authenticated ZooKeeper client to generate tokens that allow the ZooKeeper server to authenticate it, and also allows the client to authenticate the ZooKeeper server. Similar code exists on the server side, shown below.
When a client connects to the server, the server creates ajavax.security.SaslServerobject using its own authentication information derived from its startup configuration (see Configuration in the next section). This authentication information is used by the server'sSaslServerobject to exchange SASL tokens with the client'sSaslClientobject, as shown in the following code:
org.apache.zookeeper.server.ServerCnxnFactory.java
public SaslServer createSaslServer() { synchronized (loginThread) { Subject subject = loginThread.getLogin().getSubject(); if (subject != null) { // server is using a JAAS-authenticated subject: determine service principal name and hostname from zk server's subject. if (subject.getPrincipals().size() > 0) { try { final Object[] principals = subject.getPrincipals().toArray(); final Principal servicePrincipal = (Principal)principals[0]; // e.g. servicePrincipalNameAndHostname := "zookeeper/myhost.foo.com@FOO.COM" final String servicePrincipalNameAndHostname = servicePrincipal.getName(); int indexOf = servicePrincipalNameAndHostname.indexOf("/"); // e.g. servicePrincipalName := "zookeeper" final String servicePrincipalName = servicePrincipalNameAndHostname.substring(0, indexOf); // e.g. serviceHostnameAndKerbDomain := "myhost.foo.com@FOO.COM" final String serviceHostnameAndKerbDomain = servicePrincipalNameAndHostname.substring(indexOf+1,servicePrincipalNameAndHostname.length()); indexOf = serviceHostnameAndKerbDomain.indexOf("@"); // e.g. serviceHostname := "myhost.foo.com" final String serviceHostname = serviceHostnameAndKerbDomain.substring(0,indexOf); final String mech = "GSSAPI"; try { return Subject.doAs(subject,new PrivilegedExceptionAction<SaslServer>() { public SaslServer run() { try { SaslServer saslServer; saslServer = Sasl.createSaslServer(mech, servicePrincipalName, serviceHostname, null, saslServerCallbackHandler); return saslServer; } catch (SaslException e) {... return null; } } } ); }org.apache.zookeeper.server.FinalRequestProcessor.java
This class is modified to accept client packets of typeOpCode.sasl:
FinalRequestProcessor.java
case OpCode.sasl: { // client sent a SASL token: respond with our own SASL token in response. LOG.debug("FinalRequestProcessor:ProcessRequest():Responding to client SASL token."); lastOp = "SASL"; GetSASLRequest clientTokenRecord = new GetSASLRequest(); ZooKeeperServer.byteBuffer2Record(request.request,clientTokenRecord); byte[] clientToken = clientTokenRecord.getToken(); LOG.debug("Size of client SASL token: " + clientToken.length); byte[] responseToken = null; try { SaslServer saslServer = cnxn.saslServer; try { // note that clientToken might be empty (clientToken.length == 0): // in the case of the DIGEST-MD5 mechanism, clientToken will be empty at the beginning of the // SASL negotiation process. responseToken = saslServer.evaluateResponse(clientToken); if (saslServer.isComplete() == true) { cnxn.addAuthInfo(new Id("sasl",saslServer.getAuthorizationID())); } } catch (SaslException e) { LOG.warn("Client failed to SASL authenticate: " + e); if ((System.getProperty("zookeeper.maintain_connection_despite_sasl_failure") != null) && (System.getProperty("zookeeper.maintain_connection_despite_sasl_failure").equals("yes"))) { LOG.warn("Maintaining client connection despite SASL authentication failure."); } else { LOG.warn("Closing client connection due to SASL authentication failure."); cnxn.close(); .. }Note that the server uses the existingServerCnxn.addAuthInfo()function to record that a connection is authenticated, just as other Authentication Providers do.
Note also from the above that clients that fail SASL authentication will be immediately disconnectedunlessthe system propertyzookeeper.maintain_connection_despite_sasl_failureis set toyes.
conf/zoo.cfg
You need to define the SASL authentication provider class in your server config. You can also set a few optional configuration parameter for SASL:
zoo.cfg
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider# optional SASL related server-side properties:# you can instruct ZooKeeper to remove the host from the client principal name during authentication# (e.g. zk/myhost@EXAMPLE.COM client principal will be authenticated in ZooKeeper as zk@EXAMPLE.COM)# kerberos.removeHostFromPrincipal=true# you can instruct ZooKeeper to remove the realm from the client principal name during authentication# (e.g. zk/myhost@EXAMPLE.COM client principal will be authenticated in ZooKeeper as zk/myhost)# kerberos.removeRealmFromPrincipal=true
conf/java.env
java.env
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/server/jaas/file.conf"
The configuration file indicated by the system propertyjava.security.auth.login.configshould be similar to one of the following examples, depending on whether you are using DIGEST-MD5 or Kerberos as your authentication mechanism. In either case, theServerheader is required.
JAAS conf file: Kerberos authentication
JAAS configuration file, Kerberos mechanism
Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/server/keytab" storeKey=true useTicketCache=false principal="zookeeper/yourzkhostname";};Note that the keytab file given in thekeyTabsection should not be readable by anyone other than the ZooKeeper server process user.
You can see more info about the different kerberos related JAAS config parameters here:https://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
JAAS configuration file: DIGEST-MD5 authentication
JAAS configuration file, DIGEST-MD5 mechanism
Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_super="adminsecret" user_bob="bobsecret";};Note that the passwords in the above are in plain text, so the JAAS configuration file should not be readable by anyone other than the ZooKeeper server process user.
This is similar to the ZooKeeper server configuration, except there is nozoo.cfgfor the client. You can either use JVM System Properties (e.g. defining them when you start your JVM or by defining theCLIENT_JVMFLAGS environment variable that will be used by the bin/zkCli.sh file if you use the command line ZooKeeper java client), or you can set some of these parameters in the code when you create your ZooKeeper client.
conf/java.env
java.env
# REQUIRED SASL RELATED CONFIGS:# ==== java.security.auth.login.config:# Defining your client side JAAS config file path:CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Djava.security.auth.login.config=/path/to/client/jaas/file.conf"# OPTIONAL SASL RELATED CONFIGS:# ==== zookeeper.sasl.client:# You can disable SASL authentication on the client side (it is true by default):# CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.sasl.client=false"# ==== zookeeper.server.principal:# Setting the server principal of the ZooKeeper service. If this configuration is provided, then# the ZooKeeper client will NOT USE any of the following parameters to determine the server principal: # zookeeper.sasl.client.username, zookeeper.sasl.client.canonicalize.hostname, zookeeper.server.realm# Note: this config parameter is working only for ZooKeeper 3.5.7+, 3.6.0+ # CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.server.principal=zookeeper@EXAMPLE.COM"# ==== zookeeper.sasl.client.username:# Setting the 'user' part of the server principal of the ZooKeeper service, assuming the # zookeeper.server.principal parameter is not provided. When you have zookeeper/myhost@EXAMPLE.COM # defined in your server side SASL config, then use:# CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.sasl.client.username=zookeeper"# ==== zookeeper.sasl.client.canonicalize.hostname:# Assuming the zookeeper.server.principal parameter is not provided, the ZooKeeper client will try to# determine the 'instance' (host) part of the ZooKeeper server principal. First it takes the hostname provided # as the ZooKeeper server connection string. Then it tries to 'canonicalize' the address by getting# the fully qualified domain name belonging to the address. You can disable this 'canonicalization'# using the following config:# CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.sasl.client.canonicalize.hostname=false"# ==== zookeeper.server.realm:# Setting the 'realm' part of the server principal of the ZooKeeper service, assuming the # zookeeper.server.principal parameter is not provided. By default, in this case the ZooKeeper Client # will use its own realm. You can override this, e.g. when you have zookeeper/myhost@EXAMPLE.COM # defined in your server side SASL config, then use:# CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.server.realm=EXAMPLE.COM"# ==== zookeeper.sasl.clientconfig:# you can have multiple contexts defined in a JAAS.conf file. ZooKeeper client is using the section# named as 'Client' by default. You can override it if you wish, by using:# CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.sasl.clientconfig=Client"The configuration file indicated by the system propertyjava.security.auth.login.configshould be similar to one of the following examples, depending on whether you are using DIGEST-MD5 or Kerberos as your authentication mechanism. In either case, theClientheader is required.
JAAS conf file: Kerberos authentication
JAAS configuration file, Kerberos mechanism
Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/client/keytab" storeKey=true useTicketCache=false principal="yourzookeeperclient";};Note that the keytab file given in thekeyTabsection should not be readable by anyone other than the ZooKeeper client process user.
You can see more info about the different kerberos related JAAS config parameters here:https://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
JAAS configuration file: DIGEST-MD5 authentication
JAAS configuration file, DIGEST-MD5 mechanism
Client { org.apache.zookeeper.server.auth.DigestLoginModule required username="bob" password="bobsecret";};Note that (as in the server configuration) the password in the above is in plain text, so the JAAS configuration file should not be readable by anyone other than the ZooKeeper client process user.
Setting up Kerberos and SASL with ZooKeeper is a complicated process for a beginner, so I've put detailed step-by-step instructions onUp and Running with Secure ZooKeeperto quickly get a simple Kerberos and SASLized ZooKeeper setup for your evaluation.
