Hive Metastore
Hive requires a metastore service which manages metadata about the data managed by Hive. It is backed by persistent storage in the form of tables in an RDBMS. There are multiple ways to configure a metastore. A metastore service can be run in the same JVM as the hive server when it is configured as an embedded metastore. A remote metastore, on the other hand, is configured to run in a separate JVM or even on a separate host. Such a remote metastore can be shared by multiple hive server instances and is also available to be used by other applications. More about configuring a remote metastore can be found here. For achieving high availability, load balancing and scalability, more than one metastore services (all serving a consistent view of the same metadata) can be started and their URIs are listed in "hive.metastore.uris" (notice the plural "uris"). A metastore client (running from within a hive server) uses URIs specified in configuration "hive.metastore.uris" to connect to any one of the remote metastore servers specified there. In the process it may try multiple URIs, if some of them are unreachable for various reasons.Discoverying metastore services
As long as the set of metastore services remains the same, this works fine. But in real world that's not guaranteed. A metastore service can stop working, network has its own whims and the platforms hosting metastore services themselves go crazy. Neither is the demand for a service static. We may require more metastore services when the demand increases. When the set of metastore services changes, the list in "hive.metastore.uris" is required to be manually changed so that the client is aware of the new metastore servers that have come up or to avoid time wasted by a client trying to connect a failed metastore server or to avoid connecting to already loaded service. Obviously this has all the faults that any service requiring manual intervention has! There can be longer delays in detecting a failed metastore or a new metastore and also delays in updating the list of URIs. What if the client has access to a list of URIs that is always up-to-date?Metastore service comes to the Zoo
That's why metastore is admitted to the Zoo! HIVE-20794 added a Hive metastore feature to use ZooKeeper for dynamic metastore service discovery. A ZooKeeper is a centralized service for maintaining configuration information, naming, and providing group services. When a metastore server is configured to use ZooKeeper, it registers itself to the ZooKeeper when starting up. When a new metstore server is started to meet higher demand, the ZooKeeper makes a note of it. The ZooKeeper then keeps a watch on the status of metastore service and deregisters it when it becomes unavailable. Thus a ZooKeeper knows when a new metastore service is available and also when it becomes unavailable, thus keeping the list of metastore services up-to-date. A metastore client, when configured to use ZooKeeper, connects to the ZooKeeper ensemble and gets the up-to-date list of registered metastores and connects to any one of those at random balancing the load across metastores. It's not necessary that the client will connect to the first metastore in the list. The client does not It does not waste cycles trying to connect a failed metastore. Also knows about a new metastore automatically.How to use this feature
You need a single ZooKeeper or a ZooKeeper ensemble setup and running in order to use this feature. Here's how you can do that. Once it's setup, change metastore configuration as described below in all the sites where Hive servers are running and where metastore services are running. They need to be consistent at all these sites.
- set "metastore.service.discovery.mode" to "zookeeper" so that the client and server know that ZooKeeper has been configured.
- set "hive.metastore.uris" to the URI (for a single ZooKeeper) or URIs (for an ensemble) of the ZooKeeper cluster. Yes, this is the same old configuration you would use to list metastore URIs earlier. You can continue to use it in the old way if you don't want to use ZooKeeper. When all the ZooKeeper instances in an ensemble have the same port one can specify only URIs in "hive.metastore.uris" and specify the port in "metastore.zookeeper.client.port".
- metastore.zookeeper.namespace to any string and used as the root under which the metastore services are listed in the ZooKeeper hierarchial namespace. Once set users don't need to worry about it much.
- Some more configurations like metastore.zookeeper.connection.timeout, metastore.zookeeper.connection.max.retries and metastore.zookeeper.connection.basesleeptime are used as parameters to the ZooKeeper connection and you may want to leave those untouched. You may want to set different values for these at different sites depending upon the network bandwidth and other parameters at those sites.
