HIVE-29241: Distinguish the default location of the database by catalog by zhangbutao · Pull Request #6267 · apache/hive

zhangbutao · 2026-01-14T09:34:42Z

What changes were proposed in this pull request?

The main objectives of this PR are:

Allow location to be omitted when creating a catalog. At the same time, it is necessary to determine catalogs that do not require a location, such as JDBC catalogs.
For newly created catalogs, the location of their databases will be determined by the two newly added parameters: metastore.warehouse.catalog.dir and metastore.warehouse.catalog.external.dir. This helps better ensure that the location of the default Hive catalog's databases and tables remains unaffected.

For example, if metastore.warehouse.catalog.dir is hdfs://ns1/testdir, then the location for a newly created catalog named testcat would be hdfs://ns1/testdir/testcat. Consequently, the default path for a database like testdb created under this catalog would be hdfs://ns1/testdir/testcat/testdb.

Modify the semantics of catalog creation. The type parameter must be specified when creating a catalog to distinguish its type. Based on this type, it will be determined whether the catalog's databases and tables require a location, or whether the catalog type supports creating databases and tables.
CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');

Why are the changes needed?

The paths of databases and tables created under a non-default catalog may interfere with those under the default Hive catalog. We need to introduce parameters to distinguish the paths for databases and tables under non-default catalogs, ensuring that the paths of the default Hive catalog remain unaffected.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests.

Copilot

Pull request overview

This pull request introduces catalog-aware warehouse paths for Hive metastore, allowing databases and tables under non-default catalogs to have distinct storage locations. The changes ensure that catalog locations are properly isolated while maintaining backward compatibility for the default Hive catalog.

Changes:

Added new configuration parameters (WAREHOUSE_CATALOG and WAREHOUSE_CATALOG_EXTERNAL) for catalog-specific warehouse directories
Modified catalog creation to require a 'type' parameter and made location optional with automatic defaults
Updated database path resolution throughout the codebase to use catalog-aware logic via new Warehouse methods that accept catalog names

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 21 comments.

Show a summary per file

File	Description
MetastoreConf.java	Added WAREHOUSE_CATALOG and WAREHOUSE_CATALOG_EXTERNAL configuration variables
HiveConf.java	Added corresponding Hive configuration variables for catalog warehouses
Warehouse.java	Enhanced path resolution methods to accept catalog names; added deprecated markers for old methods
CatalogUtil.java	New utility class for catalog type validation (NATIVE, ICEBERG)
CreateCatalogAnalyzer.java	Updated to validate catalog type property and make location optional
CreateCatalogOperation.java	Modified to generate default location from WAREHOUSE_CATALOG config
CreateDatabaseAnalyzer.java	Added validation to check if catalog supports database creation
CreateDatabaseOperation.java	Updated path building logic to include catalog names for non-default catalogs
DDLUtils.java	Added helper method to check catalog support for database creation
HiveParser.g	Modified grammar to make catalog location optional and properties required
EximUtil.java	Updated to use catalog-aware warehouse paths for replication
Multiple test files	Updated test queries to include required 'type' property in CREATE CATALOG statements
HMSHandler.java	Updated default catalog and database creation to use catalog-aware paths
MetastoreDefaultTransformer.java	Updated to pass Database object instead of just name
Migration/Authorization files	Updated to use new Database-based path methods

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...one-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java

ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLUtils.java

common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

...one-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java

...e-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java

ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractRepository.java

...one-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java

...tore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java

ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java

parser/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g

...tore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java

common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java

dengzhhu653 · 2026-03-25T10:10:13Z

Yes. Users can specify different locations for native catalogs; if a user does not specify a location for a native catalog, the default value will be used.

This could make some confusing. For example, we declare a S3 catalog but the location is absent, then this catalog is created with a hdfs location, some databases without explicit location will be put under the hdfs, while some are on the cloud, it might be a problem if we use the FileSystem created by the catalog's properties to operate against these tables or databases.

In HMS, we can enforce the location check when creating the catalog, if it's null and the catalog is native, then we should throw the exception.

zhangbutao · 2026-03-25T12:41:03Z

This could make some confusing. For example, we declare a S3 catalog but the location is absent, then this catalog is created with a hdfs location, some databases without explicit location will be put under the hdfs, while some are on the cloud, it might be a problem if we use the FileSystem created by the catalog's properties to operate against these tables or databases.

In HMS, we can enforce the location check when creating the catalog, if it's null and the catalog is native, then we should throw the exception.

I understand what you mean. I think what we're discussing is more about usage habits.

Before the catalog capability was available, Hive used databases as a way to isolate data, and when users created a Hive database, they could omit the location, in which case the database would use the default location. If users wanted to set a value different from the default catalog location, they could specify a specific location, such as an S3 location, when creating the catalog.

This syntax of omiting the location is more flexible, and I think users might be more accustomed to the usage of having a default catalog location.

I would also like to hear other folks' thoughts on the default catalog location. :)

deniskuzZ · 2026-04-07T15:35:29Z

i'm ok with default catalog location for native catalogs if not explicitly provided

deniskuzZ · 2026-04-07T15:39:44Z

hi @zhangbutao, is the PR ready for 2nd round of review and includes the changes we discussed here?

zhangbutao · 2026-04-11T12:53:08Z

One thing to note: Currently, we have two entry points for creating catalogs—one is the SQL-based catalog creation entry involved in this PR, and the other is the HMS API-based catalog creation entry. In this PR, I intend to enforce the inclusion of the type attribute when users create catalogs via SQL, so that users can clearly distinguish between different types of catalogs. However, the HMS API side does not enforce the type attribute when creating catalogs, which would lead to inconsistency between the two entry points.
Therefore, I have two approaches to resolve this issue and ensure consistency between the two entry points:

On the HMS server side, specifically in the HMSHandler, enforce a check during catalog creation to verify whether the user has provided the type attribute. If the type is missing, return an error. This approach would introduce backward compatibility issues for catalog creation.

Modify the current PR to make the type attribute optional for SQL-based catalog creation. For both SQL-based catalog creation and HMS API-based catalog creation, if the user does not provide the type attribute, assign a default value (e.g., native) on the HMSHandler server side. This approach would not cause compatibility issues with the catalog creation syntax. However, other open-source components enforce the type attribute requirement, and the type helps users better understand the parameter requirements for the catalog, among other benefits.

What are your suggestions regarding these two approaches? Thanks in advance. cc @deniskuzZ @difin

@zhangbutao i would vote for option 2, fallback to default - native

Changes have been made to keep the type optional when creating a catalog. Thanks for the advice.

zhangbutao · 2026-04-11T12:53:41Z

hi @zhangbutao, is the PR ready for 2nd round of review and includes the changes we discussed here?

Let's go :)

deniskuzZ · 2026-04-11T12:54:03Z

itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java

      // Create a fake fs root for local fs
      Path localFsRoot = new Path(path, "localfs");
      warehousePath = new Path(localFsRoot, "warehouse");
+      warehouseCatPath = new Path(localFsRoot, "catalog");


shouldn't the catalog be under the warehousePath?

Warehouse Root └── Catalog (default or named) └── Database (.db) └── Table /warehouse/<catalog_name>/<database_name>.db/<table_name>/ # non default catalog /warehouse/hive/<database_name>.db/<table_name>/ # default catalog /warehouse/hive/default.db/<table_name>/ # default catalog & database

deniskuzZ · 2026-04-11T13:05:08Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java


+    // Initialize props if null, and set default type to native if not specified
+    if (props == null) {
+      props = new HashMap<>();


could we do this in the var initialization?

deniskuzZ · 2026-04-11T13:05:44Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java

  }
+
+  private static void checkCatalogType(Map<String, String> props) throws SemanticException {
+    String catalogType = props.get("type");


could we please create constant for that

deniskuzZ · 2026-04-11T13:07:04Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java

+    }
+    // Normalize and validate catalog type (fail fast on invalid input)
+    try {
+      props.put("type", CatalogUtil.normalizeCatalogType(catalogType));


Could we simplify by handling isNullOrEmpty in normalizeCatalogType

deniskuzZ · 2026-04-11T13:08:32Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogOperation.java

  @Override
  public int execute() throws Exception {
-    Catalog catalog = new Catalog(desc.getName(), desc.getLocationUri());
+    String catLocationUri = Optional.ofNullable(desc.getLocationUri())


minor. i think cat prefix in local var is redundant here since we are inside CreateCatalog class

deniskuzZ · 2026-04-11T13:11:36Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java

@@ -51,10 +52,9 @@ public CreateDatabaseAnalyzer(QueryState queryState) throws SemanticException {
  @Override
  public void analyzeInternal(ASTNode root) throws SemanticException {
    Pair<String, String> catDbNamePair = DDLUtils.getCatDbNamePair((ASTNode) root.getChild(0));


How about record CatalogDb(String catalog, String database) {}
please see #6379 (comment)

deniskuzZ · 2026-04-11T13:13:22Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java

  }

  private void makeLocationQualified(Database database) throws HiveException {
+    String catalogName = database.getCatalogName().toLowerCase();


Shouldn't the names be normalized here: special chars/encoding lowercase?

deniskuzZ · 2026-04-11T13:14:00Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java

  private void makeLocationQualified(Database database) throws HiveException {
+    String catalogName = database.getCatalogName().toLowerCase();
+    String dbName = database.getName().toLowerCase();
+    boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equalsIgnoreCase(catalogName);


consider moving to utils

deniskuzZ · 2026-04-11T13:17:48Z

ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java

+
+      // managed warehouse root
+      String whManagedLocatoion = MetastoreConf.getVar(conf,
+              isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE


please see #6267 (comment)

deniskuzZ · 2026-04-11T13:19:33Z

ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractRepository.java

      if (db == null) {
        LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
-        String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+        db = new Database();


looks a bit weird creating 2 Database objects in same method

deniskuzZ · 2026-04-11T13:20:34Z

...java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java

    Path root = null;
    try {
      initWh();
+      // TODO catalog. Need to determine auth root path based on catalog name.


could you please add the ticket to the EPIC
cc @saihemanth-cloudera, as he is the most experienced in this area

deniskuzZ · 2026-04-11T13:23:22Z

ql/src/test/queries/clientnegative/lockneg_try_lock_cat_db_in_use.q

 set hive.support.concurrency=true;

-CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog';
+CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog' PROPERTIES('type'='native');


is redundant

deniskuzZ · 2026-04-11T13:24:44Z

ql/src/test/queries/clientpositive/catalog.q

-- CREATE with comment
-CREATE CATALOG test_cat LOCATION 'hdfs:///tmp/test_cat' COMMENT 'Hive test catalog';
+-- CREATE with comment and default location
+CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');


same 'type'='NATIVE' is redundant

deniskuzZ · 2026-04-11T13:25:21Z

ql/src/test/queries/clientpositive/catalog.q


 -- CREATE INE already exists
-CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat';
+CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat' PROPERTIES('type'='native');


do we have tests for external catalog?

deniskuzZ · 2026-04-11T13:27:47Z

...e-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java

+
+public class CatalogUtil {
+    // Catalog type constants
+    public static final String NATIVE = "native";


no need for Strings, please use enum values.

deniskuzZ · 2026-04-11T13:28:32Z

...store/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java

    FileSystem cmFs = cmroot.getFileSystem(conf);
    cmFs.setPermission(cmroot, new FsPermission("770"));
    try {
+      // TODO catalog. The Repl function is currently only available for the default catalog path ConfVars.WAREHOUSE.


should we add the ticket to the EPIC?

deniskuzZ · 2026-04-11T13:29:25Z

...one-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java

-    }
+
+    whRootString = getRequiredVar(conf, ConfVars.WAREHOUSE);
+    whCatRootString = getRequiredVar(conf, ConfVars.WAREHOUSE_CATALOG);


please see the comment on default FS structure

deniskuzZ · 2026-04-11T13:31:45Z

...one-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java

-        return getWhRoot();
-      }
-      return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+    Database db = new Database();


are you using Database only as a DTO object? instead you could use a record CatalogDb as mentioned earlier

deniskuzZ · 2026-04-11T13:33:32Z

...etastore/metastore-rest-catalog/src/main/java/org/apache/iceberg/rest/HMSCatalogFactory.java

    if (configUri != null && !configUri.isEmpty()) {
      properties.put("uri", configUri);
    }
+    // TODO catalog. Consider adding new created native catalog warehouse(MetastoreConf.ConfVars.WAREHOUSE_CATALOG) later.


could you please elaborate, i don't quite get what you are suggesting here

sonarqubecloud · 2026-04-11T13:55:03Z

Quality Gate passed

Issues
69 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.4% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added the tests pending label Jan 14, 2026

zhangbutao requested a review from Copilot January 14, 2026 09:44

Copilot started reviewing on behalf of zhangbutao January 14, 2026 09:45 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

asf-ci-hive added tests failed and removed tests pending labels Jan 14, 2026

zhangbutao force-pushed the HIVE-29241_LOCATION branch from 5940a60 to 49c343a Compare January 15, 2026 09:29

asf-ci-hive added tests pending tests failed tests unstable and removed tests failed tests pending tests unstable labels Jan 15, 2026

zhangbutao requested a review from Copilot January 17, 2026 08:20

Copilot started reviewing on behalf of zhangbutao January 17, 2026 08:21 View session

Copilot AI reviewed Jan 17, 2026

View reviewed changes

asf-ci-hive added tests passed tests pending tests unstable and removed tests pending tests passed labels Jan 17, 2026

HIVE-29241: Distinguish the default location of the database by catalog

e299310

zhangbutao force-pushed the HIVE-29241_LOCATION branch from e29caa4 to e299310 Compare April 11, 2026 12:26

asf-ci-hive added tests pending and removed tests passed labels Apr 11, 2026

deniskuzZ reviewed Apr 11, 2026

View reviewed changes

Conversation

zhangbutao commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dengzhhu653 commented Mar 25, 2026

Uh oh!

zhangbutao commented Mar 25, 2026

Uh oh!

deniskuzZ commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deniskuzZ commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangbutao commented Apr 11, 2026

Uh oh!

zhangbutao commented Apr 11, 2026

Uh oh!

deniskuzZ Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangbutao commented Jan 14, 2026 •

edited

Loading

deniskuzZ commented Apr 7, 2026 •

edited

Loading

deniskuzZ commented Apr 7, 2026 •

edited

Loading

deniskuzZ Apr 11, 2026 •

edited

Loading

deniskuzZ Apr 11, 2026 •

edited

Loading

deniskuzZ Apr 11, 2026 •

edited

Loading

deniskuzZ Apr 11, 2026 •

edited

Loading