Aquileo | fix: updating error messages thrown to end users by vamshikrishnakyatham · Pull Request #14115 · apache/hudi

vamshikrishnakyatham · 2025-10-17T20:08:29Z

Describe the issue this Pull Request addresses

This PR just updates the error messages that needs improvement in letting the users know the issue causing the error.

Summary and Changelog

This PR just updates the error messages that needs improvement in letting the users know the issue causing the error.

Impact

Better user interaction

Risk Level

none

Documentation Update

none, its just for better usability

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

nsivabalan · 2025-10-18T04:07:02Z

hey @yihua : can you take up the review on this.

yihua · 2025-10-20T17:12:28Z

+        .orElseThrow(() -> new HoodieException(
+            "Table version mismatch detected. "
+                + "The table was created with a Hudi version that has table version code " + versionCode
+                + " but you're using an older version.\n"
+                + "Likely you are using a lower hudi binary version to read a table written using higher hudi version which is not supported. "
+                + "To fix this issue:\n"
+                + "  - Please upgrade your readers to use the same version as writers\n"
+                + "  - Table version code: " + versionCode + "\n"
+                + "  - Current supported versions: " + Arrays.toString(HoodieTableVersion.values()) + "\n"
+                + "See: https://hudi.apache.org/docs/migration_guide for more information"));


The error message does not match the actual error. The exception is thrown if the table version is not in the range of 0 to 9 (inclusive). There is no version comparison in this method.

you are right, fixed it

I am pushing out a commit to fix this.
this can only happen if X hudi lib version is used to read a table written by higher (Y) hudi version, where X and Y refers to table version.

yihua · 2025-10-20T17:14:05Z

+            "'_hoodie_is_deleted' field is missing for some of the incoming records.\n\n"
+                + "The table schema requires '_hoodie_is_deleted' to be non-null, but some records lack this field.\n\n"
+                + "To fix:\n"
+                + "  Ensure ALL records have '_hoodie_is_deleted' field set (true/false)\n\n"
+                + "Original error: " + errorMessage, e);


Avoid newline character in the error message?

yihua · 2025-10-20T17:15:39Z

+      writtenRecordCount.incrementAndGet();
+    } catch (RuntimeException e) {
+      String errorMessage = e.getMessage() != null ? e.getMessage() : "";
+      if (isRequiredFieldNullError(errorMessage) && errorMessage.contains("_hoodie_is_deleted")) {


Where does these exceptions come from? Could errorMessage.contains be avoided and the source to throw an exception of specific classes which can be caught here?

It is thrown by Apache Parquet library. It does not provide custom exception types like ParquetValidationException or RequiredFieldNullException. It only throws generic
RuntimeException and we kind of need string parsing to log proper error message in this case

guess we can avoid this fix. exiting error msg

Null-value for required field: _hoodie_is_deleted

would suffice. I just don't feel intercepting at this layer just to add more enhancement to error msg is not worth it

yihua · 2025-10-20T17:17:51Z

+        List<String> errorKeys = HoodieJavaRDD.getJavaRDD(writeStatusesOpt.get())
+            .filter(WriteStatus::hasErrors)
+            .flatMap(ws -> ws.getErrors().keySet().stream().iterator())
+            .take(10)
+            .stream()
+            .map(Object::toString)
+            .collect(Collectors.toList());
+
+        String errorSummary = String.format(
+            "%s operation failed with %d error(s).\n\nFailed records (first %d of %d):\n%s\n\n"
+                + "Check for error stacktraces in the driver logs which could give more information on the failure.",
+            writeOperationType,
+            totalErroredRecords,
+            Math.min(10, errorKeys.size()),
+            errorKeys.size(),
+            errorKeys.stream()
+                .map(k -> "  - Record Key: " + k)
+                .collect(Collectors.joining("\n")));
+
+        LOG.error(errorSummary);


We cannot log record keys which can contain PII, potentially violating data compliance.

right, updated it with just count

yihua · 2025-10-20T17:18:52Z

+                  + "  - For Hive 2.x: Use hudi-spark-bundle with 'hive2' classifier\n"
+                  + "  - For Hive 3.x: Use hudi-spark-bundle with 'hive3' classifier\n"


What does this mean?

It happens when user has Spark with Hive 3.x, but uses hudi-spark-bundle (which has Hive 2.x shaded inside)
It can happen because Spark loads Hive 3.x classes from its own JARs -> Hudi bundle expects Hive 2.x classes (but shaded/renamed) -> Class loader tries to match them but fails because packages don't match -> Throws confusing error about "class X not org.apache.hudi.org.apache.hadoop...". The above exception with error message describes the issue properly.

yihua · 2025-10-20T17:21:25Z

+      return "Cannot read Hudi table schema - required data file is missing.\n\n"
+          + "This can happen due to:\n"
+          + "  1. Aggressive cleaner retention compared to query run times\n"
+          + "  2. Manual file deletions (timeline files or data files)\n"
+          + "  3. Concurrent writers without proper locking or configurations set\n\n"
+          + "Depending on the root cause, mitigation might differ.\n\n"
+          + "Original error: " + errorMessage;


The error message does not make sense. The main cause is that the commit metadata does not have the table schema and there is no data file to check the schema either.

right, haven't put that properly previously, updated it

nsivabalan · 2025-10-24T02:41:40Z

+                    + "This table version is not recognized by this Hudi version.%n%n"
+                    + "This can happen if:%n"
+                    + "  (1) The table was created with a newer version of Hudi (upgrade your readers),%n"
+                    + "  (2) The table was created with an older version of Hudi (downgrade your readers or upgrade the table),%n"


2nd is not right.
higher versions of hudi can read older versions

nsivabalan · 2025-10-24T03:18:55Z

+    String errorMessage = e.getMessage() != null ? e.getMessage() : e.getClass().getName();
+    if (e instanceof java.io.FileNotFoundException) {
+      return String.format(
+          "Cannot read Hudi table schema.%n%n"


If we hit FileNotFoundException, then it means, the data file was deleted by some means right.

Aggressive cleaner retention compared to query run times

Manual file deletions (timeline files or data files)

Concurrent writers without proper locking or configurations set

hey @yihua : this is specifically when FileNotFoundException. not sure if the error msg lines up with that.

nsivabalan · 2025-10-24T03:20:16Z

+      writtenRecordCount.incrementAndGet();
+    } catch (RuntimeException e) {
+      String errorMessage = e.getMessage() != null ? e.getMessage() : "";
+      if (isRequiredFieldNullError(errorMessage) && errorMessage.contains("_hoodie_is_deleted")) {


guess we can avoid this fix. exiting error msg

Null-value for required field: _hoodie_is_deleted

would suffice. I just don't feel intercepting at this layer just to add more enhancement to error msg is not worth it

nsivabalan

@yihua : I have pushed a commit to address my own feedback.
feel free to review and land the patch if its good.

hudi-bot · 2025-10-24T05:12:05Z

CI report:

0349086 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan · 2025-10-26T14:18:23Z

hey @yihua : feel free to send out a follow up PR for any additional feedback you have.

--------- Co-authored-by: sivabalan <n.siva.b@gmail.com>

updating error messages thrown to end users - 1.1

8027afa

github-actions Bot added the size:M PR with lines of changes in (100, 300] label Oct 17, 2025

nsivabalan assigned yihua Oct 18, 2025

vamshikrishnakyatham added 2 commits October 17, 2025 21:21

refactor

904a8b6

refactor

d80a08c

yihua reviewed Oct 20, 2025

View reviewed changes

PR comments address 1

27e9907

github-actions Bot added size:S PR with lines of changes in (10, 100] and removed size:M PR with lines of changes in (100, 300] labels Oct 20, 2025

yihua added the release-1.1.0 label Oct 20, 2025

yihua added this to the release-1.1.0 milestone Oct 20, 2025

vamshikrishnakyatham requested a review from yihua October 20, 2025 23:54

vamshikrishnakyatham added 3 commits October 20, 2025 18:20

test fix

a8efa07

test fix

c8401d5

scalastyle fix

ece7286

github-actions Bot added size:M PR with lines of changes in (100, 300] and removed size:S PR with lines of changes in (10, 100] labels Oct 21, 2025

nsivabalan reviewed Oct 24, 2025

View reviewed changes

Addressing siva's feedback

0349086

nsivabalan reviewed Oct 24, 2025

View reviewed changes

github-actions Bot added size:S PR with lines of changes in (10, 100] and removed size:M PR with lines of changes in (100, 300] labels Oct 24, 2025

nsivabalan approved these changes Oct 24, 2025

View reviewed changes

nsivabalan merged commit 17a7a6f into apache:master Oct 26, 2025
70 checks passed

yihua pushed a commit that referenced this pull request Oct 28, 2025

fix: updating error messages thrown to end users (#14115)

90a886a

--------- Co-authored-by: sivabalan <n.siva.b@gmail.com>

		+ " - For Hive 2.x: Use hudi-spark-bundle with 'hive2' classifier\n"
		+ " - For Hive 3.x: Use hudi-spark-bundle with 'hive3' classifier\n"

Conversation

vamshikrishnakyatham commented Oct 17, 2025

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

nsivabalan commented Oct 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vamshikrishnakyatham Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vamshikrishnakyatham Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Oct 24, 2025

CI report:

Uh oh!

nsivabalan commented Oct 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vamshikrishnakyatham Oct 20, 2025 •
edited

Loading

vamshikrishnakyatham Oct 20, 2025 •
edited

Loading