Skip to content

fix: Fix predicates for base file reader in Flink FileGroup reader#14197

Merged
danny0405 merged 2 commits into
apache:masterfrom
cshuo:fix_fg_reader_base_file_filter
Nov 4, 2025
Merged

fix: Fix predicates for base file reader in Flink FileGroup reader#14197
danny0405 merged 2 commits into
apache:masterfrom
cshuo:fix_fg_reader_base_file_filter

Conversation

@cshuo

@cshuo cshuo commented Nov 3, 2025

Copy link
Copy Markdown
Collaborator

Describe the issue this Pull Request addresses

Fix issue: #14196

Summary and Changelog

The current pushed-down predicate for base file reader in flink FileGroup reader can be wrong in some cases, like

// base file: (uuid:'k1', age: 23, ts: 1003)
// log file: (uuid: 'k1', age: 25, ts: 1001)
// query filter: age = 25;
// Then the expected result should be empty, but if predicate age = 25 is pushed down
// into the parquet reader, the result would be wrong as (uuid: 'k1', age: 25, ts: 1001)

When there is log files in a file slice to merge during reading, we should make sure the predicate contains only record key fields.

Impact

Bug fix.

Risk Level

low

Documentation Update

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

@github-actions github-actions Bot added the size:M PR with lines of changes in (100, 300] label Nov 3, 2025
@cshuo cshuo force-pushed the fix_fg_reader_base_file_filter branch from 42bac3e to 3d45c81 Compare November 3, 2025 07:11
@hudi-bot

hudi-bot commented Nov 3, 2025

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands@hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@danny0405 danny0405 merged commit 95b461d into apache:master Nov 4, 2025
71 of 76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-1.1.0 size:M PR with lines of changes in (100, 300]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants