fix: Remove catalog access from SparkSQLWriter

[linliu-code]

Describe the issue this Pull Request addresses

#14081

HoodieSparkSqlWriter would access enabled catalog within Spark Datasource operations during schema resolution, when it can not get the schema from the commit metadata, table config or data files.

This behavior may cause some confusion since Spark Datasource operation may accidentally access the catalog and get the schema from a table with the same name, which may be an irrelevant table.

Summary and Changelog

We remove the catalog access from the writer, and pass the schema from SQL command. Therefore, for Spark Datasource operations, no catalog access could happen.

Impact

Removal of some unexpected behavior of Spark writer.

Risk Level

Medium.

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[nsivabalan]

can you link the offending commit @linliu-code

[nsivabalan]

hey @linliu-code : can you follow up on test failures on this

[nsivabalan]

hey @linliu-code : did you get to triage the test failures?

[nsivabalan]

hey @linliu-code : any leads on test failures.

[hudi-bot]

CI report:

• d9aa891 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

I am not convinced yet that we should contact catalog even for spark-sql writes.
can you get to the bottom of changes done in https://github.com/apache/hudi/pull/6358/files#r2425086622
and understand prior to that patch, what we were doing.
and why we had to poll catalog. for eg, prior to alexey's patch, was INSERT_INTO polling catalog for the schema.

if not, why can't we remove it completely.

in other words, within deduceWriterSchema, why can't we handle empty tables. i..e no schema for latestTableSchema.

[nsivabalan]

LGTM