Allow users to differentiate task pods from different namespaces in multi-namespace settings

[GWphua]

Description

Related to #17738.

When we have task pods running in multiple namespaces, it will be difficult to tell which task pods are being started by which cluster, especially when we have very similar job names.

Added an alias prefix tag to allow Kubernetes operators to identify which cluster the task pods are created from.

Here are some of the things I have considered when implementing the change:

1. The job name must respect the 63-character limit.
2. We can easily differentiate between tasks and namespaces.
3. The job name must be unique.
4. What if users are running everything within one namespace? Maybe users will still prefer the old way of displaying task ID.

Intuition:
k8sJobName is currently broken down into 2 segments: {taskId}-{hash(taskId)}. The hashing will take up 32 characters, and is needed to maintain uniqueness. Hence, we can only work with making changes to the taskId part. The first thought is to change taskId to namespace, so that we at least know which namespace a task pod belongs in. However, I encountered cases where namespace have long names, and similar prefix (Think data-infra-druid-cluster-ns1, data-infra-druid-cluster-ns2). This is not a good solution, given that there's a character limit to the job name and there is quite a lot of redundant information.

So, I introduced a new druid.indexer.runner. k8sTaskPodNamePrefix config to allow users to provide an alias prefix instead. Now, k8sJobName will be of the form {alias}-{hash(taskId)}. If k8sTaskPodNamePrefix is not provided, we will default to using the current {taskId}-{hash(taskId)} naming convention.

Note that this solution will not allow us to differentiate between tasks in Kubernetes. However, we can still see our tasks and their corresponding taskID via the Web Console.

Note: My approach involves changing the method of K8sTaskId, prompting change in a lot other classes. This may not be the best way, please advise if there's a more elegant solution.

Release note

You can now name your K8s job names using druid.indexer.runner. k8sTaskPodNamePrefix

---

Key changed/added classes in this PR

• KubernetesPeonLifecycle.java
• KubernetesPeonLifecycleFactory.java
• KubernetesTaskRunner.java
• KubernetesTaskRunnerConfig.java
• PeonLifecycleFactory.java
• K8sTaskId.java
• KubernetesOverlordUtils.java
• K8sTaskAdapter.java
• MultiContainerTaskAdapter.java
• PodTemplateTaskAdapter.java
• SingleContainerTaskAdapter.java

Test Files

• KubernetesPeonLifecycleTest.java
• KubernetesWorkItemTest.java
• TestPeonLifecycleFactory.java
• K8sTaskIdTest.java
• KubernetesPeonClientTest.java
• DruidPeonClientIntegrationTest.java
• K8sTaskAdapterTest.java
• PodTemplateTaskAdapterTest.java

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• been tested in a test Druid cluster.

[FrankChen021]

the name druid.indexer.runner.alias is not intuitive. Maybe druid.indexer.runner.k8sTaskNamePrefix ?
and please add document for this new configuration.

[GWphua]

hi @FrankChen021, I will work on documentation and changes after #17738 is merged. Since this PR will not make sense without it.

[GWphua]

the name druid.indexer.runner.alias is not intuitive. Maybe druid.indexer.runner.k8sTaskNamePrefix ? and please add document for this new configuration.

Hi @FrankChen021, I have changed the name to druid.indexer.runner.k8sTaskPodNamePrefix. Also, added the documentation for this new configuration. Feel free to take a look.

[Copilot]

Pull Request Overview

This PR enhances the Kubernetes task naming functionality by allowing users to specify an alias prefix for task pods via the new configuration property "druid.indexer.runner.k8sTaskPodNamePrefix". The changes update the way K8s job names are generated—incorporating a user-defined alias (when provided)—and adjust the related tests, utilities, and documentation to support the new behavior.

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
Test Files (PodTemplateTaskAdapterTest, K8sTaskAdapterTest, etc.)	Modified tests to use the new K8sTaskId constructors with an added prefix parameter.
KubernetesOverlordUtils.java	Updated job name conversion to incorporate the task pod name prefix with truncation to 30 characters.
KubernetesTaskRunnerConfig.java	Added new configuration property and updated builder methods to support specifying a task pod name prefix.
KubernetesPeonLifecycle and related classes	Updated constructors and method calls to pass the new K8sTaskId with the optional alias prefix.
docs/development/extensions-core/k8s-jobs.md	Updated documentation explaining the new property and its effect on task pod naming.

Comments suppressed due to low confidence (2)

extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerConfig.java:385

• [nitpick] Consider renaming the method 'withk8sTaskPodNamePrefix' to 'withK8sTaskPodNamePrefix' to follow standard camel case conventions for method names.

public Builder withk8sTaskPodNamePrefix(String k8sTaskPodNamePrefix)

docs/development/extensions-core/k8s-jobs.md:772

• [nitpick] Ensure that the naming and description in the documentation exactly match the implementation details in the code to avoid any configuration mismatches.

| `druid.indexer.runner.k8sTaskPodNamePrefix` | `String` |  Use this if you want to change your task name to contain a more human-readable prefix. Maximum 30 characters. Special characters `: - . _` will be ignored. | `""` | No |

[FrankChen021]

LGTM with a minor document description suggestion

[kfaraz]

Note that this solution will not allow us to differentiate between tasks in Kubernetes. However, we can still see our tasks and their corresponding taskID via the Web Console.
Note: My approach involves changing the method of K8sTaskId, prompting change in a lot other classes. This may not be the best way, please advise if there's a more elegant solution.

Yeah, @GWphua , I am not sure if this is a real improvement over what we had earlier.
With a prefix specified, all task pods launched by a cluster will end up looking the same (except for the hash part, which is not exactly human readable). It would be difficult to identify the pod for any given Druid task ID.
Plus, requiring users to specify a prefix just to be able to tell pods apart seems undesirable to me.

That said, using the substring of the task name was not all that great either.
I am not sure what a better alternative would be, but will try to give it some more thought.

Side note:
IIUC from the changes, this shouldn't have any rolling upgrade implications, i.e. task pods launched by an Overlord where the k8sTaskPodNamePrefix was not specified can still be detected by an Overlord where this config has been specified (or an Overlord where the value of the config has been updated).
@GWphua , could you please confirm if this is actually the case.

[GWphua]

IIUC from the changes, this shouldn't have any rolling upgrade implications, i.e. task pods launched by an Overlord where the k8sTaskPodNamePrefix was not specified can still be detected by an Overlord where this config has been specified (or an Overlord where the value of the config has been updated).
@GWphua , could you please confirm if this is actually the case.

Hi @kfaraz, unfortunately there will be disruptions to rolling upgrades.

The old tasks (without configured k8sTaskPodNamePrefix) will not be detected, and Overlord will restart the on-going tasks with the configured k8sTaskPodNamePrefix. From the user's perspective, the web console will show the old tasks as FAILED.

On our side, we need to notify our users to expect this to happen.

[kfaraz]

The old tasks (without configured k8sTaskPodNamePrefix) will not be detected, and Overlord will restart the on-going tasks with the configured k8sTaskPodNamePrefix. From the user's perspective, the web console will show the old tasks as FAILED.

Oh, this is problematic, @GWphua , as the Overlord is losing track of those tasks and re-launching them, even though the original tasks are still running and will finish normally. This will lead to duplicate data and possibly metadata inconsistency errors in case of streaming ingestion.

Is there any way to mitigate this?

[FrankChen021]

This is the same as the other newly introduced overlordNamespace, which requires users to configure them proply: users should have a clear pic that how his clusters are going to deploy.

Currently I think we can add some text to the doc to give a guidance on changing of this property which requires users to stop all running tasks first.

[kfaraz]

Currently I think we can add some text to the doc to give a guidance on changing of this property which requires users to stop all running tasks first.

Yes, let's update the docs and we should also clearly call this out in the release notes and upgrade notes for this feature (just adding it in the PR description should suffice).

cc: @GWphua