Add support for KubeVirt - #3133
Conversation
This commit allows cloud-api-adaptor(CAA) to support KubeVirt-based clouds. There is currently no provider in CAA that supports clouds built with standard KubeVirt. By implementing an KubeVirt provider, we can offer secure pod execution environments in a wider range of fields. Adding an inbox Provider for KubeVirt, referring to the following documentation. https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/docs/addnewprovider.md - Add and initialize the KubeVirt provider manager - Add a definition for the configuration struct - Add cloud interfaces - Add provider interfaces - Add additional files to modularize the code - Add configuration to the Helm chart configmap to include the contents of a specified file - Add relevant unit tests - Add relevant E2E tests - Update entrypoint.sh and Makefile - Added a README covering CAA deployment on KubeVirt, Pod VM creation, and e2e test steps Signed-off-by: hirai3311 <fj6082at@aa.jp.fujitsu.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a new built-in KubeVirt cloud provider to Cloud API Adaptor (CAA), including provider implementation, Helm chart support for mounting provider-specific config files, and test/provisioning scaffolding to run unit and E2E validation on KubeVirt-based environments.
Changes:
- Introduces a new
cloud-providers/kubevirtprovider (manager + provider + KubeVirt/Kubernetes client helpers) with unit tests. - Extends the Helm chart to optionally include and mount
providers/<provider>/*files into the CAA DaemonSet. - Adds KubeVirt E2E/provisioner wiring, build integration (Makefile + entrypoint), and KubeVirt deployment documentation.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 19 comments.
Show a summary per file
| File | Description |
|---|---|
| src/cloud-providers/kubevirt/types.go | Adds KubeVirt provider config struct. |
| src/cloud-providers/kubevirt/provider.go | Implements provider lifecycle (create/delete instance) for KubeVirt. |
| src/cloud-providers/kubevirt/provider_test.go | Adds unit tests for provider behaviors. |
| src/cloud-providers/kubevirt/manager.go | Registers the kubevirt provider and CLI/env parsing. |
| src/cloud-providers/kubevirt/manager_test.go | Adds tests for provider manager registration/flag parsing. |
| src/cloud-providers/kubevirt/kubevirt.go | Adds KubeVirt/Kubernetes client helpers + YAML unmarshalling helpers. |
| src/cloud-providers/kubevirt/kubevirt_test.go | Adds unit tests for client helpers and config unmarshalling. |
| src/cloud-api-adaptor/test/tools/provisioner-cli/kubevirt.go | Adds kubevirt build-tag entry for provisioner CLI. |
| src/cloud-api-adaptor/test/provisioner/kubevirt/provision.go | Registers kubevirt provisioner and chart installer factory. |
| src/cloud-api-adaptor/test/provisioner/kubevirt/provision_common.go | Implements KubeVirt provisioner + Helm chart configuration/copy logic. |
| src/cloud-api-adaptor/test/provisioner/kubevirt/kubevirt.properties | Adds example properties file for kubevirt E2E runs. |
| src/cloud-api-adaptor/test/e2e/kubevirt_test.go | Adds kubevirt-tagged E2E tests. |
| src/cloud-api-adaptor/test/e2e/kubevirt_common.go | Adds kubevirt-specific E2E assertions/helpers. |
| src/cloud-api-adaptor/Makefile | Includes kubevirt in built-in providers list. |
| src/cloud-api-adaptor/kubevirt/README.md | Documents deployment and usage of CAA on KubeVirt, plus E2E steps. |
| src/cloud-api-adaptor/install/charts/peerpods/values.yaml | Updates provider list comment to include kubevirt. |
| src/cloud-api-adaptor/install/charts/peerpods/templates/daemonset.yaml | Mounts provider file ConfigMap into the DaemonSet when present. |
| src/cloud-api-adaptor/install/charts/peerpods/templates/configmap-filelist.yaml | Adds ConfigMap template that embeds providers/<provider>/* files. |
| src/cloud-api-adaptor/install/charts/peerpods/templates/_helpers.tpl | Adds helper to detect presence of provider config file directory. |
| src/cloud-api-adaptor/install/charts/peerpods/providers/kubevirt.yaml | Adds kubevirt provider values stub for the chart. |
| src/cloud-api-adaptor/install/charts/peerpods/providers/kubevirt-secrets.yaml | Adds kubevirt secrets stub (none required). |
| src/cloud-api-adaptor/entrypoint.sh | Adds kubevirt to entrypoint provider dispatch/help. |
| src/cloud-api-adaptor/cmd/cloud-api-adaptor/kubevirt.go | Adds build-tagged kubevirt provider registration into the main binary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| package kubevirt | ||
|
|
||
| import ( | ||
| _ "github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/provisioner" | ||
| ) |
| providerConfigs: | ||
| kubevirt: {} | ||
| # CA certificate file for custom TLS (e.g. /etc/certificates/ca.crt) |
| serPath, err := expandUser(properties["path_to_serviceconfig"]) | ||
| if err != nil { | ||
| log.Infof("path_to_serviceconfig Path was not found") | ||
| return nil, nil | ||
| } |
| // Copy the kubeconfig and vmconfig to the destination directory | ||
| kubeconfigSrc := properties["path_to_kubeconfig"] | ||
| if kubeconfigSrc == "" { | ||
| return fmt.Errorf("path_to_kubeconfig is not set in properties") | ||
| } | ||
| dstPath := filepath.Join(dstDir, filepath.Base(kubeconfigSrc)) | ||
| if err := copyFile(kubeconfigSrc, dstPath); err != nil { | ||
| return fmt.Errorf("failed to copy path_to_kubeconfig from %s to %s: %w", kubeconfigSrc, dstPath, err) | ||
| } | ||
|
|
||
| vmconfigSrc := properties["path_to_vmconfig"] | ||
| if vmconfigSrc == "" { | ||
| return fmt.Errorf("path_to_vmconfig is not set in properties") | ||
| } | ||
| dstPath = filepath.Join(dstDir, filepath.Base(vmconfigSrc)) | ||
| if err := copyFile(vmconfigSrc, dstPath); err != nil { | ||
| return fmt.Errorf("failed to copy path_to_vmconfig from %s to %s: %w", vmconfigSrc, dstPath, err) |
| func NewProvider(config *Config) (provider.Provider, error) { | ||
| kubevirtClient, err := NewProviderClient() | ||
| if err != nil { | ||
| logger.Printf("Unable to create KubeVirt client: %v", err) | ||
| return nil, err | ||
| } | ||
|
|
||
| kubernetesClient, err := NewKubernetesClient() | ||
| if err != nil { | ||
| logger.Printf("Unable to create Kubernetes client: %v", err) | ||
| return nil, err | ||
| } | ||
|
|
||
| vm, err := VMconfigUnmarshal() | ||
| if err != nil { | ||
| logger.Printf("Failed to unmarshal VMconfig: %v", err) | ||
| return nil, err | ||
| } | ||
|
|
||
| var service *corev1.Service | ||
| if config.serviceconfigfile != "" { | ||
| service, err = ServiceconfigUnmarshal(config.serviceconfigfile) | ||
| if err != nil { |
| func (p *kubevirtProvider) CreateInstance(ctx context.Context, podname, sandboxID string, cloudConfig cloudinit.CloudConfigGenerator, spec provider.InstanceTypeSpec) (*provider.Instance, error) { | ||
| instancename := util.GenerateInstanceName(podname, sandboxID, maxInstanceNameLen) | ||
|
|
||
| cloudConfigData, err := cloudConfig.Generate() | ||
| if err != nil { | ||
| logger.Printf("Failed to cloudConfig Generate :%v", err) | ||
| return nil, err | ||
| } | ||
|
|
||
| vm := p.vmtemplate.DeepCopy() | ||
|
|
||
| _, err = p.kubernetesClient.CreateSecret(vm.Namespace, instancename, cloudConfigData) | ||
| if err != nil { | ||
| logger.Printf("Failed to create cloud-init Secret: %v", err) | ||
| return nil, err | ||
| } |
| {{ ( base $path ) }}: | | ||
| {{- tpl ( $currentScope.Files.Get $path ) $currentScope | trim | nindent 4 }} |
| // Build the service and return its information. | ||
| func (c *K8sclient) Getservice(namespace string, service *corev1.Service) (*corev1.Service, error) { | ||
| createservice, err := c.client.CoreV1().Services(namespace).Create(context.TODO(), service, metav1.CreateOptions{}) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("Failed to create Service: %w", err) | ||
| } | ||
| return createservice, nil | ||
| } |
| name: configlist | ||
| namespace: {{ .Values.namespace }} | ||
| labels: |
| const ( | ||
| kubeconfigpath = "/etc/config/caa/kubevirt/kubeconfig" | ||
| vmconfigpath = "/etc/config/caa/kubevirt/podvm.yaml" | ||
| cloudinitName = "cloudinit" |
|
@Satoshi-Hirai371 can you please take a look at the copilot review comments |
- Fixed the package imports in `test/tools/provisioner-cli/kubevirt.go`, which is used in e2e tests. - Corrected the return value in `test/provisioner/kubevirt/provision_common.go`, which is used in e2e tests, when an error occurs. - Ensured that the context passed from the `CreateInstance` method is included when issuing an API call to KubeVirt. - We revised the function names in `cloud-providers/kubevirt/kubevirt.go` to better match their intended purposes. - We corrected an incorrect namespace listed in `install/charts/peerpods/templates/configmap-filelist.yaml`. Signed-off-by: hirai3311 <fj6082at@aa.jp.fujitsu.com>
|
Hi maintainers and reviewers,
Note that we determined no corrections were necessary for the other issues raised, for the following reasons:
|
|
Hi, we had a discussion on the community call about the proliferation of new providers and the challenges of having code dropped into the repo and then not maintained adding burden to a small group of maintainers, as such we are looking to move to an approach of keeping provider out of tree. We have some documentation for this approach in the new providers page: https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/docs/addnewprovider.md#memo-adding-support-for-a-new-external-provider. We admit that it might be outdated and might require more clarification, but if you could consider following this approach, rather than a built-in provider then we'd love the feedback. Thanks |
Summary:
This pull request allows cloud-api-adaptor(CAA) to support KubeVirt-based clouds.
Background:
There is currently no provider in CAA that supports clouds built with standard KubeVirt.
By implementing an KubeVirt provider, we can offer secure pod execution environments in a wider range of fields.
As stated in the link below, we are focusing on implementations related to Arm CCA (Confidential Compute Architecture).
This PR provides an architecture-independent implementation.
Moving forward, we aim to execute PeerPods on KubeVirt clouds that support Arm CCA.
Links:
This PR is for the purpose and related work described in the following Issue.
#3124
Task:
Refer to the following documentation to add a Provider for KubeVirt.
https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/docs/addnewprovider.md
Implemented:
Out of scope: Future implementation: