[K8s deepdive] Operator 구축할 때 어떤 Framework를 선택하는 게 좋을까

 

구축용 프레임워크를 고를 때 

; Kubebuilder versus Operator SDK

뭐가 다를까? 어떤 상황에서 뭘 선택하는 게 좋을까?

 

https://github.com/operator-framework/operator-sdk/issues/1758

What is the difference between kubebuilder and operator-sdk · Issue #1758 · operator-framework/operator-sdk

 

Where they differ is:

• Operator SDK also has support for Ansible and Helm operators, which make it easy to write operators without having to learn Go and if you already have experience with Ansible or Helm
• Operator SDK includes integrations with the Operator Lifecycle Manager (OLM), which is a key component of the Operator Framework that is important to Day 2 cluster operations, like managing a live upgrade of your operator.
• Operator SDK includes a scorecard subcommand that helps you understand if your operator follows best practices.
• Operator SDK includes an e2e testing framework that simplifies testing your operator against an actual cluster.

 

• Kubebuilder includes an envtest package that allows operator developers to run simple tests with a standalone etcd and apiserver.
• Kubebuilder scaffolds a Makefile to assist users in operator tasks (build, test, run, code generation, etc.); Operator SDK is currently using built-in subcommands. Each has pros and cons. The SDK team will likely be migrating to a Makefile-based approach in the future.
• Kubebuilder uses Kustomize to build deployment manifests; Operator SDK uses static files with placeholders.
• Kubebuilder has recently improved its support for admission and CRD conversion webhooks, which has not yet made it into SDK.

The SDK and Kubebuilder teams work closely together, and we're planning to increase our efforts to help the kubebuilder team maintain controller-tools and controller-runtime so that the entire community has access to the latest features and bug fixes.

 

Kubebuilder Overview:

Kubebuilder is a powerful framework designed to help developers build Kubernetes operators using Go. It’s built on top of the Kubernetes controller-runtime library and aligns closely with Kubernetes conventions.

• Best for:
  • Go developers: Kubebuilder is Go-centric, so if you’re comfortable with Go or want to dive deeper into the Kubernetes ecosystem (which is written in Go), this is an ideal choice.
  • Kubernetes-native developers: Kubebuilder feels more aligned with the core Kubernetes architecture, which makes it suitable for those who want more control and a deeper integration with Kubernetes.
  • Customization and flexibility: It gives you more flexibility to customize your operator as it closely follows Kubernetes' internal patterns. If you need fine-grained control over reconciliation loops, CRD generation, and resource management, Kubebuilder might be the right choice.

Operator SDK Overview:

Operator SDK is a more general-purpose framework for building operators. While it's built on Kubebuilder's foundation (since version 1.0), it offers additional tools and abstractions, including the ability to build operators using Helm or Ansible, in addition to Go.

• Best for:
  • Multiple development options: Operator SDK supports operators written in Go, Helm, and Ansible. This makes it a good option if your team is more comfortable with Helm charts or Ansible playbooks.
  • Quick development and scaffolding: Operator SDK provides more abstractions and built-in tools (like testing, packaging, and deployment management) to make development faster and simpler.
  • Built-in lifecycle management: Operator SDK offers built-in operator lifecycle manager (OLM) support, which helps in packaging and distributing operators more easily.
  • Non-Go developers: If your team prefers not to use Go or wants to develop operators without writing custom controllers from scratch, Operator SDK can make the process easier with Helm or Ansible.

 

Comparison and Use Cases:

Feature	Kubebuilder	Operator SDK
Programming Languages	Go only	Go, Helm, Ansible
Customizability	High (closer to Kubernetes internals)	Moderate to High (depending on the language)
Learning Curve	Higher (more control but requires understanding Kubernetes concepts deeply)	Lower for Helm/Ansible, similar for Go
Speed of Development	Slower (you’re closer to the infrastructure)	Faster, especially with Helm/Ansible
Use Case	Deeply customized, complex operators	Simple-to-moderate operators, multi-language support
Community & Support	Official Kubernetes project, well-documented	Built on top of Kubebuilder but has extra layers of abstraction
OLM (Operator Lifecycle Manager) support	Available but not as deeply integrated	Fully integrated, easier packaging and management of operator lifecycle
Controller-runtime Support	Native	Built on top of controller-runtime (since SDK v1.0)

 

OLM이 뭐지

OLM(Operator Lifecycle Manager)은 Kubernetes에서 Operator의 배포, 관리, 업그레이드, 삭제 등 라이프사이클을 효율적으로 관리하기 위한 도구입니다. OLM은 Operator가 배포된 클러스터 내에서 올바르게 동작하도록 도와주며, 다양한 Operator를 안전하고 일관되게 유지할 수 있도록 지원합니다.

OLM의 주요 기능
1. **Operator 배포**: 
   OLM은 Operator를 Kubernetes 클러스터에 배포하는 작업을 자동화합니다. 사용자는 OLM을 통해 Operator의 배포 및 설정 작업을 간단히 할 수 있습니다.

2. **Operator 업데이트 관리**: 
   OLM은 새로운 버전의 Operator가 제공되면 이를 자동으로 업그레이드하거나 사용자가 수동으로 업그레이드를 할 수 있게 해 줍니다. 이를 통해 Operator의 최신 기능을 쉽게 유지할 수 있습니다.

3. **Operator 의존성 관리**: 
   여러 Operator 간에 의존성이 있을 경우 OLM은 이를 관리해 줍니다. 예를 들어, 한 Operator가 다른 Operator 또는 리소스에 의존하고 있다면 OLM은 그 의존성을 처리합니다.

4. **Operator 권한 관리**: 
   Operator가 클러스터에서 리소스를 관리하기 위해 필요한 권한을 OLM이 자동으로 처리해줍니다. 이를 통해 Operator가 필요한 역할 기반 접근 제어(RBAC)를 설정할 수 있습니다.

5. **Operator 설치 구성(CustomResource)**: 
   OLM은 Operator가 사용할 **CustomResourceDefinition(CRD)**을 관리하며, 사용자가 특정 Custom Resource를 쉽게 생성하고 설정할 수 있도록 도와줍니다.

OLM의 구성 요소
1. **CatalogSource**: 
   Operator와 관련된 패키지 및 버전 정보를 담고 있는 저장소입니다. OLM은 CatalogSource를 통해 Operator의 메타데이터를 조회하고 해당 Operator를 클러스터에 배포합니다.

2. **OperatorGroup**: 
   Operator가 작동할 수 있는 네임스페이스 범위를 지정하는 리소스입니다. Operator가 단일 네임스페이스에서 작동할지, 클러스터 전체에서 작동할지 설정할 수 있습니다.

3. **Subscription**: 
   Operator를 배포하고 유지하기 위한 구독 설정입니다. 구독을 통해 특정 Operator의 설치, 업데이트, 업그레이드를 관리합니다.

4. **InstallPlan**: 
   특정 Operator를 설치하기 위한 계획을 정의하며, OLM이 이를 자동으로 실행하여 Operator를 배포합니다.

OLM을 사용하는 이유
1. **Operator 관리 효율성**: 
   OLM은 수작업으로 Operator를 배포하고 유지 관리하는 과정을 자동화하여 관리의 복잡성을 줄여줍니다.

2. **안정성**: 
   OLM은 Operator의 버전 관리 및 업데이트를 체계적으로 처리하여 안정적인 배포를 보장합니다.

3. **의존성 처리**: 
   여러 Operator 간의 의존성을 자동으로 처리하기 때문에 사용자가 직접 의존성 문제를 해결할 필요가 없습니다.

4. **권한 관리**: 
   OLM은 Operator의 권한을 자동으로 설정하므로, 운영자가 따로 관리할 필요 없이 올바른 권한이 설정된 Operator가 배포됩니다.

언제 OLM을 사용하는 것이 좋을까?
- **다수의 Operator를 사용해야 하는 상황**: 여러 개의 Operator를 관리해야 하거나 다양한 Operator가 서로 의존 관계에 있는 경우 OLM을 사용하면 운영 효율성이 크게 향상됩니다.
- **Operator의 라이프사이클을 자동화하고자 할 때**: Operator의 배포, 업그레이드, 권한 설정을 일관되게 처리하려는 경우 OLM은 필수적인 도구입니다.

요약하면, OLM은 Kubernetes 클러스터에서 Operator의 라이프사이클을 체계적이고 자동화된 방식으로 관리하기 위한 도구입니다.

https://kubebyexample.com/learning-paths/operator-framework/operator-lifecycle-manager/adding-olm-kubernetes

https://olm.operatorframework.io/

WHAT IS OPERATOR Lifecycle manager?

: This project is a component of the Operator Framework, an open source toolkit to manage Kubernetes native applications, called Operators, in a streamlined and scalable way.

 

When to Choose Kubebuilder:

• You want fine-grained control over how your operator interacts with the Kubernetes API.
• Your team is comfortable with Go and prefers staying close to Kubernetes best practices.
• You are building complex or advanced operators with a lot of custom logic.
• You want a framework that is tightly aligned with Kubernetes and gives you flexibility to build highly customized operators.

When to Choose Operator SDK:

• Your team prefers to use Helm or Ansible and wants to avoid writing complex Go code.
• You need to build an operator quickly and easily manage the lifecycle (OLM).
• The operator you’re building involves simple-to-moderate logic, and you don’t need deep control over Kubernetes APIs.
• You want more pre-built tooling and scaffolding to accelerate development, testing, and deployment.
• You’re looking for a framework that supports multiple programming paradigms (Helm, Ansible, or Go).

Conclusion:

• Use Kubebuilder if you need maximum control, prefer Go, and are building operators with complex requirements.
• Use Operator SDK if you’re looking for faster, multi-language operator development (especially with Helm/Ansible) or want more built-in support for lifecycle management.

Ultimately, if your project involves intricate Kubernetes API interactions and your team is comfortable with Go, Kubebuilder would be the better choice. For faster development or if using Helm/Ansible is a priority, Operator SDK is more suitable.

 

 

•  CRDs are tightly coupled with operators. They are useless without one
  • Meh - yes and no. That basically becomes a semantics battle. There are plenty of controllers that use CRDs (traefik, calico, etc) that one would not normally refer to as operators. There is also the generic use case where you just want typed data stored on the API servers but you aren't necessarily reconciling any resources based off of it. More like using it as a data store.

 

해볼까 한 핸즈온

- Building a Tanant Operator using Kubebuilder : https://www.codereliant.io/build-kubernetes-operator-kubebuilder/

Hands-on Kubernetes Operator Development: Kubebuilder

 

 

컨트롤러 작성해서 리소스를 선언적으로 관리하는 것을 오퍼레이터 패턴이라고 하는데요. 오퍼레이터 패턴을 작성할 때 이런 것들을 처음부터 모두 파악하고 들어가기는 너무 어렵습니다. 여기서 팁을 드리자면, OPERATOR SDK라는 툴을 활용해 보면 좋습니다. 이 툴은 Reconcile의 내용만 구현할 수 있도록 스캐폴딩을 제공하거나 Go 코드만으로 커스텀리소스데피니션을 생성할 수 있게 해주는 등 다양한 기능을 제공하는데요. 이를 활용하면 선언적 프로그래밍인 오퍼레이터 패턴을 만드는 데 도움을 받을 수 있습니다.

 

 

 

하지만? Golang을 잘 모르고 && Helm 과 Ansible이 더 익숙한 나는 Operator SDK로 구축을 맛보기로 결정!

 

https://www.velotio.com/engineering-blog/getting-started-with-kubernetes-operators-helm-based-part-1

Getting Started With Kubernetes Operators (Helm Based) - Part 1

Helm based Operator 구축을 먼저 해보려고 한다

(따라하다가 아직 다 못 했어요 ㅎ)

 

---

 

https://opensource.com/article/20/3/kubernetes-operator-sdk

Build a Kubernetes Operator in 10 minutes with Operator SDK

↑ 10분만에 구축하는 오퍼레이터? ㄷㄷ

 

https://github.com/calvin-puram/awesome-kubernetes-operator-resources

GitHub - calvin-puram/awesome-kubernetes-operator-resources: A curated list of awesome resources: articles, books and videos abo

↑ 좋은 레퍼런스가 많다 

 

https://youtu.be/HGHYD7ejBBs?si=WUlWelF1a_xw4QIZ

 

k8s operator와 helm chart 자동화를 어떻게 하는지 알려주는 줄 알았으나

왜 하면 좋은지를 알려줬다

helm을 이용한 operator의 설명정도랄까

 

When diving into Kubernetes, it’s great to start with identifying those safe charts from well-known vendors.

Helm annotations can help you with automatic deployments, but adding operators takes it to the next level of functionality.

By using operators alongside Helm, you can really crank up the automation and keep your microservice setups running smoothly.

Plus, operators shine when it comes to managing backups and cluster configurations, making life a lot easier.

If you’re looking for a deeper dive, I can’t recommend 'Kubernetes Patterns' enough to get a solid grasp of Kubernetes and its resources.