current position:Home>How to conduct architecture design 𞓜 deeply reveal Alibaba cloud serverless kubernetes (2)

How to conduct architecture design 𞓜 deeply reveal Alibaba cloud serverless kubernetes (2)

2022-04-29 11:50:29Elastic calculation Bai Xiaosheng

(Chen Xiaoyu) , Alibaba cloud technology experts

In the last article 《 The story , from Docker Speak up | Uncover Alibaba cloud in depth Serverless Kubernetes(1)》 In the article , We introduced Serverless Kubernetes The evolution history of , In this article, we will enter Alibaba cloud Serverless Kubernetes Inside , From the perspective of architecture, let's see how Alibaba cloud implements Serverless Kubernetes Of .

The overall architecture

Serverless Kubernetes The original intention of the design is to provide a set of cloud hosting without operation and maintenance Kubernetes. therefore , We don't just have to solve Kubernetes Master(etcd、kube-apisever、kube-controller-manager) The managed , But also need to achieve Pod Hosted on the cloud , In this way, users only need to submit Yaml You can start the service , There is no need to maintain computing nodes . Based on this , We will take the whole Serverless Kubernetes The architecture is designed as follows :

The whole architecture is divided into three layers :Kubernetes Master And the virtual Kubelet、ECI Back office services and ECI Agent.

The top layer is a cloud hosted Kubernetes Master And a virtual Kubelet(Virtual Kubelet).Virtual Kubelet And standard Kubelet similar , It's just starting Pod It is no longer necessary to call local CRI Start the container , But through HTTP Method call ECI OpenAPI start-up ECI example , Every ECI It's just one. Pod.Virtual Kubelet The original intention of the design is mainly to fit k8s Native architecture : stay k8s in ,Pod By Kubelet Pull up and synchronize the status regularly .

The middle layer is ECI Background services , Responsible for resource allocation and scheduling . Such as user configured log collection ,ECI Backstage will go SLS( Alibaba cloud log service ) Create a collection log , If the user passes PVC by Pod Mount the cloud disk ,ECI The background service will create a cloud disk and mount the cloud disk to ECI On . in addition ,ECI The background is also responsible for resource scheduling , Select the appropriate physical machine node to start ECI, The specific startup method is through the deployment on each node proxy complete .

The bottom is ECI Agent, Responsible for starting the business container . above proxy Just started a safe sandbox , But what users need is a platform to run the business Pod, So we also need to pull up the user's business container in this sandbox ,Agent According to the user's Pod The definition of , Start the corresponding container , And responsible for managing the life cycle of subsequent containers , If Pod Abnormal exit ,Agent I'll pull it up again .

Next we will start from Pod Created process , Introduce the working principle of each component separately .

Hosted on the cloud Kubernetes

The user to create Pod Your request will first be sent to k8s master, So the first problem we need to solve is how to realize k8s master Hosted on the cloud .

Here we use “k8s on k8s” The plan , With the help of k8s Ability of operation and maintenance users k8s master. because k8s master There are many configuration items for , It's hard to pass a Deployment perhaps Statefulset expression , For flexible control , We use k8s CRD(CustomResourceDefinition), take k8s The cluster is abstracted into a Cluster CRD, When the user creates a ASK In clusters , The background will submit a Cluster CR. therefore ,CRD The controller will create a separate for each cluster Namespace, And here Namespace It creates Etcd and k8s master (etcd、kube-apiserver、controller manager) colony .

Careful you may find that , above k8s The management component of does not scheduler, This is because in the ASK in , There is no real computing node , No need to schedule .Virtual Kubelet It's going to be monitored all the time k8s Of apiserver, When the user submits to create Pod After the request ,Virtual Kubelet Found this Pending Of Pod after , Will call ECI OpenAPI Create directly ECI(Pod), And the timing will be Pod Status synced to apiserver.

A link between the preceding and the following ECI backstage

When Virtual Kubelet adopt HTTP Interface request creation ECI When ,ECI After receiving the request, the background will first verify the parameters . for example ECI I won't support it HostPath( Need a separate whitelist ), If the user has configured HostPath It will directly return an error , Then it will ECI Metadata warehousing , And enter the flow control queue . After leaving the team ,ECI The background will pass through the inventory system , Dispatch to a suitable physical machine . One will be deployed on each physical machine proxy, Be responsible for receiving the background creation request and starting ECI Safe sandbox . The whole process is as follows :

Make an analogy , The operating system manages the hardware resources of a machine downward and provides various system calls upward , that ECI The background manages the resources of the whole cloud data center downward and exposes them upward OpenAPI Provide ECI Management ability .

On this level ,ECI The background is the cloud operating system . such as ECI The flow control queue in the background is to ensure the fair scheduling of multiple users , Each user will have a queue , When out of the team , Take out the unscheduled... From each queue in turn ECI , This ensures that scheduling hunger does not occur .

Disguised as a Pod Of ECI-Agent

When proxy start-up ECI Behind the sandbox ,ECI-Agent Will be based on Pod The definition of , stay ECI Start the container inside . Every ECI All disguised as a Pod, So for k8s Come on , It has to satisfy Pod Your daily behavior , Such as container lifecycle management 、 health examination 、 Performance monitoring 、 perform exec/log/attach, And report your status (Status) And events (Event). So in the early days we will kubelet and containerd After simple deletion, directly insert ECI-Agent Inside .

When ECI-Agent Received Pod After creating the request , First of all, I will Pod Local persistence of information , such ECI After restart, you can recover by yourself . then ,ECI-Agent Will Pod Content sent to Kubelet, The rest is Kubelet Execute native startup Pod Logic. .Pod After successful startup ,Kubelet Will Pod Status synced to kube-apiserver, In this way, users can check and see in real time Pod Status quo .

You may have questions here , It says Virtual Kubelet , and ECI There is also a streamlined Kubelet, Is there a conflict between the two ?

Everybody knows , Native Kubelet The main job is to monitor kube-apiserver, When found Pod After scheduling to the local machine, the creation will be executed Pod( Pull up the container ), And report the status . therefore , We will be here Kubelet Disassemble into two parts : Responsible for monitoring kube-apiserver Of Virtual Kubelet and ECI-Agent It really starts Pod The status of the reporting container and the thin container Kubelet.

thus , We go through ASK Start in Pod The whole process , Opened the ASK The overall architecture . You can simply put ASK Comprehend “ Hosted on the cloud k8s master + ECI ( Elastic container )”. One side , Hosted on the cloud k8s The cluster reduces the operation and maintenance cost of users , On the other hand , Bottom ECI Use second level billing on demand to save user expenses , It's a match made in heaven .

ECI According to this Pod The use of dimension on demand is very suitable for Job The operation of the task , from Job The task starts charging until Job Stop charging when the task ends . at present ,ECI More than one million have been created every day , Support many Internet and AI companies .

In subsequent articles , We will gradually dismantle the architecture , Share more practical and internal details , Please keep your eyes on .

This article is excerpted from Chen Xiaoyu, a Alibaba cloud technology expert 《 Uncover Alibaba cloud in depth Serverless Kubernetes》 Special topic . This column will focus on how to Serverless Kubernetes Achieve second level capacity expansion in the scenario , And various technical challenges encountered in large-scale concurrent startup 、 Difficulties and Solutions , Systematically uncover Alibaba cloud Serverless Kubernetes The development of 、 Architecture and core technology .

Author's brief introduction :

Chen Xiaoyu , Alibaba cloud technology experts , Responsible for Alibaba cloud elastic containers (ECI) Bottom R & D work , Once published 《 Explain profound theories in simple language Prometheus》 and 《 Cloud computing 》.

( Article transferred from InfoQ platform , Click here : Link to the original text

copyright notice
author[Elastic calculation Bai Xiaosheng],Please bring the original link to reprint, thank you.

Random recommended