https://ops.docs.72602.space/kubernetes/serverless/kserve/index.html Install Kserve Serving Inference First Pytorch ISVC First Custom Model First Model In Minio Kafka Sink Transformer Generative First Generative Service Canary Policy Rollout Example Auto Scaling Hugo en Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/install/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/install/index.html Preliminary v 1.30 + Kubernetes has installed, if not check 🔗link Helm has installed, if not check 🔗link Installation Install By Shell Steps ArgoCD Preliminary 1. Kubernetes has installed, if not check 🔗link 2. Helm binary has installed, if not check 🔗link 1.install from script directly https://ops.docs.72602.space/kubernetes/serverless/kserve/serving/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/serving/index.html Inference First Pytorch ISVC First Custom Model First Model In Minio Kafka Sink Transformer Generative First Generative Service https://ops.docs.72602.space/kubernetes/serverless/kserve/canary/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/canary/index.html KServe supports canary rollouts for inference services. Canary rollouts allow for a new version of an InferenceService to receive a percentage of traffic. Kserve supports a configurable canary rollout strategy with multiple steps. The rollout strategy can also be implemented to rollback to the previous revision if a rollout step fails. KServe automatically tracks the last good revision that was rolled out with 100% traffic. The canaryTrafficPercent field in the component’s spec needs to be set with the percentage of traffic that should be routed to the new revision. KServe will then automatically split the traffic between the last good revision and the revision that is currently being rolled out according to the canaryTrafficPercent value. https://ops.docs.72602.space/kubernetes/serverless/kserve/auto_scaling/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/auto_scaling/index.html Soft Limit You can configure InferenceService with annotation autoscaling.knative.dev/target for a soft limit. The soft limit is a targeted limit rather than a strictly enforced bound, particularly if there is a sudden burst of requests, this value can be exceeded. apiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "sklearn-iris" namespace: kserve-test annotations: autoscaling.knative.dev/target: "5" spec: predictor: model: args: ["--enable_docs_url=True"] modelFormat: name: sklearn resources: {} runtime: kserve-sklearnserver storageUri: "gs://kfserving-examples/models/sklearn/1.0/model" Hard Limit You can also configure InferenceService with field containerConcurrency with a hard limit. The hard limit is an enforced upper bound. If concurrency reaches the hard limit, surplus requests will be buffered and must wait until enough capacity is free to execute the requests.