https://ops.docs.72602.space/kubernetes/serverless/kserve/canary/index.html KServe supports canary rollouts for inference services. Canary rollouts allow for a new version of an InferenceService to receive a percentage of traffic. Kserve supports a configurable canary rollout strategy with multiple steps. The rollout strategy can also be implemented to rollback to the previous revision if a rollout step fails. KServe automatically tracks the last good revision that was rolled out with 100% traffic. The canaryTrafficPercent field in the component’s spec needs to be set with the percentage of traffic that should be routed to the new revision. KServe will then automatically split the traffic between the last good revision and the revision that is currently being rolled out according to the canaryTrafficPercent value. Hugo en Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/canary/example/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/kserve/canary/example/index.html Create the InferenceService Follow the First Inference Service tutorial. Set up a namespace kserve-test and create an InferenceService. After rolling out the first model, 100% traffic goes to the initial model with service revision 1. kubectl -n kserve-test get isvc sklearn-iris Expectd Output NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE sklearn-iris http://sklearn-iris.kserve-test.example.com True 100 sklearn-iris-predictor--00001 46s 2m39s 70s Apply Canary Rollout Strategy Add the canaryTrafficPercent field to the predictor component Update the storageUri to use a new/updated model. kubectl apply -n kserve-test -f - <<EOF apiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "sklearn-iris" namespace: kserve-test spec: predictor: canaryTrafficPercent: 10 model: args: ["--enable_docs_url=True"] modelFormat: name: sklearn resources: {} runtime: kserve-sklearnserver storageUri: "gs://kfserving-examples/models/sklearn/1.0/model-2" EOF After rolling out the canary model, traffic is split between the latest ready revision 2 and the previously rolled out revision 1.