https://ops.docs.72602.space/kubernetes/serverless/kserve/auto_scaling/index.html Soft Limit You can configure InferenceService with annotation autoscaling.knative.dev/target for a soft limit. The soft limit is a targeted limit rather than a strictly enforced bound, particularly if there is a sudden burst of requests, this value can be exceeded. apiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "sklearn-iris" namespace: kserve-test annotations: autoscaling.knative.dev/target: "5" spec: predictor: model: args: ["--enable_docs_url=True"] modelFormat: name: sklearn resources: {} runtime: kserve-sklearnserver storageUri: "gs://kfserving-examples/models/sklearn/1.0/model" Hard Limit You can also configure InferenceService with field containerConcurrency with a hard limit. The hard limit is an enforced upper bound. If concurrency reaches the hard limit, surplus requests will be buffered and must wait until enough capacity is free to execute the requests. Hugo en