General
Why is the Workload or Application Detail page so slow or not responding?
We have identified a performance issue that happens while visiting the Workload or Application detail page, related to discovering metrics in order to show custom dashboards. This issue was originally reported here and is now tracked there.
To summarize, Kiali might be very slow to fetch some metrics from Prometheus, it might even run out of memory, and so does Prometheus.
The immediate workaround you can take in that situation is to disable dashboards discovery from config:
external_services:
custom_dashboards:
discovery_enabled: "false"
But we would also recommend that you consider a more robust setup for Prometheus, like the one described in this Istio guide (see also this Kiali blog post), in order to decrease the metrics cardinality.
As explained in the tracking issue, a modification of the Prometheus API should soon be available and, hopefully, would allow Kiali to get what it needs at a much lower cost.
What do I need to run Kiali in a private cluster?
Private clusters have higher network restrictions. Kiali needs your cluster to allow TCP
traffic between the Kubernetes API service and the Istio Control Plane namespace, for both the 8080
and 15000
ports. This is required for features such as Health and Envoy Dump to work as expected.
Make sure that the firewalls in your cluster allow the connections mentioned above.
Check section Google Kubernetes Engine (GKE) Private Cluster requirements in the Installation Guide.
How do I access Kiali UI?
See Accessing Kiali in the Installation guide.
Does Kiali support Internet Explorer?
No version of Internet Explorer is supported with Kiali. Users may experience some issues when using Kiali through this browser.
Kiali does not work - What do i do?
If you are hitting a problem, whether it is listed here or not, do not hesitate to open a GitHub Discussion to ask about your situation. If you are hitting a bug, or need a feature, you can vote (using emojis) for any existing bug or feature request found in the GitHub Issues. This will help us prioritize the most needed fixes or enhancements. You can also create a new issue.
How do I obtain the logs for Kiali?
Kiali operator logs can be obtained from within the Kiali operator pod. For example, if the operator is installed in the kiali-operator
namespace:
KIALI_OPERATOR_NAMESPACE="kiali-operator"
kubectl logs -n ${KIALI_OPERATOR_NAMESPACE} $(kubectl get pod -l app=kiali-operator -n ${KIALI_OPERATOR_NAMESPACE} -o name)
Kiali server logs can be obtained from within the Kiali server pod. For example, if the Kiali server is installed in the istio-system
namespace:
KIALI_SERVER_NAMESPACE="istio-system"
kubectl logs -n ${KIALI_SERVER_NAMESPACE} $(kubectl get pod -l app=kiali -n ${KIALI_SERVER_NAMESPACE} -o name)
Note that you can configure the logger in the Kiali Server via these settings in the Kiali CR:
log_format
supports “text” and “json”.log_level
supports “trace”, “debug”, “info”, “warn”, “error”, “fatal”.time_field_format
supports a link:https://golang.org/pkg/time/[golang time format]sampler_rate
defines a basic log sampler setting as an integer. With this setting every “sampler_rate”-th message will be logged. By default, every message is logged.
For example,
spec:
deployment:
logger:
log_level: info
log_format: text
sampler_rate: "1"
time_field_format: "2006-01-02T15:04:05Z07:00"
Which Istio metrics and attributes are required by Kiali?
To reduce Prometheus storage some users want to customize the metrics generated by Istio. This can break Kiali if the pruned metrics and/or attributes are used by Kiali in its graph or metric features.
Kiali currently requires the following metrics and attributes (note, this assumes Telemetry V2 in being used):
Metric | Notes |
---|---|
istio_requests_total | used throughout Kiali and the primary metric for http/grpc traffic graph generation |
istio_request_bytes_sum | used in metrics displays |
istio_request_duration_milliseconds_bucket | used throughout Kiali for response time calculation. used by iter8 extension. |
istio_request_duration_milliseconds_sum | used throughout Kiali for response time calculation. used by iter8 extension. |
istio_response_bytes_sum | used in metrics displays |
istio_tcp_received_bytes_total | used in metrics displays |
istio_tcp_sent_bytes_total | used throughout Kiali and the primary metric for tcp traffic graph generation |
Attribute | Metric | Notes |
---|---|---|
connection_security_policy | istio_requests_total | used only when graph Security display option is enabled |
destination_canonical_revision | istio_requests_total | |
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_tcp_sent_bytes_total | ||
destination_canonical_service | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
destination_principal | istio_requests_total | used only when graph Security display option is enabled |
istio_tcp_sent_bytes_total | ||
destination_service | istio_requests_total | |
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_tcp_sent_bytes_total | ||
destination_service_name | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
destination_service_namespace | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
destination_workload | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
destination_workload_namespace | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
grpc_response_status | istio_requests_total | used only when request_protocol=“grpc” |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
job | istio_request_duration_milliseconds_sum | used only by iter8 extension |
reporter | istio_requests_total | both “source” and “destination” metrics are used by Kiali |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
request_operation | istio_requests_total | used only when request classification is configured. “request_operation” is the default attribute, it is configurable. |
istio_request_bytes_sum | ||
istio_response_bytes_sum | ||
request_protocol | istio_requests_total | |
istio_request_bytes_sum | ||
istio_response_bytes_sum | ||
response_code | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
response_flags | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
source_canonical_revision | istio_requests_total | |
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_tcp_sent_bytes_total | ||
source_canonical_service | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
source_principal | istio_requests_total | |
istio_tcp_sent_bytes_total | ||
source_workload | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total | ||
source_workload_namespace | istio_requests_total | |
istio_request_bytes_sum | ||
istio_request_duration_milliseconds_bucket | ||
istio_request_duration_milliseconds_sum | ||
istio_response_bytes_sum | ||
istio_tcp_received_bytes_total | ||
istio_tcp_sent_bytes_total |
What are the minimum privileges to login when using RBAC?
The get namespace
privilege is required for Kiali login when using an
RBAC-enabled authentication strategy. The user needs the privilege in at least
one namespace. The Kiali Operator will provide a ClusterRole
named either
kiali
or kiali-viewer
with the needed privilege. Users can be bound to
this role.
When using a customized Role
or ClusterRole
then the following rule is
required for Kiali login:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: custom-kiali-role
rules:
- apiGroups: [""]
resources:
- namespaces
verbs:
- get
Although required for login, this privilege is not sufficient for Kiali to function well in general.
What is the License?
See here for the Kiali license.
Why isn’t my namespace in the Namespace Selection dropdown?
When deploying Kiali with the Kiali operator, by default some namespaces are excluded from the list of namespaces provided by the API and UI. Kiali filters out these namespaces and you will not see them in the Namespace Selection dropdown. You can adjust which namespaces are excluded by setting the spec.api.namespaces.exclude
field on the Kiali CR.
In addition, you must ensure that Kiali has access to the namespaces you are interested in by setting the spec.deployment.accessible_namespaces
field on the Kiali CR accordingly. Setting spec.api.namespaces.exclude
alone does not give Kiali access to the namespaces. See the Namespace Management guide for more information.
Kiali also caches namespaces by default for 10 seconds. If the cache is enabled, it might take up to the spec.kubernetes_config.cache_token_namespace_duration
in order for a newly added namespace to be seen by Kiali.
Workload “is not found as” messages
Kiali queries Deployment ,ReplicaSet, ReplicationController, DeploymentConfig, StatefulSet, Job and CronJob controllers. Deployment, ReplicaSet and StatefulSet are always queried, but ReplicationController, DeploymentConfig, Job and CronJobs are excluded by default for performance reasons.
To include them, update the list of excluded_workloads from the Kiali config.
# ---
# excluded_workloads:
# - "CronJob"
# - "DeploymentConfig"
# - "Job"
# - "ReplicationController"
#
An empty list will tell Kiali to query all type of known controllers.