This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Features

An overview of important Kiali features.

1: Topology
2: Health
3: Detail Views
4: Tracing
5: Validations
6: Wizards
7: Istio Configuration
8: Security
9: Istio Status
10: Advanced Mesh Deployment and Multi-cluster support

1 - Topology

How Kiali visualizes the mesh topology.

Kiali offers multiple ways for users to examine their mesh Topology. Each combines several information types to help users quickly evaluate the health of their service architecture.

Overview

Kiali’s default page is the topology Overview. It presents a high-level view of the namespaces accessible to Kiali, for this user. It combines service and application information, along with telemetry, validations and health, to provide a holistic summary of system behvior. The Overview page provides numerous filtering, sorting and presentation options. From here users can perform namespace-level Actions, or quickly navigate to more detailed views.

Topology namespace overview

Graph

The Kiali Graph offers a powerful visualization of your mesh traffic. The topology combines real-time request traffic with your Istio configuration information to present immediate insight into the behavior of service mesh, allowing you to quickly pinpoint issues. Multiple Graph Types allow you to visualize traffic as a high-level service topology, a low-level workload topology, or as an application-level topology.

Graph nodes are decorated with a variety of information, pointing out various route routing options like virtual services and service entries, as well as special configuration like fault-injection and circuit breakers. It can identify mTLS issues, latency issues, error traffic and more. The Graph is highly configurable, can show traffic animation, and has powerful Find and Hide abilities.

You can configure the graph to show the namespaces and data that are important to you, and display it in the way that best meets your needs.

Topology graph

Health

Colors in the graph represent the health of your service mesh. A node colored red or orange might need attention. The color of an edge between components represents the health of the requests between those components. The node shape indicates the type of component such as services, workloads, or apps.

The health of nodes and edges is refreshed automatically based on the user’s preference. The graph can also be paused to examine a particular state, or replayed to re-examine a particular time period.

Topology graph health

Side-Panel

The collapsible side-panel summarizes the current graph selection, or the graph as a whole. A single-click will select the node, edge, or box of interest. The side panel provides:

Charts showing traffic and response times.
Health details.
Links to fully-detailed pages.
Response Code and Host breakdowns.
Traces involving the selected component.

Topology graph side-panel app Topology graph side-panel service Topology graph side-panel workload

Node Detail

A single-click selects a graph node. A double-click drills in to show the node’s Detail Graph. The node detail graph visualizes traffic from the point-of-view of that node, meaning it shows only the traffic reported by that node’s Istio proxy.

You can return back to the main graph, or double-click to change to a different node’s detail graph.

Topology graph node detail

Traffic Animation

Kiali offers several display options for the graph, including traffic animation.

For HTTP traffic, circles represent successful requests while red diamonds represent errors. The more dense the circles and diamonds the higher the request rate. The faster the animation the faster the response times.

TCP traffic is represented by offset circles where the speed of the circles indicates the traffic speed.

Topology graph animation

Graph Types

Kiali offers four different traffic-graph renderings:

The workload graph provides the a detailed view of communication between workloads.
The app graph aggregates the workloads with the same app labeling, which provides a more logical view.
The versioned app graph aggregates by app, but breaks out the different versions providing traffic breakdowns that are version-specific.
The service graph provides a high-level view, which aggregates all traffic for defined services.

Topology graph type workload Topology graph type app Topology graph type versioned app Topology graph service

Replay

Graph replay allows you to replay traffic from a selected past time-period. This gives you a chance to thoroughly examine a time period of interest, or share it with a co-worker. The graph is fully bookmarkable, including replay.

Operation Nodes

Istio v1.6 introduced Request Classification. This powerful feature allows users to classify requests into aggregates, called “Operations” by convention, to better understand how a service is being used. If configured in Istio the Kiali graph can show these as Operation nodes. The user needs only to enable the “Operation Nodes” display option. Operations can span services, for example, “VIP” may be configured for both CarRental and HotelRental services. To see total “VIP” traffic then display operation nodes without service nodes. To see “VIP” traffic specific to each service then also enable the “Service Nodes” display option.

When selected, an Operation node also provides a side-panel view. And when double-clicked a node detail graph is also provided.

Because operation nodes represent aggregate traffic they are not compatible with Service graphs, which themselves are already logical aggregates. For similar reasons response time information is not available on edges leading into or out of operation nodes. But by selecting the edge the response time information is available in the side panel (if configured).

Operation nodes are represented as pentagons in the Kiali graph:

Topology graph operation

2 - Health

Health

Health Configuration

Kiali calculates health by combining the individual health of several indicators, such as pods and request traffic. The global health of a resource reflects the most severe health of its indicators.

Health Indicators

The table below lists the current health indicators and whether the indicator supports custom configuration for its health calculation.

Indicator	Supports Configuration
Pod Status	No
Traffic Health	Yes

Icons and colors

Kiali use icons and colors to indicate the health of resources and associated request traffic.

No Health Information (NA)
Healthy
Degraded
Failure

Default Values

Request Traffic

By default Kiali uses the traffic rate configuration shown below. Application errors have minimal tolerance while client errors have a higher tolerance reflecting that some level of client errors is often normal (e.g. 404 Not Found):

For http protocol 4xx are client errors and 5xx codes are application errors.
For grpc protocol all 1-16 are errors (0 is success).

So, for example, if the rate of application errors is >= 0.1% kiali will show Degraded health and if > 10% will show Failure health.

# ...
  health_config:
    rate:
      - namespace: ".*"
        kind: ".*"
        name: ".*"
        tolerance:
          - code: "^5\\d\\d$"
            direction: ".*"
            protocol: "http"
            degraded: 0
            failure: 10
          - code: "^4\\d\\d$"
            direction: ".*"
            protocol: "http"
            degraded: 10
            failure: 20
          - code: "^[1-9]$|^1[0-6]$"
            direction: ".*"
            protocol: "grpc"
            degraded: 0
            failure: 10
# ...

Configuration

Custom health configuration is specified in the Kiali CR. To see the supported configuration syntax for health_config visit Kiali CR.

Kiali applies the first matching rate configuration (namespace, kind, etc) and calculates the status for each tolerance. The reported health will be the status with highest priority (see below).

Rate Option	Definition	Default
namespace	Matching Namespaces (regex)	.* (match all)
kind	Matching Resource Types (workload\|app\|service) (regex)	.* (match all)
name	Matching Resource Names (regex)	.* (match all)
Tolerance	Array of tolerances to apply.

Tolerance Option	Definition	Default
code	Matching Response Status Codes (regex) [1]	required
direction	Matching Request Directions (inbound\|outbound) (regex)	.* (match all)
protocol	Matching Request Protocols (http\|grpc) (regex)	.* (match all)
degraded	Degraded Threshold(% matching requests >= value)	0
failure	Failure Threshold (% matching requests >= value)	0

[1] The status code typically depends on the request protocol. The special code -, a single dash, is used for requests that don’t receive a response, and therefore no response code.

Kiali reports traffic health with the following top-down status priority :

Priority	Rule (value=% matching requests)	Status
1	value >= FAILURE threshold	FAILURE
2	value >= DEGRADED threshold AND value < FAILURE threshold	DEGRADED
3	value > 0 AND value < DEGRADED threshold	HEALTHY
4	value = 0	HEALTHY
5	No traffic	No Health Information

Examples

These examples use the repo _https://github.com/kiali/demos/tree/master/error-rates_.

In this repo we can see 2 namespaces: alpha and beta (Demo design).

Alpha

Where nodes return the responses (You can configure responses here):

App (alpha/beta)	Code	Rate
x-server	200	9
x-server	404	1
y-server	200	9
y-server	500	1
z-server	200	8
z-server	201	1
z-server	201	1

The applied traffic rate configuration is:

# ...
health_config:
  rate:
   - namespace: "alpha"
     tolerance:
       - code: "404"
         failure: 10
         protocol: "http"
       - code: "[45]\\d[^\\D4]"
         protocol: "http"
   - namespace: "beta"
     tolerance:
       - code: "[4]\\d\\d"
         degraded: 30
         failure: 40
         protocol: "http"
       - code: "[5]\\d\\d"
         protocol: "http"
# ...

After Kiali adds default configuration we have the following (Debug Info Kiali):

{
  "healthConfig": {
    "rate": [
      {
        "namespace": "/alpha/",
        "kind": "/.*/",
        "name": "/.*/",
        "tolerance": [
          {
            "code": "/404/",
            "degraded": 0,
            "failure": 10,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/[45]\\d[^\\D4]/",
            "degraded": 0,
            "failure": 0,
            "protocol": "/http/",
            "direction": "/.*/"
          }
        ]
      },
      {
        "namespace": "/beta/",
        "kind": "/.*/",
        "name": "/.*/",
        "tolerance": [
          {
            "code": "/[4]\\d\\d/",
            "degraded": 30,
            "failure": 40,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/[5]\\d\\d/",
            "degraded": 0,
            "failure": 0,
            "protocol": "/http/",
            "direction": "/.*/"
          }
        ]
      },
      {
        "namespace": "/.*/",
        "kind": "/.*/",
        "name": "/.*/",
        "tolerance": [
          {
            "code": "/^5\\d\\d$/",
            "degraded": 0,
            "failure": 10,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/^4\\d\\d$/",
            "degraded": 10,
            "failure": 20,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/^[1-9]$|^1[0-6]$/",
            "degraded": 0,
            "failure": 10,
            "protocol": "/grpc/",
            "direction": "/.*/"
          }
        ]
      }
    ]
  }
}

What are we applying?

For namespace alpha, all resources
Protocol http if % requests with error code 404 are >= 10 then FAILURE, if they are > 0 then DEGRADED
Protocol http if % requests with others error codes are> 0 then FAILURE.
For namespace beta, all resources
Protocol http if % requests with error code 4xx are >= 40 then FAILURE, if they are >= 30 then DEGRADED
Protocol http if % requests with error code 5xx are > 0 then FAILURE
For other namespaces kiali apply defaults.
Protocol http if % requests with error code 5xx are >= 20 then FAILURE, if they are >= 0.1 then DEGRADED
Protocol grpc if % requests with error code match /^[1-9]$|^1[0-6]$/ are >= 20 then FAILURE, if they are >= 0.1 then DEGRADED

Alpha	Beta

3 - Detail Views

Kiali provides list and detail views for your mesh components.

Kiali provides filtered list views of all your service mesh definitions. Each view provides health, details, YAML definitions and links to help you visualize your mesh. There are list and detail views for:

Applications
Istio Configuration
Services
Workloads

Detail list apps Detail list service Detail list workload Detail list Istio config

Selecting an object from the list will bring you to its detail page. For Istio Config, Kiali will present its YAML, along with contextual validation information. Other mesh components present a variety of Tabs.

Overview Tab

Overview is the default Tab for any detail page. The overview tab provides detailed information, including health status, and a detailed mini-graph of the current traffic involving the component. The full set of tabs, as well as the detailed information, varies based on the component type.

Each Overview provides:

links to related components and linked Istio configuration.
health status.
validation information.
an Action menu for actions that can be taken on the component.
- several Wizards are available.

And also type-specfic information. For example:

Service detail includes Network information.
Workload detail provides backing Pod information.

Detail overview app Detail overview service Detail overview workload

Both Workload and Service detail can be customized to some extent, by adding additional details supplied as annotations. This is done through the additional_display_details field in the Kiali CR.

Detail overview additional details

Traffic

The Traffic Tab presents a service, app, or workload’s Inbound and Outbound traffic in a table-oriented way:

Detail traffic

Logs

Workload detail offers a special Logs tab. Kiali offers a special unified logs view that lets users correlate app and proxy logs. It can also add-in trace span information to help identify important traces based on associated logging. More powerful features include substring or regex Show/Hide, full-screen, and the ability to set proxy log level without a pod restart.

Detail logs

Metrics

Each detail view provides Inbound Metrics and/or Outbound Metrics tabs, offering predefined metric dashboards. The dashboards provided are tailored to the relevant application, workload or service level. Application and workload detail views show request and response metrics such as volume, duration, size, or tcp traffic. The service detail view shows request and response metrics for inbound traffic only.

Kiali allows the user to customize the charts by choosing the charted dimensions. It can also present metrics reported by either source or destination proxy metrics. And for troublshooting it can overlay trace spans.

Detail metric inbound Detail metrics outbound

Traces

Each detail view provides a Traces tab with a native integration with Jaeger. For more, see Tracing.

Dashboards

Kiali will display additional tabs for each applicable Built-In Dashboard or Custom Dashboard.

Built-in dashboards

Kiali comes with built-in dashboards for several runtimes, including Envoy, Go, Node.js, and others.

Envoy

The most important built-in dashboard is for Envoy. Kiali offers the Envoy tab for many workloads. The Envoy tab is actually a Built-In Dashboard, but it is very common as it applies to any workload injected with, or that is itself, an Envoy proxy. Being able to inspect the Envoy proxy is invaluable when troublshooting your mesh. The Envoy tab itself offers five subtabs, exposing a wealth of information.

Detail Envoy

Istio’s Envoy sidecars supply some internal metrics, that can be viewed in Kiali. They are different than the metrics reported by Istio Telemetry, which Kiali uses extensively. Some of Envoy’s metrics may be redundant.

Note that the enabled Envoy metrics can be tuned, as explained in the Istio documentation: it’s possible to get more metrics using the statsInclusionPrefixes annotation. Make sure you include cluster_manager and listener_manager as they are required.

For example, sidecar.istio.io/statsInclusionPrefixes: cluster_manager,listener_manager,listener will add listener metrics for more inbound traffic information. You can then customize the Envoy dashboard of Kiali according to the collected metrics.

Go

Contains metrics such as the number of threads, goroutines, and heap usage. The expected metrics are provided by the Prometheus Go client.

Example to expose built-in Go metrics:

        http.Handle("/metrics", promhttp.Handler())
        http.ListenAndServe(":2112", nil)

As an example and for self-monitoring purpose Kiali itself exposes Go metrics.

The pod annotation for Kiali is: kiali.io/dashboards: go

Node.js

Contains metrics such as active handles, event loop lag, and heap usage. The expected metrics are provided by prom-client.

Example of Node.js metrics for Prometheus:

const client = require('prom-client');
client.collectDefaultMetrics();
// ...
app.get('/metrics', (request, response) => {
  response.set('Content-Type', client.register.contentType);
  response.send(client.register.metrics());
});

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/ratings

The pod annotation for Kiali is: kiali.io/dashboards: nodejs

Quarkus

Contains JVM-related, GC usage metrics. The expected metrics can be provided by SmallRye Metrics, a MicroProfile Metrics implementation. Example with maven:

    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-metrics</artifactId>
    </dependency>

The pod annotation for Kiali is: kiali.io/dashboards: quarkus

Spring Boot

Three dashboards are provided: one for JVM memory / threads, another for JVM buffer pools and the last one for Tomcat metrics. The expected metrics come from Spring Boot Actuator for Prometheus. Example with maven:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-core</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/details

The pod annotation for Kiali with the full list of dashboards is: kiali.io/dashboards: springboot-jvm,springboot-jvm-pool,springboot-tomcat

By default, the metrics are exposed on path /actuator/prometheus, so it must be specified in the corresponding annotation: prometheus.io/path: "/actuator/prometheus"

Thorntail

Contains mostly JVM-related metrics such as loaded classes count, memory usage, etc. The expected metrics are provided by the MicroProfile Metrics module. Example with maven:

    <dependency>
      <groupId>io.thorntail</groupId>
      <artifactId>microprofile-metrics</artifactId>
    </dependency>

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/productpage

The pod annotation for Kiali is: kiali.io/dashboards: thorntail

Vert.x

Several dashboards are provided, related to different components in Vert.x: HTTP client/server metrics, Net client/server metrics, Pools usage, Eventbus metrics and JVM. The expected metrics are provided by the vertx-micrometer-metrics module. Example with maven:

    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-micrometer-metrics</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Init example of Vert.x metrics, starting a dedicated server (other options are possible):

      VertxOptions opts = new VertxOptions().setMetricsOptions(new MicrometerMetricsOptions()
          .setPrometheusOptions(new VertxPrometheusOptions()
              .setStartEmbeddedServer(true)
              .setEmbeddedServerOptions(new HttpServerOptions().setPort(9090))
              .setPublishQuantiles(true)
              .setEnabled(true))
          .setEnabled(true));

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/reviews

The pod annotation for Kiali with the full list of dashboards is: kiali.io/dashboards: vertx-client,vertx-server,vertx-eventbus,vertx-pool,vertx-jvm

Custom Dashboards

When the built-in dashboards don’t offer what you need, it’s possible to create your own. See Custom Dashboard Configuration for more information.

4 - Tracing

How Kiali integrates Distributed Tracing with Jaeger.

Kiali offers a native integration with Jaeger for Distributed Tracing. As such, users can access Jaeger’s trace visualizations. But more than that, Kiali incorporates tracing into several correlated views, making your investment in trace data even more valuable.

For a quick glimpse at Kiali tracing features, see below. For a detailed explanation of tracing in Kiali, see this 3-part Trace my mesh blog series,

Workload detail

When investigating a workload, click the Traces tab to visualize your traces in a chart. When selecting a trace Kiali presents a tab for trace detail, and a tab for span details. Kiali always tries to surface problem areas, Kiali uses a heatmap approach to help the user identify problem traces or spans.

Trace detail Span detail

Metric Correlation

Kiali offers span overlays on Metric charts. The user can simply enable the spans option to generate the overlay. Clicking any span will navigate back to the Traces tab, focused on the trace of interest.

Metrics with Tracing

Graph Correlation

Kiali users often use the Graph Feature to visualize their mesh traffic. In the side panel, When selecting a graph node, the user will be presented with a Traces tab, which lists available traces for the time period. When selecting a trace the graph will display an overlay for the trace’s spans. And the side panel will display span details and offer links back to the trace detail views.

Graph with Tracing

Logs Correlation

Kiali works to correlate the standard pillars of observability: traces, metrics and logs. Kiali can present a unified view of log and trace information, in this way letting users use logs to identify traces of interest. When enabling the spans option Kiali adds trace entries to the workload logs view. Below, in time-sorted order, the user is presented with a unified view of application logs (in white), Envoy proxy logs (in gold), and trace spaces (in blue). Clicking a span of interest brings you to the detail view for the trace of interest.

Logs with Tracing

5 - Validations

Validations Overview

Kiali performs a set of validations to the most common Istio Objects such as Destination Rules, Service Entries, and Virtual Services. Those validations are done in addition to the existing ones performed by Istio’s Galley component. Most validations are done inside a single namespace only, any exceptions, such as gateways, are properly documented.

Galley validations are mostly syntactic validations based on the object syntax analysis of Istio objects while Kiali validations are mostly semantic validations between different Istio objects. Kiali validations are based on the runtime status of your service mesh, Galley validations are static ones and doesn’t take into account what is configured in the mesh.

Istio Config Validation

Check the complete list of validations for further information.

AuthorizationPolicy

KIA0101 - Namespace not found for this rule

AuthorizationPolicy enables access control on workloads. Each policy effects only to a group of request. For instance, all requests started from a workload on a list of namespaces. The present validation points out those rules referencing a namespace that don’t exist in the cluster.

Resolution

Either remove the namespace from the list, correct if there is any typo or create a new namespace.

Severity

Warning

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
         - non-existing # warning
         - unexisting # warning
   to:
   - operation:
       methods: ["GET"]
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0102 - Only HTTP methods and fully-qualified gRPC names are allowed

An AuthorizationPolicy has an Operation field where is defined the oprations allowed for a request. In the method field are listed all the allowed methods that request can have. This validation appears when a problem is found in there. The only methods accepted are: either HTTP valid methods or fully-qualified names of gRPC service in the form of “/package.service/method”

Resolution

Either change or remove the violating method. It has to be either a HTTP valid method or a fully-qualified names of a gRPC service in the form of “/package.service/method”

Severity

Warning

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       methods:
         - "GET"
         - "/package.service/method"
         - "WRONG" # Warning
         - "non-fully-qualified-grpc" # Warning
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0104 - This host has no matching entry in the service registry

AuthorizationPolicy enables access control on workloads. Each policy effects only to a group of request going to a specific destination. For instance, allow all the request going to details host.

The present validation points out those rules referencing a host that don’t exist in the authorization policy namespace. Kiali considers services and service entries. Those hosts that refers to hosts outside of the object namespace will be presented with an unknow error.

Resolution

Either remove the host from the list, correct if there is any typo or deploy a new service or service entry.

Severity

Error

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       hosts:
         - wrong # Error
         - ratings
         - details.default
         - reviews.default.svc.cluster.local
         - productpage.outside # Unknown
         - google.com # Service Entry present. No error
         - google.org # Service Entry not present, wrong domain. Error.
       methods:
         - "GET"
         - "/package.service/method"
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

Destination rules

KIA0201 - More than one DestinationRules for the same host subset combination

This validation warning could be a result of duplicate definition of the same subsets as well as from rules that apply to all subsets. Also, a combination of one Destination Rule (DR) applying to all subsets and another defining behavior for only some subsets triggers this validation warning.

Istio silently ignores the duplicate subsets and merge these destination rules without letting the user know. Only the first seen rule (by Istio) per subset is used and information from multiple definitions is not merged. While the routing might work correctly, this is most likely a configuration error. It may lead to a undesired behavior if one of the offending rules is removed or modified and that is probably not the intention of the deployer of this service. Also, if the two offending destination rules have different policies for traffic routing the wrong one might be used.

Resolution

Either merge the settings to a single DR or split the subsets in such a way that they do not interleave. This ensures that the routing behavior stays consistent.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews-dr1
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews-dr2
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1

KIA0202 - This host has no matching entry in the service registry (service, workload or service entries)

Istio applies traffic rules for services after the routing has happened. These can include different settings such as connection pooling, circuit breakers, load balancing, and detection. Istio can define the same rules for all services under a host or different rules for different versions of the service. The host must a service that is defined in the platform’s service registry or as a ServiceEntry. Short names are extended to include ‘.namespace.cluster’ using the namespace of the destination rule, not the service itself. FQDN is evaluated as is. It is recommended to use the FQDN to prevent any confusion.

If the host is not found, Istio ignores the defined rules.

Resolution

Correct the host to point to a correct service, in this namespace or with FQDN to other namespaces, or deploy the missing service to the mesh.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: notpresent
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

KIA0203 - This subset’s labels are not found in any matching host

Subsets can override the global settings defined in the DR for a host.

If the host is not found, Istio ignores the defined rules.

If the not found subset is not referenced in any Virtual Service, the severity of this error is changed to Info.

Resolution

Correct the host to point to a correct service, in this namespace or with FQDN to other namespaces, or deploy the missing service to the mesh. If the hostname is equal to the one used otherwise in the DR, consider removing the duplicate host resolution.

Also, verify that the labels are correctly matching a workload with the intended service.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v10
  - name: v2
    labels:
      notfoundlabel: v2
  - name: v3
    labels:
      version: v3

KIA0204 - mTLS settings of a non-local Destination Rule are overridden

Istio allows you to define DestinationRule at three different levels: mesh, namespace and service level. A mesh may have multiple DRs. In case of having two DestinationRules on the first one is at a lower level than the second one, the first one overrides the TLS values of the second one.

This validation appears only when autoMtls is disabled.

Resolution

This validation aims to warn Kiali users that they may be disabling/enabling mTLS from the higher DestinationRule. Merging the TLS settings to one of the DestinationRules is the only way to fix this validation. However, this is a valid scenario so it might be impossible to remove this warning.

Severity

Warning

Example

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

KIA0205 - PeerAuthentication enabling mTLS at mesh level is missing

Istio has the ability to define mTLS communications at mesh level. In order to do that, Istio needs one DestinationRule and one PeerAuthentication. The DestinationRule configures all the clients of the mesh to use mTLS protocol on their connections. The PeerAuthentication defines what authentication methods that can be accepted on the workload of the whole mesh. If the PeerAuthentication is not found or doesn’t exist and the mesh-wide DestinationRule is on ISTIO_MUTUAL mode, all the communication returns 500 errors.

This validation appears only when autoMtls is disabled.

Resolution

Add a PeerAuthentication within the istio-system namespace without specifying targets but setting peers mtls mode to STRICT or PERMISSIVE. The PeerAuthentication should be like this.

Severity

Error

Example

# AutoMtls disabled, no PeerAuthentication at mesh-level defined
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

KIA0206 - PeerAuthentication enabling namespace-wide mTLS is missing

Istio has the ability to define mTLS communications at namespace level. In order to do that, Istio needs both a DestinationRule and a PeerAuthentication targeting all the clients/workloads of the specific namespace. The PeerAuthentication allows mTLS authentication method for all the workloads within a namespace. The DestinationRule defines all the clients within the namespace to start communications in mTLS mode. If the PeerAuthentication is not found and the DestinationRule is on STRICT mode in that namespace but there is the DestinationRule enabling mTLS, all the communications within that namespace returns 500 errors.

KIA0301 - More than one Gateway for the same host port combination

Gateway creates a proxy that forwards the inbound traffic for the exposed ports. If two different gateways expose the same ports for the same host, this creates ambiguity inside Istio as either of these gateways could handle the traffic. This is most likely a configuration error. This check is done across all namespaces the user has access to.

There is one exception: when both gateways points to a different ingress. Then the ambiguity doesn’t exist and, in consequence, no validation is shown. Kiali considers that two gateways points to the same ingress if they share the exact same selector.

Resolution

Remove the duplicate gateway entries or merge the two gateway definitions into a single one.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway # Validation shown
  namespace: bookinfo
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway-copy # Validation shown
  namespace: bookinfo2
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway-diff-ingress # No validations shown
  namespace: bookinfo
spec:
  selector:
    istio: ingressgateway-pub # Using different ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---

KIA0302 - No matching workload found for gateway selector in this namespace

This validation checks the current namespace for matching workloads as this is recommended, and potentially in the future required, by the Istio. Excluded from this check are the default “istio-ingressgateway” and “istio-egressgateway” workloads which are included in Istio by default.

Although your traffic might be correctly routed to a workload in other namespace, this is not a guaranteed behavior and thus a warning is flagged in such cases also.

Resolution

Deploy the missing workload or fix the selector to target a correct location.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  selector:
    app: nonexisting # workload doesn't exist in the namespace
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

Mesh Policies

KIA0401 - Mesh-wide Destination Rule enabling mTLS is missing

Istio has the ability to define mTLS communications at mesh level. In order to do that, Istio needs one DestinationRule and one PeerAuthentication. The DestinationRule configures all the clients of the mesh to use mTLS protocol on their connections. The PeerAuthentication defines what authentication methods can be accepted on the workload of the whole mesh. If the DestinationRule is not found or doesn’t exist and the PeerAuthentication is on STRICT mode, all the communication returns 500 errors.

This validation appears only when autoMtls is disabled.

Resolution

Add a DestinationRule with “*.cluster” host and ISTIO_MUTUAL as tls trafficPolicy mode. The DestinationRule should be like this.

Severity

Error

Example

# Make sure there isn't any DestinationRule enabling meshwide mTLS
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: DISABLE
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "enable-mtls"
  namespace: bookinfo
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Ports

KIA0601 - Port name must follow [-suffix] form

Istio requires the service ports to follow the naming form of ‘protocol-suffix’ where the ‘-suffix’ part is optional. If the naming does not match this form (or is undefined), Istio treats all the traffic TCP instead of the defined protocol in the definition. Dash is a required character between protocol and suffix. For example, ‘http2foo’ is not valid, while ‘http2-foo’ is (for http2 protocol).

Resolution

Rename the service port name field to follow the form and the traffic flows correctly.

Severity

Error

Example

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: bookinfo
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 9080
    name: wrong-http
  selector:
    app: ratings-java
    version: v1

Services

KIA0701 - Deployment exposing same port as Service not found

Service definition has a combination of labels and port definitions that are not matching to any workloads. This means the deployment will be unsuccessful and no traffic can flow between these two resources. The port is read from the Service ‘TargetPort’ definition first and if undefined, the ‘Port’ field is used as Kubernetes defaults the ‘TargetPort’ to ‘Port’. If the ‘TargetPort’ is using a integer, the port numbers are compared and if the ‘TargetPort’ is a string, the deployment’s portName is used for comparison.

Resolution

Fix the port definitions in the workload or in the service definition to ensure they match.

Severity

Warning

Example

Invalid example with port definitions unmatched:

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: ratings-java
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings-java
    version: v1
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ratings-java
  namespace: ratings-java
  labels:
    app: ratings-java
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      annotations:
         sidecar.istio.io/inject: "true"
      labels:
        app: ratings-java
        version: v1
    spec:
      containers:
      - name: ratings-java
        image: pilhuhn/ratings-java:f
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

Valid example using targetPort definition matching:

apiVersion: v1
kind: Service
metadata:
  name: ratings-java-svc
  namespace: ratings-java
  labels:
    app: ratings
    service: ratings-svc
spec:
  ports:
  - port: 9080
    targetPort: 8080
    name: http
  selector:
    app: ratings-java
    version: v1
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ratings-java
  namespace: ratings-java
  labels:
    app: ratings-java
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      annotations:
         sidecar.istio.io/inject: "true"
      labels:
        app: ratings-java
        version: v1
    spec:
      containers:
      - name: ratings-java
        image: pilhuhn/ratings-java:f
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

ServiceMesh Policies

KIA0801 - Mesh-wide Destination Rule enabling mTLS is missing

Maistra has the ability to define mTLS communications at mesh level. In order to do that, Maistra needs one DestinationRule and one ServiceMeshPolicy. The DestinationRule configures all the clients of the mesh to use mTLS protocol on their connections. The ServiceMeshPolicy defines what authentication methods can be accepted on the workload of the whole mesh. If the DestinationRule is not found or doesn’t exist and the ServiceMeshPolicy is on STRICT mode, all the communication returns 500 errors.

Resolution

Add a DestinationRule named as default with “*.cluster” host and ISTIO_MUTUAL as tls trafficPolicy mode. The DestinationRule should be like this.

Severity

Error

Example

apiVersion: "maistra.io/v1"
kind: "ServiceMeshPolicy"
metadata:
  name: default
  namespace: control-plane-ns
spec:
  peers:
  - mtls: {}

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: details-reviews-viewer
  namespace: default
spec:
  rules:
  - services: ["details.bookinfo.svc.cluster.local", "reviews.bookinfo.svc.cluster.local"]
    methods: ["GET"]
    constraints:
      - key: "destination.labels[version]"
        values: ["v1"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: bind-details-reviews
  namespace: bookinfo
spec:
  subjects:
  - user: "cluster.local/ns/bookinfo/sa/bookinfo-productpage"
  roleRef:
    kind: ServiceRole
    name: "details-reviews-viewer"

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: default
  namespace: istio-system
spec:
  workloadSelector: # Default sidecar can't have labels
    labels:
      version: v1
  egress:
  - port:
      number: 3306
      protocol: MYSQL
      name: egressmysql
    captureMode: NONE
    bind: 127.0.0.1
    hosts:
    - "bookinfo/reviews.bookinfo.svc.cluster.local"
    - "bookinfo/details.bookinfo.svc.cluster.local"

VirtualServices

KIA1101 - DestinationWeight on route doesn’t have a valid service (host not found)

VirtualService routes matching requests to a service inside your mesh. Routing can also match a subset of traffic to a certain version of it for example. Any service inside the mesh must be targeted by its name, the IP address are only allowed for hosts defined through a Gateway. Host must be in a short name or FQDN format. Short name will evaluate to VS' namespace, regardless of where the actual service might be placed.

If the host is not found, Istio ignores the defined rules. However, if a subset with a Destination Rule is not found it affects all the subsets and all the routings. As such, care must be taken that the Destination rule is available before deploying the Virtual Service.

Resolution

Correct the host to point to a correct service (in this namespace or with FQDN to other namespaces), deploy the missing service to the mesh or remove the configuration linking to that non-existing service.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: details
spec:
  hosts:
  - details
  http:
  - route:
    - destination:
        host: nonexistentsvc
        subset: v2

KIA1102 - VirtualService is pointing to a non-existent gateway

By default, VirtualService routes apply to sidecars inside the mesh. The gateway field allows to override that default and if anything is defined, the VS applies to those selected. ‘mesh’ is a reserved gateway name and means all the sidecars in the mesh. If one wishes to apply the VS to gateways as well as the sidecars, the ‘mesh’ keyword must be used as one of the gateways. Incorrect gateways mean that the VS is not applied correctly.

Resolution

Fix the possible gateway field to target all necessary gateways or remove the field if the default ‘mesh’ is enough.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: details
spec:
  hosts:
  - details
  gateways:
  - non-existent-gateway
  http:
  - route:
    - destination:
        host: details
        subset: v1

KIA1103 - VirtualService doesn’t define any route protocol

VirtualService is a defined set of rules for routing certain type of traffic to target destinations with rules. At least one, ‘tcp’, ‘http’ or ‘tls’ must be defined.

Resolution

This appears to be a configuration error. Fix the definition.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: details
spec:
  hosts:
  - details

KIA1104 - The weight is assumed to be 100 because there is only one route destination

Istio assumes the weight to be 100 when there is only one HTTPRouteDestination or RouteDestination. The warning is present because there is one route with a weight less than 100.

Resolution

Either remove the weight field or you might want to add another RouteDestination with an specific weight.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 10

KIA1105 - This subset is already referenced in another route destination

Istio allows you to apply rules over the traffic targetting to a specific service. In order to achieve that, it is necessary to add those rules into either http, tcp or tls fields in a VirtualService. In each field it is possible to specify rules for redirection or forwarding traffic. Those rules are the RouteDestination and HTTPRouteDestination structs. Each structs defines where the traffic is shifted to using the subset field. This warning message refers to the fact of referencing one subset more than one time within the same route. Galley, Istio module in charge of configuration validation, allows the subset duplicity. However, the mesh it might become broken when there are different duplicates. Also, the presented warning might help spoting a typo.

Resolution

Make sure there is only one reference to the same subset for each RouteDestination. Either HTTPRouteDestination or RouteDestination.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v2 # duplicate
      weight: 10
    - destination:
        host: reviews
        subset: v2 # duplicate
      weight: 10

KIA1106 - More than one Virtual Service for same host

A VirtualService defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for traffic of a specific protocol. If the traffic is matched, then it is sent to a named destination service (or subset/version of it) defined in the registry.

Resolution

This is a valid configuration only if two VirtualServices share the same host but are bound to a different gateways, sidecars do not accept this behavior. There are several caveats when using this method and defining the same parts in multiple Virtual Service definitions is not recommended. While Istio will merge the configuration, it does not guarantee any ordering for cross-resource merging and only the first seen configuration is applied (rest ignored). As recommended, each VS definition should have a ‘catch-all’ situation, but this can only be defined in a definition affecting the same host.

Severity

Warning

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews-cp
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

KIA1107 - Subset not found

VirtualService routes matching requests to a service inside your mesh. Routing can also match a subset of traffic to a certain version of it for example. The subsets referred in a VirtualService have to be defined in one DestinationRule.

If one route in the VirtualService points to a subset that doesn’t exist Istio won’t be able to send traffic to a service.

Resolution

Fix the routes that points to a non existing subsets. It might be fixing a typo in the subset’s name or defining the missing subset in a DestinationRule.

Severity

Error

Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: nosubset
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

KIA1108 - Preferred nomenclature: /

A virtual service may include a list of gateways which the defined routes should be applied to. Gateways in other namespaces may be referred to by /; specifying a gateway with no namespace qualifier is the same as specifying the VirtualService’s namespace.

Resolution

Move the nomenclature of the gateways into the supported Istio form: /

Example

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: bookinfo
  namespace: bookinfo
spec:
  hosts:
    - '*'
  gateways:
    - bookinfo-gateway.bookinfo.svc.cluster.local # unsupported format
    - bookinfo/bookinfo-gateway # works
  http:
    - match:
        - uri:
            exact: /productpage
        - uri:
            prefix: /static
        - uri:
            exact: /login
        - uri:
            exact: /logout
        - uri:
            prefix: /api/v1/products
      route:
        - destination:
            host: productpage
            port:
              number: 9080
---
kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
  selector:
    istio: ingressgateway

Generic

KIA0001 - Unable to verify the validity, cross-namespace validation is not supported for this field

In certain cases, Kiali is unable to validate the field since it spans another namespace to which the validator is not capable of looking at. In such cases, Kiali will mark this field with a grey icon indicating that the fields correctness could not be verified. This does not necessarily mean there is an error, but that the user should be careful and do the validation manually.

KIA0002 - More than one selector-less object in the same namespace

This validation refers to the usage of the selector. Selector-less Istio objects are those objects that don’t have the selector field specified. Therefore, objects that apply to all the workloads of a namespace (or whole mesh if the namespace is the same as the control plane namespace).

This validation warns you that you have two different objects living in the same namespace. This may leave an non-deterministic or unexpected behavior on the workloads of the namespace.

Resolution

The natural solution is to merge both objects. In case there are different behaviors you want to apply, consider to define the selector field targeting a specific set of workloads.

Severity

Error

Example

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "bookinfo"
spec:
  mtls:
    mode: STRICT
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "duplicate"
  namespace: "bookinfo"
spec:
  mtls:
    mode: STRICT

KIA0003 - More than one object applied to the same workload

This validation refers to the usage of the selector. In this field are defined the labels of the workloads that this object will be applied to. It might be one or more workloads in the same namespace.

This validation warns the scenario where there are two different objects applying to the same workload(s). This may leave an undeterministic or unexpected behavior on the workloads of the namespace.

Resolution

There isn’t a standard solution for that. It is a good practice not to have multiple rules of the same kind applying to the same workloads. Otherwise you would end up having interferences between objects and having troubles when debugging. The first approach would be to merge both objects into one if possible. The second approach would be to reorganize the objects of the same kind in a way that each one only applies to a different set of workloads. Applying no change into the objects is also an option although not desiderable.

Severity

Error

Example

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "productpage"
  namespace: "bookinfo"
spec:
  selector:
    matchLabels:
      app: productpage
  mtls:
    mode: STRICT
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "productpage-disable-80"
  namespace: "bookinfo"
spec:
  selector:
    matchLabels:
      app: productpage
      version: v1
  mtls:
    mode: STRICT
  portLevelMtls:
    80:
      mode: DISABLE

KIA0004 - No matching workload found for the selector in this namespace

This validation warns the scenario where there are not workloads matching with the selector labels. In other terms, this object doesn’t have any implication into the mesh.

Resolution

There are three scenarios: either change the labels to match an existing workload (useful with typos), deploy a workload that match with those labels or safely remove this object.

Severity

Warning

Example

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "nomatchingworkloads"
  namespace: "bookinfo"
spec:
  selector:
    matchLabels:
      app: wrong-typo
      version: v1
  mtls:
    mode: STRICT
  portLevelMtls:
    80:
      mode: DISABLE

6 - Wizards

Kiali is more than observability, it also helps you to configure, update and validate your Istio service mesh.

Istio Wizards

Kiali provides actions to create, update and delete Istio configuration, driven by wizards.

Actions can be applied to a Service

Service Details Actions

Actions can also be applied to a Workload

Workload Details Actions

Also, actions are available for an entire Namespace

Namespace Actions

Service Actions

Traffic Management: Request Routing

The Request Routing Wizard allows creating multiple routing rules.

Every rule is composed of a Request Matching and a Routes To section.
The Request Matching section can add multiple filters using HEADERS, URI, SCHEME, METHOD or AUTHORITY Http parameters.
The Request Matching section can be empty, in this case any http request received is matched against this rule.
The Routes To section can specify the percentage of traffic that is routed to a specific workload.

Request Routing

Istio applies routing rules in order, meaning that the first rule matching an HTTP request performs the routing. The Matching Routing Wizard allows changing the rule order.

Traffic Management: Fault Injection

The Fault Injection Wizard allows injecting faults to test the resiliency of a Service.

HTTP Delay specification is used to inject latency into the request forwarding path.
HTTP Abort specification is used to prematurely abort a request with a pre-specified error code.

Fault Injection

Traffic Management: Traffic Shifting

The Traffic Shifting Wizard allows selecting the percentage of traffic that is routed to a specific workload.

Traffic Shifting

Traffic Management: Request Timeouts

The Request Timeouts Wizard sets up request timeouts in Envoy, using Istio.

HTTP Timeout defines the timeout for a request.
HTTP Retry describes the retry policy to use when an HTTP request fails.

Request Timeouts

Traffic Management: Gateways

Traffic Management Wizards have an Advanced Options section that can be used to extend the scenario.

One available Advanced Option is to expose a Service to external traffic through an existing Gateway or to create a new Gateway for this Service.

Request Timeouts

Traffic Management: Circuit Breaker

Traffic Management Wizards allows defining Circuit Breakers on Services as part of the available Advanced Options.

Connection Pool defines the connection limits for an upstream host.
Outlier Detection implements the Circuit Breaker based on the consecutive errors reported.

Circuit Breakers

Security: Traffic Policy

Traffic Management Advanced Options allows defining Security and Load Balancing settings.

TLS related settings for connections to the upstream service.
Automatically generate a PeerAuthentication resource for this Service.
Load balancing policies to apply for a specific destination.

Traffic Policy

Workload Actions

Automatic Sidecar Injection

A Workload can be individually annotated to control the Sidecar Injection.

A default scenario is to indicate this at Namespace level but there can be cases where a Workload shouldn’t be part of the Mesh or vice versa.

Kiali allows users to annotate the Deployment template and propagate this configuration into the Pods.

Automatic Sidecar Injection

Namespace Actions

The Kiali Overview page offers several Namespace actions, in any of its views: Expanded, Compacted or Table.

Namespace Actions

Show

Show actions navigate from a Namespace to its specific Graph, Applications, Workloads, Services or Istio Config pages.

Automatic Sidecar Injection

When Automatic Sidecar Injection is enabled in the cluster, a Namespace can be labeled to enable/disable the injection webhook, controlling whether new deployments will automatically have a sidecar.

Canary Istio upgrade

When Istio Canary revision is installed, a Namespace can be labeled to that canary revision, so the sidecar of canary revision will be injected into workloads of the namespace.

Security: Traffic Policies

Kiali can generate Traffic Policies based on the traffic for a namespace.

For example, at some point a namespace presents a traffic graph like this:

Traffic Policies: Graph

And a user may want to add Traffic Policies to secure that communication. In other words, to prevent traffic other than that currently reflected in the Graph’s Services and Workloads.

Using the Create Traffic Policies action on a namespace, Kiali will generate AuthorizationPolicy resources per every Workload in the Namespace.

Traffic Policies: Graph

7 - Istio Configuration

Kiali is more than observability, it also helps you to configure, update and validate your Istio service mesh.

The Istio configuration view provides advanced filtering and navigation for Istio configuration objects such as Virtual Services and Gateways. Kiali provides inline config edition and powerful semantic validation for Istio resources.

Istio Config List

Validations

Check the complete list of validations for further information.

Istio Config Validation

Istio Forms

Istio Wizards provide a way to apply a Service Mesh pattern and let Kiali to generate the Istio Configuration. Kiali also offers actions to create Istio Config for Gateways and Security scenarios.

These actions are located under the Istio Config page.

Create Istio Config

Istio Security Forms

Kiali allows creation of Istio AuthorizationPolicy resources:

AuthorizationPolicy

Istio PeerAuthentication resources:

PeerAuthentication

Istio RequestAuthentication resources:

RequestAuthentication

Istio Traffic Forms

Kiali uses Istio Wizards to generate Istio Traffic config for a specific Service, but Kiali also allows creation of Gateway, ServiceEntry and Sidecar Istio resources for more generic scenarios.

Istio Gateway resources:

Gateway

Istio ServiceEntry resources:

ServiceEntry

Istio Sidecar resources:

Sidecar

8 - Security

Kiali gives support to better understand how mTLS is used in Istio meshes. Find those helpers in the graph, the masthead menu, the overview page and specific validations.

At the right side of the Masthead, Kiali shows a lock when the mesh has strictly enabled mTLS for the whole service mesh. It means that all the communications in the mesh uses mTLS.

mTLS mesh-wide enabled strictly Custom Vertx Metrics

Kiali shows a hollow lock when either the mesh is configured in PERMISSIVE mode or there is a misconfiguration in the mesh-wide mTLS configuration.

Overview locks

The overview page shows all the available namespaces with aggregated data. Besides the health and validations, Kiali shows also the mTLS status at namespace-wide. Similar to the masthead, it shows a lock when strict mTLS is enabled or a hollow lock when permissive.

Overview page: showing mTLS at namespace-wide

Graph

The mTLS method is used to establish communication between microservices. In the graph, Kiali has the option to show which edges are using mTLS and with what percentatge during the selected period. When an edge shows a lock icon it means at least one request with mTLS enabled is present. In case there are both mTLS and non-mTLS requests, the side-panel will show the percentage of requests using mTLS.

Enable the option in the Display dropdown, select the security badge.

Graph shows edges which uses mTLS

Validations

Kiali has different validations to help troubleshoot configurations related to mTLS such as DestinationRules and PeerAuthentications.

Validation supporting mTLS configuration

9 - Istio Status

Istio Component Status

The Istio service mesh architecture is comprised of several components, from istiod to Jaeger. Each component must work as expected for the mesh to work well overall. Kiali regularly checks the status of each Istio component to ensure the mesh is healthy.

Istio components status: components not healthy or found

A component status will be one of: Not found, Unreachable, Not healthy and Healthy. The Not found status means that Kiali is not able to find the deployment. The Unreachable status means that Kiali hasn’t been succesfuly able to communicate with the component (Prometheus, Grafana and Jaeger). The Not healthy status means that the deployment doesn’t have the desired amount of replicas running. The Healthy status is when the component is not in the previous ones, plus, healthy components won’t be shown in the list.

Regarding the severity of each component, there are only to options: core or add-on. The core components are those shown as errors (in red) whereas the add-ons are displayed as warnings (in orange).

By default, Kiali checks the following components installed in the control plane namespace: istiod, ingress, egress; and prometheus, grafana and jaeger accessible thought their services.

Certificates Information Indicators

In some situations, it would be useful to get information about the certificates used by internal mTLS, for example:

Know whether the default CA is used or if there is another CA configured
Check the certificates issuer and their validity timestamps to troubleshoot any issue with certificates

The certificates shown depends on how Istio is configured. The following cases are possible:

Using Istio CA certificates (default), the information shown is from a secret named istio-ca-secret
Using https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/[Plug in CA certificates, window=_blank], the information shown is from a secret named cacerts
Using https://istio.io/latest/docs/tasks/security/cert-management/dns-cert/[DNS certificates, window=_blank], the information shown is from reading many secrets found in Istio configuration

The following is an example of viewing the default case:

Certificates information

Note that displaying this configuration requires permissions to read secrets (istio-ca-secret by default, possibly cacerts or any secret configured when using DNS certificates).

Having these permissions may concern users. For this reason, this feature is implemented as a feature flag and not only can be disabled, avoiding any extra permissions to read secrets, but also a list of secrets can be configured to explicitly grant read permissions for some secrets in the control plane namespace. By default, this feature is enabled with a Kiali CR configuration equivalent to the following:

spec:
  kiali_feature_flags:
    certificates_information_indicators:
      enabled: true
      secrets:
      - cacerts
      - istio-ca-secret

You can extend this default configuration with additional secrets, remove secrets you don’t want, or disable the feature.

If you add additional secrets, the Kiali operator also needs the same privileges in order to configure Kiali successfully. If you used the Helm Charts to install the operator, specify the secretReader value with the required secrets:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    --set "secretReader={cacerts,istio-ca-secret}"
    kiali-operator \
    kiali/kiali-operator

If you installed the operator via the OperatorHub you need to update the operator privileges as a post-installation step, as follows:

$ kubectl patch $(kubectl get clusterroles -o name | grep kiali-operator) --type "json" -p '[{"op":"add","path":"/rules/0","value":{"apiGroups":[""],"resources":["secrets"],"verbs":["get"],"resourceNames":["secret-name-to-be-read"]}}]'

Replace secret-name-to-be-read with the secret name you want the operator to read and restart the Kiali operator pod after running the previous command.

10 - Advanced Mesh Deployment and Multi-cluster support

A basic Istio mesh deployment has a single control plane with a single data plane, deployed on a single Kubernetes cluster. But Istio supports a variety of advanced deployment models. It allows a mesh to span multiple primary (control plane) and/or remote (data plane only) clusters, and can use a single or multi-network approach. The only strict rule is that within a mesh service names are unique. A non-basic mesh deployment generally involves multiple clusters. See installation instructions for more detail on installing advanced mesh deployments.

Kiali v1.29 introduces experimental support for advanced deployment models. The recommended Kiali deployment model is to deploy an instance of Kiali along with each Istio primary control plane. Each Kiali instance will work as it has in the past, with the configured Kubernetes, Prometheus and Jaeger instances. It will concern itself with the istio config managed by the local primary. But, there are two new additions:

List View: Mesh Discovery

Kiali will attempt to discover the clusters configured in the mesh. And it will identify the home cluster, meaning the cluster on which it is installed and from which it presents its traffic, traces and Istio config. In the following example there are two clusters defined in the mesh, Kukulcan and Tzotz. Kukulcan is identified as the home cluster in three places: the browser tab (not shown), the masthead, and with a star icon in the clusters list:

Mesh list view

Graph View: Cluster and Namespace Boxing

Starting in v1.8 Istio provides cluster names in the traffic telemetry for multi-cluster installations. The Kiali graph can now use this information to better visualize clusters and namespaces. The Display menu now offers two new options: Cluster Boxes and Namespace Boxes. When enabled, either separately or together, the graph will generate boxes to help more easily identify the relevant nodes and edges, and to see traffic traveling between them.

Each new box type supports selection and will provide a side-panel summary of traffic. Below we see a Bookinfo traffic graph for when Bookinfo services are deployed across the Kukulcan and Tzotz clusters. The Kukulcan cluster box is selected. Because Kukulcan is the Kiali home cluster, you see traffic from the perspective of Kukulcan. Note that you do not see the internal traffic on Tzotz (the requests from Reviews to Ratings service). To see traffic from the Tzotz cluster point of view, you would open a Kiali session on Tzotz, assuming you have privileges.

Multi-cluster traffic graph

Features

1 - Topology

Overview

Graph

Health

Side-Panel

Node Detail

Traffic Animation

Graph Types

Replay

Operation Nodes

2 - Health

Health

Health Configuration

Health Indicators

Icons and colors

Default Values

Request Traffic

Configuration

Examples

3 - Detail Views

Overview Tab

Traffic

Logs

Metrics

Traces

Dashboards

Built-in dashboards

Envoy

Go

Node.js

Quarkus

Spring Boot

Thorntail

Vert.x

Custom Dashboards

4 - Tracing

Workload detail

Metric Correlation

Graph Correlation

Logs Correlation

5 - Validations

Validations Overview

AuthorizationPolicy

KIA0101 - Namespace not found for this rule

Resolution

Severity

Example

See Also

KIA0102 - Only HTTP methods and fully-qualified gRPC names are allowed

Resolution

Severity

Example

See Also

KIA0104 - This host has no matching entry in the service registry

Resolution

Severity

Example

See Also

Destination rules

KIA0201 - More than one DestinationRules for the same host subset combination

Resolution

Severity

Example

See Also

KIA0202 - This host has no matching entry in the service registry (service, workload or service entries)

Resolution

Severity

Example

See Also

KIA0203 - This subset’s labels are not found in any matching host

Resolution

Severity

Example

See Also

KIA0204 - mTLS settings of a non-local Destination Rule are overridden

Resolution

Severity

Example

See Also