#404 errors occur when multiple gateways configured with same TLS certificate

#Problem Description

#Symptom

When accessing services through Istio Ingress Gateway using HTTP/2 protocol, 404 errors occur. This is a known issue in the Istio community.

#Root Cause

Configuring multiple gateways with the same TLS certificate causes HTTP/2 browsers to generate 404 errors when accessing secondary hosts after initial connection establishment. This occurs due to HTTP/2 connection reuse in browsers.

Example Scenario:

• Domains a.example.com and b.example.com share same TLS certificate
• Configured in separate Gateway resources
• Browser accesses a.example.com then b.example.com via same connection

#Troubleshooting

#Verification Script

Execute this script on the master node of the cluster hosting Istio Ingress Gateway:

#!/bin/bash
nslist=$(kubectl get ns -o jsonpath='{.items[*].metadata.name}')
declare -A cred_map

echo "begin to check gw"
for ns in $nslist; do
  gateways=$(kubectl get gw -n $ns -o jsonpath='{.items[*].metadata.name}')
  for gateway in $gateways; do
    gateway_yaml=$(kubectl get gw -n $ns $gateway -o yaml)
    secname=$(echo "$gateway_yaml" | grep 'credentialName:' | awk '{print $2}')

    if [[ -n "$secname" ]]; then
      if [[ -n "${cred_map[$secname]}" ]]; then
        echo -e "\033[31m Duplicate TLS detected in gateway: ${cred_map[$secname]} \033[0m"
        hosts=$(kubectl get gw -n $ns $gateway -o json | jq -r '.spec.servers[] | .hosts[]')
        echo -e "\033[31m Conflict gateway: $gateway in $ns \033[0m"
        echo "Affected hosts: $hosts"
      else
        cred_map["$secname"]="$gateway~$ns"
      fi
    fi
  done
done

Expected Output:

Duplicate TLS detected in gateway: drawdb-gateway~drawdb
Conflict gateway: authory-gateway in nm-edu-authory
Affected hosts: rzzx-test.jiaxiurc.com

#Solution for Root Cause 1: Merge Gateway Resources

#Considerations

• This is the community-recommended solution
• Maintains HTTP/2 performance benefits
• Requires modification of existing Gateway configurations

#Prerequisites

1. jq v1.7+ installed on cluster nodes
2. Access to cluster with kubectl privileges

#Steps

1. Identify conflicting Gateways using verification script
2. Merge Host configurations:

   apiVersion: networking.istio.io/v1beta1
   kind: Gateway
   metadata:
     name: unified-gateway
     namespace: istio-system
   spec:
     selector:
       istio: ingressgateway
     servers:
     - hosts:
         - "a.example.com"
         - "b.example.com"
       tls:
         mode: SIMPLE
         credentialName: shared-cert
       port:
         name: https
         number: 443
         protocol: HTTPS
3. Update VirtualServices to reference merged Gateway
4. Delete redundant Gateways
5. Verify configuration:

   kubectl get gw -A
   istioctl analyze

#Solution for Root Cause 2: 421 Response Code

#Considerations

• Requires client support for 421 status code
• Compatible with Chrome/Firefox/Safari 15.1+

#Prerequisites

1. Istio version ≥ 1.12
2. Cluster admin privileges

#Steps

1. Apply EnvoyFilter:

   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: misdirected-request
     namespace: istio-system
   spec:
     configPatches:
       - applyTo: HTTP_FILTER
         match:
           context: GATEWAY
           listener:
             filterChain:
               filter:
                 name: envoy.filters.network.http_connection_manager
         patch:
           operation: INSERT_BEFORE
           value:
             name: envoy.lua
             typed_config:
               "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
               inlineCode: |
                 function envoy_on_request(request_handle)
                   local authority = request_handle:headers():get(":authority")
                   local sni = request_handle:streamInfo().requestedServerName
   
                   if sni ~= "" and sni ~= authority:match("([^:]+)") then
                     request_handle:respond({[":status"] = "421"}, "Misdirected Request")
                   end
                 end
2. Verify implementation:

   curl -I -H "Host: b.example.com" https://gateway-ip

#Preventive Measures

1. Certificate Management:
   • Use wildcard certificates (*.example.com)
   • Avoid certificate reuse across environments
2. Gateway Design:
   • Implement single gateway per domain pattern
   • Use namespace-based certificate isolation
3. Regular Audits:

   istioctl experimental precheck
   kubectl get secret --all-namespaces -o json | jq '.items[].metadata.name' | sort | uniq -d

#Related Content

HTTP/2 Connection Reuse Mechanism:

• Single TLS connection handles multiple requests
• Server uses SNI to route requests
• Mismatched SNI headers cause routing failures

Istio Documentation Reference: Istio Common Problems - 404 Errors