Troubleshoot Azure NAT Gateway connection issues (2023)

This article provides guidance on how to troubleshoot common outbound connectivity issues related to NAT gateway resources. This article also provides best practices for designing applications to make efficient use of outbound connections.

SNAT Exhaustion Due to NAT Gateway Configuration

SNAT starvation issues in NAT gateways are usually related to the NAT gateway's configuration:

• NAT gateways do not extend to enough public IP addresses.

• The NAT Gateway's configurable TCP idle timeout timer is set higher than the default of 4 minutes.

NAT gateway does not scale far enough

Each public IP address provides 64,512 SNAT ports for outbound connectivity using a NAT gateway. With available SNAT ports, the NAT Gateway can support up to 50,000 concurrent connections to the same destination endpoint. If outbound connections are dropped due to exhaustion of SNAT ports, the NAT gateway may not scale well enough to handle the workload. NAT gateways may require more public IP addresses to provide more SNAT ports for outbound connections.

The following table describes two common outbound connection failure scenarios due to scalability issues and how to validate and mitigate these issues.

TCP idle timeout timer set higher than default

The NAT Gateway TCP idle timeout timer defaults to 4 minutes, but can be configured up to 120 minutes. If the timer is set to a value higher than the default, the NAT gateway keeps flows longer andAdded pressure on SNAT port inventorycan be applied.

The following table describes scenarios where long TCP idle timeout timers cause SNAT starvation and provides mitigation steps to take.

Connection failure due to idle timeout

TCP idle timeout

from the previous sectionTCP timerYou should use TCP keepalives to refresh the idle flow and reset the idle timeout as described in TCP keepalive only needs to be enabled on one side of the connection to keep the connection alive on both sides. When a TCP keepalive is sent on one side of the connection, the other side automatically sends an ACK packet. The idle timeout timer is reset on both sides of the connection. more detailsSee TCP Idle Timeout.

UDP idle timeout

The UDP idle timeout timer is set to 4 minutes. Unlike the TCP idle timeout timer for NAT gateways, the UDP idle timeout timer is not configurable.

The following table describes common scenarios where connections are aborted due to UDP traffic idle timeout and the steps you should take to mitigate the problem.

NAT Gateway public IP not used for outbound traffic

The VM retains the old SNAT IP as an active connection after the NAT gateway is added to the virtual network.

NAT gatewaybecomes the default route to the Internet when configured on a subnet. When you migrate from Basic Outbound Access or a load balancer to a NAT gateway, a new connection is immediately created using the IP address associated with the NAT gateway resource. During migration, if the virtual machine has an established connection, it will continue to use the old SNAT IP address assigned when the connection was established.

Test and troubleshoot VMs with old SNAT IP addresses in the following ways:

• Make sure that you have established a new connection and that an existing connection is not being reused by the OS or that your browser is caching the connection. For example, when using curl in PowerShell, you must specify the -DisableKeepalive parameter for new connections to take effect. Connections may also be pooled if you are using a browser.

• There is no need to reboot virtual machines on subnets configured with NAT gateways. However, when the virtual machine reboots, the connection state is flushed. When the connection state is flushed, all connections start using the NAT gateway resource's IP address (es). This behavior is a side effect of virtual machine reboots, so it does not mean that a reboot is required.

If you still have issues, open a support case for further troubleshooting.

Virtual Appliance UDR and ExpressRoute override NAT Gateway for outbound traffic routing.

When force tunneling with a custom UDR is used to forward traffic to a virtual appliance or VPN over ExpressRoute, the UDR or ExpressRoute takes precedence over the NAT Gateway to forward internet-bound traffic. more detailsCustom UDRssee

Internet routing configuration has the following order of precedence:
Load Balancing >> Virtual Appliance UDR/ExpressRoute for Device Default Outbound Access >> NAT Gateway >> Instance Level Public IP Address >> Outbound Rules

Test and troubleshoot the virtual appliance UDR or VPN ExpressRoute overriding the NAT gateway via:

1. to outbound trafficTest if NAT Gateway public IP is useddo. If a different IP is in use, it may be due to a custom UDR. Follow the remaining steps on how to verify and remove custom UDRs.

2. Check the UDR in the virtual network's route table andView route tablesee

3. Create, change, or delete Azure route tableto remove the UDR from the route table.

When the custom UDR is removed from the route table, the NAT Gateway public IP should now take precedence for routing outbound traffic to the internet.

Private IP is used to connect to Azure services via Private Link.

Private LinkPrivately connects your Azure Virtual Network to Azure PaaS services such as Azure Storage, Azure SQL, or Azure Cosmos DB over the Azure backbone network instead of the internet. Private Link connects to these Azure platform services using the private IP address of a virtual machine instance in a virtual network instead of the NAT Gateway's public IP. So when you look at the source IP address used to connect to these Azure services, you'll see that the instance's private IP is being used. All services supported by Private Link areAzure services listed heresee

To check the Private Endpoint set up as a Private Link:

1. Search for Private Link in the search box of the Azure portal.

   (Video) Lab | Azure NAT Gateway | VNET

2. In the Private Link center, select your private endpoint or Private Link service to see the configured configuration. more detailsManage private endpoint connectionssee

You can also connect virtual networks to Azure PaaS services using service endpoints. To verify that you have a service endpoint configured for your virtual network:

1. In the Azure portal, go to your virtual network and select "Service endpoints" in settings.

2. All service endpoints created are listed along with their configured subnets. more detailsService endpoint logging and troubleshootingsee

Connection errors to public internet destinations

Connection errors on internet-facing endpoints can be caused by a number of possible factors. Factors that can affect connection success include:

• Firewall or other traffic management component of the target

• API rate limit applied on the target side

• Massive DDoS mitigation or transport layer traffic shaping

NAT Gateway in Azure Monitormetricto diagnose connection problems.

• View the number of packets from the source and destination (if available) to determine the number of connection attempts.

• Check Dropped Packets to see how many packets were dropped by the NAT gateway.

Other checks:

• SNAT consumptionto check.

• Compare by validating connections to endpoints in the same or different regions.

  (Video) Azure Networking - #12 - Azure NAT Gateway

• If you're creating a high-volume or transactional speed test, look to see if lowering the speed reduces the frequency of failures.

• If rate changes affect the failure rate, check if an API rate limit or other constraint on the target side has been reached.

Active FTP and NAT gateway

FTP uses two separate channels between client and server, command and data channels. Each channel communicates on a separate TCP connection. One is for sending commands and the other is for sending data.

In active FTP mode, the client establishes a command channel and the server establishes a data channel.

NAT gateway does not work in active FTP mode when connecting to an FTP server over the Internet. Active FTP uses the FTP client's PORT command to tell the FTP server which server's IP address and port to use on the data channel to reconnect to the client. The PORT command uses the client's private address, which cannot be changed. The client-side traffic is SNATed by the NAT gateway for Internet-based communications, so the FTP server marks the PORT command as invalid.

An alternative solution to active FTP mode when connecting to an FTP server using a NAT gateway is to use passive FTP mode instead. But if you want to use NAT gateway in passive FTP modeA few things to considerdo.

Passive FTP and NAT Gateway

In passive FTP mode, the client establishes a connection on both command and data channels. The client requests the server to start listening on the port without re-establishing the connection to the client.

Depending on your FTP server configuration, outbound passive FTP may not work for NAT gateways with multiple public IP addresses. When a NAT gateway with multiple public IP addresses sends traffic outbound, it randomly selects one of the public IP addresses for the source IP address. Depending on your FTP server configuration, FTP may fail if data and control channels use different source IP addresses.

To avoid possible passive FTP connection errors, follow these steps:

1. Make sure your NAT gateway is associated with a single public IP address instead of multiple IP addresses or prefixes.

2. Make sure that the NAT gateway's passive port range can pass through any firewalls that the target endpoint may have.

Additional network capture

If the investigation is inconclusive, open a support case for further troubleshooting and gather the following information for faster resolution: Select a single virtual machine from the NAT gateway configuration subnet to perform the following test.

• On one of the backend VMs within the virtual networkps pingto the probe port response, e.g.ps ping 10.0.0.4:3389) and record the results.

• If no response is received from these ping tests, run concurrent Netsh traces for the backend VM and virtual network test VM while running PsPing, then stop the Netsh traces.

  (Video) What is Azure NAT Gateway | SNAT Port Exhaustion | Azure Standard Logic App Static Outbound IP

Outbound Connectivity Best Practices

Azure operates very carefully, monitoring its infrastructure. Still, deployed applications may experience transient failures, and there is no guarantee that transmitted data will not be lost. A NAT gateway is the default option for outbound connectivity in Azure deployments to ensure highly reliable and resilient outbound connectivity. In addition to using NAT gateways to connect outbound, use the instructions later in the article on how to ensure that your applications are using connections efficiently.

Modifying the application to use connection pooling

When pooling connections, do not open new network connections to the same address and port calls. You can implement a connection pooling scheme in your application where requests are internally distributed over a fixed set of connections and reused where possible. This setting limits the number of SNAT ports in use and creates a predictable environment. Connection pooling helps reduce latency and resource utilization and ultimately improve the performance of your application.

For more information on HTTP connection pooling, see Using HttpClientFactory.HTTP connection poolingsee

Modify the application to reuse connections

Configure your application to reuse connections instead of creating separate atomic TCP connections for each request. Connection reuse increases the performance of TCP transactions and is particularly relevant for protocols such as HTTP/1.1 where connection reuse is the default. This reuse applies to other protocols that use HTTP as a transport, such as REST.

Modifying the application to use less aggressive retry logic

When a SNAT port exhausts or an application error occurs, exhaustion occurs or persists due to aggressive or brute force retries without delay and backoff logic. Demand on SNAT ports can be reduced by using less aggressive retry logic.

Depending on the configured idle timeout, if the retries are too aggressive, the connection may not have enough time to close and release the SNAT port to re-use it.

Additional instructions and examplesretry patternsee

Reset outbound idle timeout using keepalive

More about keepalivesTCP idle timeout timer set higher than defaultsee

Reduce SNAT port usage for connecting to other Azure services using private link

for SNAT portsto reduce demandWhere possible, you should use Private Link to connect directly from your virtual network to Azure platform services. Reducing the demand on SNAT ports can help reduce the risk of SNAT port exhaustion.

To create a Private Link, see the following quickstart guide to get started.

• Create a private endpoint

• Create a Private Link

next stage

We are always working to improve our customers' experience. If you encounter NAT gateway issues that are addressed or not addressed in this article, provide feedback via GitHub at the bottom of this page.

To learn more about NAT Gateway, see:

• Azure NAT Gateway

• NAT gateway resource

• Metrics and Alerts for NAT Gateway Resources

  (Video) NAT and NAT Gateway in Azure