7 Key Fixes for Prometheus Context Deadline Exceeded (September 2026)

You open the Prometheus targets page and see it: a target you depend on is marked DOWN, and the error column reads “context deadline exceeded.” I have been there more times than I can count, and so have hundreds of engineers on the Prometheus mailing lists and community forums. The error looks simple, but the root cause can hide in a dozen different places.

“Context deadline exceeded” means Prometheus tried to scrape your target’s metrics endpoint but the response did not arrive within the configured time window. The scrape was cancelled, and the target flipped to DOWN. That single line of text can mask a network policy blocking traffic, a slow custom collector, a DNS resolution failure, a TLS handshake that never completes, or simply a timeout value that is too aggressive for the target in question.

In this guide I walk through the complete diagnosis process for a Prometheus target stuck in the DOWN state with “context deadline exceeded.” You will get a quick triage checklist, an error-to-cause mapping, step-by-step instructions for testing from inside the Prometheus pod, Kubernetes-specific fixes with copy-paste YAML, Docker networking gotchas, and advanced pprof techniques for the stubborn cases. By the end you will have a repeatable workflow that narrows down the problem in minutes instead of hours.

Quick Diagnosis Checklist

Start here before diving deep. These ten checks cover the most common root causes I have seen across production clusters and community discussions.

• 1. Read the full error string on the Prometheus targets page. The text after “context deadline exceeded” often contains the real clue (connection refused, no route to host, TLS handshake failure).

• 2. Check if the target pod or service is actually running. Use kubectl get pods and kubectl get endpoints to confirm the target exists and has ready endpoints.

• 3. curl the metrics endpoint from inside the Prometheus pod. If curl also times out, the problem is network-level. If curl succeeds, the problem is on the Prometheus config or timeout side.

• 4. Verify the scrape URL, port, and metrics path. A wrong port or a missing /metrics path is a frequent silent killer.

• 5. Check NetworkPolicy rules. If a default-deny policy exists, Prometheus needs an explicit ingress rule on the target namespace.

• 6. Confirm DNS resolution. Run nslookup or dig for the target hostname from inside the Prometheus pod.

• 7. Review TLS and HTTPS settings. Self-signed certs, expired certs, and missing scheme: https all produce timeout-like behavior.

• 8. Check ServiceMonitor label selectors. If you use kube-prometheus-stack, a label mismatch means Prometheus never discovers the target at all.

• 9. Measure target response time. Run time curl <metrics-url> from the Prometheus pod. Anything close to your scrape_timeout is a red flag.

• 10. Check the scrape interval vs timeout ratio. Your scrape_timeout must always be less than scrape_interval.

If none of these surface the problem, move on to the detailed sections below.

What “Context Deadline Exceeded” Actually Means?

“Context deadline exceeded” is a Go standard library error. Prometheus is written in Go, and every scrape operation runs inside a context that carries a deadline. When that deadline fires before the HTTP response body is fully received, Go returns context.DeadlineExceeded and Prometheus logs it as “context deadline exceeded.”

In practical terms, the scrape lifecycle works like this. Prometheus picks up the target from its service discovery or static config, starts an HTTP GET to the configured metrics path, and starts a timer set to scrape_timeout. If the target sends every byte before the timer fires, the scrape succeeds and the target stays UP. If the timer fires first, the connection is torn down, the scrape is recorded as failed, and the target is marked DOWN for that scrape cycle.

The default scrape_timeout is 10 seconds. The default scrape_interval is also tied to the global config, often 15 to 60 seconds depending on your setup. These defaults are fine for fast exporters like node_exporter, but they fall short for targets that do heavy computation per scrape, such as a custom application metrics endpoint that queries a database on every request.

The key insight is that “context deadline exceeded” is a symptom, not a root cause. The root cause is whatever prevented the response from arriving in time: network latency, packet drops, a slow target, DNS delays, TLS negotiation overhead, or a firewall that silently drops packets instead of refusing the connection.

Error Message to Cause Mapping

The Prometheus targets page and server logs provide more detail than just “context deadline exceeded.” Here is a mapping of the error strings I see most often and what each one typically points to.

• Get “http://IP:port/metrics”: context deadline exceeded — The TCP connection itself likely succeeded but the response body took too long. Check target performance and high-cardinality exporters.

• dial tcp IP:port: connect: connection refused — The target is reachable but nothing is listening on that port. The pod may have crashed or the port declaration in the Service does not match the container.

• dial tcp IP:port: i/o timeout — Packets are being dropped by a firewall or NetworkPolicy. The SYN packet went out but no SYN-ACK came back.

• dial tcp: lookup hostname: no such host — DNS resolution failed. Check CoreDNS, the service name spelling, and the namespace.

• Get “https://IP:port/metrics”: context deadline exceeded — TLS handshake may be stalling. Verify the certificate, the CA bundle, and whether insecure_skip_verify is needed.

• server returned HTTP status 404 — Wrong metrics path. The target is reachable but metrics_path is incorrect.

• “context deadline exceeded” with no additional text — Usually a very slow target or a network path with extreme latency. Run pprof on the target if possible.

This mapping is the single most requested feature in forum threads. Print it, bookmark it, and use it as your first reference when a target goes DOWN.

Step-by-Step Diagnosis Using the Prometheus UI

The Prometheus web UI is your first and most powerful diagnostic tool. Open it at http://<prometheus-host>:9090 and navigate to Status then Targets.

Step 1: Find the failing target. Each job is listed as a collapsible group. Expand the group and look for the endpoint marked DOWN in red. The “Last Error” column shows the exact error string Prometheus recorded for the most recent failed scrape.

Step 2: Read the Last Scrape and Last Scrape Duration columns. If “Last Scrape Duration” is close to your scrape_timeout value, the target is responding but too slowly. If “Last Scrape” shows a timestamp far in the past, the target may not be discovered at all.

Step 3: Check the “Error” column for the full message. Copy the entire string, not just the “context deadline exceeded” suffix. The prefix tells you whether the failure happened at connection time, during the TLS handshake, or while reading the body.

Step 4: Cross-reference with the Prometheus server logs. Port-forward the Prometheus pod with kubectl port-forward prometheus-0 9090:9090 and tail the logs with kubectl logs -f prometheus-0. Look for lines containing scrape_timeout, error, or the target URL. The logs often reveal patterns the UI does not show, like intermittent failures that flip between UP and DOWN.

Step 5: Check the scrape config Prometheus actually loaded. Navigate to Status then Configuration in the UI. This shows the effective configuration, including all generated ServiceMonitor rules. Verify that scrape_timeout, scrape_interval, metrics_path, scheme, and the target address all match what you expect. A ServiceMonitor that silently overrides your global settings is a common surprise.

Network Diagnostics from the Prometheus Pod

The most reliable way to test connectivity is to run commands from inside the Prometheus container itself. This eliminates guesswork about firewalls, DNS, and routing because you are testing the exact network path the scraper uses.

Exec into the Prometheus pod:

kubectl exec -it prometheus-0 -n monitoring -- /bin/sh

Test the metrics endpoint with curl and timing:

time wget -qO- http://<target-ip>:<port>/metrics | head -5

If you do not have wget, use curl. The goal is to measure how long the response takes. Compare that duration against your scrape_timeout. If curl takes 12 seconds and your timeout is 10 seconds, you found the problem.

Check DNS resolution:

nslookup <target-service-name>.<namespace>.svc.cluster.local

If nslookup fails or returns the wrong IP, CoreDNS is the culprit. I have seen cases where a typo in the namespace or service name caused Prometheus to resolve a completely different pod.

Test raw TCP connectivity:

wget -O- --timeout=5 http://<target-ip>:<port>

If this times out within 5 seconds but curl from your laptop works fine, there is a firewall or NetworkPolicy between the Prometheus pod and the target. This is the scenario that trips people up most often because everything looks correct in the config.

Check for packet loss and latency:

ping <target-ip>

High latency or packet loss on the cluster network can cause intermittent “context deadline exceeded” errors that only appear under load. If ping shows 200ms+ round trips on a local cluster, investigate the CNI plugin or node networking.

Kubernetes-Specific Causes and Fixes

Kubernetes adds several layers between Prometheus and your targets, and each layer can produce a timeout. I will cover the six most common scenarios I encounter in production clusters.

NetworkPolicy Silently Blocking the Scrape

A NetworkPolicy with a default-deny ingress rule is the number one cause of “context deadline exceeded” in Kubernetes environments. The policy drops Prometheus packets silently, so Prometheus sees a timeout rather than a connection refused.

To fix this, add an ingress rule on the target namespace that allows traffic from the Prometheus namespace. Here is a working example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
  namespace: my-app
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
       matchLabels:
         name: monitoring
    ports:
    - protocol: TCP
      port: 8080

Make sure the namespaceSelector label matches how your monitoring namespace is actually labeled. A mismatch here produces the same silent timeout.

ServiceMonitor Label Mismatch

If you use the Prometheus Operator and kube-prometheus-stack, ServiceMonitor objects drive scrape config generation. A label selector mismatch means Prometheus never discovers your target, and the target simply does not appear in the targets list.

Check the Prometheus CRD’s service monitor selector:

kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.serviceMonitorSelector}'

Then verify your ServiceMonitor carries a matching label. The most common mistake is a release label like release: kube-prometheus-stack that does not match the actual Helm release name.

DNS Resolution Failure

CoreDNS can fall behind under heavy load, especially in large clusters. If Prometheus resolves the target hostname intermittently, you will see targets flipping between UP and DOWN. Check CoreDNS pods for restarts and high memory usage:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Also verify the service name and namespace in your scrape config match the actual Kubernetes service. I once spent two hours chasing a timeout caused by a service named metrics-exporter when the config referenced metric-exporter.

TLS and HTTPS Target Misconfiguration

When a target uses HTTPS, Prometheus needs the correct scheme, certificate authority, and optionally insecure_skip_verify for self-signed certs. A TLS handshake that stalls looks identical to a timeout in the UI. Add these fields to your scrape config or ServiceMonitor:

scheme: https
tlsConfig:
  insecureSkipVerify: true
  caFile: /etc/prometheus/secrets/ca.crt

If the cert is expired or the CA is wrong, the error message usually includes TLS-specific text. But in some Prometheus versions, a TLS failure surfaces as a generic deadline exceeded, which is why testing with curl -k from the pod is so important.

Target Pod Not Running or Not Ready

Sometimes the simplest explanation is correct. If the target pod is in a CrashLoopBackOff or has failed readiness checks, the Kubernetes Service has no ready endpoints. Prometheus resolves the service IP but the connection is refused or times out. Check with:

kubectl get endpoints <service-name> -n <namespace>

If the endpoints list is empty, the pod is not ready. Fix the application or readiness probe first.

Port Name Mismatch in ServiceMonitor

The ServiceMonitor’s port field must match a named port in the Service definition, not a number. If your Service defines the port as name: http but your ServiceMonitor references port: metrics, the scrape target is never generated. Check both:

kubectl get svc <service> -o jsonpath='{.spec.ports[*].name}'

Configuration Fixes: scrape_timeout and scrape_interval

Adjusting scrape_timeout and scrape_interval is the most common fix people try, and it is also the one most likely to mask the real problem. Forum users consistently report that bumping the timeout alone does not help when the root cause is networking or target performance. Still, correct configuration matters.

Rule 1: scrape_timeout must be less than scrape_interval.

If your interval is 15 seconds and your timeout is 15 seconds, Prometheus will never complete a scrape before the next one starts. Set the timeout to roughly 80 percent or less of the interval.

Rule 2: Set per-job overrides for slow targets.

Global settings apply to all jobs, but you can override them per job. This is the right approach when one target legitimately needs more time:

scrape_configs:
- job_name: 'slow-app'
  scrape_interval: 60s
  scrape_timeout: 45s
  static_configs:
  - targets: ['app.default.svc.cluster.local:8080']
  metrics_path: /metrics

Rule 3: For ServiceMonitor, use these fields:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: slow-app-monitor
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: slow-app
  endpoints:
  - port: http
    interval: 60s
    scrapeTimeout: 45s
    path: /metrics

Rule 4: Do not set scrape_timeout to absurdly high values.

A 120-second timeout means Prometheus holds a connection open for two minutes on a hung target. This wastes resources and delays alerting. If your target needs more than 60 seconds to respond, the target itself needs optimization, not a longer timeout.

Rule 5: Understand remote_write timeouts separately.

If you forward metrics to Mimir, Cortex, or Thanos using remote_write, that path has its own timeout (remote_write.timeout, default 30s). A slow remote write endpoint causes backpressure, which can eventually delay scrapes. This is a different failure mode from a target-level timeout, and the fix involves tuning the remote write queue and timeout, not the scrape config.

Advanced Debugging with pprof and Goroutine Analysis

When the target is reachable, the network is clean, and the timeout is generous, but scrapes still fail, the problem is inside the target application itself. A custom exporter or an instrumentation library may have a collector that hangs on certain queries.

If the target is a Go application, it likely exposes pprof endpoints. These endpoints let you inspect goroutines, memory, and CPU profiles in real time.

Check for stuck goroutines:

curl http://<target>:<port>/debug/pprof/goroutine?debug=1

Look for goroutines blocked on the same function call. A collector stuck waiting on a database query will show dozens of goroutines parked at the same SQL call. This is the smoking gun for a target-side hang.

Profile the metrics endpoint directly:

curl -o profile.prof http://<target>:<port>/debug/pprof/profile?seconds=30

Then analyze with go tool pprof profile.prof. If the CPU profile shows the application spending most of its time in a specific function during the scrape window, that function is your bottleneck.

Use the Prometheus debug endpoints.

Prometheus itself exposes debug endpoints at /debug. The /debug/pprof path on the Prometheus server shows whether the scraper is leaking goroutines or accumulating memory, which can happen when thousands of targets are timing out simultaneously.

Check for high-cardinality label explosions.

If your exporter emits hundreds of thousands of time series because of unbounded labels (user IDs, request paths, IP addresses), the scrape payload becomes enormous. Prometheus struggles to ingest it within the timeout, and you get “context deadline exceeded” even though the network is fine. Use prometheus_tsdb_head_series to check the active series count, and consider relabeling rules to drop high-cardinality labels before they reach Prometheus.

Docker-Specific Networking Issues

Running Prometheus in Docker adds a networking layer that produces a unique set of problems. Forum threads are full of engineers whose scrape works from the host but fails from the container.

Use the right address for host-based exporters. If Prometheus runs in a container and node_exporter runs on the host, localhost inside the container refers to the container itself, not the host. On Linux, use host.docker.internal or the Docker bridge gateway IP (usually 172.17.0.1). On Docker Desktop for Mac and Windows, host.docker.internal works out of the box.

Open the firewall port on the host. A community user reported spending hours debugging a node_exporter scrape timeout that turned out to be a host firewall rule. The port was bound to 127.0.0.1 instead of 0.0.0.0, so the Docker bridge could not reach it. Bind exporters to 0.0.0.0 and verify the host firewall allows traffic from the Docker bridge subnet.

Consider host networking mode for simplicity. For single-node setups, running Prometheus with network_mode: host eliminates the container networking layer entirely. This is not ideal for production multi-node setups, but it removes a class of networking bugs that are hard to trace.

Check Docker Compose network names. If Prometheus and the target are in different Compose services, they need to be on the same Docker network or use the service name as the hostname. A common mistake is referencing localhost when the target is a separate container.

Monitoring and Alerting for Target Downtime

Once you fix the immediate problem, set up alerts so you catch the next timeout before it causes data gaps. The up metric is your primary signal: it returns 1 for a successful scrape and 0 for a failed one.

Basic target-down alert:

alert: TargetDown
expr: up == 0
for: 2m
labels:
  severity: warning
annotations:
  summary: "Prometheus target {{ $labels.instance }} is down"
  description: "{{ $labels.job }} has been down for more than 2 minutes."

Scrape duration alert (catch slow targets before they time out):

alert: HighScrapeDuration
expr: scrape_duration_seconds > 0.8 * scrape_timeout_seconds
for: 5m
labels:
  severity: warning
annotations:
  summary: "Scrape for {{ $labels.instance }} is approaching timeout"

Note that scrape_timeout_seconds is not a built-in metric. You need to record the configured timeout as a custom metric or use a fixed threshold based on your config. A simpler approach is to alert when scrape duration exceeds a fixed value like 8 seconds if your timeout is 10 seconds.

Alert on scrape failures rate:

alert: ScrapeFailureRate
expr: rate(prometheus_target_scrape_pool_targets{state="down"}[5m]) > 0
for: 5m
labels:
  severity: critical

These alerts give you early warning. The goal is to know about a degrading scrape before it flips to DOWN, not after.

Frequently Asked Questions

Conclusion

Diagnosing a Prometheus target stuck in the DOWN state with “context deadline exceeded” comes down to a disciplined process. Read the full error string, test from inside the Prometheus pod, and work through the network, configuration, and target-performance layers in order. The quick checklist at the top of this guide covers roughly 80 percent of the cases I see in production.

The remaining 20 percent usually involves Kubernetes-specific issues like NetworkPolicy, ServiceMonitor selectors, or deep target-side problems that need pprof. Keep the error-to-cause mapping handy, set up scrape duration alerts before the next timeout hits, and remember that bumping the timeout is a last resort, not a first step.

With this workflow, you can turn a frustrating hours-long debugging session into a 15-minute diagnosis. That is the difference between a reliable monitoring stack and one with constant blind spots.