HTTPClient redirect fails with http code -7 (guessed root cause included)

[1plaintext]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP8266 NodeMCU
• Core Version: 3.1.2
• Development Env: Arduino IDE
• Operating System: Windows

Settings in IDE

• Module: NodeMCU 1.0 (ESP-12E Module)
• Flash Mode: other
• Flash Size: 2MB/1MB
• lwip Variant: v2 Lower Memory
• Reset Method: nodemcu
• Flash Frequency: 40Mhz
• CPU Frequency: 80Mhz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

When using HTTPClient setFollowRedirects(HTTPC_FORCE_FOLLOW_REDIRECTS) (follow redirect) on api.weather.gov (very popular weather source), redirect fails with http code -7
This is likely because when sending out 301, the server sends out not only headers, but http content as well (content length not 0). The guess is: the HTTPClient code itself, when making the second (redirected) request reuses connection, seemingly did not clear out the pipe enough (reason unknown), and caused subsequent reply to not look like an HTTP reply

From debug output:
Reply has content, not just headers
02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 461'
subsequent reply has gibberish in front, presumably left over from previous reply
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'est with the redundancy removed. If your client supports it, you will be redirected.",'
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: ' "instance": "https://api.weather.gov/requests/1aa0d75b"'
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: '}HTTP/1.1 200 OK'

Setting _reuse to false (new connection for redirect) solves the problem.

I am not sure api.weather.gov is violating standard by doing this, but "force follow" is supposed to throw standard out of the window, and consider api.weather.gov is very popular, this is worth fixing IMHO.

MCVE Sketch

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>
#include <WiFiClientSecureBearSSL.h>

#ifndef STASSID
#define STASSID "myapssid"
#define STAPSK "password"
#endif

ESP8266WiFiMulti WiFiMulti;

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println();
  Serial.println();
  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(STASSID, STAPSK);
  Serial.println("setup() done connecting to ssid '" STASSID "'");
}

void loop() {
  const char* headerkeys[] = {"Location"};
  if ((WiFiMulti.run() == WL_CONNECTED)) {
    std::unique_ptr<BearSSL::WiFiClientSecure> client(new BearSSL::WiFiClientSecure);
    client->setInsecure();
    HTTPClient https;
    https.collectHeaders(headerkeys, 1);
    https.setFollowRedirects(HTTPC_FORCE_FOLLOW_REDIRECTS);
    Serial.print("[HTTPS] begin...\n");
    if (https.begin(*client, "api.weather.gov", 443, "/points/40.697,-74.330"), true) {
      int httpCode = https.GET();
      if (httpCode > 0) {
        Serial.printf("[HTTPS] GET... returned code: %d\n", httpCode);
      } else {
        Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
      }
      https.end();
    } else {
      Serial.printf("[HTTPS] Unable to connect\n");
    }
  }
  Serial.println("Wait 10s before next round...");
  delay(10000);
}

Debug Messages

02:44:38.101 -> [HTTP-Client] connected to api.weather.gov:443
02:44:38.101 -> [HTTP-Client] sending request header
02:44:38.101 -> -----
02:44:38.101 -> GET /points/40.697,-74.330 HTTP/1.1
02:44:38.101 -> Host: api.weather.gov
02:44:38.101 -> User-Agent: ESP8266HTTPClient
02:44:38.101 -> Accept-Encoding: identity;q=1,chunked;q=0.1,;q=0
02:44:38.101 -> Connection: keep-alive
02:44:38.101 -> Content-Length: 0
02:44:38.101 ->
02:44:38.101 -> -----
02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 301 Moved Permanently
'
02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'Server: nginx/1.20.1
'
02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/problem+json
'
02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'Location: /points/40.697,-74.33
'
02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Request-ID: ab8a3cd6-ea1d-49f1-9b4a-bdb160497cb7
'
02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Correlation-ID: 1aa0d75b
'
02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Server-ID: vm-bldr-nids-apiapp13.ncep.noaa.gov
'
02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: *
'
02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Expose-Headers: X-Correlation-Id, X-Request-Id, X-Server-Id
'
02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Pragma: no-cache
'
02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 461
'
02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: private, must-revalidate, max-age=83944
'
02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Expires: Fri, 15 Mar 2024 09:03:43 GMT
'
02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 14 Mar 2024 09:44:39 GMT
'
02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive
'
02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Edge-Request-ID: 293e95b7
'
02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept,Feature-Flags,Accept-Language
'
02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
'
02:44:38.265 -> [HTTP-Client][handleHeaderResponse] RX: '
'
02:44:38.265 -> [HTTP-Client][handleHeaderResponse] code: 301
02:44:38.265 -> [HTTP-Client][handleHeaderResponse] size: 461
02:44:38.265 -> [HTTP-Client][sendRequest] following redirect (the same method): '/points/40.697,-74.33' redirCount: 0
02:44:38.265 -> [HTTP-Client][sendRequest] type: 'GET' redirCount: 0
02:44:38.265 -> [HTTP-Client] connect: already connected, reusing connection
02:44:38.265 -> [HTTP-Client] sending request header
02:44:38.302 -> -----
02:44:38.302 -> GET /points/40.697,-74.33 HTTP/1.1
02:44:38.302 -> Host: api.weather.gov
02:44:38.302 -> User-Agent: ESP8266HTTPClient
02:44:38.302 -> Accept-Encoding: identity;q=1,chunked;q=0.1,;q=0
02:44:38.302 -> Connection: keep-alive
02:44:38.302 -> Content-Length: 0
02:44:38.302 ->
02:44:38.302 -> -----
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'est with the redundancy removed. If your client supports it, you will be redirected.",'
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: ' "instance": "https://api.weather.gov/requests/1aa0d75b"'
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: '}HTTP/1.1 200 OK
'
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'Server: nginx/1.20.1
'
02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/geo+json
'
02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: *
'
02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Expose-Headers: X-Correlation-Id, X-Request-Id, X-Server-Id
'
02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Request-ID: 2c631c31-ab57-4ec8-ac12-8207b452a909
'
02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Correlation-ID: 1a96a60d
'
02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Server-ID: vm-bldr-nids-apiapp16.ncep.noaa.gov
'
02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept-Encoding
'
02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: public, max-age=83281, s-maxage=120
'
02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'Expires: Fri, 15 Mar 2024 08:52:40 GMT
'
02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 14 Mar 2024 09:44:39 GMT
'
02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 3098
'
02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive
'
02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Edge-Request-ID: 293e96aa
'
02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept,Feature-Flags,Accept-Language
'
02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload
'
02:44:38.465 -> [HTTP-Client][handleHeaderResponse] RX: '
'
02:44:38.465 -> [HTTP-Client][handleHeaderResponse] code: 0
02:44:38.465 -> [HTTP-Client][handleHeaderResponse] size: 3098
02:44:38.465 -> [HTTP-Client][handleHeaderResponse] Remote host is not an HTTP Server![HTTP-Client][returnError] error(-7): no HTTP server
02:44:38.465 -> [HTTP-Client][returnError] tcp stop
02:44:38.503 -> [HTTPS] GET... failed, error: no HTTP server
02:44:38.503 -> [HTTP-Client][end] tcp is closed

[1plaintext]

Jumped the gun too soon when opening the issue, it appears with the latest master I indeed no longer experience this issue. It appears to be same cause as #8871 - only part of the content body was flushed before it timed out. PR #8874 appears to resolve it.

[d-a-v]

Thanks for investigating. We need to make a new release.