Undocumented negative return values from HTTPClient::get()

[teo1978]

All I can find in the documentation about HTTPClient::get() return value is that it returns "the http code".

Sometimes it returns -1. I guess that happens when something fails while sending the request, or parsing the response (if you even got one), so there's no http code to be returned.

However

1. that should be documented
2. There needs to be a way to get information about what the error actually was. Did the client fail to connect to the server? Did it fail to send the request? Did it get an invalid response? What else?
   If there already exists a way to get that kind of information, then that needs to be documented too.

[devyte]

@teo1978 when you opened this issue, you were presented with an issue template that contains specific instructions and requested info. You have ignored this.
Closing due to lack of info.
If you think your issue is relevant, please open a new issue, read and follow the instructions, and fill out the requested info. In addition, make sure to include a MCVE sketch, and make sure it has a url that shows the issue, even if it happens only sometimes. If you used a private url, such as a webpage served from a private server that you set up yourself, then that needs to be explained in detail, including how to set up the server.

[teo1978]

This is a documentation issue, you don't need a reproducible example.

What you need to do is look at the "return -1" statement in the function's code and document them.

Well actually there's point 2 which is not just documentation, that is more of a feature request. If you read it, you'll see you don't need (nor does it make sense to ask for) a reproducible case for that either.

[Pablo2048]

1. Directly in ESP8266HTTPClient.h is
   /// HTTP client errors
   #define HTTPC_ERROR_CONNECTION_REFUSED (-1)
   #define HTTPC_ERROR_SEND_HEADER_FAILED (-2)
   #define HTTPC_ERROR_SEND_PAYLOAD_FAILED (-3)
   #define HTTPC_ERROR_NOT_CONNECTED (-4)
   #define HTTPC_ERROR_CONNECTION_LOST (-5)
   #define HTTPC_ERROR_NO_STREAM (-6)
   #define HTTPC_ERROR_NO_HTTP_SERVER (-7)
   #define HTTPC_ERROR_TOO_LESS_RAM (-8)
   #define HTTPC_ERROR_ENCODING (-9)
   #define HTTPC_ERROR_STREAM_WRITE (-10)
   #define HTTPC_ERROR_READ_TIMEOUT (-11)

2. you can also call .errorToString() to get text representation of the error code given

[teo1978]

I see, thanks (I usually don't look at the source code for documentation).

So this is HTTPC_ERROR_CONNECTION_REFUSED.
Now, being a little skeptical about the connection being actually refused that often (given I've never observed a connection being refused by the same server from any other client - actually I've very rarely if ever observed any server refusing a connection for an http request and then accepting one for an identical http request, unless under veeeeery high load which is not the case here), I've had a look at the source code of GET() and in turn sendRequest, and I see:

int HTTPClient::sendRequest(const char * type, uint8_t * payload, size_t size)
{
    // connect to server
    if(!connect()) {
        return returnError(HTTPC_ERROR_CONNECTION_REFUSED);
    }

So I wonder: is it really safe to assume that, if HTTPClient::connect() return false, it's necessarily a "connection refused" error? I strongly doubt it.

[Pablo2048]

You can enable serial port debugging and see, where in ::connect() method is source of the error...

[teo1978]

You can enable serial port debugging and see, where in ::connect() method is source of the error...

I have tried that thanks to your suggestion, and it says more or less what we already knew: "connection refused":

[HTTP-Client][begin] url: http://******
[HTTP-Client][begin] host: ******.com port: 80 url: /******
[hostByName] request IP for: ******.com
[hostByName] Host: ******.com IP: ***.***.***.***
[HTTP-Client] failed connect to ******.com:80
[HTTP-Client][returnError] error(-1): connection refused
[HTTP-Client] getStreamPtr: not connected

I strongly suspect HttpClient is wrongly reporting as "connection refused" a broader class of connection failuers, but I'd like to verify that. Do you know if there are logs on the server (it's Debian 8) where I can look, where refused connections are logged?

I have also added WIFI to the debug level (besides HTTTP_CLIENT) but that doesn't add a bit of information when this error occurs.

[Pablo2048]

Hmm, it seems like the problem is deeper than I thought (now it seems like problem in WiFiClient, or deeper). Can you please try to use direct ip address in your query so we can eliminate problem with DNS? It seems like DNS is not the source of the problem sooo - probably you have to add more debug info into int WiFiClient::connect(IPAddress ip, uint16_t port) method to see more...

[teo1978]

probably you have to add more debug info into int WiFiClient::connect()

Who's "you"? Not me, right? I'm not calling that method directly. ok I see you were talking to me; found the library files, I see I can tinker with them. Trying that out.

Anyway It definitely does look like HTTPC_ERROR_CONNECTION_REFUSED is returned whenever establishing the TCP connection fails, regardless of whether it's the connection being refused or something else.

[teo1978]

It's failing here:

int WiFiClient::connect(IPAddress ip, uint16_t port)
{
    ip_addr_t addr;
    addr.addr = ip;
    
    DEBUG_WIFI("Connecting to %s:%d\r\n", ip.toString().c_str(), port);

    if (_client)
        stop();

    // if the default interface is down, tcp_connect exits early without
    // ever calling tcp_err
    // http://lists.gnu.org/archive/html/lwip-devel/2010-05/msg00001.html
#if LWIP_VERSION_MAJOR == 1
    netif* interface = ip_route(&addr);
    if (!interface) {
        DEBUGV("no route to host\r\n");
        return 0;
    }
#endif

    tcp_pcb* pcb = tcp_new();
    if (!pcb) {
    	DEBUG_WIFI("tcp_new() failed\r\n");
        return 0;
    }

    if (_localPort > 0) {
        pcb->local_port = _localPort++;
    }

    _client = new ClientContext(pcb, nullptr, nullptr);
    _client->ref();
    _client->setTimeout(_timeout);
    int res = _client->connect(&addr, port);
    if (res == 0) {
        _client->unref();
        _client = nullptr;
        DEBUG_WIFI("ClientContext::connect() failed\r\n"); //////// <<<<<<<<<<<< HERE
        return 0;
    }

    return 1;
}

which means, ClientContext::connect() is returning zero:

// from ClientContext.h
class ClientContext
{
  // [.....]
  int connect(ip_addr_t* addr, uint16_t port)
    {
        err_t err = tcp_connect(_pcb, addr, port, &ClientContext::_s_connected);
        if (err != ERR_OK) {
            return 0; // <<<<<<<<<<<<<<<<<<<<<<<< NOTE THIS!!!
        }
        _connect_pending = 1;
        _op_start_time = millis();
        // This delay will be interrupted by esp_schedule in the connect callback
        delay(_timeout_ms);
        _connect_pending = 0;
        if (state() != ESTABLISHED) {
            abort();
            return 0; // <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< AND THIS!
        }
        return 1;
    }

So, whenever tcp_connect() returns anything other than 0, that is any error, you end up returning "connection refused".
And also (just realised, and it seems to be my case since I've found out it is not the former) when state() is not ESTABLISHED whatever that means.

[teo1978]

Note that we are actually discussing two separate issues now:

• the original one I reported, that is return values of HTTPClient::get() other than valid HTTP response codes are undocumented (a bunch of constants in a .h files is not documentation).
• the fact that the return value HTTPC_ERROR_CONNECTION_REFUSED is abused for a broad range of connect failures that are not necessarily related to the connection being refused.

They are both still valid, and the former has always been. I don't know why this report got closed.

[teo1978]

Btw, I saw some DEBUGV(/*...*/) statements in ClientContext.h, and I added some. By looking for the definition of DEBUGV, I seemed to understand that its output is supposed to go to the serial port if the debug level of the Arduino IDE includes CORE (like, for example, the output of DEBUG_HTTPCLIENT() shows up if the debug level includes HTTP_CLIENT, the output of DEBUG_WIFI() shows up if the debug level includes WIFI and so on).

However, that is not the case. Is that another bug or am I missing something? I replaced the DEBUGV() statements I added with DEBUG_WIFI() and they do show up.

[devyte]

@teo1978 this was closed because there is an issue policy in place for this repo, and because the issue template has specific instructions to follow and requests to fill out specific information, neither of which was done.
My closing comment states that if you think this issue is valid, you're invited to open a new issue and fill out the requested info.
Given the details already added here, I alternatively suggest editing your original post and replacing it with a filled out template. I will reopen and triage at that point.

[teo1978]

Well, do whatever you want.

To the part regarding lack of documentation, the template doesn't even apply. And I don't see a separate repository or bug tracker for documentation.

For the other part, maybe I'll open a separate issue, though as you say the details are all here. Either you are interested in fixing it, in which case you have all the details here, or you are not, in which case I would be wasting my time by opening a new issue and filling in the template as you request. And if you are basing your decision whether or not to fix (or triage) the issue based on whether or not I fill in a template the way you request, that in my opinion wouldn't be a sensible criteria, but of course it's up to you.

By the way, I'd suggest you to reduce the length of the template if you really expect bug reporters to read it. When I'm presented a text that long in a form I'm filling in, I'll systematically ignore it, and so will most people.