Add support for NuttX RTOS.

[centurysys]

Porting Nim to run on Apache NuttX RTOS.

Merge NuttX specific consts to posix_other_consts
SO_REUSEPORT is not supported, so use SO_REUSEADDR to avoid compilation errors.

[centurysys]

  /Users/runner/work/Nim/Nim/lib/pure/httpclient.nim(1286) downloadFile
  Error: unhandled exception: 503 Service Unavailable [HttpRequestError]
  Error: execution of an external program failed: '/Users/runner/work/Nim/Nim/pkgstemp/timezones/timezones/fetchjsontimezones 2020d --out:testdata.json'

How should I handle this error?

[ringabout]

Unrelated CI failures. I'm restarting the CI for you. Btw, nice work!

[centurysys]

Unrelated CI failures. I'm restarting the CI for you. Btw, nice work!

CI passed, thank you!

[fjpanag]

@centurysys Nice work!

Please note that CONFIG_NET_NACTIVESOCKETS is just about to be removed, see here.

After this gets merged, there may be a pool of preallocated connections, but the system may also be able to dynamically allocate more, if needed (depending on the configuration).

Either way, keep in mind that even before this change, if CONFIG_NET_ALLOC_CONNS is defined, then CONFIG_NET_NACTIVESOCKETS is not a limit. It is just the number of statically allocated connections.

[centurysys]

@fjpanag Thank you for pointing this out ! I was not aware of that change.

I see that I need to calculate the maximum value of fd according to the user's configuration.
Deriving the exact value seems to require more consideration.

[centurysys]

@fjpanag
Since other Posix-based OSes use getrlimit() to get it, as shown below, should I change it so that it can be obtained dynamically from NuttX in the same way?

      var fdLim: RLimit
      if getrlimit(RLIMIT_NOFILE, fdLim) < 0:
        raiseOSError(osLastError())
      result = int(fdLim.rlim_cur) - 1

[Araq]

Shouldn't you also update compiler/platform.nim?

[centurysys]

@Araq
Thank you for your reply !

There are some items not supported, such as dlls, but I have added in accordance with FreeRTOS. Please check them.

[fjpanag]

@centurysys As fas as I can tell, getrlimit() is not implemented.

(It may be a good idea to open issues for missing functionality, to express interest. It may not be implemented promptly, but at least it indicates which features are actually desired.)

So, regarding determining the maximum number of connections. It seems a bit difficult to get the information that you are looking for.

That is because (assuming that #7525 is merged as it is today):

• There is no limit for the available sockets. The limit is on every different protocol. So you may set up the system for let's say max 10 UDP connections, but unlimited TCP connections.
• Even if you find a limit, this may not be valid, as the system may "consume" some of these connections internally.

When this PR gets merged, the system will have 3 possible configurations for every protocol.
Assuming TCP:

Statically Allocated
If NET_TCP_PREALLOC_CONNS is not zero and NET_TCP_ALLOC_CONNS is zero.
The system works on a pool of preallocated connections only (statically allocated).
The maximum number of connections will be NET_TCP_PREALLOC_CONNS.

Dynamically Allocated
If NET_TCP_ALLOC_CONNS is non-zero and NET_TCP_MAX_CONNS is zero.
The system may use a pool of preallocated connections, but will dynamically allocate more, as needed.
There is no limit in connections number, the system will try to allocate more till heap exhaustion.

Dynamically Allocated + Limit
If both NET_TCP_ALLOC_CONNS and NET_TCP_MAX_CONNS are non-zero.
The system may use a pool of preallocated connections, but will dynamically allocate more, as needed.
The maximum number of active connections will be NET_TCP_MAX_CONNS (both preallocated and dynamic allocated) but this is not guaranteed. The available memory may limit the connections, before the limit is reached.

All in all, I see that FreeRTOS and Zephyr use FD_MAX, as seen here.

Maybe this is the way to go for NuttX too?

[centurysys]

@fjpanag Thank you for your detailed analysis.
It seems that there is no definition of "FD_MAX" in NuttX, so it might be better to return a value that is large enough (256 or so?) that an RTOS running on an MCU would not exceed, just as the Windows value of 16_700_000 seems unlikely.

What do you think?

[fjpanag]

@fjpanag Thank you for your detailed analysis. It seems that there is no definition of "FD_MAX" in NuttX, so it might be better to return a value that is large enough (256 or so?) that an RTOS running on an MCU would not exceed, just as the Windows value of 16_700_000 seems unlikely.

What do you think?

Hmm... I am not sure about this. Indeed a "large enough" number would work.
But maybe there is a better way to get a number like FD_MAX that I am not aware of.

Maybe someone in the mailing list has a better idea on this....

[fjpanag]

Or just use INT_MAX (as an fd is int), and let the system decide whether it can provide you with a new descriptor or not?

(Assuming that a failed open() or socket() are handled appropriately).

[centurysys]

In the case of FreeRTOS(+lwIP), it seems that FD_MAX reflects the user's setting.

var FD_MAX* {.importc: "CONFIG_LWIP_MAX_SOCKETS", header: "<lwipopts.h>".}: cint

In the Github issue on arduino-ESP32, there was one that said '10 is too small', so it does not seem that a very large value is set.

CONFIG_LWIP_MAX_SOCKETS=10 is too small #5699
espressif/arduino-esp32#5699

[centurysys]

So, how about deriving approximate values from CONFIG_RAM_SIZE in .config?

[Araq]

I reviewed it this way, follow up PRs are welcome.

[github-actions]

Thanks for your hard work on this PR!
The lines below are statistics of the Nim compiler built from 3b9e9fd

Hint: mm: orc; opt: speed; options: -d:release
166150 lines; 8.082s; 611.203MiB peakmem

[fjpanag]

So, how about deriving approximate values from CONFIG_RAM_SIZE in .config?

Nah, the available memory for the connections can be practically unrelated, as CONFIG_RAM_SIZE contains lots of OS stuff, static data etc.

Furthermore this config has different meaning on each arch. For example the STM32F4's have two RAM regions, and this config represents only the one of them. The second one may or may not be available, depending on other configurations.

I sincerely have no idea how the maximum number of connections can reliably be obtained (without checking a dozen configs).

I think that you should either select a "good enough" limit arbitrarily, or maybe ask in the NuttX mailing list for help, where other knowledgeable people may have any good propositions.

[centurysys]

@fjpanag
Thank you for your comment.

I sent a question on the NuttX mailing list.
And I was looking at the code and found other areas that need to be adjusted with #when defined(nuttx) block.

proc newSelector*[T](): Selector[T] =
  # Retrieve the maximum fd count (for current OS) via getrlimit()
  var maxFD = maxDescriptors()
  doAssert(maxFD > 0)
  # Start with a reasonable size, checkFd() will grow this on demand
  const numFD = 1024 <---
...
    result.fds = newSeq[SelectorKey[T]](numFD)

(this will be hard on devices with small memory)

[centurysys]

@fjpanag
Or maybe it should be changed to use ioselectors_select.
At first I was switching to use epoll because ioselectors_select looked like leaking memory.
I think putting #when defined(nuttx) in ioselectors_epoll would make the code messy.

[centurysys]

@fjpanag
We discussed this on the NuttX mailing list.
I am thinking of changing it to get it from the OS via getrlimit() like any other POSIX based OS.
(NuttX is POSIX compliant, so that would be better).
After making the change on the OS side, I will send a pull request for the change on the Nim side.