The s3 storage backend leaks a new https connection every 10 seconds

[webner]

Every time s3 is queried (over a https connection) a new Transport object is created which is not automatically garbage collected. This leads to one new established connection every 10 seconds! After some hours/days there are too many connections open and every call to s3 fails.

There is an old open issue in net/http for this here: golang/go#24739

Consider reusing the same Transport object or closing the idle connections in a deferred function.

cluster-image-registry-operator/pkg/storage/s3/s3.go

Line 170 in 99ec5c4

Transport: &http.Transport{

[dmage]

/assign @ricardomaraschini
might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1902091

[ricardomaraschini]

@webner What version have you used to verify this leak?

[webner]

I verified the issue with 4.6.8 and 4.6.12.

sh-4.4$ echo $RELEASE_VERSION
4.6.12
sh-4.4$ date; lsof -ni | wc -l
Thu Jan 21 14:12:28 UTC 2021
11066
sh-4.4$ date; lsof -ni | wc -l
Thu Jan 21 14:12:43 UTC 2021
11067
sh-4.4$ date; lsof -ni | wc -l
Thu Jan 21 14:12:57 UTC 2021
11069

When you connect to a local minio instance via http it does not leak connections, only when you use https.

[ricardomaraschini]

@webner The code you mentioned got added 18 days ago through #648 and should be present only from 4.7 on. There must be something else.

[webner]

@ricardomaraschini This referenced line is already in 4.6.

cluster-image-registry-operator/pkg/storage/s3/s3.go

Line 148 in be99ea0

Transport: &http.Transport{

I guess the issue surfaced with an go update or some other change in the operator.

[ricardomaraschini]

Is your an IPI installation or there is something special about it? I have tried again to reproduce this but I could not:

sh-4.4$  date; lsof -ni |wc -l
Wed Mar 17 16:12:32 UTC 2021
6
sh-4.4$  date; lsof -ni |wc -l
Wed Mar 17 16:14:09 UTC 2021
5
sh-4.4$  date; lsof -ni |wc -l
Wed Mar 17 16:19:24 UTC 2021
6
sh-4.4$  date; lsof -ni |wc -l
Wed Mar 17 16:20:20 UTC 2021
5
sh-4.4$

[relyt0925]

We can reproduce this in IBM Cloud. Added more notes in the bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1959563

[dmage]

The problem is fixed in 4.8. The backporting process can be tracked in the BZ.