Deprecated ruby API use - possible memory issues

[blechereyal]

Environment

Operating System

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.6 LTS
Release:        14.04
Codename:       trusty
Linux homer-eblecher 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

TinyTDS Version and Information

 Version: freetds v1.00.21
freetds.conf directory: /usr/local/etc
MS db-lib source compatibility: no
Sybase binary compatibility: no
Thread safety: yes
iconv library: yes
TDS version: 7.3
iODBC: no
unixodbc: no
SSPI "trusted" logins: no
Kerberos: no
OpenSSL: no
GnuTLS: no
MARS: no

FreeTDS Version

1.00.21

Description

I'm not positive about this, but someone with more experience here could validate/invalidate this. I noticed some fairly significant memory usage when using tiny-tds (wrapped by the activerecord sql adapter gem) within sidekiq. I profiled this activity and it turned out to be mostly creation of strings from within result.c (putting the profile here - profiled via jemalloc profiling tools).

The behaviour was mostly that when using db heavy code i would see huge bursts of memory without any subsequent downscaling of the RSS. I noticed that we're using a deprecated api with Data_Make_Struct which now looks like should be TypedData_Make_Struct.Could this be an issue in a multithreaded environment and could this new api improve things ?

profile.pdf

[larskanis]

I doubt that this is an issue in tiny_tds. It is more probably that the query result object is somewhere stored in ruby space and is therefore not freed. Especially since it is related to sidekiq and not in the rails request-response-part.

You might try to diagnose the issue with a ruby memory profiler like https://github.com/SamSaffron/memory_profiler . It doesn't show you which object prevents garbage collection of the result objects, but at least it shows you where it was allocated in your code. This should help to find the memory leak.

I'm pretty sure about the other part. The usage of Data_Make_Struct is not the root cause of this issue. I implemented TypedData_Make_Struct for the pg gem here and know the differences very well.

[blechereyal]

Hi Lars, thanks for the answer. I've already tried the ruby memory profiler path with not a lot of success. I did however see a lot of memory fragmentation even with jemalloc so I was hoping it had something to do with a combination of RUBY_TYPED_FREE_IMMEDIATELY and the new rb_gc_mark_movable. Would implementing the usage of TypedData_Make_Struct help in debugging my issue when using a memory profiler I would expect a heap dump in that case to include more exact numbers ? I'd give a try implementing it here if that were the case. Would this generally be an acceptable change here ?