Define custom memory allocation definitions

[warmwaffles]

What my aim here is to expose the amount of memory being utilized by an erlang vm process that is currently running.

I am opening this up for comment and am probably going to add in some benchmarks and some load testing scripts to go along with this.

[laszlohegedus]

Hey Matthew, our app fails with segmentation fault with these changes. I'll look into it, but can you also check what may cause it?

[warmwaffles]

Hey Matthew, our app fails with segmentation fault with these changes. I'll look into it, but can you also check what may cause it?

This is good to know. I'll need to do some more load testing with it.

[warmwaffles]

@laszlohegedus could you provide some information on what may have caused the segfault?

[laszlohegedus]

@laszlohegedus could you provide some information on what may have caused the segfault?

Sure, it actually happens on startup. I want to check it with Valgrind and see if it tells us why. We're also using libbtc_ex. I wonder if they collide with exqlite. Maybe I'll just try the two in a dummy project.

[warmwaffles]

I want to check it with Valgrind and see if it tells us why.

Let me know how that goes. I have had trouble getting valgrind to play nice with erlang and asdf installs

We're also using libbtc_ex. I wonder if they collide with exqlite. Maybe I'll just try the two in a dummy project.

That should not collide with exqlite.

[laszlohegedus]

@warmwaffles Well, it was a pain indeed. I managed to build the beam with valgrind and even "hack" the Elixir executable to use cerl, but it didn't want to work properly with our application, so I gave up.

What we can tell now is that we work with exqlite master, but not these changes. I'll dig into the code to see where it fails. At a first glance with my rarely used knowledge of C, the changes look good.

[laszlohegedus]

@warmwaffles I think I found the issue. It's in the on_unload function, when the memory allocator functions are set to NULL. We need to revert to the defaults there. I couldn't find a way to access that struct from outside of sqlite.c, but we can use SQLITE_CONFIG_GETMALLOC before we set our functions and store the defaults in a global variable. I'll add some comments to the code.

[warmwaffles]

@laszlohegedus see the latest commit, I lifted the struct method definitions to a static global context, and removed the on_unload definition. I wanted to try and be a "good citizen" and clear what was set on unload, but it probably doesn't matter in the grand scheme of things.

[warmwaffles]

The SQLITE_CONFIG_MALLOC option takes a single argument which is a pointer to an instance of the sqlite3_mem_methods structure. The argument specifies alternative low-level memory allocation routines to be used in place of the memory allocation routines built into SQLite. SQLite makes its own private copy of the content of the sqlite3_mem_methods structure before the sqlite3_config() call returns.

The SQLITE_CONFIG_GETMALLOC option takes a single argument which is a pointer to an instance of the sqlite3_mem_methods structure. The sqlite3_mem_methods structure is filled with the currently defined memory allocation routines. This option can be used to overload the default memory allocation routines with a wrapper that simulations memory allocation failure or tracks memory usage, for example.

I'm unsure which one is the one we have to set.

I think SQLITE_CONFIG_MALLOC is the one we want.

[laszlohegedus]

Looks great. Thank you.