Migrating data from InfluxDB2 to CrateDB

[ZillKhan]

CrateDB version

5.10.11

CrateDB setup information

I have setup crate DB on dockers with this specification:

sh-5.2# crash --verbose
+-----------------------+------------------+---------+-----------+---------+
| server_url            | node_name        | version | connected | message |
+-----------------------+------------------+---------+-----------+---------+
| http://localhost:4200 | Bric Costa Rossa | 5.10.11 | TRUE      | OK      |
+-----------------------+------------------+---------+-----------+---------+
CONNECT OK
CLUSTER CHECK OK
TYPES OF NODE CHECK OK
cr> SHOW TABLES;
+----------------+
| table_name     |
+----------------+
| candata        |
| sys_timeoffset |
+----------------+

Problem description

I am trying to migrate my data from influxdb2 to crateDB using this command:

ctk load table "influxdb2://bedm:YOUR_TOKEN@192.168.200.63:8087/demmel/candata?start=-1d" --cluster-url="crate://192.168.200.56:4200/doc/candata"

I am getting this error:

File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='192.168.200.63', port=8087): Read timed out. (read timeout=9.98399999999674)

The data is huge but I want to import all in one go to the CrateDB.
Please suggest me what is the best approach to migrate influxdb 2 data.
Also if we can use exported .lp files of influxdb 2 to crate db.

Steps to Reproduce

1. CREATE TABLE candata(machine TEXT,"@connectorstate" TEXT,time TIMESTAMP);
2. Run ctk command

Actual Result

File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='192.168.200.63', port=8087): Read timed out. (read timeout=9.98399999999674)

Expected Result

Write data to Crate DB.

[amotl]

Dear @ZillKhan,

thanks for writing in and for your excellent report.

The data is huge but I want to import all in one go to the CrateDB. However, we are hitting a Read timed out error.

We understand. It is likely you are hitting a bottleneck because the framework might need optimizations to process large amounts of data.

read timeout=9.98399999999674

It looks like there might be a timeout=10 parameter configured somewhere? Maybe making it configurable needs to be unlocked by propagating another parameter from ctk to influxio, which is the actual workhorse for this data nozzle.

Also if we can use exported .lp files of influxdb 2 to crate db.

Indeed, you can also import directly from ILP format:

https://cratedb.com/docs/guide/integrate/influxdb/tutorial.html#influxdb-line-protocol

Please suggest me what is the best approach to migrate influxdb 2 data.

a) We will investigate the 10s timeout you are observing. I think it is likely we can make a difference here.
b) Please also explore the option to directly import from ILP: We would love to get any feedback if that works better or not, for the same reasons to gradually improve CTK's data I/O adapters. Thank you!

With kind regards,
Andreas.

[amotl]

read timeout=9.98399999999674
It looks like there might be a timeout=10 parameter configured somewhere?
We will investigate the 10s timeout you are observing. I think it is likely we can make a difference here.

We just found 10s is the default value the InfluxDB Python client library is using, so we think the case is clear.

timeout - socket timeout in ms (default value is 10000)

-- influxdb-client-python » Client configuration

Reading on on its documentation, we can discover there is an environment variable you should be able to use right away to adjust the default value.

INFLUXDB_V2_TIMEOUT - socket timeout in ms (default value is 10000)

-- influxdb-client-python » Environment variables

So, please invoke such a command before invoking ctk load table, possibly increasing the timeout to five minutes.

export INFLUXDB_V2_TIMEOUT=300000

The client translates the parameter to urllib3 options at this spot. We don't know yet how to approach applying an infinite timeout.

We hope this helps. Let us know about your outcomes, and feel free to submit any further questions or feature requests about this topic or beyond.

[amotl]

If you would like to invoke the data transfer using time ctk load table ..., you could tell us about the results on your end, so we could possibly adjust the default value for this parameter. Maybe 60s would be a more reasonable default value than just 10s?

Other than this, we will also unlock a timeout query parameter through the data source URL passed to ctk load table, so this value can also be configured without needing to define an environment variable.

[ZillKhan]

Hello @amotl
Thank you for your response.

I have checked by adjusting the timeout using
export INFLUXDB_V2_TIMEOUT=300000

But it didn't work. Here is the complete logs>

zkhan@BEDMPK15 MINGW64 ~
$ ctk --version
Could not import ingestr
ctk, version 0.0.37

zkhan@BEDMPK15 MINGW64 ~
$ export INFLUXDB_V2_TIMEOUT=300000

zkhan@BEDMPK15 MINGW64 ~
$ ctk load table "influxdb2://bedmMytoken@192.168.200.63:8087/demmel/candata" --cluster-url="crate://192.168.200.56:4200/doc/candata"
Could not import ingestr
2025-08-18 12:33:42,873 [cratedb_toolkit.io.influxdb         ] INFO    : Running InfluxDB copy
2025-08-18 12:33:42,909 [influxio.core                       ] INFO    : Copying from http://bedm:Mytoken@192.168.200.63:8087/demmel/candata to crate://192.168.200.56:4200/doc/candata
2025-08-18 12:33:42,910 [influxio.adapter                    ] INFO    : SQLAlchemy DB URI: crate://192.168.200.56:4200/?schema=doc
2025-08-18 12:33:42,927 [influxio.adapter                    ] INFO    : Loading dataframes into RDBMS/SQL database using pandas/Dask
Traceback (most recent call last):
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connection.py", line 565, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 1428, in getresponse
    response.begin()
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Scripts\ctk.exe\__main__.py", line 7, in <module>
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\cratedb_toolkit\io\cli.py", line 69, in load_table
    cluster.load_table(source=source, target=target, transformation=transformation)
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\cratedb_toolkit\cluster\core.py", line 568, in load_table
    if influxdb_copy(str(source_url_obj), target_url, progress=True):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\cratedb_toolkit\io\influxdb.py", line 26, in influxdb_copy
    copy(source_url, target_url, progress=progress)
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxio\core.py", line 92, in copy
    sink.write(source_node)
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxio\adapter.py", line 306, in write
    for df in source.read_df():
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxio\adapter.py", line 70, in read_df
    for df in self.client.query_api().query_data_frame_stream(query=query):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\client\query_api.py", line 299, in query_data_frame_stream
    response = self._query_api.post_query(org=org, query=self._create_query(query, self.default_dialect, params,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\service\query_service.py", line 285, in post_query
    (data) = self.post_query_with_http_info(**kwargs)  # noqa: E501
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\service\query_service.py", line 311, in post_query_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\_sync\api_client.py", line 343, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\_sync\api_client.py", line 173, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\_sync\api_client.py", line 388, in request
    return self.rest_client.POST(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\_sync\rest.py", line 311, in POST
    return self.request("POST", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\influxdb_client\_sync\rest.py", line 186, in request
    r = self.pool_manager.request(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\_request_methods.py", line 143, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\_request_methods.py", line 278, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\poolmanager.py", line 459, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\util\retry.py", line 449, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\util\util.py", line 39, in reraise
    raise value
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\Users\zkhan\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='192.168.200.63', port=8087): Read timed out. (read timeout=9.85899999999674)

[ZillKhan]

Indeed, you can also import directly from ILP format:

I have tried importing data directly using an ILP file exported from InfluxDB2, and it's working.
But I am facing one issue here, as I import the complete file from InfluxDB2 it includes all the measurements from the respective bucket.

For example, when I exported my "sensor" bucket, it has all the measurements in a single .lp file, but when I tried to import the same file that had multiple measurements in the same file, it would not create multiple tables for them.

I am currently using this command for importing .lp data from a file.
ctk load table "file://influxback.lp" --cluster-url="crate://192.168.200.56:4200/doc/candata"

Let me know if there is a command that can import all the measurements into a table automatically, or do I need to split the file into multiple measurements and then import them 1 after the other in CrateDB?

[amotl]

Hi @ZillKhan. Thank you very much, this is valuable and actionable feedback. It looks like both code paths need further improvements to make them work well in the spirit of DWIM. We will look into providing relevant updates soon.

• TimeoutError with large data imports daq-tools/influxio#182
• ILP import: Load multiple measurements into different tables daq-tools/influxio#183

[amotl]

Hi. We've just released v0.0.41 including a few updates you have been suggesting. We will be happy learn if that improves the situation for you.