Getting "File doesn't exist! Unrecoverable error" on busy logstash

[benmali]

We are using the kusto output plugin on a very bust logstash with thousands of logs a minute, with many kusto output blocks (one for each table)

Sample kusto output block

            kusto {
              path => "/tmp/kusto-serviceA/%{+YYYY-MM-dd-HH-mm}.txt"
              ingest_url => "https://ingest-${ADX_CLUSTER}.${AZURE_REGION}.kusto.windows.net"
              app_id => "${APP_ID}"
              app_key => "${APP_KEY}"
              app_tenant => "${APP_TENANT}"
              database => "${ADX_DB}"
              table => "_serviceA"
              json_mapping => "_serviceA_mapping"
            }

This works most of the time, but we see around 0.5% of the events have the following error (with service name replaced):

[2025-08-16T16:09:54,263][ERROR][logstash.outputs.kusto ][pipeline][e1f976b130786e6ec4e5b77d8357144a209a9ac2cea8e1d16bb5cc679fc61617] File doesn't exist! Unrecoverable error. {:exception=>Errno::ENOENT, :message=>"No such file or directory - No such file or directory - /tmp/kusto-serviceA/2025-08-16-16-09.txt.prod.serviceA", :path=>"/tmp/kusto-serviceA/2025-08-16-16-09.txt.prod.serviceA", :backtrace=>["org/jruby/RubyFileTest.java:248:in size'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-kusto-1.0.6-java/lib/logstash/outputs/kusto/ingestor.rb:94:in upload'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-kusto-1.0.6-java/lib/logstash/outputs/kusto/ingestor.rb:86:in block in upload_async'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]}

Infra:

Image: docker.elastic.co/logstash/logstash:8.5.0
Arch: Linux/amd64
Plugin Version: v2.0.2

Can you further explain how does the file rotation happen? Is there a possibility files are uploaded and deleted while another worker is writing to them?

[benmali]

Update: This also happens in latest logstash version 9.1.2 with plugin version 2.0.9

[ag-ramachandran]

Hello @benmali , this file rotation and closing happens through a combination of the template values used and replaced by logstash (based on the datetime) and also the plugin using this to stage data.

For the file to get deleted, the data should be queued to Kusto successfully. I think it is fair to ignore the error because the delete should get triggered only in case of a successful upload. The scenario where the delete may be duplicated may be cases where there could be a flaky upload (network/throttling) where retries kick in and then on upload try to delete the source file.

Do you see any other behavior in conjunction (i.e. Duplicates or data-loss. In a worst case you may be seeing duplicates, but just wanted to confirm)

[benmali]

We are currently experiencing data loss from logstash, and suspect the output plugin, as we were able to correlate the missing data timestamps with the missing file error messages on multiple events.

Anything else we can test so we can confirm data is being uploaded successfully? We noticed some undocumented flags like upload_concurrent_count and stale_cleanup_interval but we are trying to figure out how exactly they work and if we should try change them to ensure 100% of the events are uploaded

[benmali]

We looked at the upload function. We see we are failing at the line file_size = File.size(path) by the error message.
Can there be a scenario where the file is deleted by any kind of log rotation, other than successful upload?

` def upload(path, delete_on_success)
file_size = File.size(path)
@logger.debug("Sending file to kusto: #{path}. size: #{file_size}")

  # TODO: dynamic routing
  # file_metadata = path.partition('.kusto.').last
  # file_metadata_parts = file_metadata.split('.')

  # if file_metadata_parts.length == 3
  #   # this is the number we expect - database, table, json_mapping
  #   database = file_metadata_parts[0]
  #   table = file_metadata_parts[1]
  #   json_mapping = file_metadata_parts[2]

  #   local_ingestion_properties = Java::KustoIngestionProperties.new(database, table)
  #   local_ingestion_properties.addJsonMappingName(json_mapping)
  # end

  if file_size > 0
    file_source_info = Java::com.microsoft.azure.kusto.ingest.source.FileSourceInfo.new(path, 0); # 0 - let the sdk figure out the size of the file
    @kusto_client.ingestFromFile(file_source_info, @ingestion_properties)
  else
    @logger.warn("File #{path} is an empty file and is not ingested.")
  end
  File.delete(path) if delete_on_success
  @logger.debug("File #{path} sent to kusto.")
rescue Errno::ENOENT => e
  @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
rescue Java::JavaNioFile::NoSuchFileException => e
  @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
rescue => e
  # When the retry limit is reached or another error happen we will wait and retry.
  #
  # Thread might be stuck here, but I think its better than losing anything
  # its either a transient errors or something bad really happened.
  @logger.error('Uploading failed, retrying.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
  sleep RETRY_DELAY_SECONDS
  retry
end`

[benmali]

We tried setting pipeline.workers: 1 which seems to solve this, however, this creates serious backpressure (for obvious reasons).

[ag-ramachandran]

@benmali , There is probably a race condition with the worker-thread. The comment above was the key to detecting that. A PR is out #86 for this. Local tests indicate success. Will share a gem that you can perhaps test with and let us know

[ag-ramachandran]

Hello @benmali , can you try with this specific gem please

[benmali]

Hi, thanks for the PR! can you also change the log message @logger.debug("File #{path} sent to kusto.") to logger.info? This way we can validate a file was actually processed and sent - opening and closing files are showing anyway

[ag-ramachandran]

Hello @benmali, done. Gem available on the same location for tests

[benmali]

@ag-ramachandran How did you install the plugin? Can you please provide the installation steps? I'm either getting missing metadata error or :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'kusto'. Are you sure this is correct?

[ag-ramachandran]

Hi @benmali , you can use

first uninstall

[logstash-install-dir]/bin/logstash-plugin uninstall logstash-output-kusto

copy the gem file to the machine where LS is running and then

[logstash-install-dir]/bin/logstash-plugin install logstash-output-kusto-2.1.1-java.gem

This will validate and install the plugin

[benmali]

Hi, we deployed this version. We will monitor this for the next 24 hours and check for error. I will update as soon as possible.

[benmali]

We can see that this is still happening unfortunately.

We noticed something strange:

We see that in close_stale_files the function deletes the file and then tries to send it??

Maybe logstash retention detects the file as stale before upload happens? We can't really make sense of it

[benmali]

I will try sending the logstash logs tomorrow for better understanding

[ag-ramachandran]

@benmali , will wait for the logs to see what is happening