How to detect failure to create a file?

[tesujimath]

There seems to be no way to trigger a re-run after failing to create a file.

As I develop my workflow, I find I may have a bug which causes an expected file not to be created at all. I am surprised this is not an error, since I have a File as output which doesn't exist in the filesystem.

Upon fixing my bug, subsequent runs of redun have no effect, as it seems to have happily cached the file as missing, and there is no changed state in the filesystem to trigger rerunning the task.

I think it should be an error if an output File from a task doesn't exist when the task completes?

Example workflow:

from redun import task, File
from typing import List

redun_namespace = "redun.examples.missing_after_error"


@task()
def create(path: str, content: str) -> File:
    if ok():
        with open(path, "w") as f:
            f.write(content)
    return File(path)


def ok() -> bool:
    # flipping this to True and rerunning doesn't trigger redun to generate the missing files
    return False


@task()
def main() -> List[File]:
    f1 = create("out/freddy1", "Hello Freddy 1\n")
    f2 = create("out/freddy2", "Hello Freddy 2\n")
    return [f1, f2]

I understand that the ok function is not hashed as part of the task. That's not my concern. This is about missing files.

[mattrasmus]

Thanks for submitting this issue. I agree with your explanation of how redun reacted to your bug fix and how it treats non-existent Files. A few thoughts.

Surprisingly, this case hasn't happened very often for us, but I can see how you encountered it.

Upon fixing my bug, subsequent runs of redun have no effect, as it seems to have happily cached the file as missing, and there is no changed state in the filesystem to trigger rerunning the task.

Just checking, I assume the bug fix was outside of a task? Normally such a code change would also trigger a rerun of that part of the workflow. In case its helpful, if we wanted to be reactive to code changes also in the ok() function of your example, one could use the task option hash_includes:

@task(hash_includes=[ok])
def create(path: str, content: str) -> File:
    if ok():
        with open(path, "w") as f:
            f.write(content)
    return File(path)

For more info:

• https://insitro.github.io/redun/tasks.html#task-hashing
• https://insitro.github.io/redun/tasks.html#hash-includes

Thus far we have treated non-existent Files as just another file state. So when we hash a File in a task result, it's just another hash.

• redun/redun/file.py

  Lines 464 to 476 in 147c19d

  	def get_hash(self, path: str) -> str:
  	"""
  	Return a hash for the file at path.
  	"""
  	# Perform a fast pseudo-hash of the file using O(1) proprties.
  	if self.exists(path):
  	stat = os.stat(path)
  	mtime = stat.st_mtime
  	size = stat.st_size
  	else:
  	mtime = -1
  	size = -1
  	return hash_struct(["File", "local", path, size, str(mtime)])

There is a concept in redun called "value validity", where when trying to use an cached value we check if its still valid by calling the is_valid() method. Thus far, this method for File is checking whether the hash has changed since last workflow run (non-existence state gets a hash too):

• redun/redun/file.py

  Lines 1337 to 1342 in 147c19d

  	def is_valid(self) -> bool:
  	if not self._hash:
  	self.update_hash()
  	return True
  	else:
  	return self.hash == self._calc_hash()

In theory this could be changed to also require the File exists. You could also imagine changing File.get_hash() to throw an exception if the File doesn't exist. That would cause a workflow halt even earlier during the first run, which is likely helpful.

Before actually implementing the changes listed above, I would need to think pretty hard about unintended consequences or breaking changes for use cases of the current behavior. At first glance, it does seem reasonable. Either way, if you were interested in having this new behavior for yourself right away, you could subclass File to implement it. Here is an example of subclassing done in the redun lib to provide a version of File where the hash depends on the content of the file:

• redun/redun/file.py

  Lines 1752 to 1765 in 147c19d

  	class ContentFile(File):
  	"""
  	Content-based file hashing.
  	"""
  	type_basename = "ContentFile"
  	type_name = "redun.ConentFile"
  	classes = ContentFileClasses()
  	def _calc_hash(self) -> str:
  	# Use filesystem.open() to avoid triggering a recursive hash update.
  	with self.filesystem.open(self.path, mode="rb") as infile:
  	content_hash = hash_stream(infile)
  	return hash_struct([self.type_basename, self.path, content_hash])

In our own code, we have implemented several kinds of File subclasses to customize different hashing and validity behavior.

I hope this helps.

[tesujimath]

Thank you for your detailed response!

This is not a blocker for me, so I will not attempt a quick work-around. My problem was caused by a function silently failing to create its output. I fixed it to fail properly and noisily!

I understand what you are saying about using hash_includes to trigger the task when dependent code changes, but my dependent code is rather a lot of code over several Python modules, so I don't think I'm going to be able to do that. But I'm OK with that, at least for now.

However, I do think that it is worth requiring a file to exist if it is specified as task output. I would really appreciate you considering incorporating that as new behaviour. I think it's the right thing in general.

Thanks!