Using --derivatives from Bibsnet precomputed segmentation and mask

[xxnagit]

What happened?

When it started to run apply_t2w_mask after the final_clip were completed and while Loading pkl: /work/nibabies_25_1_wf/single_subject_sub-MO101_ses-0_wf/infant_anat_fit_wf/apply_t2w_mask/result_apply_t2w_mask.pklz, it raised the errors about "image size and mask size not matching". After checking the dimensions from brain masks inside the derivates created by Bibsnet, they are of the same size as the original images.

What command did you use?

singularity run --cleanenv \
--bind /home/xna/fmri/data_M_BIDS/work:/work \
-B /home/xna/fmri/data_BF/fmriprep/.cache/fmriprep:/home/nibabies/.cache/fmriprep \
-B /home/xna/fmri/data_M_BIDS/license.txt:/opt/freesurfer/license.txt:ro \
-B /home/xna/fmri/data_M_BIDS/bibsnet_out/bibsnet:/deriv/precomputed:ro \
/home/xna/fmri/hpcScripts/nibabies_unstable_20250422.sif \
--skip_bids_validation \
--notrack \
--stop-on-first-crash \
--participant_label MO101 \
--session-id ses-0 \
--output-spaces MNIInfant:cohort-1:res-1 UNCInfant:cohort-1 \
--surface-recon-method infantfs \
-w /work \
--derivatives precomputed=/deriv/precomputed \
--anat-only \
/home/xna/fmri/data_M_BIDS/M_BIDS /home/xna/fmri/data_M_BIDS/nibabies_out \
participant -vvv

What version of NiBabies are you using?

I tried the current unstable, and 25.0.1, they ended with the same errors

Relevant log output

250422-09:59:19,47 nipype.workflow WARNING:
	 Previous output generated by version 0+unknown found.
250422-09:59:19,50 nipype.workflow IMPORTANT:
	 
    Running nibabies version 25.1.0.dev7+g91c5fc3:
      * BIDS dataset path: /home/xna/fmri/data_M_BIDS/M_BIDS.
      * Participant list: [['MO101', '0']].
      * Run identifier: 20250422-095900_c90c4edb-98ab-4e7f-8dbe-c8790bb15a8b.
      * Output spaces: MNIInfant:cohort-1:res-1 UNCInfant:cohort-1:res-native.
      * Derivatives: {'precomputed': PosixPath('/deriv/precomputed')}.
      * Pre-run FreeSurfer's SUBJECTS_DIR: /home/xna/fmri/data_M_BIDS/nibabies_out/sourcedata/freesurfer.
250422-09:59:20,65 nipype.workflow INFO:
	 Collected the following data for sub-MO101_ses-0:
Raw:
{'asl': [],
 'bold': ['/home/xna/fmri/data_M_BIDS/M_BIDS/sub-MO101/ses-0/func/sub-MO101_ses-0_task-rest_dir-AP_bold.nii.gz',
          '/home/xna/fmri/data_M_BIDS/M_BIDS/sub-MO101/ses-0/func/sub-MO101_ses-0_task-rest_dir-PA_bold.nii.gz'],
 'dwi': [],
 'flair': [],
 'fmap': ['/home/xna/fmri/data_M_BIDS/M_BIDS/sub-MO101/ses-0/fmap/sub-MO101_ses-0_task-rest_dir-AP_epi.nii.gz',
          '/home/xna/fmri/data_M_BIDS/M_BIDS/sub-MO101/ses-0/fmap/sub-MO101_ses-0_task-rest_dir-PA_epi.nii.gz'],
 'pet': [],
 'roi': [],
 'sbref': [],
 't1w': ['/home/xna/fmri/data_M_BIDS/M_BIDS/sub-MO101/ses-0/anat/sub-MO101_ses-0_T1w.nii.gz'],
 't2w': ['/home/xna/fmri/data_M_BIDS/M_BIDS/sub-MO101/ses-0/anat/sub-MO101_ses-0_T2w.nii.gz']}

Derivatives:

{'t1w_aseg': '/deriv/precomputed/sub-MO101/ses-0/anat/sub-MO101_ses-0_space-T1w_desc-aseg_dseg.nii.gz',
 't1w_mask': '/deriv/precomputed/sub-MO101/ses-0/anat/sub-MO101_ses-0_space-T1w_desc-brain_mask.nii.gz',
 't2w_aseg': '/deriv/precomputed/sub-MO101/ses-0/anat/sub-MO101_ses-0_space-T2w_desc-aseg_dseg.nii.gz',
 't2w_mask': '/deriv/precomputed/sub-MO101/ses-0/anat/sub-MO101_ses-0_space-T2w_desc-brain_mask.nii.gz',
 'transforms': {}}
250422-09:59:23,355 nipype.workflow VERBOSE:
	 nibabies config:
		[environment]
		cpu_count = 56
		exec_env = "singularity"
		free_mem = 81.3
		overcommit_policy = "heuristic"
		overcommit_limit = "50%"
		nipype_version = "1.9.2"
		templateflow_version = "24.2.2"
		version = "25.1.0.dev7+g91c5fc3"
		
		[execution]
		bids_dir = "/home/xna/fmri/data_M_BIDS/M_BIDS"
		bids_database_dir = "/work/20250422-095900_c90c4edb-98ab-4e7f-8dbe-c8790bb15a8b/bids_db"
		bids_description_hash = "028f42f5696bd02de9c5cb3080bbed8b5084af6d0546cb0563298fce257924eb"
		boilerplate_only = false
		copy_derivatives = false
		sloppy = false
		debug = []
		fs_license_file = "/opt/freesurfer/license.txt"
		fs_subjects_dir = "/home/xna/fmri/data_M_BIDS/nibabies_out/sourcedata/freesurfer"
		layout = "BIDS Layout: ...xna/fmri/data_M_BIDS/M_BIDS | Subjects: 1 | Sessions: 1 | Runs: 0"
		log_dir = "/home/xna/fmri/data_M_BIDS/nibabies_out/logs"
		log_level = 10
		low_mem = false
		md_only_boilerplate = false
		nibabies_dir = "/home/xna/fmri/data_M_BIDS/nibabies_out"
		notrack = true
		output_dir = "/home/xna/fmri/data_M_BIDS/nibabies_out"
		me_output_echos = false
		output_layout = "bids"
		output_spaces = "MNIInfant:cohort-1:res-1 UNCInfant:cohort-1:res-native"
		reports_only = false
		run_uuid = "20250422-095900_c90c4edb-98ab-4e7f-8dbe-c8790bb15a8b"
		session_id = [ "0",]
		participant_label = [ "MO101",]
		templateflow_home = "/home/nibabies/.cache/templateflow"
		unique_labels = [ [ "MO101", "0",],]
		work_dir = "/work"
		write_graph = false
		
		[workflow]
		analysis_level = "participant"
		anat_only = true
		bold2anat_dof = 6
		bold2anat_init = "auto"
		cifti_output = false
		fd_radius = 45
		fmap_bspline = false
		force_syn = false
		hires = true
		hmc_bold_frame = 16
		ignore = []
		level = "full"
		longitudinal = false
		medial_surface_nan = false
		multi_step_reg = false
		norm_csf = false
		project_goodvoxels = false
		regressors_all_comps = false
		regressors_dvars_th = 1.5
		regressors_fd_th = 0.5
		run_reconall = true
		skull_strip_fixed_seed = false
		skull_strip_template = "UNCInfant:cohort-1"
		skull_strip_anat = "force"
		slice_time_ref = 0.5
		surface_recon_method = "infantfs"
		use_bbr = false
		use_syn_sdc = false
		me_t2s_fit_method = "curvefit"
		
		[nipype]
		crashfile_format = "txt"
		get_linked_libs = false
		nprocs = 56
		omp_nthreads = 8
		plugin = "MultiProc"
		resource_monitor = false
		stop_on_first_crash = true
		
		[seeds]
		master = 47313
		ants = 16994
		numpy = 41834
		
		[execution.dataset_links]
		raw = "/home/xna/fmri/data_M_BIDS/M_BIDS"
		templateflow = "/home/nibabies/.cache/templateflow"
		precomputed = "/deriv/precomputed"
		
		[execution.derivatives]
		precomputed = "/deriv/precomputed"
		
		[nipype.plugin_args]
		maxtasksperchild = 1
		raise_insufficient = false


250422-10:28:37,240 nipype.workflow INFO:
	 [Job 56] Completed (nibabies_25_1_wf.single_subject_sub-MO101_ses-0_wf.infant_anat_fit_wf.t1w_n4_wf.final_clip).
250422-10:28:37,241 nipype.workflow DEBUG:
	 Clearing 66 from queue
250422-10:28:37,242 nipype.utils DEBUG:
	 Loading pkl: /work/nibabies_25_1_wf/single_subject_sub-MO101_ses-0_wf/infant_anat_fit_wf/apply_t2w_mask/result_apply_t2w_mask.pklz
250422-10:28:37,242 nipype.workflow ERROR:
	 Node apply_t2w_mask failed to run on host xbt085.
250422-10:28:37,243 nipype.workflow ERROR:
	 Saving crash info to /home/xna/fmri/data_M_BIDS/nibabies_out/sub-MO101/ses-0/log/20250422-095900_c90c4edb-98ab-4e7f-8dbe-c8790bb15a8b/crash-20250422-102837-xna-apply_t2w_mask-4bd297b5-362b-4dff-94e8-9e0562bbae51.txt
Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node apply_t2w_mask.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 401, in run
	    runtime = self._run_interface(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/niworkflows/interfaces/nibabel.py", line 73, in _run_interface
	    raise ValueError('Image and mask sizes do not match.')
	ValueError: Image and mask sizes do not match.


250422-10:28:37,243 nipype.workflow INFO:
	 [Node] Setting-up "nibabies_25_1_wf.single_subject_sub-MO101_ses-0_wf.infant_anat_fit_wf.coregistration_wf.coreg" in "/work/nibabies_25_1_wf/single_subject_sub-MO101_ses-0_wf/infant_anat_fit_wf/coregistration_wf/coreg".
250422-10:28:37,244 nipype.workflow CRITICAL:
	 nibabies failed: Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node apply_t2w_mask.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 401, in run
	    runtime = self._run_interface(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/niworkflows/interfaces/nibabel.py", line 73, in _run_interface
	    raise ValueError('Image and mask sizes do not match.')
	ValueError: Image and mask sizes do not match.


250422-10:28:37,258 nipype.workflow INFO:
	 [Node] Executing "coreg" <niworkflows.interfaces.fixes.FixHeaderRegistration>
Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/bin/nibabies", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nibabies/cli/run.py", line 104, in main
    nibabies_wf.run(**_plugin)
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 625, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 177, in run
    self._clean_queue(jobid, graph, result=result)
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 255, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node apply_t2w_mask.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 401, in run
	    runtime = self._run_interface(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/nibabies/lib/python3.11/site-packages/niworkflows/interfaces/nibabel.py", line 73, in _run_interface
	    raise ValueError('Image and mask sizes do not match.')
	ValueError: Image and mask sizes do not match.

Add any additional information or context about the problem here.

No response

[mgxd]

Thanks for the report - likely it's due to N4 altering the T2w's header enough to cause this mismatch. To patch this, we should preserve the original header information.

[xxnagit]

Thank you so much for your prompt reply! Yes, it happened to both T1w and T2w sometimes. I actually further checked the processed T1w from the inFS, and the x y z are swapped. Accordingly, when taking the segmentation from the bibsnet (it was generated as the same orientation as the original T1w), it yelled for the dimension not match. After I worked around this by manually swapped the orientation, "the affine was not similar enough", which can be worked around by copying the affine from the header as well. It is not ideal, but if this can be fixed in the next version, it would much improved the final results. Great work, thank you all so much!!

[xxnagit]

I have to use HPC to run this pipeline, so if there is any docker image available, I'm more than happy to try it out. Thank you so much!