Skip to content

Latest commit

 

History

History
77 lines (50 loc) · 2.99 KB

eval.md

File metadata and controls

77 lines (50 loc) · 2.99 KB

Evaluation

We provide a unified evaluation script that runs baselines on multiple benchmarks. It takes a baseline model and evaluation configurations, evaluates on-the-fly, and reports results instantly in a JSON file.

Benchmarks

Donwload the processed datasets from Huggingface Datasets and put them in the data/eval directory, using huggingface-cli:

mkdir -p data/eval
huggingface-cli download Ruicheng/monocular-geometry-evaluation --repo-type dataset --local-dir data/eval --local-dir-use-symlinks False

Then unzip the downloaded files:

cd data/eval  
unzip '*.zip'
# rm *.zip # if you don't keep the zip files

Configuration

See configs/eval/all_benchmarks.json for an example of evaluation configurations on all benchmarks. You can modify this file to evaluate on different benchmarks or different baselines.

Baseline

Some examples of baselines are provided in baselines/. Pass the path to the baseline model python code to the --baseline argument of the evaluation script.

Run Evaluation

Run the script moge/scripts/eval_baseline.py. For example,

# Evaluate MoGe on the 10 benchmarks
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/all_benchmarks.json --output eval_output/moge.json --pretrained Ruicheng/moge-vitl --resolution_level 9

# Evaluate Depth Anything V2 on the 10 benchmarks. (NOTE: affine disparity)
python moge/scripts/eval_baseline.py --baseline baselines/da_v2.py --config configs/eval/all_benchmarks.json --output eval_output/da_v2.json

The --baselies --input --output arguments are for the inference script. The rest arguments, e.g. --pretrained --resolution_level, are custormized for loading the baseline model.

Details of the arguments:

Usage: eval_baseline.py [OPTIONS]

  Evaluation script.

Options:
  --baseline PATH  Path to the baseline model python code.
  --config PATH    Path to the evaluation configurations. Defaults to
                   "configs/eval/all_benchmarks.json".
  --output PATH    Path to the output json file.
  --oracle         Use oracle mode for evaluation, i.e., use the GT intrinsics
                   input.
  --dump_pred      Dump predition results.
  --dump_gt        Dump ground truth.
  --help           Show this message and exit.

Wrap a Customized Baseline

Wrap any baseline method with moge.test.baseline.MGEBaselineInterface. See baselines/ for more examples.

It is a good idea to check the correctness of the baseline implementation by running inference on a small set of images via moge/scripts/infer_baselines.py:

python moge/scripts/infer_baselines.py --baseline baselines/moge.py --input example_images/ --output infer_outupt/moge --pretrained Ruicheng/moge-vitl --maps --ply