ECCV 2024

Every Pixel Has its Moments: Ultra-High-Resolution Unpaired Image-to-Image Translation via Dense Normalization

Ming-Yang Ho¹,   Che-Ming Wu²,   Min-Sheng Wu³,   Yufeng Jane Tseng¹

¹National Taiwan University,   ²Amazon Web Services,   ³aetherAI

[Paper (arxiv)] [Paper (official)] [Project Page]

Get started with an example

We provide a simple example (one image from the Kyoto summer2autumn dataset) for you to translate an UHR image with our DN.

Download example data

$ ./download.sh
$ unzip simple_example.zip

Environment preparation

1. Please check your GPU driver version and modify Dockerifle accordingly
2. Then, execute

   $ docker-compose up --build -d

3. Get into the docker container

   $ docker exec -it dn-env bash

Inference

1. In the docker container, please execute

   $ python3 transfer.py -c data/japan/config.yaml

2. Then, you can see a translated image at experiments/japan_CUT/test/IMG_6610/combined_dn_10.png
3. To see the image conveniently, you can leverage the provided visualization.ipynb. The setup of jupyter notebbok can be achived by
   • a. modify a port mapping setting in docker-compose.yml; e,g, - 19000:8888
   • b. install jupyter in the container
   • c. run your jupyter notebook by nohup jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root &
   • d. open the jupter notebook service on your port (19000 here)

Datasets

real2paint Dataset

For the real domain, please download the UHDM dataset from its official website. For the painting domain, we have curated a dataset of high-resolution Vincent van Gogh paintings, which can be downloaded at link1 or link2. Please note that we do not own these images; users should ensure their use does not trigger legal issues.

Kyoto-summer2autumn Dataset

Please download it at link.

ANHIR Dataset

Please download it at link. Please note that we do not own these images; users should ensure their use does not trigger legal issues.

ACROBAT Dataset

Please download it at link. Please note that we do not own these images; users should ensure their use does not trigger legal issues.

Train your model

The training of I2I model is the same as KIN. DN is a plugin for any I2I model with InstanceNorm layers.

Evaluation

Please refer to the KIN repo. All the metrics used (except the downstream task) are provided there. For the downstream task, we will provide the script recently.

Citation

@InProceedings{10.1007/978-3-031-72995-9_18,
    author="Ho, Ming-Yang and Wu, Che-Ming and Wu, Min-Sheng and Tseng, Yufeng Jane",
    title="Every Pixel Has Its Moments: Ultra-High-Resolution Unpaired Image-to-Image Translation via Dense Normalization",
    booktitle="Computer Vision -- ECCV 2024",
    year="2025",
    publisher="Springer Nature Switzerland",
    address="Cham",
    pages="312--328",
    isbn="978-3-031-72995-9"
}