The generated image is minimally affected by DWPOSE（Animate-x）Is It related to this error? How to solve it

[eavenhoo]

[Isi-dev]

I also noticed the same issue while trying to repose an image. Sometimes the image does not change at all, while at other times it does, but not as good as unianimate. But I got decent results for image to video as shown in this youtube video: https://youtu.be/OKFf8J-eMIc
I am not certain of the contribution of the above exception to the effect of dwpose on the generated image. I understand it to mean that some keys were deleted from the animate-x checkpoint to make it compatible with the UNET model. This means that some weights from the checkpoint won’t be used, potentially leading to suboptimal performance if the corresponding layers are required for better pose transfer.

[eavenhoo]

image to video is the same situation is not controlled by DWPOSE, the action is small or even not moving, Error message is the same as above, 768 resolution case is suspended，A lot of people are interested in this project. Please ask Big Brother to help solve it.

[Isi-dev]

image to video is the same situation is not controlled by DWPOSE, the action is small or even not moving, Error message is the same as above, 768 resolution case is suspended，A lot of people are interested in this project. Please ask Big Brother to help solve it.

Is your driving video a dwpose video or a raw video? A raw video is required and the code handles the conversion to a dwpose pose sequence. Please confirm.

[eavenhoo]

Is your driving video a dwpose video or a raw video? A raw video is required and the code handles the conversion to a dwpose pose sequence. Please confirm.

raw video

[Isi-dev]

Okay. I will check the code to see if there's a way to clear the error and improve the pose.

…

On Fri, 20 Dec 2024, 08:18 eavenhoo, ***@***.***> wrote: Is your driving video a dwpose video or a raw video? A raw video is required and the code handles the conversion to a dwpose pose sequence. Please confirm. raw video — Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUUXMITJNS4J34732ZO5F732GPADRAVCNFSM6AAAAABT4BLET2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWGQZDKMJTGM> . You are receiving this because you commented.Message ID: ***@***.***>

[Isi-dev]

I have updated the code to address the exception that comes up in the CLI when running Animate-X. I included a separate UNET script for Animate-x that makes the UNET model compatible with the animate-x checkpoint without the deletion of any key. Although I really can't say if this fix makes a significant improvement to the posing of images. 512 resolution is quite okay, but 768 resolution is totally off. I guess the animate-x model wasn't trained on images with width of 768. Shown below are my latest results. The rightmost image is from UniAnimate, while the next is Animate-X.

ResolutionX: 512

ResolutionX: 768

Please let me know if the issue has been resolved at your end.

[eavenhoo]

animate-x max_frames Any error over 32 will be reported，animate-x long(Error occurred when executing Animate_X_Image_Long:

DiffusionDDIM.ddim_sample_loop() got an unexpected keyword argument 'context_size')

[eavenhoo]

animate-x image to video the effect is correct

[Isi-dev]

animate-x image to video the effect is correct

Thanks for the feedback.

[Isi-dev]

animate-x max_frames Any error over 32 will be reported，animate-x long(Error occurred when executing Animate_X_Image_Long:

DiffusionDDIM.ddim_sample_loop() got an unexpected keyword argument 'context_size')

I will look into this.

[eavenhoo]

Animate_X_Image
error: The size of tensor a (65) must match the size of tensor b (33) at non-singleton dimension 0

[Isi-dev]

I have updated the code. Please let me know the result at your end. Thanks for the feedbacks.

[eavenhoo]

animate-x long It works,Thank you very much, but some video postures are not perfectly controlled

[Isi-dev]

animate-x long It works,Thank you very much, but some video postures are not perfectly controlled

You're welcome. Thanks for the feedback.