diff --git a/Docs/2024-10-25/.idea/.gitignore b/Docs/2024-10-25/.idea/.gitignore new file mode 100644 index 0000000..13566b8 --- /dev/null +++ b/Docs/2024-10-25/.idea/.gitignore @@ -0,0 +1,8 @@ +# Default ignored files +/shelf/ +/workspace.xml +# Editor-based HTTP Client requests +/httpRequests/ +# Datasource local storage ignored files +/dataSources/ +/dataSources.local.xml diff --git a/Docs/2024-10-25/.idea/inspectionProfiles/Project_Default.xml b/Docs/2024-10-25/.idea/inspectionProfiles/Project_Default.xml new file mode 100644 index 0000000..60b99ca --- /dev/null +++ b/Docs/2024-10-25/.idea/inspectionProfiles/Project_Default.xml @@ -0,0 +1,24 @@ + + + + \ No newline at end of file diff --git a/Docs/2024-10-25/.idea/inspectionProfiles/profiles_settings.xml b/Docs/2024-10-25/.idea/inspectionProfiles/profiles_settings.xml new file mode 100644 index 0000000..105ce2d --- /dev/null +++ b/Docs/2024-10-25/.idea/inspectionProfiles/profiles_settings.xml @@ -0,0 +1,6 @@ + + + + \ No newline at end of file diff --git a/Docs/2024-10-25/.idea/misc.xml b/Docs/2024-10-25/.idea/misc.xml new file mode 100644 index 0000000..812ab5a --- /dev/null +++ b/Docs/2024-10-25/.idea/misc.xml @@ -0,0 +1,7 @@ + + + + + + \ No newline at end of file diff --git a/Docs/2024-10-25/.idea/modules.xml b/Docs/2024-10-25/.idea/modules.xml new file mode 100644 index 0000000..46d6cef --- /dev/null +++ b/Docs/2024-10-25/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/Docs/2024-10-25/.idea/vcs.xml b/Docs/2024-10-25/.idea/vcs.xml new file mode 100644 index 0000000..b2bdec2 --- /dev/null +++ b/Docs/2024-10-25/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/Docs/2024-11-15/2024CVPRPaper.md b/Docs/2024-11-15/2024CVPRPaper.md new file mode 100644 index 0000000..4a98881 --- /dev/null +++ b/Docs/2024-11-15/2024CVPRPaper.md @@ -0,0 +1,805 @@ +# CVPR2024｜底层视觉相关论文汇总 +CVPR2024底层视觉(Low-Level Vision)相关的论文和代码，包括超分辨率，图像去雨，图像去雾，去模糊，去噪，图像恢复，图像增强，图像去摩尔纹，图像修复，图像质量评价，插帧，图像/视频压缩等任务，具体如下。 + + https://zhuanlan.zhihu.com/p/684196283 + + CVPR2024官网：https://cvpr.thecvf.com/Conferences/2024 + + CVPR接收论文列表：https://cvpr.thecvf.com/Conferences/2024/AcceptedPapers + + CVPR完整论文库：https://openaccess.thecvf.com/CVPR2024 + + 开会时间：2024年6月17日-6月21日 + 论文接收公布时间：2024年2月27日 + +# 相关方法概览 + 1.超分辨率(Super-Resolution) + 2.图像去雨(Image Deraining) + 3.图像去雾(Image Dehazing) + 4.去模糊(Deblurring) + 5.去噪(Denoising) + 6.图像恢复(Image Restoration) + 7.图像增强(Image Enhancement) + 8.图像修复(Inpainting) + 9.高动态范围成像(HDR Imaging) + 10.图像质量评价(Image Quality Assessment) + 11.插帧(Frame Interpolation) + 12.视频/图像压缩(Video/Image Compression) + 13.压缩图像质量增强(Compressed Image Quality Enhancement) + 14.图像去反光(Image Reflection Removal) + 15.图像去阴影(Image Shadow Removal) + 16.图像上色(Image Colorization) + 17.图像和谐化(Image Harmonization) + 18.视频稳相(Video Stabilization) + 19.图像融合(Image Fusion) + 20.其他任务(Others) + +## 1.超分辨率(Super-Resolution) +**AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution** +* Paper: https://arxiv.org/abs/2404.03296 +* Code: https://github.com/Cheeun/AdaBM + +**A Dynamic Kernel Prior Model for Unsupervised Blind Image Super-Resolution** +* Paper: https://arxiv.org/abs/2404.15620 +* Code: https://github.com/XYLGroup/DKP + +**APISR: Anime Production Inspired Real-World Anime Super-Resolution** +* Paper: https://arxiv.org/abs/2403.01598 +* Code: https://github.com/Kiteretsu77/APISR + +**Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder** +* Paper: https://arxiv.org/abs/2403.10255v1 +* Code: https://github.com/zhenshij/arbitrary-scale-diffusion + +**Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss** +* Paper: https://arxiv.org/abs/2404.01692 +* Code: https://github.com/JaehaKim97/SR4IR + +**Bilateral Event Mining and Complementary for Event Stream Super-Resolution** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Huang_Bilateral_Event_Mining_and_Complementary_for_Event_Stream_Super-Resolution_CVPR_2024_paper.html +* Code: https://github.com/Lqm26/BMCNet-ESR + +**Boosting Flow-based Generative Super-Resolution Models via Learned Prior** +* Paper: https://arxiv.org/abs/2403.10988 +* Code: https://github.com/liyuantsao/FlowSR-LP + +**Building Bridges across Spatial and Temporal Resolutions: Reference-Based Super-Resolution via Change Priors and Conditional Diffusion Model** +* Paper: https://arxiv.org/abs/2403.17460 +* Code: https://github.com/dongrunmin/RefDiff + +**CAMixerSR: Only Details Need More “Attention”** +* Paper: https://arxiv.org/abs/2402.19289 +* Code: https://github.com/icandle/CAMixerSR + +**CFAT: Unleashing Triangular Windows for Image Super-resolution** +* Paper: https://arxiv.org/abs/2403.16143 +* Code: https://github.com/rayabhisek123/CFAT + +**Continuous Optical Zooming: A Benchmark for Arbitrary-Scale Image Super-Resolution in Real World** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Fu_Continuous_Optical_Zooming_A_Benchmark_for_Arbitrary-Scale_Image_Super-Resolution_in_CVPR_2024_paper.html +* Code: https://github.com/pf0607/COZ + +**CoSeR: Bridging Image and Language for Cognitive Super-Resolution** +* Paper: https://arxiv.org/abs/2311.16512 +* Code: https://github.com/VINHYU/CoSeR + +**CDFormer: When Degradation Prediction Embraces Diffusion Model for Blind Image Super-Resolution** +* Paper: https://arxiv.org/abs/2405.07648 +* Code: https://github.com/I2-Multimedia-Lab/CDFormer + +**CycleINR: Cycle Implicit Neural Representation for Arbitrary-Scale Volumetric Super-Resolution of Medical Data** +* Paper: https://arxiv.org/abs/2404.04878 +* Code: + +**Diffusion-based Blind Text Image Super-Resolution** +* Paper: https://arxiv.org/abs/2312.08886 +* Code: https://github.com/YuzheZhang-1999/DiffTSR + +**DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF** +* Paper: https://arxiv.org/abs/2404.00874 +* Code: + +**Image Processing GNN: Breaking Rigidity in Super-Resolution** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Tian_Image_Processing_GNN_Breaking_Rigidity_in_Super-Resolution_CVPR_2024_paper.html +* Code: https://github.com/huawei-noah/Efficient-Computing/tree/master/LowLevel/IPG + +**Latent Modulated Function for Computational Optimal Continuous Image Representation** +* Paper: https://arxiv.org/abs/2404.16451 +* Code: https://github.com/HeZongyao/LMF + +**Learning Coupled Dictionaries from Unpaired Data for Image Super-Resolution** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Wang_Learning_Coupled_Dictionaries_from_Unpaired_Data_for_Image_Super-Resolution_CVPR_2024_paper.html +* Code: + +**Learning Large-Factor EM Image Super-Resolution with Generative Priors** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shou_Learning_Large-Factor_EM_Image_Super-Resolution_with_Generative_Priors_CVPR_2024_paper.html +* Code: https://github.com/jtshou/GPEMSR + +**Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning** +* Paper: https://arxiv.org/abs/2403.02601 +* Code: https://github.com/haoyuc/LWay + +**Navigating Beyond Dropout: An Intriguing Solution towards Generalizable Image Super-Resolution** +* Paper: https://arxiv.org/abs/2402.18929v2 +* Code: https://github.com/Dreamzz5/Simple-Align + +**Neural Super-Resolution for Real-time Rendering with Radiance Demodulation** +* Paper: https://arxiv.org/abs/2308.06699 +* Code: https://github.com/Riga2/NSRD + +**Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution** +* Paper: https://arxiv.org/abs/2404.04785 +* Code: https://github.com/GuangYuanKK/DiffMSR + +**SeD: Semantic-Aware Discriminator for Image Super-Resolution** +* Paper: https://arxiv.org/abs/2402.19387 +* Code: https://github.com/lbc12345/SeD + +**SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution** +* Paper: https://arxiv.org/abs/2311.16518 +* Code: https://github.com/cswry/SeeSR + +**Self-Adaptive Reality-Guided Diffusion for Artifact-Free Super-Resolution** +* Paper: https://arxiv.org/abs/2403.16643 +* Code: https://github.com/ProAirVerse/Self-Adaptive-Guidance-Diffusion + +**SinSR: Diffusion-Based Image Super-Resolution in a Single Step** +* Paper: https://github.com/wyf0912/SinSR/blob/main/main.pdf +* Code: https://github.com/wyf0912/SinSR + +**Super-Resolution Reconstruction from Bayer-Pattern Spike Streams** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Dong_Super-Resolution_Reconstruction_from_Bayer-Pattern_Spike_Streams_CVPR_2024_paper.html +* Code: https://github.com/csycdong/CSCSR + +**Text-guided Explorable Image Super-resolution** +* Paper: https://arxiv.org/abs/2403.01124 +* Code: + +**Training Generative Image Super-Resolution Models by Wavelet-Domain Losses Enables Better Control of Artifacts** +* Paper: https://arxiv.org/abs/2402.19215 +* Code: https://github.com/mandalinadagi/wgsr + +**Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary** +* Paper: https://arxiv.org/abs/2401.08209 +* Code: https://github.com/LabShuHangGU/Adaptive-Token-Dictionary + +**Uncertainty-Aware Source-Free Adaptive Image Super-Resolution with Wavelet Augmentation Transformer** +* Paper: https://arxiv.org/abs/2303.17783 +* Code: + +**Universal Robustness via Median Randomized Smoothing for Real-World Super-Resolution** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chaouai_Universal_Robustness_via_Median_Randomized_Smoothing_for_Real-World_Super-Resolution_CVPR_2024_paper.html +* Code: + +### Video Super-Resolution +**Enhancing Video Super-Resolution via Implicit Resampling-based Alignment** +* Paper: https://github.com/kai422/IART/blob/main/arxiv.pdf +* Code: https://github.com/kai422/IART + +**FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring** +* Paper: https://arxiv.org/abs/2401.03707 +* Code: https://github.com/KAIST-VICLab/FMA-Net + +**Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution** +* Paper: https://arxiv.org/abs/2403.17000 +* Code: + +**Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution** +* Paper: https://arxiv.org/abs/2312.06640 +* Code: https://github.com/sczhou/Upscale-A-Video + +**Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention** +* Paper: https://arxiv.org/abs/2401.06312 +* Code: https://github.com/LabShuHangGU/MIA-VSR +## 2.图像去雨(Image Deraining) +**Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining** +* Paper: https://arxiv.org/abs/2404.01547 +* Code: https://github.com/cschenxiang/NeRD-Rain +## 3.图像去雾(Image Dehazing) +**A Semi-supervised Nighttime Dehazing Baseline with Spatial-Frequency Aware and Realistic Brightness Constraint** +* Paper: https://arxiv.org/abs/2403.18548 +* Code: https://github.com/Xiaofeng-life/SFSNiD + +**Depth Information Assisted Collaborative Mutual Promotion Network for Single Image Dehazing** +* Paper: https://arxiv.org/abs/2403.01105 +* Code: + +**ODCR: Orthogonal Decoupling Contrastive Regularization for Unpaired Image Dehazing** +* Paper: https://arxiv.org/abs/2404.17825v1 +* Code: + +### Video Dehazing +**Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Fan_Driving-Video_Dehazing_with_Non-Aligned_Regularization_for_Safety_Assistance_CVPR_2024_paper.html +* Code: + +## 4.去模糊(Deblurring) +**A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid Transformer and Contrastive Learning** +* Paper: https://arxiv.org/abs/2403.02611 +* Code: https://github.com/PieceZhang/MPT-CataBlur + +**AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurring** +* Paper: https://github.com/INVOKERer/AdaRevD/blob/master/AdaRevD.pdf +* Code: https://github.com/INVOKERer/AdaRevD + +**Blur2Blur: Blur Conversion for Unsupervised Image Deblurring on Unknown Domains** +* Paper: https://arxiv.org/abs/2403.16205 +* Code: https://github.com/VinAIResearch/Blur2Blur + +**Fourier Priors-Guided Diffusion for Zero-Shot Joint Low-Light Enhancement and Deblurring** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Lv_Fourier_Priors-Guided_Diffusion_for_Zero-Shot_Joint_Low-Light_Enhancement_and_Deblurring_CVPR_2024_paper.html +* Code: + +**ID-Blau: Image Deblurring by Implicit Diffusion-based reBLurring AUgmentation** +* Paper：https://arxiv.org/abs/2312.10998 +* Code: https://github.com/plusgood-steven/ID-Blau + +**LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network** +* Paper: https://arxiv.org/abs/2307.09815 +* Code: https://github.com/noxsine/LDP + +**Mitigating Motion Blur in Neural Radiance Fields with Events and Frames** +* Paper: https://rpg.ifi.uzh.ch/docs/CVPR24_Cannici.pdf +* Code: https://github.com/uzh-rpg/EvDeblurNeRF + +**Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring** +* Paper: https://arxiv.org/abs/2404.13153 +* Code: https://github.com/ChengxuLiu/MISCFilter + +**Motion Blur Decomposition with Cross-shutter Guidance** +* Paper: https://arxiv.org/abs/2404.01120 +* Code: https://github.com/jixiang2016/dualBR + +**Real-World Efficient Blind Motion Deblurring via Blur Pixel Discretization** +* Paper: https://arxiv.org/abs/2404.12168 +* Code: + +**Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Spike-guided_Motion_Deblurring_with_Unknown_Modal_Spatiotemporal_Alignment_CVPR_2024_paper.html +* Code: https://github.com/Leozhangjiyuan/UaSDN + +**Unsupervised Blind Image Deblurring Based on Self-Enhancement** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chen_Unsupervised_Blind_Image_Deblurring_Based_on_Self-Enhancement_CVPR_2024_paper.html +* Code: + +### Video Deblurring +**Blur-aware Spatio-temporal Sparse Transformer for Video Deblurring** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Blur-aware_Spatio-temporal_Sparse_Transformer_for_Video_Deblurring_CVPR_2024_paper.html +* Code: https://github.com/huicongzhang/BSSTNet + +**EVS-assisted Joint Deblurring Rolling-Shutter Correction and Video Frame Interpolation through Sensor Inverse Modeling** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Jiang_EVS-assisted_Joint_Deblurring_Rolling-Shutter_Correction_and_Video_Frame_Interpolation_through_CVPR_2024_paper.html +* Code: + +**Frequency-aware Event-based Video Deblurring for Real-World Motion Blur** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Kim_Frequency-aware_Event-based_Video_Deblurring_for_Real-World_Motion_Blur_CVPR_2024_paper.html +* Code: + +**Latency Correction for Event-guided Deblurring and Frame Interpolation** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Yang_Latency_Correction_for_Event-guided_Deblurring_and_Frame_Interpolation_CVPR_2024_paper.html +****Code: + +## 5.去噪(Denoising) +**LAN: Learning to Adapt Noise for Image Denoising** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Kim_LAN_Learning_to_Adapt_Noise_for_Image_Denoising_CVPR_2024_paper.html +* Code: + +**LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising** +* Paper: https://arxiv.org/abs/2405.19718 +* Code: + +**Robust Image Denoising through Adversarial Frequency Mixup** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ryou_Robust_Image_Denoising_through_Adversarial_Frequency_Mixup_CVPR_2024_paper.html +* Code: https://github.com/dhryougit/AFM + +**Real-World Mobile Image Denoising Dataset with Efficient Baselines** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Flepp_Real-World_Mobile_Image_Denoising_Dataset_with_Efficient_Baselines_CVPR_2024_paper.html +* Code: + +**SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder** +* Paper: https://arxiv.org/abs/2403.17502 +* Code: https://github.com/zhengdharia/SeNM-VAE + +**Transfer CLIP for Generalizable Image Denoising** +* Paper: https://arxiv.org/abs/2403.15132 +* Code: + +**Unmixing Diffusion for Self-Supervised Hyperspectral Image Denoising** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zeng_Unmixing_Diffusion_for_Self-Supervised_Hyperspectral_Image_Denoising_CVPR_2024_paper.html +* Code: + +**ZERO-IG: Zero-Shot Illumination-Guided Joint Denoising and Adaptive Enhancement for Low-Light Images** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shi_ZERO-IG_Zero-Shot_Illumination-Guided_Joint_Denoising_and_Adaptive_Enhancement_for_Low-Light_CVPR_2024_paper.html +* Code: https://github.com/Doyle59217/ZeroIG + +## 6.图像恢复(Image Restoration) +**Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhou_Adapt_or_Perish_Adaptive_Sparse_Transformer_with_Attentive_Feature_Refinement_CVPR_2024_paper.html +* Code: https://github.com/joshyZhou/AST + +**Boosting Image Restoration via Priors from Pre-trained Models** +* Paper: https://arxiv.org/abs/2403.06793 +* Code: + +**CoDe: An Explicit Content Decoupling Framework for Image Restoration** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Gu_CoDe_An_Explicit_Content_Decoupling_Framework_for_Image_Restoration_CVPR_2024_paper.html +* Code: + +**Deep Equilibrium Diffusion Restoration with Parallel Sampling** +* Paper: https://arxiv.org/abs/2311.11600 +* Code: https://github.com/caojiezhang/DeqIR + +**Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks** +* Paper: https://arxiv.org/abs/2403.00644 +* Code: https://github.com/yuhaoliu7456/Diff-Plugin + +**Distilling Semantic Priors from SAM to Efficient Image Restoration Models** +* Paper: https://arxiv.org/abs/2403.16368 +* Code: + +**DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks** +* Paper: https://arxiv.org/abs/2405.04408 +* Code: https://github.com/ZZZHANG-jx/DocRes + +**HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models** +* Paper: https://arxiv.org/abs/2402.15865 +* Code: https://github.com/LiPang/HIRDiff + +**Image Restoration by Denoising Diffusion Models With Iteratively Preconditioned Guidance** +* Paper: https://arxiv.org/abs/2312.16519 +* Code: https://github.com/tirer-lab/DDPG + +**Improving Image Restoration through Removing Degradations in Textual Representations** +* Paper: https://arxiv.org/abs/2312.17334 +* Code: https://github.com/mrluin/TextualDegRemoval + +**Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Xie_Learning_Degradation-unaware_Representation_with_Prior-based_Latent_Transformations_for_Blind_Face_CVPR_2024_paper.html +* Code: + +**Learning Diffusion Texture Priors for Image Restoration** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ye_Learning_Diffusion_Texture_Priors_for_Image_Restoration_CVPR_2024_paper.html +* Code: + +**Look-Up Table Compression for Efficient Image Restoration** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Li_Look-Up_Table_Compression_for_Efficient_Image_Restoration_CVPR_2024_paper.html +* Code: + +**Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration** +* Paper: https://arxiv.org/abs/2312.02918 +* Code: + +**PFStorer: Personalized Face Restoration and Super-Resolution** +* Paper: https://arxiv.org/abs/2403.08436 +* Code: + +**Restoration by Generation with Constrained Priors** +* Paper: https://arxiv.org/abs/2312.17161 +* Code: + +**Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild** +* Paper: https://arxiv.org/abs/2401.13627 +* Code: https://github.com/Fanghua-Yu/SUPIR + +**Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model** +* Paper: https://arxiv.org/abs/2403.11157 +* Code: https://github.com/iSEE-Laboratory/DiffUIR + +**Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence** +* Paper: https://arxiv.org/abs/2404.13605 +Code: https://github.com/Riponcs/Turb-Seg-Res + +**WaveFace: Authentic Face Restoration with Efficient Frequency Recovery** +* Paper: https://arxiv.org/abs/2403.12760 +* Code: + +**Wavelet-based Fourier Information Interaction with Frequency Diffusion Adjustment for Underwater Image Restoration** +* Paper: https://arxiv.org/abs/2311.16845 +* Code: https://github.com/zhihefang/wf-diff + +## 7.图像增强(Image Enhancement) +**Color Shift Estimation-and-Correction for Image Enhancement** +* Paper: https://drive.google.com/file/d/1jZB2rW_I2WLTE5yNA4IZq9wb5p4NNOCR/view +* Code: https://github.com/yiyulics/CSEC + +**Empowering Resampling Operation for Ultra-High-Definition Image Enhancement with Model-Aware Guidance** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Yu_Empowering_Resampling_Operation_for_Ultra-High-Definition_Image_Enhancement_with_Model-Aware_Guidance_CVPR_2024_paper.html +* Code: https://github.com/YPatrickW/LMAR + +**Fourier Priors-Guided Diffusion for Zero-Shot Joint Low-Light Enhancement and Deblurring** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Lv_Fourier_Priors-Guided_Diffusion_for_Zero-Shot_Joint_Low-Light_Enhancement_and_Deblurring_CVPR_2024_paper.html +* Code: + +**FlowIE：Efficient Image Enhancement via Rectified Flow** +* Paper: https://arxiv.org/abs/2406.00508 +* Code: https://github.com/EternalEvan/FlowIE + +**Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving** +* Paper: https://arxiv.org/abs/2404.04804 +* Code: https://github.com/jinlong17/LightDiff + +**Robust Depth Enhancement via Polarization Prompt Fusion Tuning** +* Paper: https://arxiv.org/abs/2404.04318 +* Code: https://github.com/lastbasket/Polarization-Prompt-Fusion-Tuning + +**Specularity Factorization for Low Light Enhancement** +* Paper: https://arxiv.org/abs/2404.01998 +* Code: + +**Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach** +* Paper: https://arxiv.org/abs/2404.00834 +* Code: https://github.com/EthanLiang99/EvLight + +**ZERO-IG: Zero-Shot Illumination-Guided Joint Denoising and Adaptive Enhancement for Low-Light Images** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shi_ZERO-IG_Zero-Shot_Illumination-Guided_Joint_Denoising_and_Adaptive_Enhancement_for_Low-Light_CVPR_2024_paper.html +* Code: https://github.com/Doyle59217/ZeroIG + +**Zero-Reference Low-Light Enhancement via Physical Quadruple Priors** +* Paper: https://arxiv.org/abs/2403.12933 +* Code: https://github.com/daooshee/QuadPrior + +### Video Enhancement +**Binarized Low-light Raw Video Enhancement** +* Paper: https://arxiv.org/abs/2403.19944 +* Code: https://github.com/zhanggengchen/BRVE + +**UVEB: A Large-scale Benchmark and Baseline Towards Real-World Underwater Video Enhancement** +* Paper: https://arxiv.org/abs/2404.14542 +* Code: https://github.com/yzbouc/UVEB + +## 8.图像修复(Inpainting) +**Amodal Completion via Progressive Mixed Context Diffusion** +* Paper: https://arxiv.org/abs/2312.15540 +* Code: https://github.com/k8xu/amodal + +**Brush2Prompt: Contextual Prompt Generator for Object Inpainting** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chiu_Brush2Prompt_Contextual_Prompt_Generator_for_Object_Inpainting_CVPR_2024_paper.html +* Code: + +**Don’t Look into the Dark: Latent Codes for Pluralistic Image Inpainting** +* Paper: https://arxiv.org/abs/2403.18186 +* Code: + +**Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting** +* Paper: https://arxiv.org/abs/2403.19898 +* Code: https://github.com/htyjers/StrDiffusion + +### Video Inpainting +**AVID: Any-Length Video Inpainting with Diffusion Model** +* Paper: https://arxiv.org/abs/2312.03816 +* Code: https://github.com/zhang-zx/AVID + +**Towards Language-Driven Video Inpainting via Multimodal Large Language Models** +* Paper: https://arxiv.org/abs/2401.10226 +* Code: https://github.com/jianzongwu/Language-Driven-Video-Inpainting + +## 9.高动态范围成像(HDR Imaging) +**CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment** +* Paper: https://arxiv.org/abs/2404.01123 +* Code: https://github.com/hmin970922/CLIPtone/ + +**Deep Video Inverse Tone Mapping Based on Temporal Clues** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ye_Deep_Video_Inverse_Tone_Mapping_Based_on_Temporal_Clues_CVPR_2024_paper.html +* Code: https://github.com/ye3why/VITM-TC + +**Generating Content for HDR Deghosting from Frequency View** +* Paper: https://arxiv.org/abs/2404.00849 +* Code: + +**HDRFlow: Real-Time HDR Video Reconstruction with Large Motions** +* Paper: https://arxiv.org/abs/2403.03447 +* Code: https://github.com/OpenImagingLab/HDRFlow + +**Perceptual Assessment and Optimization of HDR Image Rendering** +* Paper: https://arxiv.org/abs/2310.12877v4 +* Code: https://github.com/cpb68/HDRQA/ + +**Towards HDR and HFR Video from Rolling-Mixed-Bit Spikings** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chang_Towards_HDR_and_HFR_Video_from_Rolling-Mixed-Bit_Spikings_CVPR_2024_paper.html +* Code: + +**Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network** +* Paper: https://arxiv.org/abs/2405.00244 +* Code: https://github.com/yungsyu99/Real-HDRV + +**Zero-Shot Structure-Preserving Diffusion Model for High Dynamic Range Tone Mapping** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhu_Zero-Shot_Structure-Preserving_Diffusion_Model_for_High_Dynamic_Range_Tone_Mapping_CVPR_2024_paper.html +* Code: + +## 10.图像质量评价(Image Quality Assessment) +**Blind Image Quality Assessment Based on Geometric Order Learning** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shin_Blind_Image_Quality_Assessment_Based_on_Geometric_Order_Learning_CVPR_2024_paper.html +* Code: https://github.com/nhshin-mcl/QCN + +**Boosting Image Quality Assessment through Efficient Transformer Adaptation with Local Feature Enhancement** +* Paper: https://arxiv.org/abs/2308.12001 +* Code: + +**Bridging the Synthetic-to-Authentic Gap: Distortion-Guided Unsupervised Domain Adaptation for Blind Image Quality Assessment** +* Paper: https://arxiv.org/abs/2405.04167 +* Code: + +**CLIB-FIQA: Face Image Quality Assessment with Confidence Calibration** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ou_CLIB-FIQA_Face_Image_Quality_Assessment_with_Confidence_Calibration_CVPR_2024_paper.html +* Code: + +**Contrastive Pre-Training with Multi-View Fusion for No-Reference Point Cloud Quality Assessment** +* Paper: https://arxiv.org/abs/2403.10066 +* Code: + +**Deep Generative Model based Rate-Distortion for Image Downscaling Assessment** +* Paper: https://arxiv.org/abs/2403.15139 +* Code: https://github.com/Byronliang8/IDA-RD + +**Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization** +* Paper: https://arxiv.org/abs/2403.11397 +* Code: https://github.com/YangiD/DefenseIQA-NT + +**DSL-FIQA: Assessing Facial Image Quality via Dual-Set Degradation Learning and Landmark-Guided Transformer** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chen_DSL-FIQA_Assessing_Facial_Image_Quality_via_Dual-Set_Degradation_Learning_and_CVPR_2024_paper.html +* Code: + +**EvalCrafter: Benchmarking and Evaluating Large Video Generation Models** +* Paper: https://arxiv.org/abs/2310.11440 +* Code: https://github.com/evalcrafter/EvalCrafter + +**FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment** +* Paper: https://arxiv.org/abs/2405.06887 +* Code: https://github.com/PKU-ICST-MIPL/FineParser_CVPR2024 + +**KVQ: Kwai Video Quality Assessment for Short-form Videos** +* Paper: https://arxiv.org/abs/2402.07220 +* Code: https://github.com/lixinustc/KVQ-Challenge-CVPR-NTIRE2024 + +**Learned Scanpaths Aid Blind Panoramic Video Quality Assessment** +* Paper: https://arxiv.org/abs/2404.00252 +* Code: https://github.com/kalofan/AutoScanpathQA + +**Modular Blind Video Quality Assessment** +* Paper: https://arxiv.org/abs/2402.19276 +* Code: https://github.com/winwinwenwen77/ModularBVQA + +**On the Content Bias in Fréchet Video Distance** +* Paper: https://arxiv.org/abs/2404.12391 +* Code: https://github.com/songweige/content-debiased-fvd + +**PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wild** +* Paper: https://arxiv.org/abs/2405.17765 +* Code: + +**Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models** +* Paper: https://arxiv.org/abs/2311.06783 +* Code: https://github.com/Q-Future/Q-Instruct + +## 11.插帧(Frame Interpolation) +**Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images** +Paper: https://arxiv.org/abs/2404.01464 +Code: https://github.com/jungeun122333/UVI-Net + +**IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Hu_IQ-VFI_Implicit_Quadratic_Motion_Estimation_for_Video_Frame_Interpolation_CVPR_2024_paper.html +* Code: + +**Perceptual-Oriented Video Frame Interpolation Via Asymmetric Synergistic Blending** +* Paper: https://arxiv.org/abs/2404.06692 +* Code: + +**Sparse Global Matching for Video Frame Interpolation with Large Motion** +* Paper: https://arxiv.org/abs/2404.06913 +* Code: https://github.com/MCG-NJU/SGM-VFI + +**SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation** +* Paper: https://arxiv.org/abs/2308.16876 +* Code: https://github.com/neu-vi/SportsSloMo + +**TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Cho_TTA-EVF_Test-Time_Adaptation_for_Event-based_Video_Frame_Interpolation_via_Reliable_CVPR_2024_paper.html +* Code: https://github.com/Chohoonhee/TTA-EVF + +**Video Frame Interpolation via Direct Synthesis with the Event-based Reference** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Liu_Video_Frame_Interpolation_via_Direct_Synthesis_with_the_Event-based_Reference_CVPR_2024_paper.html +* Code: + +**Video Interpolation with Diffusion Models** +* Paper: https://arxiv.org/abs/2404.01203 +* Code: + +## 12.视频/图像压缩(Video/Image Compression) +**C3: High-performance and low-complexity neural compression from a single image or video** +* Paper: https://arxiv.org/abs/2312.02753 +* Code: https://github.com/google-deepmind/c3_neural_compression + +**Generative Latent Coding for Ultra-Low Bitrate Image Compression** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Jia_Generative_Latent_Coding_for_Ultra-Low_Bitrate_Image_Compression_CVPR_2024_paper.html +* Code: + +**Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis** +* Paper: https://arxiv.org/abs/2403.16258 +* Code: + +**Learned Lossless Image Compression based on Bit Plane Slicing** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Learned_Lossless_Image_Compression_based_on_Bit_Plane_Slicing_CVPR_2024_paper.html +* Code: https://github.com/ZZ022/ArIB-BPS + +**Towards Backward-Compatible Continual Learning of Image Compression** +* Paper: https://arxiv.org/abs/2402.18862 +* Code: https://gitlab.com/viper-purdue/continual-compression + +### Video Compression +**Task-Aware Encoder Control for Deep Video Compression** +* Paper: https://arxiv.org/abs/2404.04848 +* Code: + +**Low-Latency Neural Stereo Streaming** +* Paper: https://arxiv.org/abs/2403.17879 +* Code: + +**Neural Video Compression with Feature Modulation** +* Paper: https://arxiv.org/abs/2402.17414 +* Code: https://github.com/microsoft/DCVC + +## 13.压缩图像质量增强(Compressed Image Quality Enhancement) +**CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement** +* Paper: https://arxiv.org/abs/2403.10362 +* Code: + +**Enhancing Quality of Compressed Images by Mitigating Enhancement Bias Towards Compression Domain** +* Paper: https://arxiv.org/abs/2402.17200 +* Code: + +## 14.图像去反光(Image Reflection Removal) +**Language-guided Image Reflection Separation** +* Paper: https://arxiv.org/abs/2402.11874 +* Code: + +**Revisiting Singlelmage Reflection Removal in the Wild** +* Paper: https://arxiv.org/abs/2311.17320 +* Code: https://github.com/zhuyr97/Reflection_RemoVal_CVPR2024 + +## 15.图像去阴影(Image Shadow Removal) +**HomoFormer: Homogenized Transformer for Image Shadow Removal** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Xiao_HomoFormer_Homogenized_Transformer_for_Image_Shadow_Removal_CVPR_2024_paper.html +* Code: https://github.com/jiexiaou/HomoFormer + +## 16.图像上色(Image Colorization) +**Automatic Controllable Colorization by Imagination** +* Paper: https://arxiv.org/abs/2404.05661 +* Code: https://github.com/xy-cong/imagine-colorization + +**Generative Quanta Color Imaging** +* Paper: https://arxiv.org/abs/2403.19066 +* Code: + +**Learning Inclusion Matching for Animation Paint Bucket Colorization** +* Paper: https://arxiv.org/abs/2403.18342 +* Code: https://github.com/ykdai/BasicPBC + +## 17.图像和谐化(Image Harmonization) +**Relightful Harmonization: Lighting-aware Portrait Background Replacement** +* Paper: https://arxiv.org/abs/2312.06886 +* Code: + +**Video Harmonization with Triplet Spatio-Temporal Variation Patterns** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Guo_Video_Harmonization_with_Triplet_Spatio-Temporal_Variation_Patterns_CVPR_2024_paper.html +* Code: https://github.com/zhenglab/VideoTripletTransformer +## 18.视频稳相(Video Stabilization) +**3D Multi-frame Fusion for Video Stabilization** +* Paper: https://arxiv.org/abs/2404.12887 +* Code: + +**Harnessing Meta-Learning for Improving Full-Frame Video Stabilization** +* Paper: https://arxiv.org/abs/2403.03662 +* Code: https://github.com/MKashifAli/MetaVideoStab + +## 19.图像融合(Image Fusion) +**Equivariant Multi-Modality Image Fusion** +* Paper： https://arxiv.org/abs/2305.11443 +* Code: https://github.com/Zhaozixiang1228/MMIF-EMMA + +**MRFS: Mutually Reinforcing Image Fusion and Segmentation** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_MRFS_Mutually_Reinforcing_Image_Fusion_and_Segmentation_CVPR_2024_paper.html +* Code: https://github.com/HaoZhang1018/MRFS + +**Neural Spline Fields for Burst Image Fusion and Layer Separation** +* Paper: https://arxiv.org/abs/2312.14235 +* Code: https://github.com/princeton-computational-imaging/NSF + +**Probing Synergistic High-Order Interaction in Infrared and Visible Image Fusion** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zheng_Probing_Synergistic_High-Order_Interaction_in_Infrared_and_Visible_Image_Fusion_CVPR_2024_paper.html +* Code: + +**Revisiting Spatial-Frequency Information Integration from a Hierarchical Perspective for Panchromatic and Multi-Spectral Image Fusion** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zheng_Probing_Synergistic_High-Order_Interaction_in_Infrared_and_Visible_Image_Fusion_CVPR_2024_paper.html +* Code: + +**Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion** +* Paper: https://arxiv.org/abs/2403.16387 +* Code: https://github.com/XunpengYi/Text-IF + +**Task-Customized Mixture of Adapters for General Image Fusion** +* Paper: https://arxiv.org/abs/2403.12494 +* Code: https://github.com/YangSun22/TC-MoA + +## 20.其他任务(Others) +**Close Imitation of Expert Retouching for Black-and-White Photography** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shin_Close_Imitation_of_Expert_Retouching_for_Black-and-White_Photography_CVPR_2024_paper.html +* Code: https://github.com/seunghyuns98/Decolorization + +**Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening** +Paper: https://arxiv.org/abs/2404.07543 +Code: https://github.com/Duanyll/CANConv + +**DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model** +* Paper: https://arxiv.org/abs/2311.11417 +* Code: https://github.com/PAN083/DiffSCI + +**Dual Prior Unfolding for Snapshot Compressive Imaging** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Dual_Prior_Unfolding_for_Snapshot_Compressive_Imaging_CVPR_2024_paper.html +* Code: https://github.com/ZhangJC-2k/DPU + +**Dual-Camera Smooth Zoom on Mobile Phones** +* Paper: https://arxiv.org/abs/2404.04908 +* Code: https://github.com/ZcsrenlongZ/ZoomGS + +**Dual-scale Transformer for Large-scale Single-Pixel Imaging** +* Paper: https://arxiv.org/abs/2404.05001 +* Code: https://github.com/Gang-Qu/HATNet-SPI + +**Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal** +* Paper: https://arxiv.org/abs/2403.07684 +* Code: https://github.com/scott-yjyang/DiffTTA + +**Language-driven All-in-one Adverse Weather Removal** +* Paper: https://arxiv.org/abs/2312.01381 +* Code: + +**Learning to Remove Wrinkled Transparent Film with Polarized Prior** +* Paper: https://arxiv.org/abs/2403.04368 +* Code: https://github.com/jqtangust/FilmRemovalww + +**Misalignment-Robust Frequency Distribution Loss for Image Transformation** +* Paper: https://arxiv.org/abs/2402.18192 +* Code: https://github.com/eezkni/FDL + +**On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation** +* Paper: https://arxiv.org/abs/2404.08540 +* Code: https://github.com/agneet42/lang_depth + +**ParamISP: Learned Forward and Inverse ISPs using Camera Parameters** +* Paper: https://arxiv.org/abs/2312.13313 +* Code: https://github.com/woo525/ParamISP + +**RecDiffusion: Rectangling for Image Stitching with Diffusion Models** +* Paper: https://arxiv.org/abs/2402.18192 +* Code: https://github.com/lhaippp/RecDiffusion + +**Residual Denoising Diffusion Models** +* Paper: https://arxiv.org/abs/2308.13712 +* Code: https://github.com/nachifur/RDDM + +**Real-Time Exposure Correction via Collaborative Transformations and Adaptive Sampling** +* Paper: https://arxiv.org/abs/2404.11884 +* Code: https://github.com/HUST-IAL/CoTF + +**SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image** +* Paper: https://arxiv.org/abs/2403.20018 +* Code: https://github.com/WU-CVGL/SCINeRF + +**Seeing Motion at Nighttime with an Event Camera** +* Paper: https://arxiv.org/abs/2404.11884 +* Code: https://github.com/Liu-haoyue/NER-Net + +**Shadow Generation for Composite Image Using Diffusion Model** +* Paper: https://arxiv.org/abs/2403.15234 +* Code: https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2 + +**Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification** +* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Improving_Spectral_Snapshot_Reconstruction_with_Spectral-Spatial_Rectification_CVPR_2024_paper.html +* Code: https://github.com/ZhangJC-2k/SSR + +## 参考 +相关Low-Level-Vision整理 +Awesome-CVPR2020-Low-Level-Vision +Awesome-ECCV2020-Low-Level-Vision +Awesome-Low-Level-Vision-Research-Groups +Awesome-AIGC-Research-Groups diff --git a/Docs/2024-11-15/report_1115.md b/Docs/2024-11-15/report_1115.md new file mode 100644 index 0000000..889ed3a --- /dev/null +++ b/Docs/2024-11-15/report_1115.md @@ -0,0 +1,40 @@ +# 2024-11-15感知智能组会汇报 +# 一、近期工作 + **Image Generation** vs **Image Reconstruction** + * **Image Generation**：指通过算法从零开始生成新的图像，通常基于一些输入条件（如文本描述、特定样式、语义信息）或随机噪声（如生成对抗网络的输入）。目的是创造出逼真的、具有特定特征的图像。 + * **无条件生成**是指无条件地从数据集中生成样本，即：$p(y)$； + * **条件图像生成**（子任务）是指根据标签有条件地从数据集中生成样本，即：$p(y|x)$ + * **Image Reconstruction**：指从已有的、可能损坏、不完整或压缩的图像数据中恢复原始图像，或者从观测数据中重建图像。目标是尽可能还原出真实图像。 + +| **特性** | 图像生成 (Image Generation) | 图像重建 (Image Reconstruction) | +|--------|------|--------| +| **输入** | 随机噪声、文本或条件标签等 | 受损图像、不完整数据（如低分辨率图像、部分丢失的像素、模糊图像） | +| **输出** | 全新的图像，可能是艺术性的、合成的或逼真的 | 修复后的图像，接近原始清晰图像 | +| **目标** | 目标是创造性地生成新图像，关注生成图像的多样性和真实性；注重图像的视觉质量、逼真度以及与输入条件的匹配度。 | 目标是恢复或重建图像，尽可能减少噪声、模糊和失真；注重恢复的准确性和保真度，强调与真实图像的接近程度。 | +| **技术方法** | 典型方法包括生成对抗网络（GANs）、变分自编码器（VAEs）、扩散模型、条件生成模型等 | 通常基于优化和重建技术，包括卷积神经网络（CNNs）、自编码器（Autoencoders）、逆问题求解方法（如去噪、自适应插值）、正则化技术等 | +| **应用场景** | 数字艺术与内容创作（如DALL·E、Stable Diffusion）、数据增强（为训练AI生成多样化样本）、虚拟世界构建（如游戏、元宇宙）、图像翻译（风格迁移、照片转漫画） | 医学成像（如CT/MRI图像修复）、摄影中的图像去噪、超分辨率重建、遥感影像处理（如卫星图像云层去除）、逆向工程（如压缩图片的质量恢复） | + +图像生成更侧重创新性，图像重建更注重还原性，但两者都在提升图像质量和智能处理方面发挥着重要作用。 + +我们所希望做的特征还原/图像补全/遮挡还原更倾向于**Image Reconstruction**部分的内容，但可以同时借鉴Generation和Reconstruction两个部分的内容展开。 +* 从**Image Generation**的角度：关注补全的多样性和合理性 + * 全局理解： + * MAE需要通过观察未遮挡部分来推测被遮挡区域的内容，这需要全局上下文信息的支持，与图像生成中的全局一致性思路相似。 + * 补全结果不仅要像原图，还需要自然、符合上下文逻辑。 + * 潜在表征学习： + * MAE利用自监督学习，类似生成任务中的生成网络，通过学习隐藏空间（latent space）中的表征来预测缺失区域的可能内容。 + * 去噪补全： + * 与扩散模型类似，MAE可以从不确定性的潜在空间中逐步补全遮挡区域。 +* 从**Image Reconstruction**的角度：注重还原真实性 + * 输入部分： + * MAE的输入是遮挡了部分像素的图像，这与图像重建任务中不完整图像作为输入相似。 + * 模型需要从局部上下文信息中恢复被遮挡的区域，这与图像重建强调从不完整数据中提取有用信息的思路一致。 + * 目标： + * 恢复缺失的像素，使得补全后的图像尽可能接近原始图像，注重重建的保真度。 + * MAE的设计目标是通过有效的编码和解码过程，使模型学会图像的局部与整体关系。 + +为了实现我们想要的部分，最好是找reconstruction领域去finetune的合理方法以及在generation领域的创新模块表示。除此之外，在segmentation领域找到优化的方案。 + + +# 二、未来规划 +阅读《Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting》以及所总结的和自己相关方向的论文 \ No newline at end of file diff --git a/Docs/2024-11-22/Structure_Matters:_Tackling_the_Semantic_Discrepancy_in_Diffusion_Models_for_Image_Inpainting.md b/Docs/2024-11-22/Structure_Matters:_Tackling_the_Semantic_Discrepancy_in_Diffusion_Models_for_Image_Inpainting.md new file mode 100644 index 0000000..8d5309b --- /dev/null +++ b/Docs/2024-11-22/Structure_Matters:_Tackling_the_Semantic_Discrepancy_in_Diffusion_Models_for_Image_Inpainting.md @@ -0,0 +1,26 @@ +# Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting +* Paper Link:https://openaccess.thecvf.com/content/CVPR2024/papers/Liu_Structure_Matters_Tackling_the_Semantic_Discrepancy_in_Diffusion_Models_for_CVPR_2024_paper.pdf +* Code Link:https://github.com/htyjers/StrDiffusion + +## Abstract +图像修复中的去噪扩散概率模型（DDPMs）旨在正向修复过程中向图像纹理中添加噪声，反向修复过程中利用纹理未被掩盖区域恢复被掩盖区域去噪过程。尽管进行了有意义的语义生成，但现有的方法存在被屏蔽和未屏蔽区域之间的语义差异，因为语义密集的未屏蔽纹理在扩散过程中没有被完全退化，而被屏蔽区域在扩散过程中变成了纯噪声，导致它们之间存在较大的差异。本文旨在回答未掩码语义如何指导纹理去噪过程；以及如何解决语义差异，以促进一致和有意义的语义生成。为此，本文提出了一种基于结构引导的图像修复扩散模型StrDiffusion，该模型在结构指导下对传统的纹理去噪过程进行重新表述，从而得到一个简化的图像修复去噪目标，同时揭示了： +* 语义稀疏的结构有利于早期解决语义差异问题，而稠密的纹理则能在后期生成合理的语义； +* 得益于结构语义的时变稀疏性，未掩盖区域的语义本质上为纹理去噪过程提供了时变结构指导。在去噪过程中，通过利用掩码区域和未掩码区域之间的去噪结构的一致性，训练一个结构引导的神经网络来估计简化的去噪目标。 +* 此外，设计了一种自适应重采样策略，作为结构是否能够指导纹理去噪过程的形式化标准，同时调节其语义相关性。通过大量实验验证了StrDiffusion算法相对于现有算法的优点。 +## 1.Introduction +最近，图像修复支持了广泛的应用，例如照片恢复和图像编辑，其目的是用未掩蔽区域的语义信息恢复图像的掩蔽区域，其原理主要涵盖两个方面：掩蔽区域的合理语义及其与未掩蔽区域语义的一致性。现有的工作主要涉及基于扩散的[2,8]和基于补丁的[1,4,10]方案，该方案倾向于通过简单的颜色信息输入图像的小掩模或重复图案，而无法处理不规则或复杂的掩模。为了解决这个问题，大量的注意力[15,17,31,36]已经转移到卷积神经网络（CNN）上，它致力于在掩码区域周围对局部语义进行编码，而忽略来自未掩码区域的全局信息，导致远离掩码边界的区域保持毛茸。最近，自我注意机制[5，14，16，32，35]被提出以分割图像块的形式将掩蔽区域与未掩蔽区域全局关联，增强了它们之间的语义一致性。然而，这种策略在掩蔽区域内不同斑块之间的语义相关性较差。为此，利用语义稀疏结构[3，7，9，13，18，20，27，28，33，34]来增强它们的相关性，然而，这意味着严重依赖结构和纹理之间的语义一致性，因此不可避免地在内嵌输出中带有伪影。 + +幸运的是，去噪扩散概率模型（DDPM）[12,25]已成为强大的生成模型，在语义生成和模式收敛方面取得了显著进展，从而很好地弥补了图像修复语义生成不佳的问题[22,23,29,38]。具体来说，[22]建议采用预训练的DDPM作为先验，而不是专注于训练过程，并开发一种重采样策略来调节推理过程中的反向去噪过程。此外，[23]试图通过利用未掩蔽区域的语义来模拟图像修复的扩散过程，从而为去噪过程提供最佳的反向解决方案。这些方法大多表现出具有DDPM优势的语义有意义的修复结果，而忽略了掩蔽和未掩蔽区域之间的语义一致性。直觉是，语义密集的无掩模纹理退化为无掩模结构和高斯噪声的组合，而掩模区域在扩散过程中变成了纯噪声，导致它们之间存在很大差异（见图1（a）和图2（a）），并引发了以下问题： +* 1）*无掩模语义如何指导图像修复的纹理去噪过程？*我们的激励实验建议，当未掩蔽的语义与噪声相结合变得更稀疏时，例如，利用掩蔽图像的灰度或边缘图作为替代，差异问题在很大程度上得到了缓解，同时，在修复结果中损失了大量的语义信息；见图2（b）（c）。因此，未掩蔽区域随时间变化的不变语义无法很好地指导纹理去噪过程。 +* 2）在 (1) 之后，人们自然会想知道*如何产生具有一致且有意义的语义的去噪结果*。很明显，稀疏结构在早期阶段有利于语义一致性，而密集纹理在去噪过程中倾向于在后期生成有意义的语义，这意味着去噪结果的一致性和合理语义之间的平衡。为了进一步产生理想的结果，我们将结构的引导视为纹理去噪过程的辅助；见图1(B) +## 2.Structure-Guided Texture Diffusion Models + +### 2.1 Preliminaries: Denoising Diffusion Probabilistic Models for Image Inpainting + +### 2.2 Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting + +#### 2.2.1 Structure Matters: Sparse Semantics benefits the Semantic Consistency + +#### 2.2.2 Optimal Solution to the Denoising Process Under the Guidance of the Structure + +### 2.3 Structure-Guided Denoising Process: How does the Structure Guide the Texture Denoising Process? \ No newline at end of file