UMat: Uncertainty-Aware Single Image High Resolution Material Capture

UMat estimates high-resolution surface normals, specularity, and roughness from a single diffuse image of a material, taken with a flatbed scanner. UMat introduces a novel framework to quantify the model’s confidence about its prediction at test time, the first of its kind in material digitization.

Poster

Abstract

We propose a learning-based method to recover normals, specularity, and roughness from a single diffuse image of a material, using microgeometry appearance as our primary cue. Previous methods that work on single images tend to produce over-smooth outputs with artifacts, operate at limited resolution, or train one model per class with little room for generalization. In contrast, in this work, we propose a novel capture approach that leverages a generative network with attention and a U-Net discriminator, which shows outstanding performance integrating global information at reduced computational complexity. We showcase the performance of our method with a real dataset of digitized textile materials and show that a commodity flatbed scanner can produce the type of diffuse illumination required as input to our method. Additionally, because the problem might be illposed –more than a single diffuse image might be needed to disambiguate the specular reflection– or because the training dataset is not representative enough of the real distribution, we propose a novel framework to quantify the model’s confidence about its prediction at test time. Our method is the first one to deal with the problem of modeling uncertainty in material digitization, increasing the trustworthiness of the process and enabling more intelligent strategies for dataset creation, as we demonstrate with an active learning experiment.

Model Design & Uncertainty Quantification

UMat is a GAN specifically tailored for material digitization. We use Monte Carlo dropout to measure material digitization uncertainty, which we leverage for an active learning experiment.

Dataset

Evaluation

Our model provides sharp, accurate and artifact-free estimations. Our uncertainty metric correlates with render error. Using uncertainty sampling, we can build datasets more efficiently.

Qualitative Results

UMat outputs accurate SVBRDFs at very high resolutions.

Interactive Visualization

Please use the sliders to check the model estimations.

Citation

@inproceedings{Rodriguez-Pardo_2023_CVPR,
    author    = {Rodriguez-Pardo, Carlos and Dominguez-Elvira, Henar and Pascual-Hernandez, David and Garces, Elena},
    title     = {UMat: Uncertainty-Aware Single Image High Resolution Material Capture},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {5764-5774}
}

Acknowledgements

Elena Garces was partially supported by a Juan de la Cierva - Incorporacion Fellowship (IJC2020-044192-I). We thank Jorge Lopez-Moreno and Dan Casas for valuable discussions, and Sofía Domínguez for helping build the dataset.