News
- June, 2022 : SeamlessGAN is accepted as an invited presentation AI for Content Creation Workshop on CVPR 2022.
- May, 2022 : We presented SeamlessGAN as an invited talk on MLT __init__.
- March, 2022 : SeamlessGAN post on Reddit r/MachineLearning.
- January, 2022 : Web launched.
Abstract
We present SeamlessGAN, a method capable of automatically generating tileable texture maps from a single input exemplar. In contrast to most existing methods, focused solely on solving the synthesis problem, our work tackles both problems, synthesis and tileability, simultaneously. Our key idea is to realize that tiling a latent space within a generative network trained using adversarial expansion techniques produces outputs with continuity at the seam intersection that can be then be turned into tileable images by cropping the central area. Since not every value of the latent space is valid to produce high-quality outputs, we leverage the discriminator as a perceptual error metric capable of identifying artifact-free textures during a sampling process. Further, in contrast to previous work on deep texture synthesis, our model is designed and optimized to work with multi-layered texture representations, enabling textures composed of multiple maps such as albedo, normals, etc. We extensively test our design choices for the network architecture, loss function and sampling parameters. We show qualitatively and quantitatively that our approach outperforms previous methods and works for textures of different types.
Resources
- CVPR 2022 Poster (AI for Content Creation Workshop) [PDF, 3.78 MB]
- MLT __init__ Youtube Talk
- Reddit r/MachineLearning post
- Slides
- TVCG Link
- Arxiv Link
- Preprint High-Resolution [PDF, 25.2 MB]
- Preprint Low-Resolution [PDF, 5.6 MB]
- Supplementary material [PDF, 34.6 MB]
Presentation
-
We presented this paper at MLT __init__. Please use this link to see the slides and video. Raw 4K presentation is included below:
Poster
Results
We present SeamlessGAN, a fully self-supervised generative model capable of generating multiple tileable texture stacks from a single input example. To achieve this, follow a two-step process. First, based on recent work on generative models for texture synthesis, we train a GAN using an adversarial expansion scheme, which learns to generate textures which double the spatial extent of their inputs. Our generator can output multiple spatially-coherent texture maps at the same time. The training objetive is a combination of pixel-wise, adversarial and perceptual losses.-
Once the GAN has been fully trained, which, thanks to different training optimizations, takes one order of magnitude less time than previous methods, we can generate tileable texture stacks. To achieve this, we feed the generator with an input crop of the original texture, obtain a latent space, which we tile horizontally and vertically. This generates a latent field, which is fed to the rest of our generator. The generated output contains 4 copies of a generated texture, with seamless borders. Cropping the central area, we obtain a single of those copies, which is the output tileable texture stack. The discriminator provides pixel-wise quality estimations of the quality of the generated textures, which we use as guidance for a novel sampling algorithm. This sampling can generate multiple tileable texture stacks from the same inputs, while being able to discard low-quality samples.
-
We extensively study our design choices, including the size of our discriminator and generator, their internal architecture and the loss function used:
-
Our framework works for textures of multiple levels of regularity, stochasticity and semantics, surpassing previous methods on different quantitative metrics.
Related Projects
Bibtex
Acknowledgements
Elena Garces was partially supported by a Torres Quevedo Fellowship (PTQ2018-009868).