We gratefully acknowledge support from
the Simons Foundation and member institutions.


New submissions

[ total of 10 entries: 1-10 ]
[ showing up to 2000 entries per page: fewer | more ]

New submissions for Thu, 25 Nov 21

[1]  arXiv:2111.12159 [pdf, other]
Title: Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure
Subjects: Graphics (cs.GR); Machine Learning (cs.LG)

Synthesizing human motion with a global structure, such as a choreography, is a challenging task. Existing methods tend to concentrate on local smooth pose transitions and neglect the global context or the theme of the motion. In this work, we present a music-driven motion synthesis framework that generates long-term sequences of human motions which are synchronized with the input beats, and jointly form a global structure that respects a specific dance genre. In addition, our framework enables generation of diverse motions that are controlled by the content of the music, and not only by the beat. Our music-driven dance synthesis framework is a hierarchical system that consists of three levels: pose, motif, and choreography. The pose level consists of an LSTM component that generates temporally coherent sequences of poses. The motif level guides sets of consecutive poses to form a movement that belongs to a specific distribution using a novel motion perceptual-loss. And the choreography level selects the order of the performed movements and drives the system to follow the global structure of a dance genre. Our results demonstrate the effectiveness of our music-driven framework to generate natural and consistent movements on various dance types, having control over the content of the synthesized motions, and respecting the overall structure of the dance.

[2]  arXiv:2111.12549 [pdf, ps, other]
Title: Interpolating Rotations with Non-abelian Kuramoto Model on the 3-Sphere
Journal-ref: Advanced Technologies, Systems, and Applications VI. IAT 2021. Lecture Notes in Networks and Systems, vol 316. Springer, Cham
Subjects: Graphics (cs.GR)

The paper presents a novel method for interpolating rotations based on the non-Abelian Kuramoto model on sphere S3. The algorithm, introduced in this paper, finds the shortest and most direct path between two rotations. We have discovered that it gives approximately the same results as a Spherical Linear Interpolation algorithm. Simulation results of our algorithm are visualized on S2 using Hopf fibration. In addition, in order to gain a better insight, we have provided one short video illustrating the rotation of an object between two positions.

Cross-lists for Thu, 25 Nov 21

[3]  arXiv:2111.12448 (cross-list from cs.CV) [pdf, other]
Title: 3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG)

Learning a disentangled, interpretable, and structured latent representation in 3D generative models of faces and bodies is still an open problem. The problem is particularly acute when control over identity features is required. In this paper, we propose an intuitive yet effective self-supervised approach to train a 3D shape variational autoencoder (VAE) which encourages a disentangled latent representation of identity features. Curating the mini-batch generation by swapping arbitrary features across different shapes allows to define a loss function leveraging known differences and similarities in the latent representations. Experimental results conducted on 3D meshes show that state-of-the-art methods for latent disentanglement are not able to disentangle identity features of faces and bodies. Our proposed method properly decouples the generation of such features while maintaining good representation and reconstruction capabilities.

[4]  arXiv:2111.12480 (cross-list from cs.CV) [pdf, other]
Title: Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG)

Autoregressive models have proven to be very powerful in NLP text generation tasks and lately have gained popularity for image generation as well. However, they have seen limited use for the synthesis of 3D shapes so far. This is mainly due to the lack of a straightforward way to linearize 3D data as well as to scaling problems with the length of the resulting sequences when describing complex shapes. In this work we address both of these problems. We use octrees as a compact hierarchical shape representation that can be sequentialized by traversal ordering. Moreover, we introduce an adaptive compression scheme, that significantly reduces sequence lengths and thus enables their effective generation with a transformer, while still allowing fully autoregressive sampling and parallel training. We demonstrate the performance of our model by comparing against the state-of-the-art in shape generation.

[5]  arXiv:2111.12488 (cross-list from cs.CV) [pdf, other]
Title: Intuitive Shape Editing in Latent Space
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG)

The use of autoencoders for shape generation and editing suffers from manipulations in latent space that may lead to unpredictable changes in the output shape. We present an autoencoder-based method that enables intuitive shape editing in latent space by disentangling latent sub-spaces to obtain control points on the surface and style variables that can be manipulated independently. The key idea is adding a Lipschitz-type constraint to the loss function, i.e. bounding the change of the output shape proportionally to the change in latent space, leading to interpretable latent space representations. The control points on the surface can then be freely moved around, allowing for intuitive shape editing directly in latent space. We evaluate our method by comparing it to state-of-the-art data-driven shape editing methods. Besides shape manipulation, we demonstrate the expressiveness of our control points by leveraging them for unsupervised part segmentation.

[6]  arXiv:2111.12503 (cross-list from cs.CV) [pdf, other]
Title: Extracting Triangular 3D Models, Materials, and Lighting From Images
Authors: Jacob Munkberg (1), Jon Hasselgren (1), Tianchang Shen (1,2,3), Jun Gao (1,2,3), Wenzheng Chen (1), Alex Evans (1), Thomas Müller (1), Sanja Fidler (1,2,3) ((1) NVIDIA, (2) University of Toronto, (3) Vector Institute)
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR)

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers).

Replacements for Thu, 25 Nov 21

[7]  arXiv:2101.11529 (replaced) [pdf, other]
Title: NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions
Comments: First two authors contributed equally. Code repository at this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Human-Computer Interaction (cs.HC); Multimedia (cs.MM)
[8]  arXiv:2111.10520 (replaced) [pdf, other]
Title: StylePart: Image-based Shape Part Manipulation
Comments: 10 pages, Project page: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR)
[9]  arXiv:2111.11426 (replaced) [pdf, other]
Title: Neural Fields in Visual Computing and Beyond
Comments: Equal advising: Vincent Sitzmann and Srinath Sridhar
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG)
[10]  arXiv:2111.12077 (replaced) [pdf, other]
Title: Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
Comments: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR)
[ total of 10 entries: 1-10 ]
[ showing up to 2000 entries per page: fewer | more ]

Disable MathJax (What is MathJax?)

Links to: arXiv, form interface, find, cs, recent, 2111, contact, help  (Access key information)