PhD Proposal: Dynamics-Inspired Hidden Parameter Learning for Simulation-Based Virtual Try-On

Junbang Liang
03.31.2020 13:00 to 15:00

E-Commerce has been growing at a rapid pace in recent years. People are now more likely to shop online than to go to physical stores. Digital try-on systems, as one important way to improve the user experience and popularize online garment shopping, has drawn attention of many researchers. However, the technology is still far from being practical and easy-to-use to replace physical try-on, mostly due to the gap in modeling and in demonstrating garment fitting between the digital and the real worlds.There are several reasons why customers still prefer physical try-on. First, consumers are unsure if what they buy online will fit their bodies well. Although there exist general sizing systems for individuals, its lack of consistency and standardization across different brands and garment materials can often make it difficult to sizing the clothes, especially for those with non-standard body shapes and proportion. Accurate estimation of human body shapes is the key to make digital try-on work. Second, the fabric material is usually one of the key considerations when shopping for clothes. Different fabric materials affect how the garments look and fit on a body, how customers would wear it, and whether or not they would buy it. However, the correspondences between the actual material and its digital representation are not well understood, not to mention an accurate material estimation and digital cloning from the real-world examples.Visual effects from the customers' view is as critical as other factors. There are two common presentations of garments: 2D image-based and 3D mesh with photo-realistic rendering. They have different advantages and drawbacks, but both need a large garment database for support. While creating a 3D garment model takes considerable labor, 2D images often suffer from the lack of variation and it is much more difficult to make customized changes. In either case, the try-on system would need a user-friendly design and manipulation backend to meet the customer's needs. Last, but not least, a fast and vivid animation of the garments in motion, along with the body movement, can considerably improve the user experience. Although it is not as critical as other factors, realistic visual rendering could effectively reduce the perceptual gap between the real-world and the virtual garments for online shopping.Although previous methods have made some progresses on these under-constrained problems, learning-based approaches have shown tremendous potential in making notable impact. We propose to address the key open research issues above by adopting machine learning and optimization techniques. These include: (a) fast and realistic visual rendering of animated try-on results; (b) accurate reconstruction of human shapes and sizes through consumer devices; (c) Faithful estimation of garment materials via learning and optimization; and (d) user-friendly recovery of garment geometry for rapid prototyping.Examining Committee:

Chair: Dr. Ming Lin Dept rep: Dr. David Jacobs Members: Dr. Dinesh Manocha Dr. Tom Goldstein Dr. Soheil Feizi