By: Shesh Narayan Gupta, Nicholas Bear Brown
3D software are now capable of producing highly realistic images that look nearly indistinguishable from real images. This raises the question: can real datasets be enhanced with 3D-rendered data? We investigate this question. In this paper, we demonstrate the use of 3D-rendered data and procedural, data for the adjustment of bias in image datasets. We perform error analysis of images of animals which shows that the misclassification of some animal breeds is largely a data issue. We then create procedural images of the poorly classified breeds and that model further trained on procedural data can better classify poorly performing breeds on real data. We believe that this approach can be used for the enhancement of visual data for any underrepresented group, including rare diseases, or any data bias potentially improving the accuracy and fairness of models. We find that the resulting representations rival or even outperform those learned directly from real data, but that good performance requires care in the 3D-rendered procedural data generation. 3D image datasets can be viewed as a compressed and organized copy of a real dataset, and we envision a future where more and more procedural data proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such a future.