We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.SE

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Software Engineering

Title: Robustness Evaluation of Stacked Generative Adversarial Networks using Metamorphic Testing

Abstract: Synthesising photo-realistic images from natural language is one of the challenging problems in computer vision. Over the past decade, a number of approaches have been proposed, of which the improved Stacked Generative Adversarial Network (StackGAN-v2) has proven capable of generating high resolution images that reflect the details specified in the input text descriptions. In this paper, we aim to assess the robustness and fault-tolerance capability of the StackGAN-v2 model by introducing variations in the training data. However, due to the working principle of Generative Adversarial Network (GAN), it is difficult to predict the output of the model when the training data are modified. Hence, in this work, we adopt Metamorphic Testing technique to evaluate the robustness of the model with a variety of unexpected training dataset. As such, we first implement StackGAN-v2 algorithm and test the pre-trained model provided by the original authors to establish a ground truth for our experiments. We then identify a metamorphic relation, from which test cases are generated. Further, metamorphic relations were derived successively based on the observations of prior test results. Finally, we synthesise the results from our experiment of all the metamorphic relations and found that StackGAN-v2 algorithm is susceptible to input images with obtrusive objects, even if it overlaps with the main object minimally, which was not reported by the authors and users of StackGAN-v2 model. The proposed metamorphic relations can be applied to other text-to-image synthesis models to not only verify the robustness but also to help researchers understand and interpret the results made by the machine learning models.
Comments: 8 pages, accepted at the 6th International Workshop on Metamorphic Testing (MET'21)
Subjects: Software Engineering (cs.SE); Computer Vision and Pattern Recognition (cs.CV)
DOI: 10.1109/MET52542.2021.00008
Cite as: arXiv:2103.02870 [cs.SE]
  (or arXiv:2103.02870v2 [cs.SE] for this version)

Submission history

From: Chun Yong Chong [view email]
[v1] Thu, 4 Mar 2021 07:29:17 GMT (3180kb,D)
[v2] Sat, 2 Oct 2021 08:01:53 GMT (3180kb,D)

Link back to: arXiv, form interface, contact.