Reusing Deep Learning Models: Challenges and Directions in Trustworthy Software Engineering

Principal Investigator: Jamie Davis

The development and training of deep learning models have become increasingly costly and complex. Consequently, software engineers are adopting pre-trained models (PTMs) for their downstream applications. The content, dynamics, and effective use of the PTM supply chain remain largely unexplored. This project seeks to characterize the associated engineering processes and artifacts, in order to identify and mitigate failure modes. We then develop tools, e.g. automation, to optimize the reuse of PTMs. We integrate research methods from human factors, mining software repositories, and machine learning. We are focused on major model registries such as HuggingFace and PyTorchHub, and interoperability infrastructure such as the Open Neural Network eXchange (ONNX).

Personnel

Other PIs: Yung-Hsiang Lu (Purdue) George K Thiruvathukal (Loyola University Chicago)

Students: Wenxin Jiang, PhD student Purvish Jajal, PhD student ~20 Purdue undergraduate students from several majors, through the VIP program

Representative Publications

• Analysis of Failures and Risks in Deep Learning Model Converters: A Case Study in the ONNX Ecosystem.
  Jajal, Jiang, Tewari, Woo, Lu, Thiruvathukal, and Davis.
  arXiv 2023.

• An Empirical Study of Artifacts and Security Practices in the Pre-trained Model Supply Chain.
  Jiang, Synovic, Sethi, Indarapu, Hyatt, Schorlemmer, Thiruvathukal, and Davis.
  Proceedings of the 1st ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED) 2022.

• Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability.
  Montes, Peerapatanapokin, Schultz, Guo, Jiang, and Davis.
  Proceedings of the 30th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering: Ideas, Visions, and Reflections track (ESEC/FSE-IVR) 2022.

• PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software. Jiang, Yasmin, Jones, Synovic, Kuo, Bielanski, Tian, Thiruvathukal, and Davis. Proceedings of the 21st Annual Conference on Mining Software Repositories (MSR’24) 2024.

• Reusing Deep Learning Models: Challenges and Directions in Software Engineering.
  Davis, Jajal, Jiang, Schorlemmer, Synovic, and Thiruvathukal.
  Proceedings of the IEEE John Vincent Atanasoff Symposium on Modern Computing (JVA’23) 2023.

• An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry.
  Jiang, Synovic, Hyatt, Schorlemmer, Sethi, Lu, Thiruvathukal, and Davis.
  Proceedings of the ACM/IEEE 45th International Conference on Software Engineering (ICSE) 2023.

• PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software. Jiang, Yasmin, Jones, Synovic, Kuo, Bielanski, Tian, Thiruvathukal, and Davis. Proceedings of the 21st Annual Conference on Mining Software Repositories (MSR’24) 2024.

• Exploring Naming Conventions (and Defects) of Pre-trained Deep Learning Models in Hugging Face and Other Model Hubs.
  Jiang, Cheung, Thiruvathukal, and Davis.
  arXiv 2023.

Keywords: machine learning, reproducibility, software engineering