Differential Privacy in the Era of Foundation Models

Differential Privacy in the Era of Foundation Models

Abstract:

In recent years, foundation models, such as GPT, LLaMA, Dall-E, or Stable Diffusion, have transformed the field of machine learning, particularly in large-scale tasks like natural language processing and computer vision. These models, trained on vast datasets, are capable of transferring their learned knowledge to a wide range of applications, making them incredibly powerful and versatile. However, this also raises significant privacy concerns when sensitive data is involved.

This seminar will explore how differential privacy (DP), the leading standard for privacy protection, can be applied to foundation models to mitigate these risks. DP ensures that changes in individual data points in a model’s training data minimally affect the overall model predictions, providing a safeguard for privacy even in the most data-intensive models. We will dive into the fundamentals of both DP and foundation models, study how they intersect, and explore strategies for integrating privacy guarantees into these cutting-edge systems. Key topics will include the theory behind DP, practical privacy-preserving mechanisms, and case studies of DP implementation in advanced foundation models.

 

Learning Objective:

There are two main learning objectives of this course. 

1) Learning the foundations of Differential Privacy, Foundation Models, what they are, how they play together, how we can leverage them to achieve privacy preservation in machine learning.

2) Getting a glimpse into how to be a successful researcher. As part of research, you have to read papers, understand what they are about, and be able to apply what they talk about, in the best case to your own research ideas. Additionally, you will learn how to give a good (research) presentation, how to identify the relevant questions, ask and answer them, and how to do scientific writing.

 

Time:

The seminar will take place on Wednesdays 4:05 PM-6:00 PM in the CISPA building (Stuhlsatzenhaus 5, 66123 Saarbrücken). Please make sure to be on time, we start at 16:05 sharp.

 

Rooms, Dates, and Topics:

15.10.2025 (Room 0.07):  Introduction: Presentation of Seminar Topics, and "How-To" give a presentation 

22.10.2025 (Room 0.02): Topic 1: Introduction to Foundation Models & The Pre-train/Adapt Paradigm

29.10.2025 (Room 0.02): Topic 2: Introduction to Differential Privacy

5.11.2025 (Room 0.02): Topic 3: Privacy Risks in Foundation Models (Data Extraction)

12.11.2025 (Room 0.02): Topic 4: Privacy Risks in Foundation Models (Membership Inference)

19.11.2025 (Room 0.02): Topic 5: Memorization in Foundation Models

26.11.2025 (Room 0.02): Topic 6: Privately Pre-Training Diffusion Models

7.1.2026 (Room 0.02): Topic 7: Privately Training Large Language Models

14.1.2026 (Room 0.02): Topic 8: Other Private Language Model Adaptations

21.1.2026 (Room 0.02): Topic 9: Differential Privacy Auditing

28.1.2026 (Room 0.02): Topic 10: Unlearning

4.2.2026 (Room 0.02): Topic 11: Problems and Open Research Directions in Privacy-Preserving Machine Learning in Foundation Models

4.2.2026 (Room 0.02): Topic 12: Technical and Societal Impact of Foundation Model Privacy

20.2.2026: Report Due

 

 

Papers:

Topic 1: Introduction to Foundation Models & The Pre-train/Adapt Paradigm

Muhammad Shayan: Diffusion Models: Denoising Diffusion Probabilistic Models (https://arxiv.org/abs/2006.11239)

Ivo: Large Language Models: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (https://arxiv.org/abs/1810.04805)

Summary of the course in G-Doc: Nuren, Melih

 

Topic 2: Introduction to Differential Privacy 

Harsha: Differential Privacy: Differential privacy (https://www.comp.nus.edu.sg/~tankl/cs5322/readings/dwork.pdf)

Maitri: Differential Privacy in Machine Learning: Deep learning with differential privacy (https://arxiv.org/abs/1607.00133)

Summary of the course in G-Doc: 

 

Topic 3: Privacy Risks in Foundation Models (Data Extraction)

Geronimo:  Data extraction from Language Models: Extracting training data from large language models (https://www.usenix.org/system/files/sec21-carlini-extracting.pdf)

Michel: Data extraction from Diffusion Models: Extracting training data from diffusion models (https://www.usenix.org/system/files/usenixsecurity23-carlini.pdf)

Summary of the course in G-Doc: 

 

 

Topic 4: Privacy Risks in Foundation Models (Membership Inference)

Teodora:  Membership inference against LLMs and shortcomings: Do membership inference attacks work on large language models?  (https://arxiv.org/pdf/2402.07841). Please all also read briefly: https://arxiv.org/pdf/1610.05820

Ved Rahul: Stronger membership inference attacks: Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models  (https://arxiv.org/pdf/2505.18773)

Summary of the course in G-Doc: 

 

Topic 5: Memorization in Foundation Models

Ivan: Memorization Scaling Laws: How much do language models memorize? (https://arxiv.org/pdf/2505.24832)

Sayali: Predicting Memorization: Emergent and predictable memorization in large language models (https://proceedings.neurips.cc/paper_files/paper/2023/file/59404fb89d6194641c69ae99ecdf8f6d-Paper-Conference.pdf)

Summary of the course in G-Doc: 

 

Topic 6: Privately Pre-Training Diffusion Models

XX: Pre-training using semantics: PrivImage: Differentially private synthetic image generation using diffusion models with semantic-aware pretraining (https://www.usenix.org/system/files/usenixsecurity24-li-kecen.pdf)

Lavanya Ratna Sirisha: Pre-training using advanced noise composition: dp-promise: Differentially private diffusion probabilistic models for image synthesis (https://www.usenix.org/system/files/sec24fall-prepub-1157-wang-haichen.pdf)

Summary of the course in G-Doc: 

 

Topic 7: Privately Training Large Language Models

Adarsh Kumar Reddy: Private Pretraining: Large-scale differentially private BERT (https://arxiv.org/pdf/2108.01624)

XX: Private Fine-Tuning: Large language models can be strong differentially private learners (https://arxiv.org/pdf/2110.05679)

Summary of the course in G-Doc: 

 

Topic 8: Other Private Language Model Adaptations

Nuren: Private Low Rank Training: Differentially private fine-tuning of language models (https://arxiv.org/pdf/2110.06500)

Rithvika: Private Prompting: Flocks of stochastic parrots: Differentially private prompt learning for large language models (https://proceedings.neurips.cc/paper_files/paper/2023/file/f26119b4ffe38c24d97e4c49d334b99e-Paper-Conference.pdf)

Summary of the course in G-Doc: 

 

Topic 9: Differential Privacy Auditing

Vishnuvasan: Efficient Audits: Privacy auditing with one (1) training run (https://proceedings.neurips.cc/paper_files/paper/2023/file/9a6f6e0d6781d1cb8689192408946d73-Paper-Conference.pdf)

XX: Baysian Estimation: Bayesian Estimation of Differential Privacy (https://proceedings.mlr.press/v202/zanella-beguelin23a/zanella-beguelin23a.pdf)

Summary of the course in G-Doc: 

 

Topic 10: Unlearning

CATHERIN: Unlearning in LLMs: Rethinking machine unlearning for large language models (https://arxiv.org/pdf/2402.08787)

Nima: Evaluating Unlearning: Inexact unlearning needs more careful evaluations to avoid a false sense of privacy (https://arxiv.org/pdf/2403.01218)

Summary of the course in G-Doc: 

 

Topic 11: Problems and Open Research Directions in Privacy-Preserving Machine Learning in Foundation Models

Yashashri: Problems in private LLMs: What does it mean for a language model to preserve privacy? (https://dl.acm.org/doi/pdf/10.1145/3531146.3534642) 

OMKAR RAJEEV: Position on privacy in the pretrain-adapt paradigm: Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (https://openreview.net/pdf?id=ncjhi4qAPV)

Summary of the course in G-Doc: 

 

Topic 12: Technical and Societal Impact of Foundation Model Privacy

Kashan: Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice (https://arxiv.org/pdf/2412.06966?)

XX: Extracting memorized pieces of (copyrighted) books from open-weight language models (https://arxiv.org/pdf/2505.12546)

Summary of the course in G-Doc: 

 

Peer Groups:

Always the two students who present on the same day form the peer group for that same day.

 

Questions:

Questions can be posed here: https://docs.google.com/document/d/1JoY78ReTzjzcNYWgxvrDQpB3WSd2bF0VwmGF6H4-VQ0/edit?usp=sharing 

All presentations should be uploaded here: https://drive.google.com/drive/folders/1yWcWRWdICfMCOCLhzO4-HML0IEHdB1fq?usp=sharing

 

Topic signup sheet (Until 22.10., 8PM): https://docs.google.com/spreadsheets/d/1nwtRpc1EwCkqtdcDM79RNBVl0LOJ20SQTH4TdE8XhQ4/edit?gid=0#gid=0

 

Requirements:

This seminar is open to senior Bachelor, Masters, and doctoral students. Ideally, students should have a solid background in mathematics through the base lectures, and a strong interest in deep learning.

 

TL;DR What you need to do:

As a participant:

• Read the two papers every week.
• Write your questions until Monday in the Google Doc to give the speaker enough time to prepare.
• Attend n-1 seminars.
• If you cannot attend, you need to write a summary (1 page, i.e., half page per paper) on the papers you miss. Things build on each other, so it is important to catch up. Email this summary to boenisch@cispa.de.
• At the end of the semester, write your report and hand it in on time through CMS.

As a speaker:

• Read your paper extra carefully, know all concepts you talk about.
• Prepare your presentation (30 min).
• Review the questions and integrate, if possible in your presentation.
• Meet with your peer (person who presents the same day) and send the filled peer feedback for to boenisch@cispa.de until a week before (i.e., the Wednesday before).

 

 

Administration and LSF Registration:

A registration to LSF is required if you want the points. I have myself as an instructor no access to the system, hence, questions need to be posed to the administration: studium@cs.uni-saarland.de A guide for registration can be found here: https://saarland-informatics-campus.de/studium-studies/#lehrveranstaltungen-ansprechpartnerInnen