Master Seminar Deep Learning — WS 2026/27
Master Seminar Deep Learning, Winter Semester 2026/27
Schedule: Tuesday, 14:30–16:00
Location: Room 2522.00.82
First meeting: Tuesday, October 13th, 2026 — paper selection and organizational arrangements
Participants: up to 15
Format
The aim of the seminar is to learn as much as possible about research topics at the frontier of deep learning.
- Talk: about 1 hour per topic, followed by a discussion.
- Depth: the talk must be technical enough that everyone understands the methods in depth.
- Reimplementation: whenever possible, reimplement the method, if necessary as a toy version at small scale. We agree on the implementation task in advance, based on the topic and what is feasible. GPUs can be requested from the group cluster.
- Training report: the talk includes a report on how the reimplementation and training went: training logs and curves, the key parts of the code, all important hyperparameters, and what worked and what did not.
- Results: the talk shows both qualitative and quantitative results. Qualitative: a few examples shown in detail, visually where applicable. Quantitative: tables with the performance achieved on the full dataset.
- Slides: polished slides are not the priority, but the presentation must be solid.
- Scheduling: talks are scheduled as we go. Once you have a topic, send me your preferred date.
- Review: send me your slides one week before your talk for review.
- Hand-in: after your talk, hand in your final slides and your code. The code should be a Jupyter notebook or a single-file or few-file implementation, written to be as pedagogical as possible.
- Grading: a grade is only issued if you attended at least six other talks and took part in the discussion with the presenter.
Schedule
One talk per session. Free slots are taken in the order requests arrive. No sessions during the Christmas break (23 Dec – 3 Jan).
| Date | Topic | Presenter |
|---|---|---|
| Tue 20 Oct 2026 | free | |
| Tue 27 Oct 2026 | free | |
| Tue 3 Nov 2026 | free | |
| Tue 10 Nov 2026 | 5. Auto-research with coding agents | Ones Khaldi |
| Tue 17 Nov 2026 | 4. Looped transformers | Maximilian Huft |
| Tue 24 Nov 2026 | free | |
| Tue 1 Dec 2026 | 9. World models: generative simulators | Oleksii Bykov |
| Tue 8 Dec 2026 | 2. Linear attention | Shaoxiao Langlotz |
| Tue 15 Dec 2026 | 12. Reasoning in latent space | Paul Warta |
| Tue 5 Jan 2027 | 10. Diffusion language models: LLaDA and TiDAR | Oliver Kleisch |
| Tue 12 Jan 2027 | 11. RL training of agent swarms | Bjarne Schmitz |
| Tue 19 Jan 2027 | 13. Soft thinking | Mohamed Arslen Ben Ghorbal |
| Tue 26 Jan 2027 | 10. Diffusion language models: Simplex Diffusion Models | Torben Heckes |
Topics
Each topic is one talk. Where a topic lists several papers, you can present a subset of them, so that the scope of the talk stays manageable. We agree on the subset when we discuss the implementation task.
1. Conditional memory: Engram, Lngram and memory layers
- Cheng et al. 2026: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram)
- Zheng et al. 2026: Lngram: N-gram Conditional Memory in Latent Space
- Berges et al. 2024: Memory Layers at Scale
2. Linear attention: Gated DeltaNet and Kimi Linear
- Yang, Kautz & Hatamizadeh 2024: Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025)
- Kimi Team 2025: Kimi Linear: An Expressive, Efficient Attention Architecture
3. On-policy distillation
- Agarwal et al. 2023: On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024)
- Zhao et al. 2026: Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
- Li et al. 2026: On-Policy Self-Distillation without Any Supervision (U-OPSD)
4. Looped transformers
- Geiping et al. 2025: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Zhu et al. 2025: Scaling Latent Reasoning via Looped Language Models (Ouro)
- Prairie et al. 2026: Parcae: Scaling Laws For Stable Looped Language Models
5. Auto-research with coding agents
- Karpathy 2026: autoresearch (code repository)
- Novikov et al. 2025: AlphaEvolve: A coding agent for scientific and algorithmic discovery
- Yamada et al. 2025: The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
6. Introduction to Lean
- de Moura & Ullrich 2021: The Lean 4 Theorem Prover and Programming Language (CADE 2021)
- The mathlib Community 2019: The Lean Mathematical Library (CPP 2020)
- Avigad & Massot: Mathematics in Lean (tutorial)
7. Zero RL at scale
8. World models: agents and latent prediction
- Hafner, Yan & Lillicrap 2025: Training Agents Inside of Scalable World Models (Dreamer 4)
- Assran et al. 2025: V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
9. World models: generative simulators
- Bruce et al. 2024: Genie: Generative Interactive Environments (ICML 2024)
- Alonso et al. 2024: Diffusion for World Modeling: Visual Details Matter in Atari (NeurIPS 2024)
- Valevski et al. 2024: Diffusion Models Are Real-Time Game Engines (ICLR 2025)
10. Diffusion language models
- Lou, Meng & Ermon 2023: Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (SEDD, ICML 2024)
- Sahoo et al. 2024: Simple and Effective Masked Diffusion Language Models (NeurIPS 2024)
- Nie et al. 2025: Large Language Diffusion Models (LLaDA)
- Liu et al. 2025: TiDAR: Think in Diffusion, Talk in Autoregression
- Deschenaux et al. 2026: Simplex Diffusion Models
11. RL training of agent swarms
- Kimi Team 2026: Kimi K2.5: Visual Agentic Intelligence (Agent Swarm / PARL part)
- Park et al. 2025: MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning (ACL 2025)
12. Reasoning in latent space
- Hao et al. 2024: Training Large Language Models to Reason in a Continuous Latent Space (Coconut)
- Zhu et al. 2025: Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought (NeurIPS 2025)
- Shen et al. 2025: CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation (EMNLP 2025)