Master Seminar Deep Learning — WS 2026/27

Master Seminar Deep Learning, Winter Semester 2026/27

Schedule: Tuesday, 14:30–16:00

Location: Room 2522.00.82

First meeting: Tuesday, October 13th, 2026 — paper selection and organizational arrangements

Participants: up to 15

Format

The aim of the seminar is to learn as much as possible about research topics at the frontier of deep learning.

  • Talk: about 1 hour per topic, followed by a discussion.
  • Depth: the talk must be technical enough that everyone understands the methods in depth.
  • Reimplementation: whenever possible, reimplement the method, if necessary as a toy version at small scale. We agree on the implementation task in advance, based on the topic and what is feasible. GPUs can be requested from the group cluster.
  • Training report: the talk includes a report on how the reimplementation and training went: training logs and curves, the key parts of the code, all important hyperparameters, and what worked and what did not.
  • Results: the talk shows both qualitative and quantitative results. Qualitative: a few examples shown in detail, visually where applicable. Quantitative: tables with the performance achieved on the full dataset.
  • Slides: polished slides are not the priority, but the presentation must be solid.
  • Scheduling: talks are scheduled as we go. Once you have a topic, send me your preferred date.
  • Review: send me your slides one week before your talk for review.
  • Hand-in: after your talk, hand in your final slides and your code. The code should be a Jupyter notebook or a single-file or few-file implementation, written to be as pedagogical as possible.
  • Grading: a grade is only issued if you attended at least six other talks and took part in the discussion with the presenter.

Schedule

One talk per session. Free slots are taken in the order requests arrive. No sessions during the Christmas break (23 Dec – 3 Jan).

Date Topic Presenter
Tue 20 Oct 2026 free
Tue 27 Oct 2026 free
Tue 3 Nov 2026 free
Tue 10 Nov 2026 5. Auto-research with coding agents Ones Khaldi
Tue 17 Nov 2026 4. Looped transformers Maximilian Huft
Tue 24 Nov 2026 free
Tue 1 Dec 2026 9. World models: generative simulators Oleksii Bykov
Tue 8 Dec 2026 2. Linear attention Shaoxiao Langlotz
Tue 15 Dec 2026 12. Reasoning in latent space Paul Warta
Tue 5 Jan 2027 10. Diffusion language models: LLaDA and TiDAR Oliver Kleisch
Tue 12 Jan 2027 11. RL training of agent swarms Bjarne Schmitz
Tue 19 Jan 2027 13. Soft thinking Mohamed Arslen Ben Ghorbal
Tue 26 Jan 2027 10. Diffusion language models: Simplex Diffusion Models Torben Heckes

Topics

Each topic is one talk. Where a topic lists several papers, you can present a subset of them, so that the scope of the talk stays manageable. We agree on the subset when we discuss the implementation task.

1. Conditional memory: Engram, Lngram and memory layers

  1. Cheng et al. 2026: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram)
  2. Zheng et al. 2026: Lngram: N-gram Conditional Memory in Latent Space
  3. Berges et al. 2024: Memory Layers at Scale

2. Linear attention: Gated DeltaNet and Kimi Linear

  1. Yang, Kautz & Hatamizadeh 2024: Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025)
  2. Kimi Team 2025: Kimi Linear: An Expressive, Efficient Attention Architecture

3. On-policy distillation

  1. Agarwal et al. 2023: On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024)
  2. Zhao et al. 2026: Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
  3. Li et al. 2026: On-Policy Self-Distillation without Any Supervision (U-OPSD)

4. Looped transformers

  1. Geiping et al. 2025: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
  2. Zhu et al. 2025: Scaling Latent Reasoning via Looped Language Models (Ouro)
  3. Prairie et al. 2026: Parcae: Scaling Laws For Stable Looped Language Models

5. Auto-research with coding agents

  1. Karpathy 2026: autoresearch (code repository)
  2. Novikov et al. 2025: AlphaEvolve: A coding agent for scientific and algorithmic discovery
  3. Yamada et al. 2025: The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

6. Introduction to Lean

  1. de Moura & Ullrich 2021: The Lean 4 Theorem Prover and Programming Language (CADE 2021)
  2. The mathlib Community 2019: The Lean Mathematical Library (CPP 2020)
  3. Avigad & Massot: Mathematics in Lean (tutorial)

7. Zero RL at scale

  1. Tang et al. 2026: Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

8. World models: agents and latent prediction

  1. Hafner, Yan & Lillicrap 2025: Training Agents Inside of Scalable World Models (Dreamer 4)
  2. Assran et al. 2025: V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

9. World models: generative simulators

  1. Bruce et al. 2024: Genie: Generative Interactive Environments (ICML 2024)
  2. Alonso et al. 2024: Diffusion for World Modeling: Visual Details Matter in Atari (NeurIPS 2024)
  3. Valevski et al. 2024: Diffusion Models Are Real-Time Game Engines (ICLR 2025)

10. Diffusion language models

  1. Lou, Meng & Ermon 2023: Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (SEDD, ICML 2024)
  2. Sahoo et al. 2024: Simple and Effective Masked Diffusion Language Models (NeurIPS 2024)
  3. Nie et al. 2025: Large Language Diffusion Models (LLaDA)
  4. Liu et al. 2025: TiDAR: Think in Diffusion, Talk in Autoregression
  5. Deschenaux et al. 2026: Simplex Diffusion Models

11. RL training of agent swarms

  1. Kimi Team 2026: Kimi K2.5: Visual Agentic Intelligence (Agent Swarm / PARL part)
  2. Park et al. 2025: MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning (ACL 2025)

12. Reasoning in latent space

  1. Hao et al. 2024: Training Large Language Models to Reason in a Continuous Latent Space (Coconut)
  2. Zhu et al. 2025: Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought (NeurIPS 2025)
  3. Shen et al. 2025: CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation (EMNLP 2025)

13. Soft thinking

  1. Zhang et al. 2025: Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
  2. Wu et al. 2025: LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking