When: May 20, 2025, 10:00 – 12:00

Where: VU Campus, NU building, room NU-5A47

Directions to NU building: https://vusec.net/directions

This mini workshop features a strong line-up of leading systems security researchers from around the world. The event is freely accessible to everyone on a first come, first serve basis.

Speakers

Workshop program (May 20 at VU, NU-5A47)

10:00	Coffee and tea
10:25	Opening by Cristiano Giuffrida (AMSec)
10:30	Speaker: Marcel Böhme Title: Benchmarks are our measures of progress. Or are they? Abstract: How do we know how well our tool solves a problem, like bug finding, compared to other state-of-the-art tools? We run a benchmark. We choose a few representative instances of the problem, define a reasonable measure of success, and identify and mitigate various threats to validity. Finally, we implement (or reuse) a benchmarking framework, and compare the results for our tool with those for the state-of-the-art. For many important software engineering problems, we have seen new sparks of interest and serious progress made whenever a (substantially better) benchmark became available. Benchmarks are our measure of progress. Without them, we have no empirical support to our claims of effectiveness. Yet, time and again, we see practitioners disregard entire technologies as "paper-ware"---far from solving the problem they set out to solve. In this keynote, I will discuss our recent efforts to systematically study the degree to which our evaluation methodologies allow us to measure those capabilities that we aim to measure. We shed new light on a long-standing dispute about code coverage as a measure of testing effectiveness, explore the impact of the specific benchmark configuration on the evaluation outcome, and call into question the actual versus measured progress of an entire field (ML4VD) just as it gains substantial momentum and interest. Bio: Marcel Böhme is a faculty member at the Max Planck Institute for Security and Privacy (MPI-SP) in Germany where he leads the Software Security research group. His group has made foundational contributions to fuzzing which has become one of the most successful techniques for the automatic discovery of security flaws in practice. Marcel was awarded an 2024 ERC Consolidator grant for his project on in-vivo software security analysis at scale which will develop the empirical foundations of program analysis. To find out more about the research in his group, head over to https://mpi-softsec.github.io Marcel is a Guest Editor-in-Chief (Registered Papers) and Associate Editor for the ACM TOSEM, the flagship journal in software engineering, and a PC Chair for the upcoming instances of two major conferences, ASE'25 and ISSTA'26. Marcel received his PhD from the National University of Singapore where, 10 years later, he received an Outstanding Young Computing Alumni Award.
11:15	Speaker: Victor van der Veen Title: Real-World Problems Bio: Victor van der Veen is a security architect in Qualcomm. Before joining Qualcomm, he obtained his PhD in the VUSec group at Vrije Universiteit Amsterdam. He was among the first to publicly report Rowhammer bit flips in mobile devices. At Qualcomm, he continued his work on this fundamental issue in modern DRAM. In his ongoing attempts to bring academia and industry closer together, he helped some of our best next-generation scientists to publish their seminal Rowhammer research.
12:00	Closing remarks