Twelve test generators for twelve programming languages

Dear all,

The end of our seminar is nearing, and we would appreciate your feedback on the course. Please spend a few minutes to give your feedback via this link:

https://qualis.uni-saarland.de/eva/?l=164532&p=p4cfz2

Please note:

• While the evaluation form lists "Prof. Dr. Andreas Zeller" as lecturer, your comments regarding "the lecturer" should apply to both Alexi and Andreas.
• You can also skip the questions regarding oral presentations.
• While we are happy about quantitative assessments, we especially appreciate your text comments on what you liked and what you didn't like, and what we could do better next time.

The above link will be available until July 8, 2026. Very much looking forward to your feedback!

- Alexi + Andreas

Hey all!

As promised:
1. I invited you all to a (currently private) repo with the work-in-progress parametric compiler testing (PCT) constraints. You should accept this invitation, and importantly focus on the 'prototype' branch. The repo also contains a reworked example of the C language grammar from the compiler testing case study in the FANDANGO-RS paper, with the generic type checking and def-use constraints (btw, this is what I showed in class). I added detailed comments especially in the type checking part for C. You should focus mainly on the C language example for reference.
2. I updated the C and JavaScript examples in the org; C with def-use, and JavaScript with a mini type checking example. It's not very deep or complete, but should be an additional reference. The C language example in PCT is your go-to for an exemplar.
3. I forked my grammar testing linter tool in the org as well; please take a look.

For Friday, I will try to expand this to function types (should be fairly straightforward), and we can go over that and any pain points. Also, if you have a hard time with the generic constraints, as a fallback option you can consider adapting the "caveman" constraints directly from the FANDANGO-RS C compiler study; there are more constraints, but there's less abstractions so you can see directly how things work.

I appreciate your patience! Stay cool and hydrated,
Alexi

Hey all!

I just pushed two example repos to the org: one for JavaScript, and one for C, with a few constraints and examples in each, including the structure for setting up the fuzzing experiments as cargo examples. Shoot me an email if you can't see them, I may have misconfigured the org.

I will push a few more changes to those tomorrow. If you haven't already, I'd recommend getting your code migrated soon!

Best,
Alexi

Hey all!

You should have received invites to the shared repo; if not, shoot me an email.

I will post in a few minutes material from this week's meeting, with a "checklist" of what exactly to put in your repo. You should have repo creation privilege, so please go ahead and create a repo in the org titled according to your programming language.

Best,
Alexi

Hey all!

Since it's a holiday on Thursday, we will hold the seminar meeting over Zoom in case y'all wanted to head out of town and make it a long weekend. I'll post the material we discussed last day soon, and look forward to catching up with you!

I will share a Zoom link a little later, after I've posted the rest of the news.

Best,
Alexi

Hey all!

I've been considering different options for how to allow y'all to see each other's progress and share your compiler testers, and I've settled on setting up a GitHub organization for the seminar and inviting you all to it. I'll post an "assignment" in which you can submit your GitHub username, I will then invite you to the organization, and then you can create a repo for your language and migrate your code there. 

Also I have a few of your emails still to process, sorry again about the delay! 

Best,
Alexi

Hey all!

Hope you all had a nice long weekend.

In a moment I will post some new material, a document with pointers for some C language constraints that should give you some inspiration and guidance when building up some simpler constraints for your language. I'll take a peek at the forum tomorrow and answer any outstanding posts, and catch up on your emails as well. We will share a larger, shared repo with you tomorrow as well, as discussed on Friday.

Best,
Alexi

Howdy!

Sorry for the short notice, but tomorrow we will meet at the usual time, but on Zoom: https://cispa-de.zoom-x.de/j/61800509104?pwd=6wv00wiDUDl8MKHlS7k20fxgt2sMmV.1

Looking forward to hearing about your adventures with your programming languages! I will get to some outstanding emails shortly.

See ya tomorrow,
Alexi

Hey all!

I just added two new documents: one about how to compose constraints together and set up a basic NSGA-2 evolver (i.e., the multi-objective optimization algorithm, as when we have multiple constraints, we want to optimize for all of them), as well as one about how to set generation goals.

I also updated the grammar document with some of the discussion we had last Friday.

For this week, I'd like you to focus on getting your grammar working (if you haven't already), and running some preliminary experiments by generating lots of programs, trying to compile them, and seeing what sort of issues the compilers find with the programs. This will help you prioritize what behaviour to constrain, as well as help us identify general, generic issues that many languages face. Our discussion on Friday will centre around this. Also, if you're feeling particularly ambitious, you could try writing some really basic constraints to get more familiar with the design patterns. Remember: every line of Rust, every constraint you write will be easier than the last!

Best,
Alexi

Howdy!

I will add some new material in a few minutes which goes over some issues you may face when adapting a grammar into fandango-rs. The document has a walkthrough of me taking the JavaScript ANTLR grammar, using the fandango-py tooling to convert it, and step-by-step fixing issues and trying to shed some light on the opaque error messages you will no doubt face. 

Hey all!

More info on developing your grammars is forthcoming, as well as a summary of Friday's discussion.

As promised, here is the list of languages people are initially working on:

Lea: Julia
Syed: Lua
Johannes Haegele: Squirrel
Jashmanpreet: Rust
Jan: Swift
Chi: Go
Johannes Victor Weissmann: Typst (or C# if Typst is rough)
Haoyang: Java
Farbod: Pascal
Niklas: Kotlin
Waleed: Dart
Aadrish: OCaml
Philipp: C++

BTW, let me know if you'd like to switch languages, from here on it'll be on a first come, first serve basis for language changes. 

Howdy!

Action Item: Please share your programming language rankings with us by Wednesday via submission in CMS. We'd like to finalize the language assignments by Friday.

Hope you had a nice weekend! Besides the above action item, a few minor updates:

1. Updated the material for the Week 1 & 2 Tasks (a few little fixes), and shared a PDF version of the slides instead of Keynote.

2. I will share a few more tutorial documents throughout the week, namely discussing combining constraints, generation goals, fixers, and the more advanced evolutionary algorithm. 

3. The plan for this Friday is to first discuss the Week 1 & 2 Tasks, discuss programming language assignments, and I will then present some material on how to prepare a grammar for the language you will have chosen.

4. There's also now a forum set up in CMS.

Best,

Alexi

Hey all!

As promised in the kick-off meeting, I've just posted two documents in "Materials":

1. The slides from the meeting, and 

2. A much more detailed document, walking you through setting up your first fandango-rs crate, as well as three tasks to complete, one thinking, one reading, and one implementing. Please read and work on these tasks before the next seminar meeting.

While you go about getting your first crate up and running, think of a preferably ranked list of at least three programming languages you'd like to fuzz in this course. We will likely ask you to submit list before the seminar meeting next week so that we can ensure a good spread of unique programming languages. Besides that, if you encounter any insurmountable difficulties when following the tasks, particularly if the instructions are unclear or you struggle with implementing the extra constraint, let us know! Now, let's find tons of compiler bugs.

Dear all,

Please be reminded that you need to formally register for the seminar "exam" in LSF by May 15, 2026. Do not forget to do so; we'd suggest that you register right now.

You can also always un-register by that date, should you find the seminar is not to your taste (and register again if your mood changes). However, be also reminded that we're always here to help you make your participation a success :-)

Looking forward to working with you,

Andreas + Alexi