# Selected Topics in Automated Testing and Debugging Rahul Gopinath + Rafael Dutra + Andreas Zeller

## News

14.05.2021

#### Abstract Submissions for Language of Program Behaviors

Dear Students,

As we mentioned during the previous week, the next seminar will be on "Language of Program Behaviors". The following publications provide the basic background.

* Input Algebras
Gopinath, Nemati, Zeller

Dear Students,

As we mentioned during the previous week, the next seminar will be on "Language of Program Behaviors". The following publications provide the basic background.

* Input Algebras
Gopinath, Nemati, Zeller
https://publications.cispa.saarland/3208/7/gopinath2021input.pdf
Talk: Osama Altamar

*  Learning the Language of Error
Chapman, Chockler, Kesseli, Kroening, Strichman, Tautschnig http://www.cprover.org/learning-errors/learning-the-language-of-error-including-a-proof-supplement.pdf
Talk: Reza Zamiri

We expect a short abstract, no more than one page, capturing the essential ideas. As before you can use the following latex snippet to determine what fits in a single page and what doesn't.


\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage{blindtext}

\title{My Title}
\begin{document}
\section{Abstract}
\blindtext
\blindtext

\blindtext
\blindtext
\end{document}


Please upload your summary as a PDF to the CMS submission slot (you can use the above latex code to generate the PDF). The submission slot for evocative patterns abstracts is here: https://cms.cispa.saarland/testdebug/submissions/view/3

Regards,

Rahul

04.05.2021

#### Abstract Submissions for Evocative Patterns for Behavioral Abstraction

Dear Students,

As we mentioned during the previous week, the next seminar will be on evocative or failure inducing patterns for behavioral abstraction. The following publications provide the basic background.

* Abstracting Failure-Inducing Inputs

Dear Students,

As we mentioned during the previous week, the next seminar will be on evocative or failure inducing patterns for behavioral abstraction. The following publications provide the basic background.

* Abstracting Failure-Inducing Inputs
Gopinath, Kampmann, Havrikov, Soremekun, Zeller
https://publications.cispa.saarland/3103/7/issta2020-language-of-failure.pdf

Talk: Benjamin Peters

(You are encouraged to read the Debugging book chapter on this subject: https://www.debuggingbook.org/beta/html/DDSetDebugger.html which details a simpler algorithm than that given in the paper, and understand the trade offs.)

* When does my Program do this? Learning Circumstances of Software Behavior

Kampmann, Havrikov, Soremekun, Zeller
https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf

Talk: Ali Alhasani

We expect a short abstract, no more than one page, capturing the essential ideas. As before you can use the following latex snippet to determine what fits in a single page and what doesn't.


\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage{blindtext}

\title{My Title}
\begin{document}
\section{Abstract}
\blindtext
\blindtext

\blindtext
\blindtext
\end{document}


Please upload your summary as a PDF to the CMS submission slot (you can use the above latex code to generate the PDF). The submission slot for evocative patterns abstracts is here: https://cms.cispa.saarland/testdebug/submissions/view/2

Regards,

Rahul

02.05.2021

#### Abstract Submissions for Delta Debugging

Dear Students,

As we mentioned during the previous week, the first seminar will be on Delta Debugging. The following publications provide the basic background.
* Simplifying and isolating failure-inducing input
Zeller, Hildebrandt

Dear Students,

As we mentioned during the previous week, the first seminar will be on Delta Debugging. The following publications provide the basic background.
* Simplifying and isolating failure-inducing input
Zeller, Hildebrandt
https://www.st.cs.uni-saarland.de/papers/tse2002/tse2002.pdf
* Reducing Failure-Inducing Inputs
https://www.fuzzingbook.org/html/Reducer.html

We expect a short abstract, no more than one page, capturing the essential ideas of the delta-debugging. You can use the following latex snippet to determine what fits in a single page and what doesn't.


\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage{blindtext}

\title{My Title}
\begin{document}
\section{Abstract}
\blindtext
\blindtext

\blindtext
\blindtext
\end{document}


Please upload your summary as a PDF to the CMS submission slot (you can use the above latex code to generate the PDF). The submission slot for delta-debugging abstracts is here: https://cms.cispa.saarland/testdebug/submissions/view/1

Regards,

Rahul

28.04.2021

#### Please register in LSF for the Seminar before May 10

Dear all,

The CS examination office tells us that all students are asked to register for the exams for the summer semester 2021. For (pro)seminars, registration or cancellation is possible up to three weeks after the topic assignment or kick-off meeting. In our... Read more

Dear all,

The CS examination office tells us that all students are asked to register for the exams for the summer semester 2021. For (pro)seminars, registration or cancellation is possible up to three weeks after the topic assignment or kick-off meeting. In our case, this was April 19, 2021, which means that you have to register in LSF by May 10 at the latest.

In case of problems wih the exam registration, students of the MI faculty should contact: studium@cs.uni-saarland.de. Other students have to contact their examination office.

If you are not a student of Computer Science or Cybersecurity, please read on:

These students are not able to register in HISPOS:

• VSIMint
• Master IT und Recht
• Bachelor-Plus MINT
• Erasmusstudents
• guest students

Currently the following students of the Computer Science department cannot register in HISPOS:

• Bachelor of Computer Science, StO 2020
• Bachelor Cyber Security, StO 2020
• Bachelor Media Informatics, StO 2020
• Master Media Informatics, StO 2020
• Bachelor Mathematics and Computer Science, StO 2020
• Teacher of computer science, StO 2020

If you fall into any of these categories, let us know, and we will forward your names and grades to the examination office after the seminar. We hope that the courses will be implemented in HISPOS by the start of the exam period and that exam registration will be possible then.

All the best,

Andreas Zeller

26.04.2021

#### Slides of "How to give a good research talk" now available...

... at https://dl.cispa.de/s/q9EfYZ5AqMrmSc7. Enjoy!

Andreas Zeller

22.04.2021

#### Details of presentations and grading

Dear Students,
As we discussed in our meeting on Monday, our seminar will continue to take place regularly on Monday 17:00 hours. The seminar web page is at: https://cms.cispa.saarland/testdebug/

Dear Students,
As we discussed in our meeting on Monday, our seminar will continue to take place regularly on Monday 17:00 hours. The seminar web page is at: https://cms.cispa.saarland/testdebug/

We will use the following zoom link: https://cispa-de.zoom.us/j/99579531702?pwd=eStVWTZXbkNDc2RnVHByZkNZcDUwZz09
(This is the same zoom link we used in our kickoff meeting)

We will start our seminar with a talk by Prof. Dr. Zeller on "How to Give a Talk". This talk will help you organize your presentations (especially the last one). So, don't miss it.

From the following week onward, we will have regular recurring sessions. We will have topics along with relevant publications assigned for reading each week. The format is for two students to volunteer for a short (5 minutes), ungraded introductory talk on the subject and publications, followed by discussions on the given publications possible limitations, published advancements, possible avenues of improvement, and possible applications other than what is envisioned in the paper. The other students will be required to read the publications, and prepare abstracts of the publications. These abstracts will be graded (pass/fail).

At the end of the course, each student will pick a topic, and provide a detailed (20 minutes) presentation. This presentation can include not only the publication, but also interesting advancements, avenues of improvement etc. The discussion on the paper by the class can be used for material in the final presentation. New related material can also be added. You will be graded on your final presentation.

Most papers are associated with artifacts/tools that you could run. Students are encouraged to play with those tools to form their own opinions on them (especially for the last presentation).

0) "How to give a talk": Prof. Dr. Andreas Zeller

1) Delta Debugging
* Simplifying and isolating failure-inducing input
Zeller, Hildebrandt
https://www.st.cs.uni-saarland.de/papers/tse2002/tse2002.pdf
* Reducing Failure-Inducing Inputs
https://www.fuzzingbook.org/html/Reducer.html

2) Evocative Patterns for Behavioral Abstraction
* Abstracting Failure-Inducing Inputs
Gopinath, Kampmann, Havrikov, Soremekun, Zeller
https://publications.cispa.saarland/3103/7/issta2020-language-of-failure.pdf

* When does my Program do this? Learning Circumstances of Software Behavior
Kampmann, Havrikov, Soremekun, Zeller
https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf

3) Language of Program Behaviors
* Input Algebras
Gopinath, Nemati, Zeller
https://publications.cispa.saarland/3208/7/gopinath2021input.pdf
*  Learning the Language of Error
Chapman, Chockler, Kesseli, Kroening, Strichman, Tautschnig http://www.cprover.org/learning-errors/learning-the-language-of-error-including-a-proof-supplement.pdf

4) Mining Dynamic Invariants
* Dynamically discovering likely program invariants to support program evolution
Ernst, Cockrell, Griswold, Notkin
https://homes.cs.washington.edu/~mernst/pubs/invariants-tse2001.pdf

* Mining Function Specifications
https://www.fuzzingbook.org/html/DynamicInvariants.html

5) Mining Program Input Grammars
*  Mining Input Grammars from Dynamic Control Flow
Gopinath, Mathis, Zeller
https://publications.cispa.saarland/3101/1/fse2020-mimid.pdf

* Mining Input Grammars from Dynamic Taints
Höschele, Zeller
https://www.fuzzingbook.org/html/GrammarMiner.html

6) A review of reverse debugging (Time-travel Debugging)
Engblom

7) Compiler Validation via Equivalence Modulo Inputs (EMI)
Le, Afshari, Su
https://www.cs.ucdavis.edu/~su/publications/emi.pdf

8) MultiSE: Multi-Path Symbolic Execution using Value Summaries (MultiSE)
Sen, Necula, Gong, Choi
https://people.eecs.berkeley.edu/~ksen/papers/multise.pdf

9) Smart Greybox Fuzzing (AFLSmart)
Pham, Böhme, Santosa, Caciulescu, Roychoudhury
https://arxiv.org/pdf/1811.09447.pdf

We expect a short abstract, no more than one page, capturing the essential ideas of the topic. You can use the following latex snippet to determine what fits in a single page and what doesn't.


\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage{blindtext}

\title{My Title}
\begin{document}
\section{Abstract}
\blindtext
\blindtext

\blindtext
\blindtext
\end{document}


Please upload your summary as a PDF to the CMS submission slot (you can use the above latex code to generate the PDF). The submission slot will open at least a day before the talk. If you face problems, please let us know.

The topics would be assigned to earliest volunteers of that topic. So, please send rahul.gopinath@cispa.de a mail regarding your choices as early as possible.

Regards,
Rahul + Rafael + Zeller

12.04.2021

#### Kickoff Meeting on Monday, April 19, 17:00

Welcome to the seminar on "Selected Topics in Automated Testing and Debugging”! We’re happy you’re with us, and we look forward to an exciting and inspiring seminar.

We have a kick-off meeting on Monday, April 19, 17:00in which we will discuss details of the... Read more

Welcome to the seminar on "Selected Topics in Automated Testing and Debugging”! We’re happy you’re with us, and we look forward to an exciting and inspiring seminar.

We have a kick-off meeting on Monday, April 19, 17:00in which we will discuss details of the seminar. This will also likely be the recurrent meeting time of the seminar, so try to keep this space free of other events. Details of the meeting are sent to you by mail.

Looking forward to see you, and best wishes,

Rahul + Rafael + Andreas

## Selected Topics in Automated Testing and Debugging

Description: Wouldn't it be cool if one could leave the task of testing and debugging code to a computer? In this seminar, we will discuss current results and new problems in the domain of _automated testing and debugging_ based on relevant scientific papers. The focus will be on techniques available in the Fuzzing Book (https://www.fuzzingbook.org) and the Debugging Book (https://beta.debuggingbook.org), as well as recent advances.

The general process will be as follows: Each week, you get 1-2 reading assignments and write an abstract about them. You may also be asked to give an (ungraded) five-minute mini presentation to kick off the discussion and improve your presentation skills. At the end of the seminar, you give a 15-20 minute presentation on one of the techniques, preferably including small experiments or demonstrations on how well they work; these will then be graded.

Requirements: Prior knowledge in automated testing and debugging techniques will be beneficial. For experiments and evaluations, programming knowledge will be helpful, too. The Fuzzing Book and Debugging Book infrastructures are implemented in Python, and we recommend doing experiments and evaluations in Jupyter Notebooks – so don't be afraid of these two.