Generating Software Tests Andreas Zeller, Rahul Gopinath

News

08.12.2018

Project 2

Hi all,

Project 2 has been published!
 
The goal of this project is to implement an algorithm that constructs a CFG from an arbitrary regular expression.
 
The project notebook (Project_Grammar.ipynb) is available at https://fuzzingbook.cispa.saarland or for... Read more

Hi all,

Project 2 has been published!
 
The goal of this project is to implement an algorithm that constructs a CFG from an arbitrary regular expression.
 
The project notebook (Project_Grammar.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
If you download the project make sure to also download the data (data/regex) folder as well!

You should all have access to https://fuzzingbook.cispa.saarland by now. In case, you do not have access, forward us your Github username, then we can grant you access to the system.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS: https://cms.cispa.saarland/fuzzing1819/students/view#submissions.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: January 7th 2019, 07:00 CET

30.11.2018

Project 1 - Results

Hi all,

 

Results from Project 1 are out!

Your grade is based on your coverage with the following bonus being applied:

  • +50 LOC for finding an error in the library (the bonus is applied 1x irrespectively of the number of exceptions found. Note that XML... Read more

Hi all,

 

Results from Project 1 are out!

Your grade is based on your coverage with the following bonus being applied:

  • +50 LOC for finding an error in the library (the bonus is applied 1x irrespectively of the number of exceptions found. Note that XML syntax validation exceptions are not errors but intended behavior).
  • +50 LOC for reaching an overall coverage of more than 5500 unique statements throughout the 5 runs.

 

Regards,

Nataniel

29.11.2018

Talk Friday 10:15 Caroline Lemieux (UC Berkeley) on "Expanding the Reach of Fuzz Testing"

Dear all,

This Friday, we have a visiting researcher in the field of software test generation and security testing as our guest.  Come visit her talk
 
Expanding the Reach of Fuzz Testing 
Caroline Lemieux, UC Berkeley
 
Friday, November 30,... Read more

Dear all,

This Friday, we have a visiting researcher in the field of software test generation and security testing as our guest.  Come visit her talk
 
Expanding the Reach of Fuzz Testing 
Caroline Lemieux, UC Berkeley
 
Friday, November 30, 10:15
CISPA (E9 1), Seminar Room 308
 
Abstract.  Fuzz testing, in particular feedback-directed fuzz testing (FDF) popularized by AFL, has gained prominence in recent years in both industry and academia thanks to its bug-finding power and apparent simplicity. A natural question is whether the FDF algorithm is well-suited to achieve other testing goals. In this talk, I will discuss my work on expanding the reach of fuzz testing, centering around three projects that altered FDF to: achieve higher program coverage, discover performance problems, and test programs with highly-structured inputs. In spite of these differing goals, these projects all made similar tweaks to FDF, and I will summarize these common themes. 
 
Bio.  Caroline Lemieux is a 3rd year PhD student at UC Berkeley advised by Koushik Sen. Her current research focuses on expanding the reach of fuzz testing to novel testing objectives, with judicious use of user feedback. Her broader research interests are in developing automated methods for software testing and comprehension relying on the power of dynamic validation. She received her B.Sc. in Mathematics and Computer Science at the University of British Columbia. 

 

Looking forward to see you, and best wishes,
 
Andreas Zeller
 

 

27.11.2018

Course Registration

Hi all,

 

Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are... Read more

Hi all,

 

Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
 
To be approved in the course it is necessary to pass all 3 projects and your final grade will be based on the grade of your projects.
 
We currently have 25 registrations and 45 submissions.
 
Students for the Masters in Cybersecurity which are having problems to register at HISPOS, as well as ERASMUS students: please send me an email with your name and matriculation number until December 1st.
 
Regards,
 
Nataniel
24.11.2018

Project submission

Hi all,

 

Since it seems that there was some doubt about the project submission I'm sending this message.

The project should be submitted in the course page submissions tab.

 

Regards,

Nataniel

23.11.2018

Course Registration

Hi all,
 
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are... Read more
Hi all,
 
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
 
To be approved in the course it is necessary to pass all 3 projects and your final grade will be based on the grade of your projects.
 
Regards,
 
Nataniel
11.11.2018

Project 1 Tips

Hi all, 
 
I received with questions about how to reach more than 4300 LOC. If you are having problems, it may be in one of the following items:
 
- A mix between purely random mutations (without converting the SVG into a tree) and... Read more
Hi all, 
 
I received with questions about how to reach more than 4300 LOC. If you are having problems, it may be in one of the following items:
 
- A mix between purely random mutations (without converting the SVG into a tree) and specialised mutations helps the fuzzer to explore other code segments, such as XML validations.
 
- The timeout for the FunctionCoverageRunner may be too small, try overwriting the coverage runner to increase the timeout a bit. That’s a valid solution, as long as the timeout is reasonable [3-5 seconds should do the trick in most modern computers or in the fuzzingbook.cispa.saarland server]. Have a look here https://cms.cispa.saarland/askbot/fuzzing1819/question/29/project-1-evaluation/ to see how to extend the class.
 
A last tip to improve your coverage is to let the purely random (or your own) fuzzer run for, like 50.000 iterations and see which new children (i.e. children which reaches code locations you could not with 10.000 iterations) were created. This may help you to come up with new mutation operators.
 
Finally, note that the final evaluation will be executed in the fuzzingbook.cispa.saarland server, thus, it is recommended that you check/evaluate your results there before submission. If you still don’t have access to the server, just send your GitHub data by email to nataniel.borges at cispa.saarland.
 
Regards,
 
Nataniel
31.10.2018

Project 1

Hi all, Project 1 is out!
 
The goal of this project is to fuzz a SVG processing library using the mutation fuzzing techniques we saw in the lecture.
 
The project notebook (Project_MutationFuzzing.ipynb) is available at ... Read more
Hi all, Project 1 is out!
 
The goal of this project is to fuzz a SVG processing library using the mutation fuzzing techniques we saw in the lecture.
 
The project notebook (Project_MutationFuzzing.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
If you download the project make sure to download the data (data/svg-full) folder as well!
 
To access https://fuzzingbook.cispa.saarland you need to forward us your Github username, then we can grant you access to the system.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: November 26th 07:00 CET
23.10.2018

Another week, another set of chapters!  The next two chapters to work on this week are "Getting Coverage" and "Mutation-Based Fuzzing".  After that, we will slow down to one chapter per week.

Bug bounties are still on offer – feel free to report bugs... Read more

Another week, another set of chapters!  The next two chapters to work on this week are "Getting Coverage" and "Mutation-Based Fuzzing".  After that, we will slow down to one chapter per week.

Bug bounties are still on offer – feel free to report bugs anytime.

16.10.2018

Bug Bounties for "Generating Software Tests"

Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:

  • 0.20€ for every new bug in the writing (i.e. text cells)
  • 0.50€ for every new bug found in the code.

Here, a "bug" is any condition leading to a change.  

Bugs are... Read more

Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:

  • 0.20€ for every new bug in the writing (i.e. text cells)
  • 0.50€ for every new bug found in the code.

Here, a "bug" is any condition leading to a change.  

Bugs are best resolved interactively.  Come and see Professor Zeller with a printout or a list, preferably after the lecture.  We will have leaderboards of students reporting the most bugs.

The fine print: Offer can be revoked at any time, all further liability is excluded, decisions are final, there is no recourse to the courts, yada yada yada.

16.10.2018

Book "Generating Software Tests"

The book "Generating Software Tests" can be found here: https://beta.fuzzingbook.org

The chapters to work on this week are the first two: "Introduction to Software Testing" and "Fuzzing: Breaking Things with Random Inputs"
 

Show all
 

Security Testing

Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing.  In this course, we explore these techniques – in theory and in code.

Course Organization

Every week, you will be provided with Jupyter Notebooks every week that teach a particular topic and illustrate it using plenty of runnable Python code.  In the notebook, you can edit the code as you like, run your own experiments, and re-use and extend the code to your liking.  Your task will be to use these techniques (and their code) to build a series of fuzzers (i.e. test generators) that find bugs in a number of challenging settings.

This course uses the "inverted classroom" principle – you learn at home, and discuss issues with your instructor.  In our weekly meeting, we use the gathering in the lecture hall to introduce the notebooks and projects as well as to discuss the ongoing ones as well as general questions.

Topics and Lectures

  • Intro
  • Fuzzing
  • Simplifying Tests
  • Testing with Mutations
  • Testing with Grammars
  • Grammar Inference
  • Taint Analysis
  • Coverage-Driven Testing
  • Search-Based Testing
  • Solving Constraints
  • Inferring Preconditions
  • Mutation Analysis
  • Automated Repair and Protection
  • Test Carving and Decomposition

Advanced course (6 CP)

  • 12-14 lectures
  • Tuesday 16-18 @ CISPA Lecture Hall
  • Start: Oct 16


Privacy Policy | Legal Notice
If you encounter technical problems, please contact the administrators