Generating Software Tests Andreas Zeller, Rahul Gopinath

News

12.02.2019

Project 3 Tips

Hi all,

I have received some questions about how to solve Project 3 and the implementation policy for the project. I will like to clarify some of them here.

In this project, you are allowed to inport the fuzzingbook module, any of the inbuilt python modules... Read more

Hi all,

I have received some questions about how to solve Project 3 and the implementation policy for the project. I will like to clarify some of them here.

In this project, you are allowed to inport the fuzzingbook module, any of the inbuilt python modules (https://docs.python.org/3/py-modindex.html), and the regex module (https://pypi.org/project/regex/). The `regex` module is important because the `re` module does not support the "\p{}" syntax. However, do not install or import any other module other than the aforementioned modules, and do not use (parts of) any open source code, e.g. found online.

Do not make any assumption on the nature of the validation regex, the only valid assumption is that each field would be validated with a regex. Hence, focus on developing fitness function(s) for any arbitrary regex, you are allowed to use any solution you deem fit for your fitness function.

The goal of this project is to search the input space, starting from the provided initial input till you reach a valid input that matches the regex. This goal is to be achieved only via your mutation, selection, crossover and fitness functions. We have used regex-based validators because of its familiarity from Project 2. Simply generating a string that matches the expected regex will be penalized.

Lastly, over the last two weeks some students have raised some questions and issues in the project. The most recent version of the project clarifies all issues raised thus far. Kindly pull the most recent version of the project notebook (Project_Search_Based_WebFuzzer.ipynb) available at https://fuzzingbook.cispa.saarland or for download https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks

 

Regards,

Ezekiel

11.02.2019

Project 1 Sample Solution

Hi all,

 

Sample solution for Project 1 has been published. It describes the tree and string mutations from three of the top solutions (> 5000 LOC) from Project 1 (Project 1 - Mutation Fuzzing.ipynb). It can be downloaded here:... Read more

Hi all,

 

Sample solution for Project 1 has been published. It describes the tree and string mutations from three of the top solutions (> 5000 LOC) from Project 1 (Project 1 - Mutation Fuzzing.ipynb). It can be downloaded here: https://cms.cispa.saarland/fuzzing1819/dl/1/Project_1_Sample_Solution_Mutation_Fuzzing.pdf

 

Regards,

Ezekiel & Nils

02.02.2019

Project 2 - Results

Hi all,

Results for Project 2 has been published.

Your grade is based on the coverage achieved by the test inputs generated by your grammar, with one point obtainable for each set of inputs reaching the minimum specified coverage. The evaluation was performed... Read more

Hi all,

Results for Project 2 has been published.

Your grade is based on the coverage achieved by the test inputs generated by your grammar, with one point obtainable for each set of inputs reaching the minimum specified coverage. The evaluation was performed with 86 regexes (including all 26 initial regexes in the Project), consisting of 23 regexes for the basic syntax category, 58 regexes for the complex syntax category and five regexes for the bonus regex category.

For grade understanding and reproducibiltiy, the evaluation code, the details of the final secret_set_of_regex and the minimum coverage used for the evaluation can be found in the evaluation project notebook (Project_Grammar_eval.ipynb). This is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks

Note on result reproducibility: The displayed results in (Project_Grammar_eval.ipynb) is provided as a sample result sheet. If you want to reproduce the points obtained by your implementation using the evaluation notebook (i.e. Project_Grammar_eval.ipynb), copy and paste your implementation in “Your code” section of the notebook, and run all cells, then your result will be automatically generated.

If you have any questions or require feedback for your grade, kindly send an email or drop by my office.  

Regards,

Ezekiel

29.01.2019

Project 3

Hi all,

Project 3 has been published!
 
The goal of this project is to implement a Search-based Web Fuzzer.

The project notebook (Project_Search_Based_WebFuzzer.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at... Read more

Hi all,

Project 3 has been published!
 
The goal of this project is to implement a Search-based Web Fuzzer.

The project notebook (Project_Search_Based_WebFuzzer.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
You should all have access to the server (https://fuzzingbook.cispa.saarland) by now. In case you do not have access yet (or lost your access), forward us your Github username, then we can grant you access.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS: https://cms.cispa.saarland/fuzzing1819/students/view#submissions.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: February 26th 2019, 07:00 CET

08.12.2018

Project 2

Hi all,

Project 2 has been published!
 
The goal of this project is to implement an algorithm that constructs a CFG from an arbitrary regular expression.
 
The project notebook (Project_Grammar.ipynb) is available at https://fuzzingbook.cispa.saarland or for... Read more

Hi all,

Project 2 has been published!
 
The goal of this project is to implement an algorithm that constructs a CFG from an arbitrary regular expression.
 
The project notebook (Project_Grammar.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
If you download the project make sure to also download the data (data/regex) folder as well!

You should all have access to https://fuzzingbook.cispa.saarland by now. In case, you do not have access, forward us your Github username, then we can grant you access to the system.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS: https://cms.cispa.saarland/fuzzing1819/students/view#submissions.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: January 7th 2019, 07:00 CET

30.11.2018

Project 1 - Results

Hi all,

 

Results from Project 1 are out!

Your grade is based on your coverage with the following bonus being applied:

  • +50 LOC for finding an error in the library (the bonus is applied 1x irrespectively of the number of exceptions found. Note that XML... Read more

Hi all,

 

Results from Project 1 are out!

Your grade is based on your coverage with the following bonus being applied:

  • +50 LOC for finding an error in the library (the bonus is applied 1x irrespectively of the number of exceptions found. Note that XML syntax validation exceptions are not errors but intended behavior).
  • +50 LOC for reaching an overall coverage of more than 5500 unique statements throughout the 5 runs.

 

Regards,

Nataniel

29.11.2018

Talk Friday 10:15 Caroline Lemieux (UC Berkeley) on "Expanding the Reach of Fuzz Testing"

Dear all,

This Friday, we have a visiting researcher in the field of software test generation and security testing as our guest.  Come visit her talk
 
Expanding the Reach of Fuzz Testing 
Caroline Lemieux, UC Berkeley
 
Friday, November 30,... Read more

Dear all,

This Friday, we have a visiting researcher in the field of software test generation and security testing as our guest.  Come visit her talk
 
Expanding the Reach of Fuzz Testing 
Caroline Lemieux, UC Berkeley
 
Friday, November 30, 10:15
CISPA (E9 1), Seminar Room 308
 
Abstract.  Fuzz testing, in particular feedback-directed fuzz testing (FDF) popularized by AFL, has gained prominence in recent years in both industry and academia thanks to its bug-finding power and apparent simplicity. A natural question is whether the FDF algorithm is well-suited to achieve other testing goals. In this talk, I will discuss my work on expanding the reach of fuzz testing, centering around three projects that altered FDF to: achieve higher program coverage, discover performance problems, and test programs with highly-structured inputs. In spite of these differing goals, these projects all made similar tweaks to FDF, and I will summarize these common themes. 
 
Bio.  Caroline Lemieux is a 3rd year PhD student at UC Berkeley advised by Koushik Sen. Her current research focuses on expanding the reach of fuzz testing to novel testing objectives, with judicious use of user feedback. Her broader research interests are in developing automated methods for software testing and comprehension relying on the power of dynamic validation. She received her B.Sc. in Mathematics and Computer Science at the University of British Columbia. 

 

Looking forward to see you, and best wishes,
 
Andreas Zeller
 

 

27.11.2018

Course Registration

Hi all,

 

Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are... Read more

Hi all,

 

Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
 
To be approved in the course it is necessary to pass all 3 projects and your final grade will be based on the grade of your projects.
 
We currently have 25 registrations and 45 submissions.
 
Students for the Masters in Cybersecurity which are having problems to register at HISPOS, as well as ERASMUS students: please send me an email with your name and matriculation number until December 1st.
 
Regards,
 
Nataniel
24.11.2018

Project submission

Hi all,

 

Since it seems that there was some doubt about the project submission I'm sending this message.

The project should be submitted in the course page submissions tab.

 

Regards,

Nataniel

23.11.2018

Course Registration

Hi all,
 
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are... Read more
Hi all,
 
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
 
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
 
To be approved in the course it is necessary to pass all 3 projects and your final grade will be based on the grade of your projects.
 
Regards,
 
Nataniel
11.11.2018

Project 1 Tips

Hi all, 
 
I received with questions about how to reach more than 4300 LOC. If you are having problems, it may be in one of the following items:
 
- A mix between purely random mutations (without converting the SVG into a tree) and... Read more
Hi all, 
 
I received with questions about how to reach more than 4300 LOC. If you are having problems, it may be in one of the following items:
 
- A mix between purely random mutations (without converting the SVG into a tree) and specialised mutations helps the fuzzer to explore other code segments, such as XML validations.
 
- The timeout for the FunctionCoverageRunner may be too small, try overwriting the coverage runner to increase the timeout a bit. That’s a valid solution, as long as the timeout is reasonable [3-5 seconds should do the trick in most modern computers or in the fuzzingbook.cispa.saarland server]. Have a look here https://cms.cispa.saarland/askbot/fuzzing1819/question/29/project-1-evaluation/ to see how to extend the class.
 
A last tip to improve your coverage is to let the purely random (or your own) fuzzer run for, like 50.000 iterations and see which new children (i.e. children which reaches code locations you could not with 10.000 iterations) were created. This may help you to come up with new mutation operators.
 
Finally, note that the final evaluation will be executed in the fuzzingbook.cispa.saarland server, thus, it is recommended that you check/evaluate your results there before submission. If you still don’t have access to the server, just send your GitHub data by email to nataniel.borges at cispa.saarland.
 
Regards,
 
Nataniel
31.10.2018

Project 1

Hi all, Project 1 is out!
 
The goal of this project is to fuzz a SVG processing library using the mutation fuzzing techniques we saw in the lecture.
 
The project notebook (Project_MutationFuzzing.ipynb) is available at ... Read more
Hi all, Project 1 is out!
 
The goal of this project is to fuzz a SVG processing library using the mutation fuzzing techniques we saw in the lecture.
 
The project notebook (Project_MutationFuzzing.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
If you download the project make sure to download the data (data/svg-full) folder as well!
 
To access https://fuzzingbook.cispa.saarland you need to forward us your Github username, then we can grant you access to the system.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: November 26th 07:00 CET
23.10.2018

Another week, another set of chapters!  The next two chapters to work on this week are "Getting Coverage" and "Mutation-Based Fuzzing".  After that, we will slow down to one chapter per week.

Bug bounties are still on offer – feel free to report bugs... Read more

Another week, another set of chapters!  The next two chapters to work on this week are "Getting Coverage" and "Mutation-Based Fuzzing".  After that, we will slow down to one chapter per week.

Bug bounties are still on offer – feel free to report bugs anytime.

16.10.2018

Bug Bounties for "Generating Software Tests"

Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:

  • 0.20€ for every new bug in the writing (i.e. text cells)
  • 0.50€ for every new bug found in the code.

Here, a "bug" is any condition leading to a change.  

Bugs are... Read more

Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:

  • 0.20€ for every new bug in the writing (i.e. text cells)
  • 0.50€ for every new bug found in the code.

Here, a "bug" is any condition leading to a change.  

Bugs are best resolved interactively.  Come and see Professor Zeller with a printout or a list, preferably after the lecture.  We will have leaderboards of students reporting the most bugs.

The fine print: Offer can be revoked at any time, all further liability is excluded, decisions are final, there is no recourse to the courts, yada yada yada.

16.10.2018

Book "Generating Software Tests"

The book "Generating Software Tests" can be found here: https://beta.fuzzingbook.org

The chapters to work on this week are the first two: "Introduction to Software Testing" and "Fuzzing: Breaking Things with Random Inputs"
 

Show all
 

Security Testing

Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing.  In this course, we explore these techniques – in theory and in code.

Course Organization

Every week, you will be provided with Jupyter Notebooks every week that teach a particular topic and illustrate it using plenty of runnable Python code.  In the notebook, you can edit the code as you like, run your own experiments, and re-use and extend the code to your liking.  Your task will be to use these techniques (and their code) to build a series of fuzzers (i.e. test generators) that find bugs in a number of challenging settings.

This course uses the "inverted classroom" principle – you learn at home, and discuss issues with your instructor.  In our weekly meeting, we use the gathering in the lecture hall to introduce the notebooks and projects as well as to discuss the ongoing ones as well as general questions.

Topics and Lectures

  • Intro
  • Fuzzing
  • Simplifying Tests
  • Testing with Mutations
  • Testing with Grammars
  • Grammar Inference
  • Taint Analysis
  • Coverage-Driven Testing
  • Search-Based Testing
  • Solving Constraints
  • Inferring Preconditions
  • Mutation Analysis
  • Automated Repair and Protection
  • Test Carving and Decomposition

Advanced course (6 CP)

  • 12-14 lectures
  • Tuesday 16-18 @ CISPA Lecture Hall
  • Start: Oct 16


Privacy Policy | Legal Notice
If you encounter technical problems, please contact the administrators