News
Help us tailoring automated debugging to your needsWritten on 16.09.21 by Andreas Zeller Dear former course participants, Thank you for your interest in software testing and debugging! We're developing Alhazen, a tool that automatically finds out when and why your program fails. We are heavily relying on your opinion as professional developers. Please take part in our user study to… Read more Dear former course participants, Thank you for your interest in software testing and debugging! We're developing Alhazen, a tool that automatically finds out when and why your program fails. We are heavily relying on your opinion as professional developers. Please take part in our user study to help us tie our approach to your needs: https://tinyurl.com/debuggingstudy Thanks a lot! Andreas Zeller + Team |
Searching for Tutors for Generating Software Tests - WS19/20Written on 24.10.19 by Nataniel Borges Jr. Hi all,
We're lookig for tutors for the new iteration of our current interation of the "Generating Software Tests" lecture. If you passed the course in the last semester and are interested in becoming a tutor for this semester, please contact Rahul Gopinath (rahul.gopinath@cispa.saarland)… Read more Hi all,
We're lookig for tutors for the new iteration of our current interation of the "Generating Software Tests" lecture. If you passed the course in the last semester and are interested in becoming a tutor for this semester, please contact Rahul Gopinath (rahul.gopinath@cispa.saarland) or Michael Mera (michael.mera@cispa.saarland).
Regards, Nataniel |
Project 3 Results & Final GradeWritten on 12.04.19 by Ezekiel Soremekun Hi all, Results for Project 3 has been published, as well as the final grades for the course. Project 3's final grade was computed with one bonus point for greater than average validity, one bonus point for lower than average number of generations and one bonus point for greater than average… Read more Hi all, Results for Project 3 has been published, as well as the final grades for the course. Project 3's final grade was computed with one bonus point for greater than average validity, one bonus point for lower than average number of generations and one bonus point for greater than average reachability. For point reproducibility, the evaluation code for the project containing the details of the time limits, final set of regex and the final population can be found in the evaluation notebook (final_Project_Search_Based_WebFuzzer_eval.ipynb). This is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks If you want to reproduce the points obtained by your implementation using the evaluation notebook (i.e. final_Project_Search_Based_WebFuzzer_eval.ipynb), copy and paste your implementation in “Your code” section of the notebook, and run all cells, then your result will be automatically generated. The course final grade was computed as the average of the three project grades. In addition, to pass the course, you are required to pass all three projects with a minimum grade of 4.0. If you have any questions or require feedback for your grade, kindly send an email or drop by my office (E 9.1, room 2.06). Regards, Ezekiel |
Project 3 TipsWritten on 12.02.19 (last change on 12.02.19) by Ezekiel Soremekun Hi all, I have received some questions about how to solve Project 3 and the implementation policy for the project. I will like to clarify some of them here. In this project, you are allowed to inport the fuzzingbook module, any of the inbuilt python modules… Read more Hi all, I have received some questions about how to solve Project 3 and the implementation policy for the project. I will like to clarify some of them here. In this project, you are allowed to inport the fuzzingbook module, any of the inbuilt python modules (https://docs.python.org/3/py-modindex.html), and the regex module (https://pypi.org/project/regex/). The `regex` module is important because the `re` module does not support the "\p{}" syntax. However, do not install or import any other module other than the aforementioned modules, and do not use (parts of) any open source code, e.g. found online. Do not make any assumption on the nature of the validation regex, the only valid assumption is that each field would be validated with a regex. Hence, focus on developing fitness function(s) for any arbitrary regex, you are allowed to use any solution you deem fit for your fitness function. The goal of this project is to search the input space, starting from the provided initial input till you reach a valid input that matches the regex. This goal is to be achieved only via your mutation, selection, crossover and fitness functions. We have used regex-based validators because of its familiarity from Project 2. Simply generating a string that matches the expected regex will be penalized. Lastly, over the last two weeks some students have raised some questions and issues in the project. The most recent version of the project clarifies all issues raised thus far. Kindly pull the most recent version of the project notebook (Project_Search_Based_WebFuzzer.ipynb) available at https://fuzzingbook.cispa.saarland or for download https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
Regards, Ezekiel |
Project 1 Sample SolutionWritten on 11.02.19 by Ezekiel Soremekun Hi all,
Sample solution for Project 1 has been published. It describes the tree and string mutations from three of the top solutions (> 5000 LOC) from Project 1 (Project 1 - Mutation Fuzzing.ipynb). It can be downloaded here:… Read more Hi all,
Sample solution for Project 1 has been published. It describes the tree and string mutations from three of the top solutions (> 5000 LOC) from Project 1 (Project 1 - Mutation Fuzzing.ipynb). It can be downloaded here: https://cms.cispa.saarland/fuzzing1819/dl/1/Project_1_Sample_Solution_Mutation_Fuzzing.pdf
Regards, Ezekiel & Nils |
Project 2 - ResultsWritten on 02.02.19 (last change on 10.02.19) by Ezekiel Soremekun Hi all, Results for Project 2 has been published. Your grade is based on the coverage achieved by the test inputs generated by your grammar, with one point obtainable for each set of inputs reaching the minimum specified coverage. The evaluation was performed with 86 regexes (including all 26… Read more Hi all, Results for Project 2 has been published. Your grade is based on the coverage achieved by the test inputs generated by your grammar, with one point obtainable for each set of inputs reaching the minimum specified coverage. The evaluation was performed with 86 regexes (including all 26 initial regexes in the Project), consisting of 23 regexes for the basic syntax category, 58 regexes for the complex syntax category and five regexes for the bonus regex category. For grade understanding and reproducibiltiy, the evaluation code, the details of the final secret_set_of_regex and the minimum coverage used for the evaluation can be found in the evaluation project notebook (Project_Grammar_eval.ipynb). This is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks Note on result reproducibility: The displayed results in (Project_Grammar_eval.ipynb) is provided as a sample result sheet. If you want to reproduce the points obtained by your implementation using the evaluation notebook (i.e. Project_Grammar_eval.ipynb), copy and paste your implementation in “Your code” section of the notebook, and run all cells, then your result will be automatically generated. If you have any questions or require feedback for your grade, kindly send an email or drop by my office. Regards, Ezekiel |
Project 3Written on 29.01.19 by Ezekiel Soremekun Hi all, Project 3 has been published! Hi all, Project 3 has been published! |
Project 2Written on 08.12.18 by Ezekiel Soremekun Hi all, Hi all, |
Project 1 - ResultsWritten on 30.11.18 by Nataniel Borges Jr. Hi all,
Results from Project 1 are out! Your grade is based on your coverage with the following bonus being applied:
Hi all,
Results from Project 1 are out! Your grade is based on your coverage with the following bonus being applied:
Regards, Nataniel |
Talk Friday 10:15 Caroline Lemieux (UC Berkeley) on "Expanding the Reach of Fuzz Testing"Written on 29.11.18 by Andreas Zeller Dear all, This Friday, we have a visiting researcher in the field of software test generation and security testing as our guest. Come visit her talk
Expanding the Reach of Fuzz Testing
Caroline Lemieux, UC Berkeley
Friday, November 30, 10:15
CISPA (E9 1), Seminar Room… Read more Dear all, This Friday, we have a visiting researcher in the field of software test generation and security testing as our guest. Come visit her talk
Expanding the Reach of Fuzz Testing
Caroline Lemieux, UC Berkeley
Friday, November 30, 10:15
CISPA (E9 1), Seminar Room 308
Abstract. Fuzz testing, in particular feedback-directed fuzz testing (FDF) popularized by AFL, has gained prominence in recent years in both industry and academia thanks to its bug-finding power and apparent simplicity. A natural question is whether the FDF algorithm is well-suited to achieve other testing goals. In this talk, I will discuss my work on expanding the reach of fuzz testing, centering around three projects that altered FDF to: achieve higher program coverage, discover performance problems, and test programs with highly-structured inputs. In spite of these differing goals, these projects all made similar tweaks to FDF, and I will summarize these common themes.
Bio. Caroline Lemieux is a 3rd year PhD student at UC Berkeley advised by Koushik Sen. Her current research focuses on expanding the reach of fuzz testing to novel testing objectives, with judicious use of user feedback. Her broader research interests are in developing automated methods for software testing and comprehension relying on the power of dynamic validation. She received her B.Sc. in Mathematics and Computer Science at the University of British Columbia.
Looking forward to see you, and best wishes,
Andreas Zeller
|
Course RegistrationWritten on 27.11.18 by Nataniel Borges Jr. Hi all,
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
To be approved in the course it… Read more Hi all,
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
To be approved in the course it is necessary to pass all 3 projects and your final grade will be based on the grade of your projects.
We currently have 25 registrations and 45 submissions.
Students for the Masters in Cybersecurity which are having problems to register at HISPOS, as well as ERASMUS students: please send me an email with your name and matriculation number until December 1st.
Regards,
Nataniel
|
Project submissionWritten on 24.11.18 by Nataniel Borges Jr. Hi all,
Since it seems that there was some doubt about the project submission I'm sending this message. The project should be submitted in the course page submissions tab.
Regards, Nataniel |
Course RegistrationWritten on 23.11.18 by Nataniel Borges Jr. Hi all,
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
To be approved in the course it… Read more Hi all,
Registration and de-registration to the course are open until December 1st on https://www.lsf.uni-saarland.de/.
The grades of the Project 1 (Mutation Fuzzing) will be published after December 1st, only for students which are registered.
To be approved in the course it is necessary to pass all 3 projects and your final grade will be based on the grade of your projects.
Regards,
Nataniel
|
Project 1 TipsWritten on 11.11.18 by Nataniel Borges Jr. Hi all,
I received with questions about how to reach more than 4300 LOC. If you are having problems, it may be in one of the following items:
- A mix between purely random mutations (without converting the SVG into a tree) and specialised mutations helps the fuzzer… Read more Hi all,
I received with questions about how to reach more than 4300 LOC. If you are having problems, it may be in one of the following items:
- A mix between purely random mutations (without converting the SVG into a tree) and specialised mutations helps the fuzzer to explore other code segments, such as XML validations.
- The timeout for the FunctionCoverageRunner may be too small, try overwriting the coverage runner to increase the timeout a bit. That’s a valid solution, as long as the timeout is reasonable [3-5 seconds should do the trick in most modern computers or in the fuzzingbook.cispa.saarland server]. Have a look here https://cms.cispa.saarland/askbot/fuzzing1819/question/29/project-1-evaluation/ to see how to extend the class.
A last tip to improve your coverage is to let the purely random (or your own) fuzzer run for, like 50.000 iterations and see which new children (i.e. children which reaches code locations you could not with 10.000 iterations) were created. This may help you to come up with new mutation operators.
Finally, note that the final evaluation will be executed in the fuzzingbook.cispa.saarland server, thus, it is recommended that you check/evaluate your results there before submission. If you still don’t have access to the server, just send your GitHub data by email to nataniel.borges at cispa.saarland.
Regards,
Nataniel
|
Project 1Written on 31.10.18 by Nataniel Borges Jr. Hi all, Project 1 is out!
The goal of this project is to fuzz a SVG processing library using the mutation fuzzing techniques we saw in the lecture.
The project notebook (Project_MutationFuzzing.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at… Read more Hi all, Project 1 is out!
The goal of this project is to fuzz a SVG processing library using the mutation fuzzing techniques we saw in the lecture.
The project notebook (Project_MutationFuzzing.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
If you download the project make sure to download the data (data/svg-full) folder as well!
To access https://fuzzingbook.cispa.saarland you need to forward us your Github username, then we can grant you access to the system.
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS.
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
Deadline: November 26th 07:00 CET
|
|
Written on 23.10.18 (last change on 23.10.18) by Andreas Zeller Another week, another set of chapters! The next two chapters to work on this week are "Getting Coverage" and "Mutation-Based Fuzzing". After that, we will slow down to one chapter per week. Bug bounties are still on offer – feel free to report bugs anytime. |
Bug Bounties for "Generating Software Tests"Written on 16.10.18 (last change on 23.10.18) by Andreas Zeller Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:
Here, a "bug" is any condition leading to a change. Bugs are best resolved interactively. Come… Read more Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:
Here, a "bug" is any condition leading to a change. Bugs are best resolved interactively. Come and see Professor Zeller with a printout or a list, preferably after the lecture. We will have leaderboards of students reporting the most bugs. The fine print: Offer can be revoked at any time, all further liability is excluded, decisions are final, there is no recourse to the courts, yada yada yada. |
Book "Generating Software Tests"Written on 16.10.18 (last change on 23.10.18) by Andreas Zeller The book "Generating Software Tests" can be found here: https://beta.fuzzingbook.org The chapters to work on this week are the first two: "Introduction to Software Testing" and "Fuzzing: Breaking Things with Random Inputs" |
Security Testing
Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing. In this course, we explore these techniques – in theory and in code.
Course Organization
Every week, you will be provided with Jupyter Notebooks every week that teach a particular topic and illustrate it using plenty of runnable Python code. In the notebook, you can edit the code as you like, run your own experiments, and re-use and extend the code to your liking. Your task will be to use these techniques (and their code) to build a series of fuzzers (i.e. test generators) that find bugs in a number of challenging settings.
This course uses the "inverted classroom" principle – you learn at home, and discuss issues with your instructor. In our weekly meeting, we use the gathering in the lecture hall to introduce the notebooks and projects as well as to discuss the ongoing ones as well as general questions.
Topics and Lectures
- Intro
- Fuzzing
- Simplifying Tests
- Testing with Mutations
- Testing with Grammars
- Grammar Inference
- Taint Analysis
- Coverage-Driven Testing
- Search-Based Testing
- Solving Constraints
- Inferring Preconditions
- Mutation Analysis
- Automated Repair and Protection
- Test Carving and Decomposition
Advanced course (6 CP)
- 12-14 lectures
- Tuesday 16-18 @ CISPA Lecture Hall
- Start: Oct 16