Generating Software Tests

Dear former course participants,

Thank you for your interest in software testing and debugging! We're developing Alhazen, a tool that automatically finds out when and why your program fails. We are heavily relying on your opinion as professional developers. Please take part in our user study to help us tie our approach to your needs:

    https://tinyurl.com/debuggingstudy

Thanks a lot!

Andreas Zeller + Team

Hi all,

We're lookig for tutors for the new iteration of our current interation of the "Generating Software Tests" lecture.

If you passed the course in the last semester and are interested in becoming a tutor for this semester, please contact Rahul Gopinath (rahul.gopinath@cispa.saarland) or Michael Mera (michael.mera@cispa.saarland).

Regards,

Nataniel

Hi all,

Results for Project 3 has been published, as well as the final grades for the course. Project 3's final grade was computed with one bonus point for greater than average validity, one bonus point for lower than average number of generations and one bonus point for greater than average reachability.

For point reproducibility, the evaluation code for the project containing the details of the time limits, final set of regex and the final population can be found in the evaluation notebook (final_Project_Search_Based_WebFuzzer_eval.ipynb). This is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks

If you want to reproduce the points obtained by your implementation using the evaluation notebook (i.e. final_Project_Search_Based_WebFuzzer_eval.ipynb), copy and paste your implementation in “Your code” section of the notebook, and run all cells, then your result will be automatically generated.

The course final grade was computed as the average of the three project grades. In addition, to pass the course, you are required to pass all three projects with a minimum grade of 4.0.

If you have any questions or require feedback for your grade, kindly send an email or drop by my office (E 9.1, room 2.06).

Regards,

Ezekiel

Hi all,

I have received some questions about how to solve Project 3 and the implementation policy for the project. I will like to clarify some of them here.

In this project, you are allowed to inport the fuzzingbook module, any of the inbuilt python modules (https://docs.python.org/3/py-modindex.html), and the regex module (https://pypi.org/project/regex/). The `regex` module is important because the `re` module does not support the "\p{}" syntax. However, do not install or import any other module other than the aforementioned modules, and do not use (parts of) any open source code, e.g. found online.

Do not make any assumption on the nature of the validation regex, the only valid assumption is that each field would be validated with a regex. Hence, focus on developing fitness function(s) for any arbitrary regex, you are allowed to use any solution you deem fit for your fitness function.

The goal of this project is to search the input space, starting from the provided initial input till you reach a valid input that matches the regex. This goal is to be achieved only via your mutation, selection, crossover and fitness functions. We have used regex-based validators because of its familiarity from Project 2. Simply generating a string that matches the expected regex will be penalized.

Lastly, over the last two weeks some students have raised some questions and issues in the project. The most recent version of the project clarifies all issues raised thus far. Kindly pull the most recent version of the project notebook (Project_Search_Based_WebFuzzer.ipynb) available at https://fuzzingbook.cispa.saarland or for download https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks

Regards,

Ezekiel

Hi all,

Sample solution for Project 1 has been published. It describes the tree and string mutations from three of the top solutions (> 5000 LOC) from Project 1 (Project 1 - Mutation Fuzzing.ipynb). It can be downloaded here: https://cms.cispa.saarland/fuzzing1819/dl/1/Project_1_Sample_Solution_Mutation_Fuzzing.pdf

Regards,

Ezekiel & Nils

Hi all,

Results for Project 2 has been published.

Your grade is based on the coverage achieved by the test inputs generated by your grammar, with one point obtainable for each set of inputs reaching the minimum specified coverage. The evaluation was performed with 86 regexes (including all 26 initial regexes in the Project), consisting of 23 regexes for the basic syntax category, 58 regexes for the complex syntax category and five regexes for the bonus regex category.

For grade understanding and reproducibiltiy, the evaluation code, the details of the final secret_set_of_regex and the minimum coverage used for the evaluation can be found in the evaluation project notebook (Project_Grammar_eval.ipynb). This is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks

Note on result reproducibility: The displayed results in (Project_Grammar_eval.ipynb) is provided as a sample result sheet. If you want to reproduce the points obtained by your implementation using the evaluation notebook (i.e. Project_Grammar_eval.ipynb), copy and paste your implementation in “Your code” section of the notebook, and run all cells, then your result will be automatically generated.

If you have any questions or require feedback for your grade, kindly send an email or drop by my office.  

Regards,

Ezekiel

Hi all,

Project 3 has been published!
 
The goal of this project is to implement a Search-based Web Fuzzer.

The project notebook (Project_Search_Based_WebFuzzer.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
You should all have access to the server (https://fuzzingbook.cispa.saarland) by now. In case you do not have access yet (or lost your access), forward us your Github username, then we can grant you access.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS: https://cms.cispa.saarland/fuzzing1819/students/view#submissions.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: February 26th 2019, 07:00 CET

Hi all,

Project 2 has been published!
 
The goal of this project is to implement an algorithm that constructs a CFG from an arbitrary regular expression.
 
The project notebook (Project_Grammar.ipynb) is available at https://fuzzingbook.cispa.saarland or for download at https://github.com/uds-se/fuzzingbook/tree/master/docs/beta/notebooks
 
If you download the project make sure to also download the data (data/regex) folder as well!

You should all have access to https://fuzzingbook.cispa.saarland by now. In case, you do not have access, forward us your Github username, then we can grant you access to the system.
 
You can develop your solution as you wish, but for submission you have to write it down in the provided notebook and upload it to the CMS: https://cms.cispa.saarland/fuzzing1819/students/view#submissions.
 
More instructions on how to solve the project are in the notebook. If you have any doubt, just send us a question on the course AskBot.
 
Deadline: January 7th 2019, 07:00 CET

Hi all,

Results from Project 1 are out!

Your grade is based on your coverage with the following bonus being applied:

• +50 LOC for finding an error in the library (the bonus is applied 1x irrespectively of the number of exceptions found. Note that XML syntax validation exceptions are not errors but intended behavior).
• +50 LOC for reaching an overall coverage of more than 5500 unique statements throughout the 5 runs.

Regards,

Nataniel

Dear all,

Hi all,

Andreas Zeller offers bounties for bugs found in the notebooks distributed every week:

• 0.20€ for every new bug in the writing (i.e. text cells)
• 0.50€ for every new bug found in the code.

Here, a "bug" is any condition leading to a change.  

Bugs are best resolved interactively.  Come and see Professor Zeller with a printout or a list, preferably after the lecture.  We will have leaderboards of students reporting the most bugs.

The fine print: Offer can be revoked at any time, all further liability is excluded, decisions are final, there is no recourse to the courts, yada yada yada.