Selected Topics in Specification and Testing

Dear seminar participants,

Thank you for your interest in software testing and debugging! We're developing Alhazen, a tool that automatically finds out when and why your program fails. We are heavily relying on your opinion as professional developers. Please take part in our user study to help us tie our approach to your needs:

    https://tinyurl.com/debuggingstudy

Thanks a lot!

Andreas Zeller + Team

Dear all,

please prepare an annotated version of your slides and submit it until September 14th (I created a submission slot for that).

The annotations should comprise anything essential that you are planning to say during your presentation which is not obvious from the slides. In particular, this refers to illustration-heavy slides, but also any other interesting remarks that you plan to only express verbally.

You can, for instance, upload a PowerPoint / Keynote / Impress document with comments for each slide, or a PDF export. It's also OK to create a ZIP file with the slides and a text document containing the annotations, as long as it is clear how to associate annotations to slides.

Thank you very much!

Best,
Dominic

Dear all,

as usual for all courses at Saarland University, you have the opportunity to evaluate the seminar and provide feedback – all anonymously, of course. Please do us a favor and fill out the form at

    https://qualis.uni-saarland.de/eva/?l=130530&p=4mizc2

Filling out the form takes only five minutes. Please provide your feedback by July 14!

Best wishes,

Dominic + Andreas

Dear all,

as you can already tell from the title of this message, this is about some totally unrelated things which I wanted to tell you about (but not in separate messages).

In the materials section (auxiliary material), you find a link to the "pFuzzer" approach for systematically covering the input space of a parser. These inputs can, e.g., be used to derive high-quality grammars using Mimid.

Since we already have set our presenters for session #8, there remains no degree of freedom for choosing the presenters of the final 5-min-presentations, which will be Eric and Kev. I'll provide you with more information about the paper we'll discuss at least one week before the session, as usual.

I also wanted to write down Andreas' remark on presenting evaluation results: You have to state the corresponding research question or goal before presenting the actual numbers, since otherwise, those numbers have no real meaning. This is, of course, all the more important when writing a paper (and not presenting it). If you conduct an evaluation without a precise question to answer, it's questionable whether you're conducting anything at all.

Last but not least, some words on the assignments of papers for the long presentations at the end of the seminar: Since by now, most papers are known to everyone, and soon everyone will know about all available papers, we will not have a first-come-first-serve assignments procedure for the long talks to avoid any struggles ;) Instead, I'm going to collect preferences from you and will try to achieve a fair distribution. More about this will follow in due time.

Best Regards,
Dominic

Dear all,

Albeit quite late in the seminar, I'd like to give you some advice on how to write summaries and give talks in the seminar, following Kevin's suggestion. As said before, I want to avoid an over-formalization here, like giving you a Skeleton document which you then fill with words. Rather, I'll provide you with an incomplete list of DO's and DON'Ts that hopefully help you when preparing your summaries and talks.

Summaries

DO:

• Mention the motivation of the work
• Outline crucial technical contributions. What makes this approach special?
• Flexibly go a little more in depth for contributions you find particularly interesting
• Always use your own words for describing the paper's content
• Possibly end with a brief, personal reflection on the paper. You can also mention open questions, such that we can address them in the seminar.

DON'T:

• Focus on formalities, like the authors' affiliations
• Copy-paste content from the paper, or use the wording / justification / explanation of the author's
• Write sentences whose meaning you don't understand. Better mention that this part is difficult to understand / add a question, then we'll clarify it in the seminar!
• Simply write a sentence for each section / subsection in the paper. DO detail the content that YOU deem interesting!
• Skip huge (relatively to the paper / contribution size) parts of the paper which you don't mention at all. If you don't understand them, maybe add a question, but try to understand everything.

Presentations

DO:

• Explain along examples. It's best if you find your own ones, but good examples from the paper are also fine
• Explain all relevant details of the paper. It's your job to decide what's relevant if there's too much content for five minutes
• Rehearse your presentation. Your presentation should take exactly 5min, not much longer, not much shorter
• Raise the interest of your audience! Unicorn pictures are also fine, if they are not too time consuming
• Find the right tradeoff between too many technical details and staying too much on the surface
• Use adequate visualizations
• Proof-read your slides! Avoid typos at all means

DON'T:

• Miss the really important contributions
• Use small fonts, flood your slides with text or illegible graphics. DO use adequate visualizations.
• Attain the author's view. You're an objective observer!
• Add unexplained content to your slide. Everything there should be easy to understand given the previous part of the presentation and your current explanations.
• Use too many bullet points
• Use ugly presentation templates :)

 

I hope this helps! As always, ask if anything is unclear.

Also ask if you don't understand something that might be vital in a paper, especially if you're the next presenter! I'm happy to help out. If it's not critical, the seminar session is always there to clarify everything, of course. Remember that we also have a forum in the CMS, you can use this to discuss with other seminar participants before the sessions.

Sincerely,
Dominic

Dear all,

we already have three presenters for the aforementioned seminar sessions:

May 17th: Rutuja
May 31st: Johannes (QuickSpec) and Kevin (Summaries for String Loops).

Although we only have one paper for May 17th, we still need a second presenter! I apologize for not having mentioned this during the session just now.

If you would like to volunteer for May 17th, please write me an e-mail. Although I have mixed feelings about first-come-first-serve, I think it's acceptable for this occasion (and we've anyway already been practicing this throughout the seminar) ;) However, I will prefer volunteers not having presented so far. Remember that after the seminar, everybody needs to have given two short presentations.

Best Regards,
Dominic

Dear all,

the papers for our sessions at May 17th and 31st are already set. Please note that May 24th is a public holiday, and our seminar consequently will not take place that week. This means that you have more time for reading and summarizing the two papers for May 31st. I would recommend you to use that time, since one of the papers we'll discuss that week is quite lengthy (though not particularly dense).

The paper for May 17th is "Your Proof Fails? Testing Helps to Find the Reason" by Petiot et al. It is about the interesting use case of testing specifications that were not meant to be tested, but have been written for formal (deductive) verification. The authors propose a testing methodology that helps debugging failed proof attempts.

This concludes the first part of the seminar about testing specifications. Next, we will have a look at how to derive them.

The papers for May 31st are "Quick Specifications for the Busy Programmer" by Smallbone et al., and "Computing Summaries of String Loops in C for Better Testing and Refactoring" by Kapus et al. Both discover equational specifications, however using very different methodologies and for different use cases. The QuickSpec paper is a long journal paper, but quite easy to read and not very dense, while the paper about string loops is written in two-column layout and is somewhat more formal in my opinion. Due to the public holiday, you have one additional week for preparing the summaries for these papers.

Please submit your summaries before the day the respective seminar session takes place. Submission does not close before 10:00 AM at the day of the seminar, but this is only meant to counteract technical problems etc. Personally, I'd suggest you already submit the Fridays before, then you have a free weekend!

I have already uploaded all papers to the materials section and also updated the seminar program page.

Best Regards,
Dominic

Dear all,

The CS examination office tells us that all students are asked to register for the exams for the summer semester 2021. For (pro)seminars, registration or cancellation is possible up to three weeks after the topic assignment or kick-off meeting. In our case, this was April 19, 2021, which means that you have to register in LSF by May 10 at the latest.

In case of problems wih the exam registration, students of the MI faculty should contact: studium@cs.uni-saarland.de. Other students have to contact their examination office.

If you are not a student of Computer Science or Cybersecurity, please read on:

These students are not able to register in HISPOS:

• Wirtschaftsinformatiker (Business Informatics)
• VSIMint
• Master IT und Recht
• Bachelor-Plus MINT
• Erasmusstudents
• guest students

Currently the following students of the Computer Science department cannot register in HISPOS:

• Bachelor of Computer Science, StO 2020
• Bachelor Cyber Security, StO 2020
• Bachelor Media Informatics, StO 2020
• Master Media Informatics, StO 2020
• Bachelor Mathematics and Computer Science, StO 2020
• Teacher of computer science, StO 2020

If you fall into any of these categories, let us know, and we will forward your names and grades to the examination office after the seminar. We hope that the courses will be implemented in HISPOS by the start of the exam period and that exam registration will be possible then.

All the best,

Andreas Zeller

Dear all,

in our fourth seminar session (the second "regular" one) which takes place at May 3rd, 4:00 PM, we will again focus on property-based testing. This time, two papers will be discussed, which both propose approaches to steer random input generation for property-based testing into the right directions.

The titles of these papers, which you find in the "Materials" section, are "Semantic Fuzzing with Zest" and "Coverage Guided, Property Based Testing".

This means that for May 3rd, you will have to submit two abstracts, and we will have two short talks on different papers (in contrast to our April 26th session, where we'll have two talks on the same paper). Please submit your abstracts before May 3rd.

We already have two volunteers for the short talks, which are Hannah and Florian.

Best Regards,
Dominic

Hi everybody,

Johannes and Divesh were the first two volunteers for giving short presentations on QuickCheck during our April 26th seminar session. Looking forward to their 5-minute talks!

Brief reminder: We expect short summaries on QuickCheck by everybody, not only Johannes and Divesh.

Best Regards,
Dominic

Hi again,

of course, I found out how to create a forum directly after having told you that I don't know how to do it. You should fine a "Forum" link in the navigation bar for this course. Feel free to use the forum for any questions that in your opinion might be interesting also for the other participants. Otherwise, please write an e-mail to Andreas or myself.

Best Regards,
Dominic

Hi all,

as discussed in our meeting today, our seminar will from now on regularly take place on Monday at 4:00 PM. The Zoom link will stay the same as for today's meeting.

Next week (April 19th), Andreas will give a talk on how to give a talk, so we start a little meta ;)

The week after, at April 26th, we will have our first "regular" seminar session. The topic of that session is Property-Based Testing, in particular the QuickCheck approach. I have already uploaded the paper, you should be able to access it in the materials section. If there are any problems with that, please let me know. Additionally, I added a link to the Fuzzing Book chapter about specification mining, and to the documentation of hypothesis, a package for quick checking python programs. I encourage you to have look at this chapter and to try out hypothesis, it's worth it!

We would like to ask each of you to provide a short abstract / summary of this paper until before Monday 26th. The length of the summary should be about half a page, and be no longer than one page (using some reasonable font settings). Please upload your summary using the CISPA CMS page as a PDF or text file; I just created a corresponding submission slot. If you are facing any problems with this, please also let me know.

So far, we have not chosen the two presenters for the April 26th session. If there are any volunteers, please send me a mail. The two quickest volunteers get the job. In case I don't receive at least two mails, you will get the chance to volunteer during our next session at April 19th. And yes, these will be two presentations on the very same paper ;) Each of these should take five minutes. As a reminder: Neither the summaries nor the short presentations will be graded! Notwithstanding, you should summarize papers for at least 8 out of 10 sessions, and give two short presentations.

If you have any other questions, concerns or ideas, just drop me a mail! I couldn't find a way to create some kind of forum within the CISPA CMS so far, so mails are the preferred way to communicate in between the seminar sessions, at least for now.

Apart from that, I'm very much looking forward to an interesting seminar!

Best Regards,
Dominic

Welcome to the seminar on "Selected Topics in Specification and Testing”! We’re happy you’re with us, and we look forward to an exciting and inspiring seminar.

We have a kick-off meeting on Tuesday, April 13, 17:00, in which we will discuss details of the seminar. This will also likely be the recurrent meeting time of the seminar, so try to keep this space free of other events. Details of the meeting are sent to you by mail.

Looking forward to see you, and best wishes,

Dominic + Andreas