Workshop on the International Planning Competition (WIPC)

1. Overview

The International Planning Competition (IPC) is held every few years in the context of ICAPS. It empirically evaluates state-of-the-art planning systems on a number of benchmark problems. The goals of the IPC are to promote planning research, highlight challenges in the planning community and provide new and interesting problems as benchmarks for future research. The IPC has an important role in the ICAPS community, being a forum to compare different algorithmic ideas and implementations, and setting standards for research and evaluation in the area.

Similar to the lineage of IPC workshops organised at ICAPS 2003, 2007, 2012, and 2015 this workshop aims to review the current status of the IPC, analyze the results of the last IPC (2018), and provide a venue for discussing aspects that will be helpful for preparing forthcoming competitions.

1. Proceedings

The proceedings of the workshop are available here

1. Schedule

09:00	Session 1: Interpreting Results Opening Remarks Performance Robustness of AI Planners to Changes in Software Environment (15 minutes) Chris Fawcett, Mauro Vallati, Alfonso E. Gerevini, and Holger H. Hoos Insights from the 2018 IPC Benchmarks (30 minutes) Isabel Cenamor and Alberto Pozanco An Analysis of the Probabilistic Track of the IPC 2018 (30 minutes) Florian Geißer, David Speck, and Thomas Keller
10:30	Coffee Break
11:00	Session 2: Metrics and Portfolios Cost-Optimal Planning in IPC 2018: Symbolic Search and Pattern Databases vs. Portfolio Planning (15 minutes) Stefan Edelkamp and Ionut Moraru Democratizing Usage of Planning Systems by Facilitating Research in Algorithm Selection for Planning (5 minutes) Michael Katz and Silvan Sievers Benchmarks Old and New: How to compare domain independence for cost-optimal classical planning? (5 minutes) Ionut Moraru and Stefan Edelkamp Planner Metrics Should Satisfy Independence of Irrelevant Alternatives (15 minutes) Jendrik Seipp Open Discussion on Portfolios and Benchmark Design
12:30	Lunch Break
14:00	Session 3: Rules and Tracks The Role of IPC in Setting Standards for Experimental Evaluation in Planning Research (15 minutes) Michael Katz and Silvan Sievers Hierarchical Planning in the IPC (30 minutes) Gregor Behnke, Daniel Höller, Pascal Bercher, Susanne Biundo, Damien Pellier, Humbert Fiorino, and Ron Alford Open Discussion on Tracks Closing Remarks

1. Topics

Topics include, but need not be limited to:

• Past, present and future of the IPC
• Tracks: Proposing new and/or revising the current tracks
• Domain Definition Languages: benefits and limitations. Should larger subsets of PDDL or RDDL be considered in next editions of the IPC? Should the input languages evolve in some way?
• Benchmark Domains: Discussion/Analysis of existing domains. Are IPC benchmarks a good standard for evaluation of planners? How should instances be chosen for the competition?
• Evaluation Metrics: Should we adapt the metrics used for evaluating the performance of planners? How should the winners of the competition be decided (e.g. analysis of advantages and disadvantages of IPC 2018 vs. SPARKLE Planning challenge 2019)?
• Results: interpretation of past results, new ideas about how to analyze/compare the performance of planners beyond the score of the competition.
• Rules: Should portfolios be given a special treatment? Should there be any limits on re-using code from others?
• Tools: Can we automate large parts of the IPC to make organising an IPC track easier?

1. Submissions

We encourage submissions in three formats. All submissions should be accompanied by a paper in AAAI style (see instructions at http://www.aaai.org/Publications/Author/author.php). In all tracks you can chose the length of your submission freely between 1-8 pages + one page for references. Please choose a length appropriate for the format of your submission.

• long presentations have a 30 minute slot, including discussions. We expect most papers for this format will make use of all 8 pages.
• short presentations have a 15 minute slot, including discussions. We expect the average paper in this format will have around 4 pages.
• super-short presentations have a 5 minute slot, including discussion. We expect submissions to consist of an idea that can be presented on 1-2 slides but lead to some discussion, like a suggestion for a new track or metric. Submissions in this track should consist of a title and abstract, so no PDF submission is required (though it is also welcomed).

As suggested by the list of topics, we encourage the submission of position papers as well as other submissions that do not fit the mold of a typical conference paper. Paper Submissions should be made through OpenReview. The review process will be single blind for reviewers and double blind for the general public. All reviews will be made public via the OpenReview system.

1. Important Dates

• Submission deadline: March 25 (extended to April 12)
• Notification: April 30
• Camera Ready: May 31
• Workshop: July 11 or 12

1. Organizers

• Alvaro Torralba (Saarland University)
• Florian Pommerening (University of Basel)
• Thomas Keller (University of Basel)
• Amanda Coles (King's College London)
• Andrew Coles (King's College London)