Challenge Rules

• General Terms: This challenge is governed by the General ChaLearn Contest Rule Terms, the Codalab Terms and Conditions, and the specific rules set forth.
• Announcements: To receive announcements and be informed of any change in rules, the participants must provide a valid email.
• Conditions of participation: Participation requires complying with the rules of the challenge. Prize eligibility is restricted by US government export regulations, see the General ChaLearn Contest Rule Terms. The organizers, sponsors, their students, close family members (parents, sibling, spouse or children) and household members, as well as any person having had access to the truth values or to any information about the data or the challenge design giving him (or her) an unfair advantage, are excluded from participation. A disqualified person may submit one or several entries in the challenge and request to have them evaluated, provided that they notify the organizers of their conflict of interest. If a disqualified person submits an entry, this entry will not be part of the final ranking and does not qualify for prizes. The participants should be aware that ChaLearn and the organizers reserve the right to evaluate for scientific purposes any entry made in the challenge, whether or not it qualifies for prizes.
• Dissemination: The challenge is part of the official selection of the NeurIPS 2021 conference. Top ranking participants will be invited to contribute to a collective paper on the results and analysis of the challenge, to appear in the Proceedings of Machine Learning Research (PMLR).
• Registration: The participants must register to Codalab and provide a valid email address. Teams must register only once and provide a group email, which is forwarded to all team members. Teams or solo participants registering multiple times to gain an advantage in the competition may be disqualified.
• Anonymity: The participants who do not present their results at the workshop can elect to remain anonymous by using a pseudonym. Their results will be published on the leaderboard under that pseudonym, and their real name will remain confidential. However, the participants must disclose their real identity to the organizers to claim any prize they might win. See our privacy policy for details.
• Submission method: The results must be submitted through this CodaLab competition site. The number of submissions per day and maximum total computational time are restrained and subject to change, according to the number of participants. Using multiple accounts to increase the number of submissions in NOT permitted. In case of problem, send email to metadl@chalearn.org. The entries must be formatted as specified on the Instructions page.
• Reproducibility: The participant should make efforts to guarantee the reproducibility of their method, for example by fixing all random seeds involved. The organizers reserve the right to eliminate entries with abnormally large variance in terms of performance or to run multiple times the same entry and select the run yielding the WORST result.
• Final phase qualification: Only participants having observed the rules, and having beaten the baseline (fo-MAML) results in the Feedback phase, will qualify for the Final Phase.
• Prizes: The three top ranking participants in the Final phase blind testing may qualify for prizes. The last valid submission in Feedback Phase will be automatically submitted to the Final Phase for final evaluation. The participant must fill out a fact sheet (TBA) briefly describing their methods. There is no other publication requirement. The winners will be required to make their code publicly available under an OSI-approved license such as, for instance, Apache 2.0, MIT or BSD-like license, if they accept their prize, within a week of the deadline for submitting the final results. Entries exceeding the time budget will not qualify for prizes. In case of a tie, the prize will go to the participant who submitted his/her entry first. Non winners or entrants who decline their prize retain all their rights on their entries and are not obliged to publicly release their code.

Credits

This challenge would not have been possible without the help of many people:

• Adrian El Baz (Université Paris-Saclay, France)

• Isabelle Guyon (Université Paris-Saclay; INRIA, France and ChaLearn, USA)

• Jennifer He (4Paradigm, China)

• Mike Huisman (Leiden University, the Netherlands)

• Zhengying Liu (Université Paris-Saclay, France)

• Yui Man Lui (University of Central Missouri)

• Felix Mohr (Universidad de la Sabana)

• Jan N. van Rijn (Leiden University, Netherlands)

• Sebastien Treguer (Université Paris-Saclay, France)

• Wei-Wei Tu (4Paradigm, China)

• Jun Wan (Chinese Academy of Sciences, China)

• Lisheng Sun (Université Paris-Saclay, France)

• Haozhe Sun (Université Paris-Saclay; LISN, France)

• Phan Anh Vu (Université Paris-Saclay; LISN, France)

• Ihsan Ullah (Université Paris-Saclay; LISN, France)

• Joaquin Vanschoren (Eindhoven University, the Netherlands)

• Jun Wan (Chinese Academy of Science, China)

• Benjia Zhou (Chinese Academy of Sciences, China)

The challenge is running on the Codalab platform, administered by Université Paris-Saclay and maintained by:

• Tyler Thomas (CKCollab, USA)

• Loic Sarrazin (Artelys, France)

• Anne-Catherine Letournel (UPSaclay, France)

• Adrien Pavao (UPSaclay, France)

ChaLearn is the challenge organization coordinator. Microsoft and Google are the primary sponsors of the challenge. 4Paradigm donated prizes, datasets, and contributed to the protocol, baselines methods and beta-testing. Other institutions of the co-organizers provided in-kind contributions, including datasets, data formatting, baseline methods, and beta-testing.

Contact the organizers.

Data & Starting Kit

During the challenge, each phase (i.e. Feedback Phase or Final Phase) has 5 meta-datasets, each from a different image domain. Each meta-dataset is used to generate few-shot learning tasks via the same API as the Meta-dataset repo. For more details, please refer to the provided starting kit which can be downloaded/cloned in the following.

Download/Clone the Starting Kit

This starting kit provides a walkthrough to making a valid submission with an example meta-dataset: Omniglot.

For a quick submission, you can directly download this mysubmission.zip file and submit it by clicking on the button "Upload a Submission". This file can also be generated by going through the tutorial notobook in the starting kit.

All images in the challenge are of shape [128,128,3].

Feedback data: The data is kept hidden but you can get quick feedback on the leaderboard.

Final data: This phase will start at the very end of the challenge. The data is kept hidden.