Official Sponsor

Sponsored by ARIA-VALUSPA

Official Sponsor

audeering-AVEC2016

AVEC 2017 menu

Challenge guidelines

Details about the data, performance measure, baseline features and prediction results are provided in the baseline paper, which is now available for download (Updated: 27/05/2016). Please note that the content of this paper may change until the camera ready deadline.

To get started

Please contact Dr. Mercedes Torres Torres by email (Mercedes.Torrestorres@nottingham.ac.uk) to register your team. This email should include:

  • Team name, e.g., name of your research institute/university
  • Team short name (
  • Team members, including affiliation
  • Team contact person
  • Signed EULA(s)

The list of team member must be the same as those provided on the EULAs (see below), and due to the sensitive nature of the data it is not allowed to share the data with other people, including lab mates. Upon registering your team you will be given a link, username and a password to access the files.

The EULA for the Depression Sub-Challenge can be downloaded from the DAICWOZ website.

The EULA for the Emotion Sub-Challenge can be downloaded from the RECOLA website.

After downloading the data you can directly start your experiments with the train and development sets. Once you found your best method you should write your paper for the Workshop. At the same time you can compute your results per instance of the test set and upload them. We will then let you know your performance result. See below for more information on the submission process and the way performance is measured.

Depression Sub-Challenge data

The organisers provide the following data:

  • Audio data (.wav) –scrubbed
  • Audio features (.csv)
  • Transcripts (.csv) –scrubbed
  • Video features (sampled at 30FPS from videos)
    • Pixel coordinates for 68 facial landmarks (.txt)
    • World coordinates for 68 3D facial landmarks (.txt)
    • Gaze vector (.txt)
    • Headpose vector (.txt)
    • HOG features (.bin)
    • Emotion and AU labels (.csv)
  • Depression labels and score per participant, based on PHQ8 questionnaire for the train and development splits (.csv)
  • Gender info for train, development and test splits.

More details can be found in relevant documentation at the DAICWOZ website.

Emotion Sub-Challenge data

The organisers provide the following data:

  • Audio data (PCM/wav)
  • Electrocardiogram data (raw-filtered/csv)
  • Electrodermal data (raw-filtered/csv)
  • Video data (h264/mp4)
    • Audiovisual recordings
    • Pixel coordinates of face bounding box
    • Pixel coordinates of 49 facial landmarks
  • Dimensional affective labels (arousal, valence) for the train and development partitions.
    • Individual labels per recording from each rater (6)
    • Averaged labels per recording from all 6 raters (gold standard used for performance evaluation).

Results submission

Each participant has up to five submission attempts. You can submit results until the final results deadline, which is before the camera ready version deadline. Your best results will be used in determining the winner of the challenge. Please send submissions by email to Stefan Scherer for the Depression Sub-Challenge, and to Fabien Ringeval for the Emotion Sub-Challenge. Participants’ results should be sent as a single zip that includes the name of your team, and the number of this attempt, e.g. results_TEAMNAME_1.zip.

For the Depression Sub-Challenge, the zip file should contain only one directory: “depression”. The data in the results files themselves should be formatted the exact same way as the training/development gold standard label files, that is, one CSV (comma separated value) file named “test_prediction.csv” containing two attributes: participant_ID and the prediction in binary format, i.e. 0 or 1. The organisers will provide the average F1 score for both classes “depressed” and “not_depressed”, which will be used to rank participants. The overall accuracy, average precision, and average recall will be provided as well, which can be used by the authors to further discuss their results in the paper accompanying their submission. Participants in the challenge are also strongly encouraged to provide confusion matrix results on the development set to discuss precision of the algorithms on either classes.

For the Emotion Sub-Challenge, the zip file should contain only two directories: “arousal” and “valence”. The data in the results files themselves should be formatted the exact same way as the training/development gold standard label files, that is, one ARFF file per subject, containing three attributes: Instance_name, frameTime and the prediction in ASCII values. Their filenames should also be formatted in exactly the same way. The organisers will provide for each dimension the Concordance Correlation Coefficient, which will be used to rank participants. The Pearson’s correlation coefficient and the RMS error will be provided as well, which can be used by the authors to further discuss their results in the paper accompanying their submission. Those metrics of performance will be computed on the concatenation of all development/test instances (i.e., gold standard and prediction).

Important: the top-two performers of the challenge will be asked to submit their program to us (Depression Sub-Challenge: Stefan Scherer, University of Southern California, Emotion Sub-Challenge: Fabien Ringeval, University of Passau) to verify the results, both on the original test set and extra hold-out data. The program may be delivered (partly) as an executable or e.g. encrypted Matlab code, and we will endeavour to cater for all possible variations in operating systems, etc., but we do ask you to be available in the period of 14 September – 5 October to work with our team in validating your results.

Paper submission

All papers must be formatted according to ACM proceedings style, and should be no more than 8 pages long. Reviewing will be double-blind. Latex and word templates can be downloaded from the following links:

Latex files (Sample file, Style file, PDF sample file Bib Sample file, and Graph sample file)

Word files

Formatting instructions for other tools

In your submission, please refer to the baseline paper for details about the dataset and baseline results. This makes for a more readable set of papers, compared to each workshop paper repeating the same information. The baseline paper will be made available on the 30th of April.

Papers must be submitted as PDF files in Letter size through the online paper submission system. Submissions must strictly adhere to page limits. Papers exceeding the page limits will be rejected without review. The maximum allowed file size is 10 MB.

Papers should be submitted through the AVEC 2016′s easychair submission site. Deadline is 1 July.

Frequently asked questions/made comments:

Where can I find information about the AVEC 2016 Challenge? Information about the challenge, the data we are releasing, performance metrics and baseline features can be found in the baseline paper, which is now available for download  (Updated: 09/05/2015). Please note that the content of this paper may change until the camera ready deadline.

I have already downloaded the RECOLA database, shall I send another EULA for AVEC 2016? Yes please. The username and password used for RECOLA won’t be the same for accessing the AVEC 2016 dataset, moreover, you may want to change the names of the lab mates that will use the data.

How many times can I submit my results? You can submit results five times. We will not count badly formatted results towards one of your five submissions.

I’ve submitted a set of results, when will I receive my scores? During an active challenge we strive to return scores within 24hrs during typical working days (Monday – Friday), however please try to be patient as this is subject to workload. Under no circumstances should you spam the organisers as this simply delays the process for all teams.

Can I have access to the mailing list of participants? Can you tell me the results of other teams? Absolutely not – registrations and results are not for public view. If there is sufficient demand, we may consider offering a separate opt-in mailing list for teams to discuss the challenge.

Can we use other datasets to pre-train the model? The use of external sources is allowed. Any other dataset can be used for training your models. Please however consider the description of these additional datasets when writing your paper.

Can we use the development and test data during training? Both training and development partitions can be used for training your models, as well as additional resources – see above. The test partition, for which the labels are unknown, must solely be used for testing your system. It is strictly forbidden to perform any annotation on the test partition! The top-two performers of the challenge will be asked to submit their program to us to verify the results, both on the original test set and extra hold-out data.

I have submitted all my results on the test partition, can I know my rank now? The rank of the participants will be announced at the end of the Workshop day, no need to send personal requests, we will keep the final results private until the end.

Will we be ranked with the mean of our best attempt, or will you first select our best scores for the Arousal and Valence independently and then compute a mean? The ranking will be based on the mean of the best CCC score obtained on arousal and valence independently of the attempt. So no need to submit a trial with the best systems tuned for each dimension. Comments on why a specific architecture might work significantly better than another are thus strongly encouraged in case such observations occur.