Multi-dataset Time Series Anomaly Detection

Anomaly Detection in Time Series

In recent years there has been an explosion of papers on time series anomaly detection appearing in SIGKDD and other data mining, machine learning and database conferences. Most of these papers test on one or more of a handful of benchmark datasets, including datasets created by NASA, Yahoo, Numenta and Tsinghua-OMNI (Pei’s Lab) etc.

While the community should greatly appreciate the efforts of these teams to share data, a handful of recent papers [a], have suggested that these are unsuitable datasets for gauging progress in anomaly detection.

In brief, the two most compelling arguments against using these datasets are:

· Triviality : Almost all the benchmark datasets mentioned above can be perfected solved, without the need to look any at any training data, and with decade-old algorithms.

· Mislabeling : The possibility of mislabeling for anomaly detection benchmarks can never be completely eliminated. However, some of the datasets mentioned above seem to have a significant number of false positives and false negatives in the ground truth. Papers have been published arguing that method A is better than method B, because it is 5% more accurate on benchmark X. However, a careful examination of benchmark X suggests that more that 25% of the labels are wrong, a number that dwarfs the claimed difference between the algorithms being compared.

Beyond the issues listed above, and the possibility of file drawer effect [b] and/or cherry-picking [c], we believe that the community has been left with a set of unsuitable benchmarks. With this in mind, we have created new benchmarks for time series anomaly detection as part of this contest.

The benchmark datasets created for this contest are designed to mitigate this problem. It is important to note our claim is “mitigate”, not “solve”. We think it would be wonderful for a large and diverse group of researchers to address this issue, much in the spirit of CASP [d].

In the meantime, the 250 datasets that are part of this challenge reflect more than 20 years work surveying the time series anomaly detection literature and collecting datasets. Beyond the life of this competition, we hope that they can serve as a resource for the community for years to come, and to inspire deeper introspection about the evaluation of anomaly detection.

We hope you will enter the contest, and have lots of fun! Please use this forum to share, ask questions and collaborate.

Best wishes,

Prof. Eamonn Keogh, UC Riverside and Taposh Roy, Kaiser Permanente

[a] [2009.13807] Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress. Wu and Keogh

[b] Publication bias - Wikipedia

[c] Cherry picking - Wikipedia

[d] CASP - Wikipedia

1 Like

Hi, nice challenge!
One question regarding the deadlines: what exactly is the difference between phase 1 and 2? What is the deadline for submitting a final solution? Thanks in advance!

2 Likes

Thank you and welcome to the competition!

Dates as mentioned on the completion site:
Phase 1 : March 15- April 7th.
Phase 2: April 8th - June 1st

  • Evaluation for this competition will be done based on the outcomes of Phase II only. Phase 1 allows you to prepare and evaluate yourself before Phase 2.
  • Phase I (warm-up) will have only 25 files, while Phase II (graded competition) will have 200 files. Your scores from Phase I in the leaderboard will cleared out for Phase II.

Please go through the links below for the rules, evaluation and getting started video

https://compete.hexagon-ml.com/practice/competition/39/#rules

https://compete.hexagon-ml.com/practice/competition/39/#evaluation

https://compete.hexagon-ml.com/practice/competition/39/#getting_started

Thank you!

2 Likes

Thank you for organizing this nice challenge.I wonder if we will have a multivariate time series ?

1 Like

Hi Seif,

Univariate time series is most challenging, since there is no contextual information from other variables that can be obtained. This competition will focus on univariate for anomaly detection.

In the future we may think of multivariate, but we dont have plans at present.

1 Like

How do I invite members into my team ?
Should they have been registered for the competition (they have already registered
on the hexagon-ml platform, but not for the competition) ?
Since when I searched their username in the “Invite Others” tab, it displays
“error, try again”. I also tried searching their email-ids.

Thank You

To invite members they should be both registered to Hexagon-ml and have joined the competition. Once they are then you can search and add them using their display name. Emails will not work.

2 Likes

Thank you for clearing it up. I have a related question.

How many files of the 200 are evaluated for the public leaderboard in phase 2? Thank you.

Phase I is warm-up with 25 files.
The current public leaderboard will be cleared when we start phase 2.
Phase II has 200 files. These will result in the Public leader board.

After we review the code we will select the private leaderboard. Code is required for top 15 places. Others are highly recommended to share source code. We have not seen real code being shared yet.

Thank you for the reply.

After we review the code we will select the private leaderboard.

Does it mean the private leaderboard in phase 2 is based on the same 200 files as in the public leaderboard? If so, it is a bit concerning that people would just overfit public leaderboard in phase 2.

2 Likes

企业微信截图_16164905744513

When I submit the result, I get this prompt, but the file I submitted is a csv file.

I checked the file format (column name, type), it seems to be ok ???

submit submissionsample.csv Same problem

Very strange, but no problem with firefox

Important Note:

At the beginning of Phase II we will clear the leaderboard. We have to do this, because we have reasonable doubt that there are a lot of duplicate accounts and hand labeling done.

Also, we will be requesting two new things:

  1. Affiliations - We request you to provide your school or work email addresses as affiliations. You can also participate un-affiliated, however you cannot change from un-affiliated to school or work.

  2. Submission of code : We will be requiring submission of code along with each submission you make in a day.

Please note: We reserve the rights to remove participants whom we find reasonable doubts of hand labeling or having multiple accounts. It is against the spirit of the competition.

Some Questions & Answers:

Q) Why must you submit code with every attempt?
(A) Recall the goal of the contest is not to find the anomalies. The goal of the contest is to produce a single algorithm that can find the anomalies [*]. If your submission turns out to be competitive, your submitted code allows you to create a post-hoc demonstration that it was the result of a clever algorithm.

(Q) Why must you use an official university or company email address to register?
(A) Experience in similar contests suggest that otherwise individuals may register multiple times to glean an advantage. It is hard to prevent multiple registrations, but this policy goes someway to limit the utility of an unfair advantage.

[*] Of course, the “single algorithm” can be an ensemble or a meta-algorithm that automatically chooses which algorithm and which parameters to use. However, having a human to decide which algorithm, or which parameters on a case-by-case basis is not allowed. This is not a test of human skill, this is a test of algorithms.

2 Likes

Will the submitted code (required for each submission in the Phase II) be made publicly available or will it only be used for internal review by the committee?

Can I decide whether to make my submission visible on the public leaderboard?

The goal of the contest is to produce a single algorithm (ensemble or a meta-algorithm) that can find the anomalies. The spirit of the competition is to provide open source code for this noble task. While we may not release code for each submission during the competition. We may choose to review and disqualify if we find reasonable doubts of hand labeling, multiple accounts and any other cheating.

“re-register” means that we need to register a new account in this platform, then joined this competition again? and the older account can’t be used, is right?

We will make it easy for the competitors and you may not have to re-register in the platform. Only during submission provide details.

1 Like

thanks for your reply, another question is that if we need to add details(school/work email) in each submission? or just for final 2 submissions.
thanks!

it would be better not to put all 200 datasets for phase II.

If one submit multiple times, he can guess the location of the anomalies. Thus, he can use these labeles…the paradigm changes and we move to the full supervised one…

I suggest to not give a ground truth for at least 30 datasets…

1 Like