KDD 2019 | Policy Learning for Malaria Elimination

#236

thank you all for your participation

0 Likes

#237

Kudos to @oetbent @sekou and hexagon-ml team for running this awesome kdd cup 2019 competition.
You guys truly did an awesome job.

Thank you all contestants for creating this vibrant community !! Keep learning, sharing and participating !

0 Likes

#238

Hi @oetbent . First of all, I think it’s a wonderful and challenge task for RL community. During my participation, I have learned a lot of knowledge about exploration techniques. From your position rather than changing the position of our team, just running on a task can hardly persuade us that it is a general solution as accidentally overfitted to one task might influence our judgment on the algorithm. So, I suggest that you officially run all submitted solutions on these three environments and give an official and fair leaderboard on these all three tasks, which can help us comprehensively understand the generalizability of these submitted solutions. @taposhdr @oetbent @vlad @hzk123 @thyrixyang

1 Like

#239

What does evaluating on an environment that you have had unlimited access to have to do with generalizability? :slight_smile:
The whole point of the competition (not only to me, this directly corresponds to how our submissions were evaluated) was to develop an algorithm that performs well on the unseen environment, not to optimize the score on the training and validation environments. It makes no sense to change the competition’s metric after it is already over.

1 Like

#240

I am not intended to change the competition’s metric. I just suggest that officially evaluate all our solutions on these three tasks and publish all results. A well-designed solution should at least has the same level performance on train, validation and test tasks. Otherwise, if the solution works nearly the same as the random policy on training or validation task but it performances very well on the test task, it is hard to believe it is a good solution. Additionally, maybe all our submitted solutions are bad on the test environment, so what’s the difference with lottery. @oetbent @taposhdr

2 Likes

#241

Our solutions with optimized Q-learning:

3 Likes

#242

overfitting can be avoided by cross validation.

0 Likes

#243

Is this question solved?

1 Like

#244

Could you please give some explanation about why my submission is judged as invalid? I haven’t received any feedback about that and I think this doesn’t make sense. Thanks in advance.

1 Like

#245

In the rules…

If submitted code cannot be run, the team may be contacted

4 Likes

#246

Hi @oetbent @sekou,

We have a weird issue, Our team name was “A^2” and we don’t seem to appear in the final leaderboard. There seems to be a team named “A2” in the private leaderboads, which isn’t present in the public one. Are we looking at some special char escaping?

It would be amazing if you could confirm if “A2” is actually us. Thanks for the awesome competition.

0 Likes

#247

I think not. The moderators are probably working on publishing the final environment. But this issue is still pretty serious if you ask me.

3 Likes

#249

Hi @apoorvagni

Thanks for participating in the competition. We confirm that A2 is A^2 and we will soon update the leaderboard. We are very sorry for any inconvenience.

0 Likes

#250

Hi everyone. This is a wonderful RL competition. For the purpose of learning, I have carefully ran Hzk’s solution and a random policy on SeqDecChallenge and ProveChallenge environments, the results is interesting:
#########################################################################
SeqDecChallenge
Random policy:
172.32,133.12,162.65,132.93,222.99,183.59,103.73,183.36,197.64,251.10
Hzk:
314.98,399.96,283.30,198.65,319.13,236.39,207.44,302.99,355.52,244.36
#########################################################################
ProveChallenge:
Random policy:
309.54,76.22,250.78,311.18,170.90,110.88,280.46,267.38,141.74,202.42
Hzk:
296.90,50.55,81.40,114.33,146.08,169.06,99.87,80.92,161.13,14.51
#########################################################################
On SeqDecChallenge, Hzy’s solution is better than Random policy. However, on ProveChallenge, it nearly have same performance as random policy, which makes me confused about the final leaderboard. @oetbent @taposhdr .

6 Likes

#251

Same confusion as you! @vlad @taposhdr @oetbent @ikki407 @super11 @thyrixyang

2 Likes

#252

I can confirm your experiment results, if the final test environment has such a large variance, the leaderboard may not be valid.
We are all waiting for the test environment to be published.

2 Likes

#253

Is this post abandoned? We have not heard anything for a while …

2 Likes

#254

I think so. But…where is the test environment?

1 Like

#255

About when will the final environment be opened?

2 Likes

#256

Same Question! :face_with_raised_eyebrow:

1 Like