3 Rankings

Leaderboard

Attention

2024-10-22: Code and Documentation

For eligibility to final prizes, please prepare your repo/code and documentation.
(See email sent to team leaders)

2024-10-17T18:16Z Final phase has started!

Submission shall be send for the 158149 flight_id’s as in

competition-data/final_submission_set.csv

for eligibility to final prize.

The following table shows the best result per team’s submission.

To see all submissions’s RMSE’s please use https://datacomp.opensky-network.org/api/stats

To see all rankings (in case the table below doesn’t work/refresh) please use https://datacomp.opensky-network.org/api/rankings

(Submissions are processed automatically every 30 minutes.)

ranking_url = "https://datacomp.opensky-network.org/api/rankings"
ranking = await aq.loadJSON(ranking_url)

thresholds = [
  { percent: 0.01, threshold: 956.5586 },
  { percent: 0.02, threshold: 1913.117 },
  { percent: 0.03, threshold: 2869.676 },
  { percent: 0.05, threshold: 4782.793 },
  { percent: 0.1, threshold: 9565.586 },
  { percent: 0.15, threshold: 14348.38 }
]
ll = new Map(thresholds.map((i) => [100 * i.percent, i.threshold]))
formatTime = d3.utcFormat("%Y-%m-%d %H:%M");

Inputs.table(ranking
    .orderby("rank")
    .select({ team_name: "team" }, "rank", { file_version: "version" }, "rmse", { submission_date: "date"})
    .derive({ date: (d) => aq.op.parse_date(d.date) })
    .derive({ version: (d) => +aq.op.replace(d.version, "v", "") })
    .derive({
      mean_error: aq.escape((d) => {
        if (0 <= d.rmse && d.rmse < ll.get(1)) return 1;
        if (ll.get(1) <= d.rmse && d.rmse < ll.get(2)) return 2;
        if (ll.get(2) <= d.rmse && d.rmse < ll.get(3)) return 3;
        if (ll.get(3) <= d.rmse && d.rmse < ll.get(5)) return 5;
        if (ll.get(5) <= d.rmse && d.rmse < ll.get(10)) return 10;
        if (ll.get(10) <= d.rmse && d.rmse < ll.get(15)) return 15;
        return 16;
      })
    })
    .derive({
      error: aq.escape((d) => {
        if (d.mean_error > 15) return "> 15%";
        return "≤ " + d.mean_error + "%";
      })
    })
    .select("team", "rank", "version", "rmse", "error", "date"),
  {
    rows: 50,
    maxWidth: 960,
    // remove checkboxes to select rows
    select: false,
    format: {
      team: (name) =>
        htl.html`<a href="https://ansperformance.eu/study/data-challenge/teams.html#${encodeURIComponent(
          name
        )}" target="_blank">${name}</a>`
    }
  }
)

Criteria for Ranking Intermediate Submissions

Attention

2024-10-17T18:16Z Final phase has started!

Submission shall be send for the 158149 flight_id’s as in

competition-data/final_submission_set.csv

Intermediate submissions are ranked on RMSE of the submitted tow with respect to the (hidden) “ground truth for the flight_id’s in submission_set.csv”.

There are 105959 flight_id’s in submission_set.csv.

Criteria for Ranking of Final Submission

Attention

2024-10-17T18:16Z Final phase has started!

Submission shall be send for the 158149 flight_id’s as in

competition-data/final_submission_set.csv

Final ranking will be based on

RMSE from competition-data/final_submission_set.csv (it will be posted around 1 week before the deadline) which includes an additional 52190 flights compared to competition-data/submission_set.csv

Eligibility for Prize

The winning solutions will only be eligible for the awards if

all source code and additional data sets used to generate the results from the measurement data will be made available to be published under the GNU GPLv3 license on the OpenSky Network’s github account.
sufficient documentation is provided to understand and reproduce the results.
the solution is original: teams must use their own original solutions. Re-using any existing implementation is only allowed if the original authors grant you the rights to use their solution and if you made significant modifications to the algorithm or model. In particular, simply re-using existing code and rewriting the data input and output mechanism is not sufficient. Adding parameters to the model and modifying filters to match the specific peculiarities of the data, however, can be considered sufficient.

Final Prize

The Selection Committee will review and discuss the top ranking teams and select final Awardees considering

their final ranking score,
the originality of the proposed model
the open availability of the code and the relevant documentation provided.

By participating in the Challenge, you agree to be bound by the final decision of the PRC Data Challenge selection committee.