![\"Flight](\"https://storage.googleapis.com/kaggle-media/competitions/gequest/3521/media/flightstats_logo_dark.png\"){kind=logo}
The code linked above is provided subject to the limited License to Code contained in the Competition Rules. See the included license.txt for more information.
\r\nFiles used and needed by the Flight Simulator can be found on the Flight Quest Data wiki.
\r\nFlight Quest Phase 2 data is temporally split into 4 datasets that will be released over the course of the competition. Flight data in each dataset is for the following time periods:
\r\nThe leaderboard will be based on data from the following time periods:
\r\nGuide To Files
\r\nNaming convention: Revised files will have \"RevX\" in the name where X represents the revision number. Previous versions of the file will be removed.
\r\nFor a fuller description of the data in AggregateTrain and AggregateTestFeatures, visit the Flight Stats Data page.
\r\nVisit the Timeline to learn about when data will be released.
\r\nDatasets listed here are provided, in part, by Flight Stats, Inc.
\r\nPlease see the Flight Stats Data wiki page for another source of approved weather data.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Description", "content": "Did you know airlines are constantly looking for ways to make flights more efficient? From gate conflicts to operational challenges to air traffic management, the dynamics of a flight can change quickly and lead to costly delays.
\r\nThere is good news. Advancements in real-time big data analysis are changing the course of flight as we know it. Imagine if the pilot could augment their decision-making process with “real time business intelligence” — information available in the cockpit that would allow them to make adjustments to their flight patterns.
\r\nYour challenge, should you decide to accept it:
\r\nUse the different data sets found on this page under Get the Data to develop a usable and scalable algorithm that delivers a real-time flight profile to the pilot, helping them make flights more efficient and reliably on time. For background on the competition structure, visit the Basic Structure page.
\r\nBe sure to check the Forums regularly to stay on top of the latest competition news.
\r\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "In this Flight Quest, you are tasked with creating algorithms to generate optimal flight plans (routes) given a flight's origin and destination airports, current air traffic conditions, and weather information.
\r\nEach route, represented as a series of points (latitude and longitude) with airspeed and altitude, will be evaluated against the Cost Function.
\r\nThe Cost Function, a measure of the relative cost in dollars of the flight given a submitted flight plan, seeks to quantitatively capture the direct and indirect costs of flights. See the Simulator code for the exact Cost Function.
\r\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "The complete Official GE Flight Quest 2 rules are available in PDF form.
\r\nClick here to view or download the complete Official GE Flight Quest 2 rules. Dates have changed. See Timeline page.
\r\nContest ends at 11:59:59 p.m. UTC on February 23, 2014. Must submit at least one Entry and final model codebase before 11:59:59 p.m. UTC on January 18, 2014 to be eligible. Must be 18 years of age or older to participate. Free GEQuest website or Kaggle account required. No purchase necessary. Void where prohibited.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "\r\n
Phased Launch of Flight Quest 2
\r\nThis will be a phased launch. On Monday August 5, the Flight Quest pages will open for data downloads. Submissions will be accepted and the leaderboard will be revealed by the end of the following week.
\r\nUpdates to the Flight Simulator
\r\nKaggle may release an updated version of the Simulator following the Milestone deadline sometime during the first 2 weeks in October. This release will incorporate bug fixes and may include additional constraints to the competition, such as no-fly zones and inclement weather. Details will be provided in the Forums, so please sign up to watch them.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Basic Structure Of Fq2", "content": "The goal of Flight Quest 2 is to develop an Agent, a computer program that recommends the route (defined by latitude, longitude, altitude, and airspeed) a plane should take for the remainder of each flight provided in the test set.
\r\nEach route will be simulated through our Flight Simulator, which calculates the fuel burn during the course of the flight, while also accounting for things like weather and changing aircraft weight.
\r\nThe goal is to optimize the Cost Function, which quantifies the costs related to fuel burn, aircraft costs, and other passenger costs.
\r\nThe Agent that produces a set of routes with the lowest average cost will be at the top of the leaderboard.
\r\nGetting Started
\r\n1. Each data release covers a particular time period. The first data release consists of 4 weeks of training data (June 7 - July 4) and 2 weeks of test data (July 5-18).
\r\nThe training data provides all available information about flights: their destination and arrival airports, scheduled and actual departure and arrival times, position during the course of the flight, filed flight plans and changes to flight plans, etc. Also provided is weather information related to the flights, such as runway conditions, sky conditions, visibility, airport alerts (e.g., fog, icing), etc.
\r\nThe test data is used to create the test set collection of flights and the associated test feature set. We ask participants to submit routes for the test flights, identified by flight id and listed in the file TestFlights.csv. To create this collection of flights, we pick a random cutoff time for each day in the test set. We filter for flights that are above 18,000 feet, have destination and arrival airports in the contiguous US, are more than 75 miles from the destination airport, are flying a medium-range aircraft, and are in the air at the random cutoff time. Your job as participants is to “fly” these planes to their destinations in the most efficient way that minimizes the cost function. The test feature set provides information about the test flights up to the cutoff time. After the cutoff time, no additional information is available, which mimics real-life situations where the future is not known. Your models must be built using existing information and be robust to the uncertainty of the future.
\r\n2. Each flight is optimized independently; your instructions for a flight are only used when the cost of that flight is being simulated. That flight reverts to its real-life path (and may for example affect airport congestion) when another flight is evaluated.
\r\nLet’s say we have a set of 10 flights. You need to develop optimal routes for each of the flights. The Simulator then evaluates each flight in turn. For each flight, the remaining 9 flights continue to exist and contribute to overall conditions (congestion at the destination airport, for example) but you do not have direct control over those flights. So part of building optimal routes will include developing models to account for the current conditions a specific plane may experience.
\r\n3. A flight is simulated by moving towards the Agent’s indicated waypoints at a consistent time interval. At each step in the simulation, the Simulator moves the plane based on pertinent conditions (e.g., weather and ground conditions at airports) for that time and location. Participants only have the current conditions up to the cutoff time for the test flights. After the cutoff time, the conditions that will affect the rest of the flight are \"future information\" which the Agent you develop will not have.
\r\n4. The best way to get started is to select a few flights from the training data set, run them through the Simulator, and understand what the Simulator is doing and what are its required inputs. Determine which inputs you have in the test set and which you do not have and then develop a strategy for routing planes using the inputs available.
\r\n5. The Simulator also incorporates some constraints and penalties which are necessary to model aspects of the real world, to make the competition interesting, or to make scoring submissions feasible. These aspects can be discovered in the Simulator code and comments. One constraint worth mentioning here is the maximum allowed number of instructions for each flight is 200 instructions.
\r\nPlease be sure to watch the forums for the latest on Flight Simulator code releases.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Faq", "content": "Click here for the Complete Competition Rules
\r\nWhat are the Industrial Internet Quests?
\r\nThe Industrial Internet Quests are a challenge to developers and data scientists to create the best new algorithms to reduce air travel delays. The Quest will span several months and submissions will be reviewed and winners judged based on automated rules (scored by a leaderboard). Winners will receive a total prize pool of $250,000 in cash rewards as well as the opportunity to have their submission turned into a working simulation, with the ultimate goal of creating an onboard flight management application supported by GE.
\r\nProposals will be submitted online via www.gequest.com
\r\nThe Flight Quest 2, launched in collaboration with Alaska Airlines, is part of an ongoing series of Industrial Internet challenges. GE is partnering with Kaggle, a leader in big data challenges, to execute the Quests.
\r\nWhy is GE issuing these Industrial Internet Quests?
\r\nGE believes that the Industrial Internet represents the next frontier of systemic technical advances and innovations that enable breakthrough levels of efficiency and productivity. By creating an open platform and engaging with the developer community and data scientists to develop Industrial Internet expertise, we will be able to eliminate waste and increase efficiency to invent the next generation of industrial services.
\r\nThe Quests:
\r\nWho is eligible to enter?
\r\nEntries are open to anyone 18 years old or older, and all legally formed entities, except a bankrupt person or organization. The Quests are global, except where prohibited by law.
\r\nAre GE employees eligible to enter?
\r\nOfficers, directors, employees and advisory board members (and their immediate families and members of the same household) of GE, Kaggle Inc. and their respective affiliates, subsidiaries, contractors, agents, judges and advertising and promotion agencies also are not eligible to participate in the Competition .
\r\nAre there any age parameters?
\r\nEntries are open to anyone 18 years old or older.
\r\nWhat makes GE qualified to issue the Quests?
\r\nAt GE, we’re working on exciting and promising Industrial Internet technologies to help create more efficient air travel. As part of this, we are looking to engage with data scientists to create an open platform, cultivate a community and develop a widespread expertise in the Industrial Internet. GE has partnered with Alaska Airlines to help develop new technologies.
\r\nDoes GE plan to announce additional Industrial Internet Quests in the future?
\r\nGE plans to launch a series of “Industrial Internet Quests” in 2013. These quests would embrace the goals of open collaboration with our customers, developers, data scientists and designers to help GE in transforming various industries across Healthcare, Aviation, Rail, Oil & Gas and Power.
\r\n
Entry and judging process:
What is the entry and judging process for Flight Quest?
The second phase of Flight Quest asks participants to develop agents that will provide the optimal route for planes in mid-flight. The Flight Quest will use Kaggle’s live competition platform to submit and analyze predictive models. Visit the competition pages for more details on the submission and scoring process.
\r\nWhat information will GE provide for the Flight Quest?
\r\nThe information will include: flight number, origin, destination, take-off time, arrival time, latitude and longitude at frequent interim waypoints along the journey, and weather and wind data. Based on this information, entrants are challenged to design an algorithm for a flight management system that advises pilots as to the optimal route. Data is provided by FlightStats, Inc.
\r\nCan an individual submit more than one entry?
\r\nYes, multiple entries are permitted. Each participant will select their best entry for final evaluation.
\r\nCan individuals collaborate to submit a single entry?
\r\nYes, multiple individuals or entities may collaborate to submit a single entry. In the Flight Quest, all team members may submit entries on behalf of the team.
\r\nAre you working with any outside partners, or firms to manage the entry process?
\r\nTo execute the Flight Quest, GE has partnered with Kaggle, the leader in big data challenges and analysis.
\r\nCan entrants enter other similar challenges?
\r\nYes.
\r\nWho will comprise the judging panel?
\r\nThe Flight Quest will be scored by Kaggle’s live leaderboard.
\r\nHow much money will be awarded?
\r\nThe combined winners from the second phase of Flight Quest will receive a total prize pool of $250,000 from GE.
\r\nAre there specific countries you are targeting?
\r\nThis is a global challenge and we encourage and welcome entries from everyone who meets the eligibility requirements.
\r\nIs there a contact name or number for people with questions?
\r\nAll of the information for the Industrial Internet Quests will be available at www.gequest.com. The specific terms for the Flight Quest can be found at www.gequest.com/flight2.
\r\nWhere can I find more answers?
\r\nThe competition will have Forums for live discussion and addressing questions that arise during the course of the Quest.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Flight Stats", "content": "
PDF version of this data overview. Visit the Data page for a complete data dictionary.
\r\nThis document serves as a guide to the files provided by FlightStats for the GE Flight Quest contest. The data set includes information for:
\r\nOn a given day, between 80,000 and 105,000 commercial aviation flights operate worldwide. The flight_history table represents information on these operated flights with data collected from airlines, airports, radar, government and civil aviation authorities. For a particular flight record, the following information is available:
\r\nThe flight_history_events table lists the changes made to a given flight_history entity over time. Interpreting this table correctly is important to representing the state of a given flight at a moment in time. There are a number of considerations to consider when interpreting the data. First, the date_time_recorded field is provided in Pacific Time. The initial state of a flight consists of its published and scheduled gate times. As new information is reported about the flight, a change is made to the record and encapsulated in a row in the flight_history_event table. Details about the change are stored in a data_updated field. This field includes a commaseparated encoding of updated flight_history fields. Each entry in the data_updated field begins with the abbreviation of the field updated followed by the old value (if a previous value was present) and the new value.
\r\nFor example, a data_updated value of:
\r\nARD- New=10/25/12 14:35, STATUS- Old=S New=A
\r\nrepresents a change to ARD (Actual Runway Departure) setting the field' initial value to 2012-10-25T 14:35 and a change to the STATUS of the flight from S (Scheduled) to A (Active).
\r\nThe following table lists various encodings.<.p>
\r\n| Code | \r\nField Name | \r\n
| SGD | \r\nScheduledGateDeparture | \r\n
| EGD | \r\nEstimatedGateDeparture | \r\n
| AGD | \r\nActualGateDeparture | \r\n
| SRD | \r\nScheduledRunwayDeparture | \r\n
| ERD | \r\nEstimatedRunwayDeparture | \r\n
| ARD | \r\nActualRunwayDeparture | \r\n
| SGA | \r\nScheduledGateArrival | \r\n
| EGA | \r\nEstimatedGateArrival | \r\n
| AGA | \r\nActualGateArrival | \r\n
| SRA | \r\nScheduledRunwayArrival | \r\n
| ERA | \r\nEstimatedRunwayArrival | \r\n
| ARA | \r\nActualRunwayArrival | \r\n
| DGATE | \r\nDepartureGate | \r\n
| AGATE | \r\nArrivalGate | \r\n
| DTERM | \r\nDepartureTerminal | \r\n
| ATERM | \r\nArrivalTerminal | \r\n
| BAGGAGE | \r\nBaggageClaim | \r\n
| STATUS | \r\nFlightHistoryStatusCode | \r\n
| AEQP | \r\nActualAircraftType | \r\n
| SEQP | \r\n@ScheduledAircraftType | \r\n
| PD | \r\nActualAircraftType | \r\n
| AEQP | \r\nActualAircraftType | \r\n
The data set includes flight plans for commercial aviation filed with the FAA. The flight plans are enumerated in the file asdi_flight_plans. Plans include information provided to the FAA including the original and estimated runway times, route (departure airport and arrival airport) and where
applicable a link back to the Flight History record. The Flight Plan also contains several structures in separate files that describe the intended path of the plane. There are numerous descriptions of this data including airways, centers, fixes, and sectors; however, the useful file will be asdifp_waypoints which describes the path in latitude and longitude.
The actual path taken by an aircraft is described in the asdi_positions file. Each row in this file represents a radar observed position of an aircraft and contains:
\r\nWeather affects aviation and is a cited reason for delays. The data set includes several weather products:
\r\nMETAR refers to the current conditions at an airport weather station. The information is typically published hourly, but may be more or less frequent. Additional information about the interpretation of a METAR report can be found here. The information in a METAR report is broken into a number of files for easier handling:
\r\nThe file metar_reports contains the overall report and standard items broken into specific fields including:
\r\nThe file metar_present_conditions contains the weather phenomena reported in the METAR (for example: +SH can be interpreted as heavy showers and -RA as light rain). The sky conditions describes the vertical visibility and cloud cover at different elevation (for example: SKC refers to sky clear and FEW100 indicates few clouds at 10000 feet.) Few METAR reports will contain information about runway groups (observations pertinent to a specific runway).
\r\nThe TAF (terminal aerodrome forecast) is a forecast of expected weather published four times daily for most airports. NOAA’s aviation weather center includes a good description of the TAF report found here. The data set includes the files:
\r\nAIRMETs (Airmen's Meteorological Information) and SIGMETs (Significant Meterological Information) are reports of various hazards to aviation. They are represented in the files airsigmet and airsigmet_area. Additional information about the content in these reports can be found here and here. The AIRSIGMET file includes:
\r\nThe bounding polygon describing the AIRSIGMET can be found in the airsigmet_area file and includes:
\r\nFlight Efficiency Services - GE Aviation
\r\nGE is leading the next wave of productivity revolution with its \"industrial internet\" solutions that will transform global industrial. One such solution is GE's Flight Efficiency Services.
\r\nThrough a comprehensive, collaborative approach, GE Flight Efficiency Services goes beyond the power plant to provide airlines with insight into their operations, policies and processes. Our experts will work with our customers team to identify and select fuel savings projects, implementation and ongoing support to ensure fuel savings are sustainable.
\r\nGE Aviation's Flight Efficiency Services uses proprietary algorithms to evaluate airline operations data and identify opportunities that could reduce fuel consumption by more than 2 percent. If an average-sized airline (70 aircraft, $1 billion annual fuel costs) implemented Flight Efficiency Services to achieve 2 percent improvement in fuel consumption, their annual fuel expenditure would decrease by $20 million and reduce 60 metric tons of CO2 emissions - which is equivalent to removing more than 10,000 passenger vehicles from the road or planting more than 1.3 million trees.
\r\n![\"\"](\"https://storage.googleapis.com/kaggle-media/competitions/gequest/3209/media/fcs-image-sm.png\")
Prizes will be awarded based on leaderboard performance on the final evaluation set as follows
\r\n1st Prize $100,000
2nd Prize $50,000
3rd Prize $40,000
4th Prize $30,000
Milestone Prize $30,000
\r\nTotal Prize Pool $250,000
\r\nThe Milestone Prize will be awarded based on the private leaderboard standings on 11:59:59 p.m. UTC September 25, 2013. To claim the prize, the winner will publish the ideas (not code), methodology, and documentation for their agent on the competition forum within one week of the Prize announcement. The winner of the Milestone Prize will be announced on October 9, 2013 at the GE Minds and Machines conference.
\r\nEligibility for all prizes is determined by the private leaderboard standings and compliance with the Rules. To claim a prize, Entrants must provide the final model (agent) code, documentation, and also complete, sign and return a Declaration of Eligibility and an Intellectual Property Assignment of Rights, as described in the Rules.
\r\nOnce your Entry has been verified by Kaggle as a potential winning Entry, GE shall have the right to have that Entry and the underlying code scanned and/or reviewed by Black Duck Software Inc., 8 New England Executive Park, Burlington, MA 01803 or a third party of GE’s choosing to determine whether the Entry and underlying code are consistent with these Competition Rules. If the results of the scan show that the Entry contains intellectual property risk and/or security vulnerability issues that need to be remediated, GE may at its discretion take either of the following actions: i. disqualify your Entry; or ii. require that you remediate within one (1) week all issues discovered in your Entry (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions.)
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Submission Instructions", "content": "Submissions
\r\nFor each FlightId, provide the recommended route for the remainder of the flight.
\r\nThe submission format is as follows:
\r\nFlightId,Ordinal,Latitude,Longitude,Altitude,Airspeed\r\n
287383197,1,39.9,-81.08,41000,514.0
287383197,2,39.9,-80.90,40100,447.8
287383197,3,39.9,-80.72,39100,447.7
...\r\n
Model Submission
\r\nWhen you submit an Entry, you also will be able to upload your model to the Website. Your final model must contain all code and parameter settings necessary to evaluate your models on new data, and include a README file with instructions on how to do so. The purpose of this is to ensure this Competition is fair and no answers were looked up for the Final Evaluation Set period. This model submission is required to be eligible to win a prize. This final model will be used solely for the purpose of verifying eligibility for prizes. You may upload either the archive itself or a SHA-256 hash of the archive.
\r\nKaggle will check the final submissions from the Entrants at the top of the Final Evaluation Leaderboard to make sure that Entrant’s final submission was created using that Entrant’s final model submission. If a potential winner did not use the final model codebase to create the final submission Entry, that potential winner will be disqualified and Kaggle will evaluate the final submission of the next Entrant on the Leaderboard as a potential winner.
\r\nFor an entry to be eligible to win, any open source or third party code or other materials used in connection with that Entry must be limited to code that GE will be entitled to commercially use and exploit, and any open-source-related or third party licensor requirements and restrictions must be disclosed as part of the model submission.
\r\nPlease do not upload Kaggle-provided data files, but mention which ones are used and how to link them to your agent in the readme.
\r\nRegarding non-Kaggle data (Previous leaderboard GRIB files): Zip this data and paste its hash into a text file called 'Data Hash.txt'. Place this text file into your model archive. If your model is in-the-money, we'll contact you to retrieve the non-Kaggle data and validate the hash.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Winners", "content": "![\"\"](\"https://storage.googleapis.com/kaggle-media/competitions/gequest/3611/media/Milestone_Roman_Prokopenko.jpg\")
Team id created an agent that optimized the cost function by selecting one of three actions for each time. Visit his forum post to learn more details.
\r\nRoman Prokopenko is a researcher at the Russian Academy of Sciences in the Institute of Higher Nervous Activity and Neurophysiology. He received his doctorate in physics and mathematics from Moscow Institute of Physics and Technology, State University in 2005. Having specialized in machine learning and control theory, Roman entered Flight Quest 2 for fun. Also, since Roman's brother is a professional pilot, he was enthusiastic to start an optimization problem in aviation.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Fq2 Milestone", "content": "To find the pages for the Milestone phase of this competition, use this link. Forums can be found here.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }