menu

Lockheed Martin

 253,280

AlphaPilot – Lockheed Martin AI Drone Racing Innovation Challenge

AlphaPilot is the first large-scale open innovation challenge of its kind focused on advancing artificial intelligence (AI) and autonomy.

This challenge is closed

stage:
COVID-19 Cancellation
prize:
$1,000,000

This challenge is closed

Partners
more
Summary
Teams
Timeline
Updates13
Forum7
Teams2.9K
Leaderboard
Press
Resources
FAQ
Sponsors
2019 Virtual Qualifier Tests
Summary

Overview

 

 

Calling all coders, gamers, race fans, and drone enthusiasts

 

Launching the AlphaPilot Innovation Challenge, Lockheed Martin and The Drone Racing League (DRL) have challenged teams of coders, engineers, and technologists to take on developing artificial intelligence (AI) for high-speed racing drones.  

 

Of over the 420 teams that applied in 2019, only 9 finalists earned the opportunity to compete in mastering autonomous flight and win more than $1,250,000 in cash prizes.

 

The AlphaPilot challenge tests teams of up to 10 participants on their ability to design an AI framework - powered by the NVIDIA Xavier GPU for autonomous systems - capable of flying a drone through three-dimensional race courses using only machine vision and guidance, navigation, and control algorithms. Teams will not have prior knowledge of the exact course layouts, nor will they benefit from GPS, data offboarding, or human intervention of any kind. Designed by DRL, the autonomous drone will provide the sensing, agility, and speed needed for competitive racing as part of DRL’s new Artificial Intelligence Robotic Racing (AIRR) Circuit, starting in fall 2019.

 

 

Why Drones?

Drone racing is a futuristic sport and a big draw for millennials and K-12 students with an interest in technology — many of whom will become future STEM professionals, drone pilots, and engineers. Lockheed Martin recognizes the important role in helping to develop a workforce with skills to compete in a 21st century high-tech economy. Lockheed Martin and DRL targeted U.S. undergraduate and graduate students to apply for AlphaPilot; however, the competition was open to drone enthusiasts, coders and technologists of all ages from around the world.

 

Why is Lockheed Martin doing this?

For more than 100 years, Lockheed Martin has been redefining flight — from the fastest speeds, to the edge of space, to unmatched maneuverability, and stealth. AI-enabled autonomy promises to fundamentally change the future of flight, and we are actively developing disruptive new AI technologies that will help our customers accomplish their most important missions – from reaching Mars to fighting wildfires.

ABOUT LOCKHEED MARTIN

Headquartered in Bethesda, Maryland, Lockheed Martin is a global security and aerospace company that employs approximately 105,000 people worldwide and is principally engaged in the research, design, development, manufacture, integration and sustainment of advanced technology systems, products and services. For more information, please visit www.lockheedmartin.com/alphapilot.

 

 

 

ABOUT DRONE RACING LEAGUE

DRL is the professional drone racing circuit for elite FPV pilots around the world. A technology, sports and media company, DRL combines world-class media and proprietary technology to create thrilling 3D drone racing content with mass appeal. The 2019 DRL Allianz World Championship Season will feature races in iconic venues across the globe that'll stream on Twitter and Youku and air on the best sports programs worldwide, including NBC, Sky Sports and ProSieben. For more information, please visit www.drl.io

View legal agreement


Guidelines

The AlphaPilot Innovation Challenge will push the boundaries of AI “edge computing” far beyond current achievements. It takes humans years to master high-speed, precision drone flight. Lockheed Martin and DRL are aiming to exceed human capabilities in under two years.

 

The Challenge

The AlphaPilot challenge tests teams of up to 10 participants on their ability to design an AI framework - powered by the NVIDIA Xavier GPU for autonomous systems - capable of flying a drone through three-dimensional race courses using only machine vision and guidance, navigation, and control algorithms. Teams will not have prior knowledge of the exact course layouts, nor will they benefit from GPS, data off-boarding or human intervention of any kind. Designed by DRL, the autonomous drone will provide the sensing, agility and speed needed for competitive racing as part of DRL’s new Artificial Intelligence Robotic Racing (AIRR) Circuit, starting in fall 2019.

This is a competition of AI quality – all other racing variables, including the drone hardware, are controlled.

 

Prizes

AlphaPilot will award more than $2,000,000 in prizes for its top performers through 2020!

  • The first AlphaPilot Team to beat a human-piloted drone in head-to-head competition will take home an extra $250,000.
  • The 2019 AlphaPilot World Champion will take home $1 million.

How can I Participate?

In the program’s inaugural season, AlphaPilot Teams were challenged to a barrage of tests in the Virtual Challenge Qualifier including development of image classification algorithms and performance in simulated racing environments, where their technical strategies were evaluated by a panel of industry experts.

For this year, the final 9 teams have been chosen and the program is now closed to new applicants. To follow the final qualifying teams through the challenge and get the latest news on the 2019 AIRR circuit and 2020 AlphaPilot open applications, hit the “Follow” button in the top right corner.  

 

The 2019 Virtual Qualifier

The 2019 Challenge Qualifier consisted of three different tests of which their combined scores determined who moved forward in the competition.  

Test 1: The first component of the AlphaPilot Virtual Qualifier - a series of questions focused around the team’s strategic approach to autonomy programming, tested each team’s knowledge, experience, readiness, and cohesion. The goal was to inspire confidence in the team’s ability to develop novel approaches to drone racing, bring the resources needed to compete, and mitigate real-world uncertainty.

Test 2: The second component of the AlphaPilot qualification focused on machine vision - challenging teams to develop an image classifier algorithm capable of detecting key racing elements. Drone racing requires efficient, low-latency visual processing in order to autonomously navigate through gates at high speeds. Therefore, effective machine vision approaches are critical for teams to be successful at AlphaPilot.

Test 3: The third component of the AlphaPilot qualification focused on a team’s ability to design algorithms for the guidance, navigation, and control of an autonomous drone. This test utilized a simulator framework that provided users with the tools needed to test their drone racing algorithms using realistic dynamics and exteroceptive sensors. These skills are essential for competition in AlphaPilot and the test was considered a precursor to work conducted by finalist teams in preparation for each AIRR race event.

 

Who can Participate?

Lockheed Martin and DRL have targeted U.S. undergraduate and graduate students to apply for AlphaPilot; however, the competition was open to drone enthusiasts, coders and technologists ages 18+ from around the world. The final qualifying teams in 2019 have come from all across the globe including Sweden, the Netherlands, South Korea, Brazil, Poland, Switzerland, and three different states within the U.S.

AlphaPilot Teams will work with professional mentors and industry-leading technical partners. Lockheed Martin engineers and AI specialists will serve as mentors to AlphaPilot Teams. DRL will supply AlphaPilot teams with standardized drones, and their drone experts will collaborate with teams and Lockheed Martin technical mentors throughout the multi-year partnership. Contestants will also have access to products from NVIDIA, the leader in AI computing.

 

Teams
Timeline
Updates13

Challenge Updates

COVID-19 Cancellation

April 21, 2020, 7:11 a.m. PDT by Keith Lynn

Dear HeroX Community,

Over the past few weeks, the rapid spread of COVID-19 and its wide-ranging impacts have caused severe disruptions around the world. The COVID-19 is affecting us all – at work, at home, and in our communities. Lockheed Martin believes that resolving this challenge should take precedent over our other challenge, AlphaPilot. In this spirit, we have decided to cancel our plans for a 2020 season. Health and safety are our top priority. We encourage everyone following AlphaPilot to use the time and effort they would have put into a 2020 season to instead help the world defeat COVID-19. Your creative spark and desire to solve difficult problems is exactly what the world needs right now.

Lockheed Martin would like to thank those who followed and participated in the 2019 season. Together, we accomplished amazing things and brought new energy to the future of autonomous flight. This is not an ending; it’s just the beginning. 

 

Stay safe, stay healthy, and keep innovating,

The Lockheed Martin AlphaPilot Team


Television Times

Dec. 9, 2019, 9:32 a.m. PST by Keith Lynn

Check out highlights from the 2019 AIRR Season during The Drone Racing League broadcasts on NBC Sports (NBCSN) and Twitter at the following times:

 

Feature

Broadcast Date

Broadcast Time (UTC-5)

Network

 

Intro to AlphaPilot & AIRR

6-Nov

6:00 PM

Twitter

 

7-Nov

1:00 AM & 1:00 PM

NBCSN

 
Overview of AlphaPilot Teams

13-Nov

6:00 PM

Twitter

 

14-Nov

1:00 AM & 1:00 PM

NBCSN

 
AI Competition /Orlando Feature Overview 

4-Dec

6:00 PM

Twitter

 

21-Nov

1:00 AM & 1:00 PM

NBCSN

 
RacerAI Feature 

11-Dec

6:00 PM

Twitter

 

28-Nov

1:00 AM & 1:00 PM

NBCSN

 
DC Recap 

5-Dec

1:00 AM & 1:00 PM

NBCSN

 
Baltimore Recap 

12-Dec

1:00 AM & 1:00 PM

NBCSN

 
Season Recap 

29-Dec

4:00 PM

Twitter

 

29-Dec

4:00 PM

NBC

 

MAVLab vs. Gab707

Dec. 9, 2019, 9:20 a.m. PST by Keith Lynn

The 2019 AlphaPilot and AIRR seasons finished with a head-to-head race between our winning team, MAVLab, and DRL's representative human pilot, Gab707. Through several course runs, MAVLab clocked a best time of 11.5 seconds; shaving a half-second off their winning time from the AIRR Championship. Their top speed was measured at 14.6 miles per hour. Manually piloting the RacerAI drone, Gab707 achieved a 6.9 second finish time, reaching 24.7 miles per hour.  Pilot vs. Programmer: Round 1 goes to Gab707...but by a much smaller margin than anticipated! 

We hope to build on this friendly competition in 2020 by giving our top programmers another chance to outperform the world's top human drone pilots. 

Gabriel Kocher

AIRR World Championship - Final Results

Dec. 9, 2019, 9:11 a.m. PST by Keith Lynn

Congrats, Team MAVLab, for taking home the $1,000,000 prize with a winning time of 12 seconds! A tight competition with Team UZH Robotics & Perception, who clocked a best finishing time of 15 seconds.

 

Lockheed Martin is pleased to recognize Team MAVLab for their fantastic achievement in the 2019 AIRR Season. Selected among 424 team entries in the AlphaPilot Innovation Challenge, Team MAVLab has demonstrated commendable teamwork, agility, and innovation. Lockheed Martin launched AlphaPilot using benefits from the Tax Cut and Jobs Act of 2017. Our goal was to drive innovation and foster the development of cutting-edge AI applications. AI is an exponential technology with great potential benefits for humanity. The rapid progress of 9 Finalist teams over the season is evidence of how quickly technology can advance with the right talent. Together, our teams are helping make history, as we strive to achieve the “Deep Blue Moment” for autonomous flight – outperforming a human drone pilot. Your work is monumental to building a better future.


Special Thanks to Beemo!

Dec. 9, 2019, 8:49 a.m. PST by Keith Lynn

Thank you, Beemo, for playing at our AIRR World Championship Race! Beemo is an Orlando-based folk band comprised of several Lockheed Martin engineers ....their very own Tony Mickle (base guitar) is a member of the AlphaPilot program team!

Thanks for the great performance and for showing the amazing diversity of our workforce!

https://www.mynameisbeemo.com/

2019_april4.jpg

Forum7
Teams2.9K
Leaderboard
Press
Resources
FAQ
Sponsors
2019 Virtual Qualifier Tests

2019 Virtual Qualifier Tests

AlphaPilot Virtual Qualifier

 

Test #1 – Team Video & Questionnaire

Overview:

The first component of AlphaPilot qualifications will focus on team knowledge, experience, readiness, and cohesion. This is your chance to convince AlphaPilot judges that you have what it takes to tackle a difficult, year-long challenge. Your goal is to inspire confidence in your team’s ability to develop novel approaches to drone racing, bring the resources needed to compete, and mitigate real-world uncertainty.

Format: 

Test #1 will consist of two parts: written-response and video submission. Both parts must be submitted together via the Test #1 entry form. Teams may submit one and only one entry form, to be received from the Team Captain. Deadline for submission is March 11th at 5:00PM EST. To begin your entry, click the orange "Begin Entry" button or "Accept Challenge" button above.

Written-Response

This section will consist of five questions addressing team experience with technical challenges, predictions for AlphaPilot technology development, and proposed development methodologies. Each written question has a limit of 4000 characters, including spaces.

Video Submission

AlphaPilot is interested in the human story behind autonomy and artificial intelligence. To help us tell your story, we are asking each team to submit a video (up to 3 minutes in length) demonstrating what makes your team qualified, unique, and compelling. What you include is up to you – this is your story! A scoring rubric is outlined in the following section and can be used as a content guideline. Please note that teams will not be judged based on video production value; the quality of your message is what counts. At a minimum, video entries must cover the following:

  • Describe your team’s composition, background, and qualifications as they relate to the AlphaPilot challenge and associated technologies. Why is your team the one to beat? *Note: teams may add or modify members following video submission, so long as all changes comply with AlphaPilot Rules & Regulations. However, teams will be assessed only on the members presented in their video.
  • Have any team members participated in prior competitions focused on artificial intelligence and autonomy? Has your team worked together in the past on technical challenges with comparable scope or complexity? Please give examples and describe the results.

Other suggested topics include:

  • What will make your team a success? What is your advantage?
  • What makes your team unique?
  • Where do you work or go to school?
  • Who are your mentors, advisors, role-models?
  • What about AlphaPilot inspires you?
  • What will you do if you win AlphaPilot?

Scoring:

A panel of AlphaPilot judges will review and score your submitted Test #1 form. Each entry will be reviewed by multiple judges. The questions will be graded based on a maximum point value, with your score for each reflecting the quality of your answer as determined by the judging panel. Videos will be assessed based on the following criteria:

  • Team Experience: team demonstrates experience working on similar technical challenges as part of their career, academic study, challenge programs, or personal hobbies.
  • Team Knowledge: team demonstrates suitable knowledge of subject matter, including supporting evidence such as research, work experience, degrees, publications, and accolades.
  • Team Cohesion: team demonstrates comfort working together and complimentary skillsets that could lead to a winning solution. There are no obvious logistic or social challenges that would impede a team’s performance.
  • Team Resources: team demonstrates enough resources to develop, train, and test their solutions. This includes access to enabling hardware and software, support from organizations, and mentorship from thought-leaders.
  • Team Perspective: team communicates a compelling perspective and driving force behind their participation in AlphaPilot. Team displays energy and passion for artificial intelligence and autonomous systems. Team members are doing interesting things in the field and have commendable aspirations.

Test #1 - Frequently Asked Questions (FAQs) 

What information are you looking for in the Test 1 video? We have a team of 10 people, what should we prioritize given we only have three minutes.

AlphaPilot is a year-long endeavor. Aside from technical performance in Test #2 and Test #3, we want to know that your team is a solid investment. Do you have the cohesion and commitment to see the challenge through? We don’t want teams that will break-up, become distracted, or lose interest half-way through. Do you appreciate the difficulty of AlphaPilot and the level of effort that will be required? Have you thought through possible setbacks? Can you reference some examples of difficult software problems and how you handled them? What makes your team unique – i.e. more than a group of individuals? In summary, make us feel confident in you as a team.

 

 

 

Test #2 – Machine Vision

Overview:

Test #2, “Eye Exam”, focuses on effective machine vision, as this will be critical for success in AlphaPilot. Drone racing requires efficient, low-latency visual processing in order to autonomously navigate through gates at high speeds. Therefore, to qualify for AlphaPilot, teams will first need to pass the Eye Exam!

Evaluation:

Goal

The Drone Racing League (DRL) has developed unique racing gates for use in AIRR, their new autonomous racing league attached to AlphaPilot. These gates are equipped with visual markings (e.g. colors, patterns, logos) that will provide fiducials to aid in guidance through the course. Teams are tasked with:

(1) Developing a gate detection algorithm

(2) Describing their algorithm in a 2-page Technical Report

This gate detection algorithm needs to be capable of detecting the flyable region of AIRR Race Gates and producing a quadrilateral around its edge (see Figure 1) with a high degree of accuracy and speed.

Figure 1: Shows sample AIRR gate images from the training dataset both without (left) and with (right) the flyable region outlined in red.

Test 2 Scoring

Each team will receive a Test #2 score (max 100 points) that combines an objective score from their gate detection performance with a qualitative review of their written submission:

  • Algorithm Score: 70% of total Test 2 score
  • Technical Report Score: 30% of total Test 2 score

Algorithm Score

Each team’s gate detector will receive an Algorithm Score (max 70 points) that is based on a metric evaluating their algorithm’s ability to find the flyable region of AIRR gates. This is computed using the accuracy of the reported bounding coordinates compared to the labelled coordinates (referred to as “ground-truth”), along with a measure of average execution time of the algorithm in seconds (avg_time). The accuracy of all the labels is evaluated according to the Mean Average Precision (MAP) metric, and the measure of execution time is assessed according to the wall clock time for the gate detector to read an image, extract the corners of the flyable region of the gate, and output the label.

The total Algorithm Score is then calculated by subtracting the average wall clock time from the weighted MAP score and multiplying by 35 to get a maximum of 70 points:

 

Note: If a team’s execution time is longer than 2 seconds, it is possible for a team’s Algorithm Score to be negative. If this occurs, a team will be given 0 points for their Algorithm Score.

For more information on the implementation of this metric used for AlphaPilot, read more here: https://arxiv.org/abs/1405.0312

Technical Report Score

Each team’s Technical Report Score (max 30 points) is based on a rubric evaluated by judges from academia, industry, and government. The judges will review each report for technical merit, applicability to AlphaPilot, and presentation and clarity of the approach.

Resources:

Data

The data used for EyeExam consists of images of AIRR Racing Gates from various distances, angles, and lighting profiles in JPG format. The AIRR gates are square with parallel edges. The internal clearance is 8ft x 8ft, the external dimensions are 11ft x 11ft, and the depth is 1ft. However, it is very likely these dimensions could change a bit for the final AlphaPilot Challenge as the drone architecture and race courses get finalized. This data has been divided into 3 sets:

  1. Training Dataset
  2. Leaderboard Testing Dataset
  3. Final Sequestered Testing Dataset

The Training Dataset (can be found in DataTraining.zip) contains roughly 9,300 images totaling 2.8GB. This will be the primary resource available to teams for development. A JSON file containing the ground truth labels for this training dataset is now available as training_GT_labels_v2.json.

The Leaderboard Testing Dataset (can be found in Data_LeaderboardTesting.zip) is now available as a practice exam!! This dataset contains roughly 1,000 images totalling 360MB. This test dataset should be used to see how well a team’s algorithm performs on unseen images. For the Leaderboard Testing Dataset, the ground truth is not provided; it is the team’s job to predict these labels. With this dataset, teams may submit a JSON file of their labels for scoring and placement on a leaderboard.

The Final Sequestered Testing Dataset and the corresponding ground truth will not be released to teams. This test set will be used to evaluate each team’s algorithm and determine their final EyeExam score.

See the ‘Testing’ section below for more details on Leaderboard and Final Testing.

Sample Submission

At this time, the final version of the sample submission and scoring code is available for teams to use. These scripts provide a framework on which to build and develop algorithms so that they meet all the submission requirements. The starter scripts folder (starter_scripts_v2.zip) contains code to help teams create submissions:

  • generate_results.py – Sample submission with example class by which to define a solution. Also reads and plots images and labels.
  • generate_submission.py – Test script that reads all test images, imports and calls a team’s algorithm, and outputs labels for all images in a JSON file
  • random_submission.json – Sample JSON file of labels; this JSON is also the output of generate_submission.py and meets the submission requirements

The scorer scripts folder (scorer_scripts_v3.zip) contains code to calculate a team’s Algorithm Score:

  • score_detections.py – Test script which calls a team’s JSON file, evaluates it against the ground-truth JSON file, and outputs a MAP score

To run the sample submission, configure the environment to match the environment described in the ‘Testing’ section below. Teams will also need to install the following libraries which can be installed using pip:

  • Shapely
  • NumPy

Please note that these libraries should be compatible with Python 3.5.2.

Labels

For the training dataset, the Training Dataset Ground-Truth Labels file (training_GT_labels_v2.json) contains a label for each image provided. This JSON file contains the coordinates of quadrilaterals that best fit around the flyable region of the AIRR gates (see Figure 1). For each image tested, the JSON file contains an object corresponding to the label for that image. The labels are provided as a dictionary where keys are image_ID and values are numbers that correspond to the four (x,y) coordinates of the corners of the quadrilateral (e.g. "IMG_####.JPG": [[x1, y1, x2, y2, x3, y3, x4, y4]]). The four coordinates of the quadrilateral need to be in clockwise ordering; preferably starting with the upper-left most corner (see Figure 1).

JSON files are structured such that there is an array of values for each gate in an image. An image with 2 gates should have 2 arrays of coordinates. If there is no gate in the image, the values should be empty for that image_ID. Teams will not be tested on images with multiple gates in Test 2 though.

Here are a few examples:

Single Gate: "IMG_3668.JPG": [[370.3913304338685, 377.2051599830667, 7.742989976402814, 13.058747191455566, 246.50017198915896, 321.3924492511484, 342.11494534552276, 35.65517009139904]]

Two Gates: "IMG_3668.JPG": [[370.3913304338685, 377.2051599830667, 7.742989976402814, 13.058747191455566, 246.50017198915896, 321.3924492511484, 342.11494534552276, 35.65517009139904], [370.3913304338685, 377.2051599830667, 7.742989976402814, 13.058747191455566, 246.50017198915896, 321.3924492511484, 342.11494534552276, 35.65517009139904]]

No Gate: "IMG_3668.JPG": [[]]

The Training Dataset Ground-Truth Labels are crowd-sourced, and as often happens, are not perfect data. Above all, follow the guidance given for accurately identifying correct gate labels in images. Some of the training data ground-truth labels do not follow these rules accurately, and that should be considered when developing machine vision algorithms. Please address any data curation done in the Technical Report as AlphaPilot judges would like to know this when considering a team’s technical approach.

Please be assured that the Leaderboard and Final Testing ground-truth labels are very accurate, and that is what teams will be tested on.

Camera Calibration Images

A set of images of a checkerboard pattern taken with the same camera and lens as the training and testing data is available to teams in Cam_Calibration.zip. All images are taken with Canon DSLR with 18mm lens.

File Descriptions

Data_Training.zip – Training dataset images

Data_LeaderboardTesting.zip – Leaderboard Testing images

training_GT_labels_v2.json – Training dataset labels

Cam_Calibration.zip – Calibration dataset images. The square sizes are 19 x 19 mm.

starter_scripts_v2.zip – Final version of starter scripts for submission creation

scorer_scripts_v3.zip – Final version of scorer scripts for submission evaluation

submission.zip - Sample submission.zip archive

Submission Requirements:

For Test 2 submissions, each Team Captain should upload 2 items as attachments via the HeroX Test 2 submission form:

  1. 1 PDF file describing their algorithm in a 2-page Technical Report
  2. 1 zipped archive named ‘submission’ with maximum upload size of 1GB

In the folder named ‘submission’, at the highest level of the directory, please include the following:

  1. generate_results.py - Within this script, teams need to define their algorithm within the GenerateFinalDetections() class in the predict(self,img) function. This function is called within the generate_submission.py test script that reads all test images, imports and calls a team’s algorithm, and outputs labels with confidence score for all images in a JSON file.
  2. requirements.txt - A file which lists libraries that need to be installed for an algorithm’s source code to run.
  3. Other code - Teams can include additional code in the directory as needed for your algorithms to run. This can be compiled libraries, source code, etc. Teams SHOULD NOT include any of the following in their submitted archive:
    1. Any code not needed for their algorithms to run
    2. We decided to make OpenCV 4.0 available in the testing environment for teams. So teams will not need to include OpenCV in their submission.
    3. generate_submission.py as AlphaPilot will run our own version of this code
    4. Scoring scripts as AlphaPilot will run our own version of this code

A sample submission with example class and predict function is included in the starter scripts, and these help teams to format and define a solution. A sample submission.zip archive has also been made available to help teams make sure their submissions are formatted correctly.

Test 2 Source Code and IP Concerns:

As a reminder, the judges will have access to teams’ submitted source code to help them understand the technical approach taken for Test 2. The AlphaPilot team suggests that if there are any concerns with IP, then that IP should be compiled into an executable that is called from your source code and that executable should be included in your archive. However, by abstracting functionality away in executables, it makes it more difficult for the judges to verify that teams have the technical skills and capabilities to succeed during the AlphaPilot Competition. As a result, please balance your team’s individual needs to obscure IP with building transparency of your approach.

Algorithm Requirements:

Test 2 Algorithms need to detect a single polygon per gate in the image. While there will not be multiple gates in any test images in the Eye Exam, there may be some test images with no gates. Multiple gates in view is something teams will have to deal with during Test 3 and the AlphaPilot Challenge, and the judges will be interested in hearing how teams will extend their Test 2 approach. Please address in Technical Report.

Each gate should be labeled with a single polygon with 4 corners around the flyable region (even if it’s partially obstructed). When all 4 corners of the flyable region of the gate are visible, teams are expected to find them (even if part of the flyable region is not visible). If that is not the case (where one or more corners is not visible in the image), teams will not be tested on these images, and they have been removed from the testing data. This is largely due to the fact that it is very difficult to accurately label the ground-truth for those cases.

Within the generate_results.py script provided, teams should define their algorithm within the GenerateFinalDetections() class in the predict(self,img) function. This function is called within the generate_submission.py test script that reads all test images, imports and calls a team’s algorithm, and outputs labels for all images in a JSON file. For an example, teams can view the random_submission.json included in, and generated by, the starter scripts.

The predict function should input an OpenCV image object and output an array of values for each gate in an image followed by a confidence score (e.g. "IMG_####.JPG": [[x1, y1, x2, y2, x3, y3, x4, y4, CS]]). There exist two distinct measurements needed in object detection: (1) Whether the image is correctly classified (i.e. if an object exists in the image) and (2) how well the object has been localized. Simple metrics introduce biases and so it is important to assess the risk of misclassifications. Thus, the “confidence score” is used. An image with 2 gates should have 2 arrays of coordinates. If there is no gate in the image, the values should be empty.

Here are a few examples:

Single Gate: "IMG_3668.JPG": [[370.3913304338685, 377.2051599830667, 7.742989976402814, 13.058747191455566, 246.50017198915896, 321.3924492511484, 342.11494534552276, 35.65517009139904, 0.5]]

Two Gates: "IMG_3668.JPG": [[370.3913304338685, 377.2051599830667, 7.742989976402814, 13.058747191455566, 246.50017198915896, 321.3924492511484, 342.11494534552276, 35.65517009139904, 0.5], [370.3913304338685, 377.2051599830667, 7.742989976402814, 13.058747191455566, 246.50017198915896, 321.3924492511484, 342.11494534552276, 35.65517009139904, 0.5]]

No Gate: "IMG_3668.JPG": [[]]

Algorithms must be compatible with Python 3.5.2. Teams may leverage open-source algorithms and data libraries to assist in their design. Use of TensorFlow is recommended but not required. No other specific software and hardware is required, but teams should review the testing information below for further considerations. Further, teams can choose any approach for their algorithms (i.e. teams don’t have to use machine learning).

Teams will need to list dependencies in a requirements.txt file included in their submitted archive which can be automatically generated. The requirements.txt file only needs to list libraries that need to be installed for an algorithm’s source code to run. This will be installed using pip with the following command:

>> pip install -r requirements.txt

Given this, all libraries listed in requirements.txt must be installed via pip. If teams want to use libraries that do not fit within this constraint, they can add their compiled versions of their libraries into the submitted archive. However, proceed with the latter option at risk, because the AlphaPilot team can only guarantee successful install of libraries using pip.

Note: GPU and CUDA drivers will already be installed in the testing environment. The following instance will be used at the basis for the testing environment: https://aws.amazon.com/marketplace/pp/B077GCZ4GR

Technical Report Requirements:

In the Technical Report, teams must document how their algorithm works and how they conducted analysis, including any training and data curation. Teams should detail any libraries or well-known approaches used. In addition to describing their approach, teams must also address:

  • How the team plans to build onto or modify their gate detection algorithm for use in the AlphaPilot competition should they qualify.
  • Any technical issues the team ran into and how they overcame them.

Reports need to be in PDF format and 2-pages maximum, single-spaced. Font size should be minimum 11pts. Insertion of text, equations, images, figures, plots, code, and pseudo-code is accepted but will be included in the 2-page limitation. The only exception to the 2-page limitation will be a list of references.

Testing:

The Leaderboard and Final Exam are now out!

Leaderboard Testing

The Test 2 Leaderboard is now open for participants to test their algorithms on a practice exam. At this time, the Leaderboard Testing Dataset is now available for teams to download, test their algorithms against, and evaluate their algorithm’s performance. This will give teams a rough idea of how well they will perform on the final testing though actual results might differ slightly.

To participate in the Leaderboard Test, a team can upload to HeroX AlphaPilot their JSON file containing the labels and confidence score for each image in the set. Each day, a cut-off is implemented for uploading and testing. If a team uploaded a JSON since the last cut-off, their most recently uploaded JSON file will be used to compare against the ground-truth labels for the practice images and calculate a team’s MAP value. The leaderboard shows the team’s most recent score and is updated with the top teams according to their most recent MAP value.

The AlphaPilot team has decided to increase the frequency of the Leaderboard refreshes until the Test 2 deadline. These will now be 3 times a day with the new cut-offs and subsequent refreshes being at:

  • 7:59AM PST
  • 3:59PM PST
  • 11:59PM PST

Final Testing

By the final deadline, teams must submit source code for testing and evaluation as previously described. During testing, there will not be internet access, and so their submissions will not be able to perform actions that require internet.

The testing environment is an instance of Ubuntu 16.04.5 LTS running in AWS type p3.2xlarge. The following instance will be used at the basis for the testing environment: https://aws.amazon.com/marketplace/pp/B077GCZ4GR. 

OpenCV 4.0, Python 3.5.2, numpy, and shapely have also been already installed in the environment for teams.

The final algorithm testing will be conducted as such:

          1. A new Linux user home directory is created. The team’s archive will be unzipped and ‘submission’ folder will be unpacked into the user’s home directory. A virtual testing environment is setup in the user’s home directory.

          2. Team dependencies will be installed according to the libraries listed in a requirements.txt file. See ‘Algorithm Requirements’ section for more details on this file. This will be installed using pip with the following command:

                    >> pip install -r requirements.txt

          3. An internal version of the testing script, generate_submission.py, will run through all available image data, implement the predict(self,img) function call to get the labels and execution time of team algorithms for a given image, and store the results for each image in a JSON file named ‘random_submission.json’. The internal version of generate_submission.py is functionally comparable to the version gives to teams and only differs in that it contains additional checks performed on algorithms.

          Note: there is a cut-off for execution time for a team’s algorithm, and it is 10 seconds. If a team’s algorithm takes longer than this to output a label, the script will be stopped, and the team will be given an Algorithm Score of 0 points.

          4. Once all results are stored in a JSON, the scoring script, score_detections.py, is run to compare the outputted JSON file to a master JSON containing the ground-truth labels. The total algorithm score is then calculated using the output from the scoring script as well as the execution time outputted from running generate_submission.py.

          Note: there is a cut-off for the total submission evaluation time (Final Testing steps 3 and 4) and that is 2 hours. If a team’s algorithm takes longer than this to be evaluated, the testing will be stopped, and the team will be given an Algorithm Score of 0 points.

Separately, judges will read and score each team’s technical report which will be subsequently used to calculate their total final score for Test 2.

Test 2 Algorithm Submission Checking

The final algorithm testing is completed automatically, and therefore, it is essential that teams follow the above requirements exactly. To reduce the possibility of small, technical errors causing issues for the teams, the AlphaPilot team has provided all the tools needed to check and validate Test 2 algorithm submissions. To check that your submitted source code will function correctly, we suggest teams follow the above testing process using the starter and scorer scripts provided. 

Test #2 - Frequently Asked Questions (FAQs) 

How do we deal with edges cases in the images (i.e. where gates might be partially or fully obstructed or not visible for whatever reason)?

Each gate should be labeled with a single polygon with 4 corners around the flyable region (even if it’s partially obstructed). When all 4 corners of the flyable region of the gate are visible, teams are expected to find them (even if part of the flyable region is not visible). If that is not the case (where one or more corners is not visible in the image), teams will not be tested on these images, and they have been removed from the testing data. This is largely due to the fact that it is very difficult to accurately label the ground-truth for those cases.

How do you recommend we use the ground truth labels in the training dataset?

Above all, follow the guidance given in the Test 2 description for accurately identifying correct gate labels in images. Some of the training data ground-truth labels do not follow these rules accurately, and that should be considered when developing machine vision algorithms. Handling real-world challenges effectively will be critical for success in the AlphaPilot Competition, and similarly, teams need to deal with some flawed ground-truth in Test 2 and sensor noise, control drift, and modeling errors in Test 3. Address how your team plans to deal with these in the Technical Reports, because AlphaPilot judges would like to know this when considering a team’s approach. 

What are some safe assumptions that can be made about the AIRR gates?

The gate is square with parallel edges. The internal clearance is 8ft x 8ft, the external dimensions are 11ft x 11ft, and the depth is 1ft. However, it is very likely these dimensions could change for the final AlphaPilot Challenge (as the drone architecture and race courses get finalized). If your team qualifies, we would like to know how your team plans to deal with these real-life variations (please address in Technical Report).

 

How is execution time of algorithms calculated?

The measure of execution time is assessed according to the wall clock time for the gate detector to read an image, extract the corners of the flyable region of the gate, and output the label. So, 2 seconds per image. Please also note the additional time restrictions during testing (see ‘Testing Section’ of Eye Exam).

 

What are some safe assumptions that can be made about the camera that took all the training and testing images?

The camera used for training and testing images was a canon DSLR with an 18mm lens. A set of images that can be used for calibration are available here (Cam_Calibration.zip). The squares are 19 x 19 mm.

 

How are teams supposed to deal with multiple gates in view?

Test 2 Algorithms need to detect a single polygon per gate in the image. While there will not be multiple gates in any test images in the Eye Exam, there may be some test images with no gates. Multiple gates in view is something teams will have to deal with during Test 3 and the AlphaPilot Challenge, and the judges will be interested in hearing how teams will extend their Test 2 approach. Please address in Technical Report.

 

What are the requirements for the leaderboard submission json?

JSON files are structured such that there is an array of data for each gate in an image. So an image with 2 arrays of coordinates would indicate an image with 2 gates. An image with an empty array would indicate an image with no gate.

For further info, review the ‘Submission Requirements: Algorithm Requirements’ section above. The random_submission.json included in, and generated by, the starter scripts give an example of the format, and this will also have the same format as the ground-truth JSON file.

 

What is the required version of Python?

AlphaPilot has modified the Python requirements. Please note that all Test #2 algorithms must be compatible with Python 3.5.2.

 

Is there a test instance on your platform we can use to see that our code will work when you score it?

The starter scripts and scorer scripts provided for Test 2 represent how each team’s source code will be tested.

We will not be providing the exact test instance for teams. However if a team defines a GenerateFinalDetections() class with a predict(self,img) function that runs smoothly in generate_submission.py (as shown in the starter scripts), this provides the sanity checks needed on source code.

 

How do the ground truth labels work in the testing dataset? Is there a “fudge factor” that will score correct if we’re a few pixels off?

The ground-truth (GT) labels represent the 4 coordinates which define a polygon that maximizes the flyable region of the AIRR gates. The primary measure used in the MAP score is the Intersection over Union (IoU). The IoU is computed by dividing the area of overlap between the GT and predicted bounding boxes by the area of union. For more information on the implementation of this metric used for AlphaPilot, read more here: https://arxiv.org/abs/1405.0312

 

Is calling a C/C++ library from Python allowed for Test 2? I'm working under assumption that it is allowed, since almost all Python libraries, where performance is an issue are written in C/C++ or other higher performance language.

This is allowed. However, please make sure your code is still compatible with the specified testing environment and requirements.

 

Can a 3D model of the gate used in Test 2 be made available?

Unfortunately, we won't be able to provide any mechanical drawings of the gates.

 

Is there any difference in the distribution of the public and private leaderboard data for Test 2?

No

 

Do you account for inference time?

The measure of execution time is assessed according to the wall clock time for the generate_submission.py function to run finalDetector.predict(img) as defined by your class. That is, the time for the gate detector to read an image, extract the corners of the flyable region of the gate, and output the label.

The total Algorithm Score is then calculated by subtracting the average wall clock time from the weighted MAP score and multiplying by 35 to get a maximum of 70 points:

 

Do we need to detect gates obscured by pillars?

Each gate should be labeled with a single polygon with 4 corners around the flyable region (even if it’s partially obstructed). When all 4 corners of the flyable region of the gate are visible, teams are expected to find them (even if part of the flyable region is not visible). If that is not the case (where one or more corners is not visible in the image), teams will not be tested on these images, and they have been removed from the testing data. This is largely due to the fact that it is very difficult to accurately label the ground-truth for those cases.

 

Could you elaborate on the MAP score for test 2?

Please read the paper included in the Test 2 overview. We also suggest teams do a bit of their own research into the score. This is a very common metric for data science competitions, and we utilize a common implementation in AlphaPilot.

 

Execution time for test #2 will be benchmarked on multicore, hyperthreaded CPU where my task execution, due to primarily cache thrashing, may be randomly affected by another task running on that CPU at the same time. Are you aware of that?

Each team’s algorithm will be tested on the server sequentially (one at a time), and no other tasks will be running on the server during that process.

 

Can you expand on the confidence score?

  1. There exist two distinct measurements needed in object detection:
    1. Whether the image is correctly classified (i.e. if an object exists in the image)
    2. How well the object has been localized
  2. Simple metrics introduce biases and so it is important to assess the risk of misclassifications. Thus, the “confidence score” is used. For more information on how this is used in AlphaPilot, please doing some reading into the “mean Average Precision” on the MAP metric and how the confidence score is considered mathematically.

 

 

 

Test #3 – Guidance, Navigation & Control

Overview:

The third component of AlphaPilot qualifications will focus on a team’s ability to design algorithms for the guidance, navigation, and control (GNC) of an autonomous drone. The test will utilize a simulator framework that provides users with the tools needed to test their drone racing algorithms using realistic dynamics and exteroceptive sensors. These skills are essential for competition in AlphaPilot and the test is considered a precursor to work conducted by Finalist teams in preparation for each AIRR race event.

 

Evaluation:

Goal

Teams must develop GNC algorithms to fly a simulated drone through a structured test environment utilizing a typical ACRO/RATE flight mode for control inputs and exteroceptive sensors for feedback. Teams are tasked with:

  1. Developing GNC algorithms to pilot an autonomous drone through FlightGoggles
  2. Describing their GNC algorithms in a 2-page Technical Report

The GNC algorithms must be capable of navigating through gates (see Figure 1) in a FlightGoggles challenge course. The objective is to pass through all gates in the defined order, as quickly as possible, without crashing.

 

Figure 1: Shows a screenshot of an example drone race course in MIT’s FlightGoggles Simulation. 

Test 3 Scoring

Each team will receive a Test #3 score (max 100 points) that combines an objective score from their racing performance with a qualitative review of their written submission:

  • Algorithm Score: 70% of total Test 3 score
  • Technical Report Score: 30% of total Test 3 score

Algorithm Score

Each team’s drone performance will receive an Algorithm Score (max 70 points) that is based on a metric evaluating the ability of their submitted algorithms to navigate through race gates. Total lap time will be the primary measure of performance, along with points awarded for each successful gate fly-through. The Reporter node in FlightGoggles tracks these metrics and outputs this score per “run”:

Run Score = +10 pts per gate successfully passed – race completion time (in seconds)

Since algorithms are tested on 25 slight variations (“runs”) of the same challenge course, the Racing Score is then calculated by looking at the best 5 runs from the 25 and averaging them. Since it is possible to hit obstacles and crash, simulator runs with a crash or object strike will receive 0 pts.

This averaged Racing Score is reported on the Leaderboard and scaled slightly for the Final Algorithm Score, so the best team receives the maximum allowable 70 pts.

Technical Report Score

Each team’s Technical Report Score (max 30 pts.) is based on a rubric evaluated by judges from academia, industry, and government. The judges will review each report for technical merit, applicability to AlphaPilot, and presentation and clarity of the approach.

 

Resources:

Test #3 is built on the Massachusetts Institute of Technology (MIT) FlightGoggles simulator, a first-of-its-kind virtual reality environment for drone research and development. MIT has modified FlightGoggles to include software for a drone racing simulation environment based on the Unity3D game engine. It includes Robotic Operating System (ROS) binding for integrating autonomy software and Unity assets for environments that resemble drone racing scenarios.

The FlightGoggles simulator and several training courses are now available open-source to the public and are hosted on GitHub. You can find the source code for the simulator, more information about how to get started, and some example challenge files here: http://flightgoggles.mit.edu/

 

Vehicle Model:

The simulator emulates a high-performance drone racer with high pitch rate and collective thrust command inputs. Note that this drone is not an exact model of the one used by AlphaPilot in AIRR race events. A Vehicle Dynamics node will keep track of the simulation clock and adjust the simulation clock ratio, which can be seamlessly lowered in real-time if the host computer is unable to provide real-time performance due to high-load or can be raised if faster real-time simulation is desired. More details about the drone (vehicle dynamics, sensor models, specifications, etc.) will be added to the GitHub repo.

 

Training Courses & Course Developer:

The simulator environment includes several pre-developed challenge courses and an open-map course creator for easy development and testing. Teams can use both resources for practice and may submit performance data on a Leaderboard Challenge Course to be included on a HeroX Leaderboard. The Final Challenge Course files will be developed by the AlphaPilot administration and kept on a separate instance. This course will be used for scoring submitted algorithms and determining teams final Test 3 Algorithm Scores.

The simulator is run with a Challenge File in YAML format which corresponds to a challenge course, and it will produce a Results YAML File after it is done. The Challenge File describes the following details:

  • Initial position and orientation of the drone
  • Challenge Name
  • Timeout Specification
  • A list of gate names
  • Width of all gates
  • Location of all the gates, determined by four points that make a rectangle

Simulator Output:

The simulator is equipped with a Reporter node that tracks ground truth information and outputs, via command line, and yields metrics on algorithm performance as a Results YAML File for each run on a Challenge YAML File. The file will contain metrics on drone performance attributes useful in logging, evaluating, and visualizing algorithm performance.

Additionally, the Simulator Reporter will output a YAML file with details about the run including output one of the following results:

  • Interrupted: If the grader process is interrupted, e.g., via ctrl+c.
  • Timeout: The timeout on the challenge file is reached
  • Crashed: The drone crashed into an obstacle in the sim
  • Completed: The drone passed through the last gate in the sequence.

The Reporter will also keep track of when the drone reaches each gate and record the time. If a gate in order is skipped, then the grader will put down “Success: False” for that gate.

With the Leaderboard Challenge Course files, teams have scripts to help with scoring the Results YAML file and outputting a Scores YAML file. See the included ‘Scorer Scripts’ in the challenge files as well as the README_v2.md file for more details on running these scripts. Teams may submit their Scores YAML file of their summarized results for scoring and placement on the Leaderboard.

The Final Challenge Course files will not be released to teams and will be used to evaluate each team’s algorithm and determine their final FlightGoggles score. See the ‘Testing’ section below for more details on Leaderboard and Final Testing.

Tech Support:

Teams may file a report on the simulator GitHub page if they encounter any bugs or errors. Issues will be addressed by the MIT team as quickly as possible. Any specific questions about the AlphaPilot Test 3 challenge should be posted and addressed on the HeroX Test 3 Forum page.

Scorer Scripts

Challenge_Leaderboardtest.zip\ChallengeLeaderboardtest\launch – scripts to score Results YAML file and output Scores YAML file. 

README_v2.md – contains challenge installation instructions, scorer usage guide, and list of allowed and prohibited information for Test 3.

 

Submission Requirements:

Teams will submit two attachments via the Test #3 entry form:

(1) Autonomous drone racer source code in a zipped archive named ‘submission’ with maximum upload size of 1GB

(2) Technical Report of the drone racer in PDF, maximum 2 pages.

Teams will have internet access during installation, but there will not be internet access during testing. Teams will not have root access at any point. In your archived named ‘submission’, at the highest level of the directory, please include the following:

  1. scorer.launch
    Team archives must contain and edit `scorer.launch` to include all ROS nodes required for completing the challenges. This will be the launch file used to run your team’s algorithm.
  2. install.bash
    If desired, teams may include an install.bash file which can be run to automatically install algorithm dependencies. Your install.bash file can run commands such as catkin build, rosdep, etc. as needed. Alternatively, teams can include compiled libraries in their archive as noted below. Please note the limitation on installation time of 1 hour.
  3. catkin_ws
    Teams must include a catkin workspace with the team’s own ROS packages separate from the flightgoggles catkin workspace. It should be named ‘catkin_ws’ and include the following folders:
    1. src ‘src’ folder should include all source code for the team’s ROS packages
    2. devel – the ROS packages should already be compiled into the ‘devel’ folder, the development space, and should include setup.bash (which is usually autogenerated when you build with catkin)
  4. Compiled Libraries and External Code
    Teams can include additional code anywhere in the archive as needed for your algorithms to run. This can be compiled libraries, source code, etc. Note that the testing environment will have ROS Kinetic, OpenCV 3.4, and FlightGoggles already installed for teams so no need to include those.

Test 3 Source Code and IP Concerns

As a reminder, the judges will have access to teams’ submitted source code to help them understand the technical approach taken for Test 3. The AlphaPilot team suggests that if there are any concerns with IP, then that IP should be compiled into an executable that is called from your source code and that executable should be included in your archive. However, by abstracting functionality away in executables, it makes it more difficult for the judges to verify that teams have the technical skills and capabilities to succeed during the AlphaPilot Competition. As a result, please balance your team’s individual needs to obscure IP with building transparency of your approach.

Algorithm Requirements:

Teams must develop GNC algorithms to fly the simulated drone through a structured test environment utilizing a typical ACRO/RATE flight mode for control inputs and only on-board sensory feedback. As is typical in drone racing, teams know the approximate locations of their starting point and the gate locations, and they must deal with slight variations during the actual race. In Test 3, teams are challenged to create a single algorithm that can be tested on 25 slight variations of the same Challenge Course.

Teams are given Challenge YAML files on which teams will be tested that indicate the location of the initial conditions for each exam. For Leaderboard Testing, teams are given all 25 Challenge YAML Files and self-report their scores. For Final Testing, teams are not given the variations on the Challenge YAML Files, and their algorithms will be run and scored by HeroX.

Algorithms are not allowed to use ground-truth state data; only inputs from their sensors. These vehicle and sensor feeds are modelled to include real-world errors like thrust-limited motors and rate-based control drift. Teams will have access to this noisy data, as well as some visual processing outputs. In summary, the allowable sensory inputs for algorithms include:

  • Photorealistic 60Hz RGB camera feeds – mono and stereo allowed
  • “Perfect” (not noisy) gate detection algorithm output (polygon coordinates) in the camera frame
  • “Perfect” (not noisy) IR marker locations in the camera frame
  • Noisy IMU data
  • Noisy downward-facing laser range-finder to aid altitude estimation. The laser range finder points in the negative z direction. Its measurements are defined in the drone body frame, and thus range measurements will have a negative value.

Using these sensory inputs, teams are permitted to conduct any processing that they desire. No visual-based obstacle avoidance is necessary (course paths are mostly clear of obstacles). That being said, it is possible to crash into things and fail the run.

  • Note: AlphaPilot is aware that it will be possible to back-out the exact global 3D locations of gates. Teams are not allowed to utilize this data in their algorithms, and any teams that do so will be flagged during judging.

Algorithm entries for the Final Challenge Course (see more details about ‘Final Testing’ below) must be submitted as source code before the final deadline. The final submission form will contain fields for attachment of a zip file of your code archive and technical report.

Entries to both the Leaderboard and Final Exam must be submitted using the Test #3 entry form provided on the AlphaPilot HeroX website. Entries for the Leaderboard Challenge Course must be a single Scores YAML file and submitted as an attachment. This file contains scores from all 25 Results YAML files and is used by HeroX to calculate teams Leaderboard Algorithm Score (where top 5 scores are found and averaged).

Algorithm entries for the Final Challenge Course (see more details about ‘Final Testing’ below) must be submitted executable programs as well as source code before the final deadline. The final submission form will contain fields for attachment of a zip file of your executable code, source code, and technical report.

See the ‘Testing’ section for more details about how algorithms will be tested.

Note: GPU and CUDA drivers will already be installed in the testing environment. The following instance will be used at the basis for the testing environment: https://aws.amazon.com/marketplace/pp/B077GCZ4GR

 

Technical Report Requirements:

In the Technical Report, teams must document how their algorithms work and how they conducted testing and analysis. Teams should detail their use of any libraries or well-known approaches. In addition to describing their approach, teams must also address:

  • How the team plans to build onto or modify their drone racing algorithms for use in the AlphaPilot competition should they qualify.
  • Any technical issues the team ran into and how they overcame them.

Reports need to be in PDF format and 2-pages maximum, single-spaced. Font size should be minimum 11pts. Insertion of text, equations, images, figures, plots, code, and pseudo-code is accepted but will be included in the 2-page limitation.

 

Testing:

The Leaderboard and Final Challenge Course are now out! The Leaderboard represents a practice exam and will give teams a rough idea of how well they will ultimately perform. Please note HeroX AlphaPilot will only accept leaderboard submissions at this time, and there is no need to submit source code until the final deadline.

Overview

Challenge 3 is modeled to closely resemble a real-world FPV drone racing scenario. As such, participants know the nominal gate locations from their practice rounds. However, at race time, the actual locations of the gates may be slightly perturbed (unknown to the participants).  Similar to human FPV racers, autonomy algorithms should also be able to navigate even when the gates are slightly perturbed. For the challenge, this robustness is addressed as follows: The nominal gate locations are available for the participants to use in their algorithms. But for the evaluation of their algorithms, small perturbations will be added to these nominal gate locations. The resulting perturbed gate locations are used in the evaluation runs, but are not known a priori to the autonomy algorithms. The bounds on the gate location perturbations are known to the contestants and can be read from the ROS parameter server (see README_v2.md). 

The figure below shows a snapshot of the Test 3 Challenge Course. Contestants will start in a known starting location in the upper right corner of the figure. Contestants must traverse gates in order along the course marked in red and must finish by passing through the gate in the lower left corner of the figure (gate 6) within a known time limit. Missing a gate does not lead to disqualification, however if contestants miss a gate in the ordering, they forgo the points for that gate and cannot traverse it (for reward) after passing a gate later in the gate ordering. Gate 6 (the finish line) is unskippable and must be traversed in order to complete the course. In the figure below, gate IDs belonging to gates along the race path are marked in yellow. Gate IDs in blue are not part of the race course and will not change in position. The ordering of the gates to traverse will not change. Other obstacles in the environment will not change. For Leaderboard score submission, teams are given all 25 gate perturbation layouts, and they will self-report their scores to HeroX. For Final Testing, teams are not given gate perturbations, and their algorithms will be run and scored by HeroX. The course order is shown in Table 1. 

 

CourseGate 10, 21, 2, 13, 9, 14, 1, 22, 15, 23, 6.

Table 1. The ordered list of gates to traverse to successfully complete the course.

Figure 2: Shows a screenshot of the Leaderboard/Final Testing Challenge Course in MIT’s FlightGoggles Simulation.

 

Leaderboard Testing

Teams can now download the Leaderboard Challenge Course files (challenge_leaderboardtest.zip) and submit their Scores YAML file to the leaderboard for evaluation.

The Leaderboard Challenge Course folder contains the 25 YAML files which are slight variations on the above Challenge Course. Teams are expected to test their algorithms on each one of these YAML files and output a Results YAML file per run (as done by the Reporter node). Using the scoring scripts provided, teams can generate the Scores YAML file to submit to the Leaderboard. The Scores YAML file will read, the top 5 scores will be found and averaged, and this final score will be posted on the Leaderboard.

The first round on the leaderboard will close at Monday, Feb. 18th at 11:59PM PST, and any submissions will appear on the leaderboard by 9AM PST the following day. After this, the leaderboard will refresh daily until the final deadline. Teams can update their submission multiple times per day, but only their most recently uploaded YAML files will be evaluated and scored.

 

Final Testing

The Final Challenge Course is also based on the above Challenge Course, but teams will not receive the 25 Challenge YAML files, which again are slight variations on the initial conditions. Teams may assume that the stochastic changes will mirror those provided in Leaderboard Challenge Course Challenge YAML files.

By the final deadline, teams must submit source code for evaluation on these 25 sequestered Challenge YAML files. The testing environment is an instance of Ubuntu 16.04.5 LTS running in AWS type p3.2xlarge. The following instance will be used as the basis for the testing environment: https://aws.amazon.com/marketplace/pp/B077GCZ4GR

Additionally, this environment will have ROS Kinetic, OpenCV 3.4, and FlightGoggles already installed for teams.

The final algorithm testing process will be conducted as such:

  1. The Flight Goggles catkin workspace with the addition of “scorer.py” in flightgoggles/launch will be sourced.
  2. A new Linux user home directory is created. The team’s archive will be unzipped and ‘submission’ folder will be unpacked into the user’s home directory.
  3. A symbolic link “scorer.launch” will be added to flightgoggles/launch that links to “scorer.launch” in the submission root directory
  4. Team dependencies will be added by running the following command in the submission root directory:
    >> ./install.bash
    Note: there is a cut-off for installation time for a team’s algorithm, and it is 1 hour. If a team’s source code takes longer than this to install dependencies, the script will be stopped, and the team will be given an Algorithm Score of 0 points.
  5. The scorer will run the launch file 25 times with the perturbed gates in `flightgoggles/config/gate_locations_x.yaml` and accumulate the results from the reporter in a `results` folder. At the end of the evaluation, `scorer.py` is run which generates a `scores.yaml` which contains the individual scores for each run. This will be implemented in testing using the following command:
    >> source catkin_ws/devel/setup.bash && rosrun flightgoggles scorer.sh
  6. The score.yaml file will be read, the top 5 scores will be found and averaged, and this final result will be used for the algorithm score.
    Note: there is a cut-off for the total submission evaluation time (Final Testing steps 5 and 6) and that is 2 hours. If a team’s algorithm takes longer than this to be evaluated, the testing will be stopped, and the team will be given an Algorithm Score of 0 points.

Separately, judges will read and score each team’s technical report which will be subsequently used to calculate their total final score for Test 3.

Test 3 Submission Checking

It is essential that teams follow the above submission requirements exactly. However, in the event your team makes a small mistake, we have implemented a Submission Checker for Test 3. Each team has the opportunity to try to fix their submission as many times as needed before Monday, April 1st at 12PM EST if their submission fails the algorithm submission testing described above. In the event of a submission failure, your team will receive an email with the error and will have a short window to re-submit. If we suspect teams are abusing this system, it will be investigated and scored accordingly.

Test #3 - Frequently Asked Questions (FAQs)

Can teams utilize the global 3D locations of the gates?

As is typical in drone racing, teams know approximately the initial drone and gate locations and must deal with slight variations during the actual race. Teams are challenged to create a single algorithm that can be tested on 25 slight variations of the same Challenge Course.

Teams are given Challenge YAML files that indicate the initial location of the drone and gates for each exam and on which team algorithms will be tested. For Leaderboard Testing, teams are given all 25 Challenge YAML Files and self-report their scores. For Final Testing, teams are not given the variations on the Challenge YAML Files, and their algorithms will be run and scored by HeroX.

So in general, no, teams cannot utilize the exact global 3D locations of the gates. However, they can approximate it from visual processing and other on-board feedback.

Note: AlphaPilot is aware that there is a hack that makes it possible to back-out the exact global 3D locations of gates. Teams are not allowed to utilize this data in their algorithms, and any teams that do so will be flagged during judging.

Will we have to do obstacle avoidance?

No. Visual-based obstacle avoidance is not needed (course paths are mostly clear of obstacles). That being said, it is possible to crash into things and fail the run.

Can teams utilize the stereo cameras?

Yes. You can use the photorealistic 60Hz RGB camera feeds – mono and stereo allowed.

Will you give teams information about the drone model?

More details about the drone (vehicle dynamics, sensor models, specifications, etc.) are added to the GitHub repo.

What are some good resources to learn ROS and learn how to add our nodes onto the flightgoogles?

Please refer to the official ROS tutorials: http://wiki.ros.org/ROS/Tutorials

I couldn't find the API documentation for FlightGoggles to send http messages over to the simulation. Can I use any ROS API to do that?

FlightGoggles uses standard ROS message types for communication. As such please refer to the ROS message publisher/subscriber tutorial to familiarize yourself with this message passing concept: http://wiki.ros.org/ROS/Tutorials/WritingPublisherSubscriber%28c%2B%2B%29

Will FlightGoggles be updated to depend on the current recommended ROS version ( --rosdistro melodic v.s. --rosdistro kinetic)?

We decided to start with Kinetic for stability. MIT will update and maintain flightGoggles in the future. We will consider an update to melodic in the future. However, it will happen after test 3 is completed.

Are we allowed to change the camera angle in the FlightGoggles simulator? Can you set the camera angle to something greater than 0?

No, we are not allowing teams to change the camera angle. All teams are challenged to use the same hardware setup in AlphaPilot.

What data inputs can be used in Test 3?

Please see the README file for Test 3 for a complete list of allowed and prohibited inputs.

Are the gates disturbances in the IR beacons sensor or in the real position of the gates in map?

The gate disturbances are from the nominal position of the gates in the map.

Will there by a left/right camera version of the IR beacons in Flight Goggles?

Please see the description provided by alphapilot for allowed topics and params: https://www.herox.com/alphapilot/resource/320

In Test 3, it's hard to know when we fly through a gate, without ground-truth data. Can we get an event for us to subscribe to?

Unfortunately, we will not provide this for teams as this is not information you would realistically have during a drone race.

Will all target gates be vertical, or will there be horizontal gates (flying through vertically)?

All target gates in the virtual qualifiers will be vertical for both Test #2 and Test #3.

For test 3, will the gates locations (not including the perturbations) be changed for the final test?

No, the nominal positions of the gates are the same for both the Leaderboard practice and final tests.

How will the race be timed? Will the timing start from begin of code launch or will it start when flying through the first gate or leaving a bounding box? (technically: how will time to initialize estimators etc. be penalized?)

The race will be timed from the moment of the drone's first takeoff movement to the time when the drone arrives at the finish gate.

From the latest update it would seem that the ROS server is sending a message about the location of each gate.  Is the messages sent from the given ROS node providing a x,y from the camera view from the drone's perspective?

The "/uav/camera/left/ir_beacons" ROS topic publishes all unoccluded IR beacons within the camera's field of view. This message contains IR marker detections in the image space of the left camera.

What are minimum requirements for FiightGoggles video card wise?

  • Minimum Local Hardware Requirements
    • A Vulkan-supported GPU with >=2.1GB of VRAM.
    • Native Ubuntu 16.04 installation with ROS Kinetic on x86/x64.
    • Virtual machines do not have native access to the GPU and therefore are not supported.
    • nvidia-docker2 does not currently support the Vulkan API and therefore cannot run the FlightGoggles renderer binary (see workarounds in issue #46).
  • We have tested this project on two different setups: High end Desktop computer with:
    • Processor: Intel i9 extreme (i9-7980XE)
    • RAM: 32Gb
    • GPU: Titan V
  • We have also tested on the following AWS instances:
    • p3.2xlarge
    • g3s.xlarge
  • Please see https://github.com/mit-fast/FlightGoggles/wiki/Prerequisites-and-Testing-Setup

Does the flightgoggles simulator take .yaml files as an input or will we have to convert the files?

Teams can use the Challenge YAML files as input to FlightGoggles for their own development and testing.

However, users of the FlightGoggles simulator should not have to read or modify any YAML files for correct operation of the simulation system.

All relevant information to the FlightGoggles simulator is exposed through the ROS Parameter API: https://github.com/mit-fast/FlightGoggles/wiki/ros-params

Can you simulate more than one drone?

We intentionally did not put support for multi-drone simulation in order to avoid confusion and to keep the initial FlightGoggles code as simple as possible. Before the challenge is done, we will not provide the software for simulating multiple drones. That said, you are able to run multiple instances of FlightGoggles if you would like to conduct several tests, of course.

Keras is now a part of tensorflow. Can we use keras to develop the AI model.

Solutions need to be compatible with the algorithm and testing requirements. Otherwise, there are no restrictions on the approach and libraries teams can use to develop their solutions.