2024-07-10 –, While Loop (4.2)
We present RunwayPNPSolve.jl, a framework for uncertainty-aware pose estimation for visual landing applications with multiple methods including real-time least-squares minimization + resampling, Monte-Carlo Markov-Chain, and a linear approximation, by leveraging the existing Julia package ecosystem.
The package further provides a framework of useful primitives to build simultaneously differentiable, unitful and coordinate-system-aware interfaces, and an interactive visualization pipeline.
Autononomous visual landing systems for personal and commercial aircraft have typically only been available given the use of expensive sensors with a moderate safety certification level.
With the recent emergence of high-performing computer vision models using deep learning, a new avenue for such landing systems has opened, and is being pursued by multiple companies such as Acubed (an Airbus company), Daedalean, XWing, Reliable Robotics, and others.
However, it is an open problem to construct such systems under the high safety and certification requirements in aviation.
Thus, deep learning models are typically only used for image feature estimation (including measures of uncertainty), and more traditional techniques are used for the actual pose estimation and processing of uncertainty estimates.
In this talk we will focus on the latter, and showcase our package RunwayPNPSolve.jl which leverages Julia’s extensive ecosystem for processing pixel features, for example provided as a Normal distribution, to construct a probability distribution over our pose and orientation.
In particular, we first present a framework of data primitives tailored for visual landing setting, including
- statically typed coordinate systems (e.g.
XYZ,ImageProjection,LongitudeLatitude) that integrate withGeodesy.jlandCoordinateTransformations.jl; - defining the external and (most of the) internal API through
Unitful.jlinterfaces (m,pxl,degrees), while maintaining type flexibility for e.g.ForwardDiff.jl’sDualtypes; - a live satelite-view of pose estimation on real runways that integrates with OpenStreetMaps through
Tyler.jland a custom runway database; and - interactive visualization capabilities through
Makie.jl.
After presenting the framework, we present several approaches implemented in RunwayPNPSolve.jl for the actual uncertainty-aware pose estimation, including
- a mathematically sound approach using Markov-Chain Monte-Carlo backed by
Turing.jl; - a resampling based approach built on top of
NonlinearSolve.jl(although we are happy to discuss our attempts to useLsqFit.jl,Optim.jl, andNLSolve.jl); - a somewhat uninformed approach that directly differentiates through the iterative least-squares solver from
NonlinearSolve.jlusingForwardDiff.jl; - an efficient linear-approximation requiring only a single (automatic) derivative.
If time permits, we will also discuss how we can estimate the quality of the resulting uncertainty estimates by measuring calibration in a variety of way, including a novel method we plan to publish soon.
The package including the discussed topics can be found under https://github.com/sisl/RunwayPNPSolve.jl and https://github.com/sisl/MultivariateCalibration.jl.
- PhD Student at the Stanford Intelligent Systems Lab (SISL), developing certification guidelines for employing data-driven algorithms in safety-critical applications.