Nature Inspired Basic Research at Eglin
6 October 2014
Ric Wehling & Dr. Jennifer Talley Research Biological Scientist RWWI Integrity Service Excellence Air Force Research Laboratory
The Need for Agile Autonomy
• Report on Technology Horizons highlights a class of problems RWW has been addressing for years: how to build systems capable of autonomous behavior? • The need is for ‘smart’ systems that can navigate, identify and track ‘targets,’ and perform the mission without having to rely on external aids (GPS, a human in the loop, etc.) • We are looking to Nature for inspiration for the sensor suites and information processing to enable agile autonomous systems
DISTRIBUTION A. Approved for public release; distribution unlimited. 2
Guided Flying Machines Have Common Core Instrumentation
ATTITUDE, ATTITUDE RATE Strain gauges, hair cells Gyroscopes Halteres distributed across body
Compound Eyes, Ocelli Antennae: Acoustic sensors VISION Gravity sensors Olfactory sensors Seeker Compound Eyes Geomagnetic sensors Air speed sensors etc. DISTRIBUTION A. Approved for public release; distribution unlimited. 3
Biologically-Inspired Unmanned Autonomous System: GN&C
NOVEL Acoustic sensors BIO-INSPIRED Ocelli SENSORS AND Olfactory Sensors PROCESSING Mechanosensors Antennae
NAVIGATION, WFOV STATE IMAGE SEEKER VECTOR CONTROL PRO- GUIDANCE AUTOPILOT SENSOR ESTIMATOR AUTHORITY CESSOR LAW
IMU: Optic flow Mode Sensing INERTIAL SENSOR
Power Materials Structures Airframe Signatures Understand and apply the basic principles; not emulate flying insects
DISTRIBUTION A. Approved for public release; distribution unlimited. 4
In-house Instrumentation Development
• Electro Retinography Rig • Stargate (512 UV, blue, green interdigitated 16 x 32 deg fov LED arrays) • Automated goniometer • Flight path characterization • Histology • Environmental monitoring for providing realistic environments during measurements • UV-vis spectral / polarization camera
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 5
In-house Instrumentation
• Available associated capabilities – KHILS panoramic Biodome (RGB 120 degree by 240 degree presented imagery, designed to train humans) – Scanning Electron Microscopes
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 6
New In-house Activities
• Building capability – Magnetosensing – Antennal investigations – Hex grid array geometry – Compressive sensing (bioprincipic)
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 7
Comparative Approach
Same Sensors Same Behavior Different Sensors • Predatory air strikes • Compound Eyes • Fast • Ocelli Robber Fly • Agile • Antenna • Halteres
Dragonfly • Compound Eyes (& Damselfly) • Ocelli • Antenna Seeking understanding of sensory integration • Compound Eyes Owlfly and the production of • Antenna behavior. DISTRIBUTION A. Approved for public release; distribution unlimited. 8
Understand Visual Sensors and the Production of Behavior
• Spectral Measurements of compound eyes and ocelli. – Equalization of intensity across the spectrum – Stimulate from 300 to 600 nm – Record electroretinogram • Field of view measurements of compound eyes and ocelli. – Using LEDs of 3 different wavelengths – Sample 16 points in space around the insect – Record electroretinogram • Flicker fusion frequency of compound eyes and ocelli. – Using LEDs of 3 different wavelengths at one point in space – Record electroretinogram • High speed recordings of free flight. – Confined to 2 x 1 x 1 meters – Indoors vs outdoors • Head rotation recordings to oscillating horizon.
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 9
Optics Train for Spectral Measurements
(PIE)
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 10
Spectral Intensity Calibration
Shorter wavelengths have low signal to noise ratio because the near infrared signal saturates the spectrometer
Noise from spectrometer
Unfiltered xenon lamp spectrum shows different intensities at different wavelengths.
DISTRIBUTION A. Approved for public release; distribution unlimited. 11
Spectral Intensity Calibration
Structure from xenon lamp
Noise from spectrometer
Combination of monochromator and photon intensity equalizer (PIE) produce a consistent intensity across the spectrum.
DISTRIBUTION A. Approved for public release; distribution unlimited. 12
Spectral Measurements (Owlfly Ascaloptynx appendiculata)
DISTRIBUTION A. Approved for public release; distribution unlimited. 13
Pyranometer & Pyrheliometer Data from Sunlight
10
9
8 cloud Eppley Laboratory 7 Precision Spectral Pyranometer (PSP)
6
5 psp295 psp695 4
Output(mV) nip (295 filter)
3 sunset 2 Eppley Laboratory Normal sunrise Incidence Pyrheliometer (NIP) 1
0 0:00:00 4:48:00 9:36:00 14:24:00 19:12:00 24:00:00 28:48:00 33:36:00 38:24:00 -1 Hours
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 14
LED Panel Calibration to Sunlight
Measured during sunniest part of the day on rooftop sun Diffuse between
diffuse = global – direct * cosine (solar zenith angle) 295 and 695 nm = 198 W/m2 Pyranometer Pyrheliometer measures global measures direct d2
d3
d1
Total energy from LED panel = (d2*d3)÷d12÷2π*diffuse
Total LED panel power required to match full sunlight = 3.9 W/m2
DISTRIBUTION A. Approved for public release; distribution unlimited. 15
Scaling the Spectral Content of LEDs to Sunlight Dr. Dennis Norton AFRL/RW
374 nm peak
467 nm peak
517 nm peak
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 16
LED panel output controlled by two parameters Dr. Arunava Banerjee UF Varying PWM [0, 255] Varying I [0, 255] 0.25 0.25
0.2 0.2
0.15 0.15
0.1 0.1
millivolts millivolts millivolts millivolts 0.05 0.05 0 0 0 50 100 150 200 250 300 0 50 100 150 200 250 300 -0.05 PWM I
퐼 푃푊푀 푃푊푀 푃푊푀 훼 ∗ * f 푈푉 , 퐺푅 , 퐵퐿 255 255 255 255
I = ceiling current control PWM = pulse width modulation control level
DISTRIBUTION A. Approved for public release; distribution unlimited. 17
Pyranometer data taken of 8 hours of LED panel operation.
Dr. Arunava Banerjee UF
Pyranometer output output (mV) Pyranometer output (mV) Pyranometer
Time (seconds) Time (seconds)
24.5 0.00003
Effect of ambient 24 0.000025 0.00002 temperature from HVAC 23.5 0.000015 temperature cycling on pyranometer 23 output 0.00001 PSP 295 no LEDs on
22.5 0.000005 Temperature (degrees C) (degrees Temperature
22 (mV) output Pyranometer 0 9:07:12 10:19:12 11:31:12 12:43:12 13:55:12 Time (HH:MM:SS)
DISTRIBUTION A. Approved for public release; distribution unlimited. 18
Compute the resting value of each LED panel display Dr. Arunava Banerjee UF
a + b (1 – exp (t/Ƭ)) Ƭ = 0
Pyranometer output output (mV) Pyranometer output (mV) Pyranometer
Time (seconds) Time (seconds) Calculate scalar change for each intensity 푃푊푀 푃푊푀 푃푊푀 0.138 ∗ 푈푉+ 0.035 * 퐺푅 + 0.051 * 퐵퐿 255 255 255
PWMUV = 7 PWMGR = 109 PWMBL = 38
DISTRIBUTION A. Approved for public release; distribution unlimited. 19
LED Panel Measurements
Stimulation
All UV G B
8 seconds DISTRIBUTION A. Approved for public release; distribution unlimited. 20
High Speed Video Owlfly Flight
DISTRIBUTION A. Approved for public release; distribution unlimited. 21
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 22
Tracking software progress
David Forester Kaitlin Fair Michael (David) Richards
DISTRIBUTION A. Approved for public release; distribution unlimited. 23
Head Rotation to Oscillating Horizon Jessica Thompson Celina Calma
Panel field of view = 45 x 22.5 degrees
Ischnura ramburii
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 24
KHILS Projector (half) Dome
Jessica Thompson Celina Calma
Field of view = 240 x 120 degrees
Ischnura ramburii
DISTRIBUTION A. Approved for public release; distribution unlimited. 25
KHILS Projector Spectral Characteristics
Blue filter
sky
Green filter “White”
ground Red filter “Black”
DISTRIBUTION A. Approved for public release; distribution unlimited. 26
Damselfly Movie in KHILS Projector through City Scene
Jessica Thompson Celina Calma DISTRIBUTION A. Approved for public release; distribution unlimited. 27
Damselfly Movie in KHILS Projector to Moving Horizon
Jessica Thompson Celina Calma 7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 28
Analysis of Head Rotation
Jessica Thompson Celina Calma 8°
48°
7 November 2014
DISTRIBUTION A. Approved for public release; distribution unlimited. 29
LED panel versus WFOV stimulus and window stimulus
• LED panel (16 by 32 deg at design distance) may not be wide field enough to stimulate horizon following. • LED panel cannot behaviorally attract insects unlike natural sunlight through a window. • Phenomena under investigation.
DISTRIBUTION A. Approved for public release; distribution unlimited. 30
Histology
Bridget Lyons Laura Nelson Whole brain section
Retinal Tissue Retinal Tissue Ocellar Nerve
Optic Lobe
Top Row: Janus Green stained sample of entire Sarcophagid sample. Two sections of retinal tissue sections in sarcophagus fly. Bottom Row: Sarcophagid in whole mount procedure, using different fluorescing cubes for tissue identification
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 31
Compressive Sensing Motivation Kaitlin Fair
DSP: Collect Compress Reconstruct Analyze/Act
Sparsely CS: Reconstruct Analyze/Act Sample
Sparsely Sample Analyze/Act Biology: Sufficient Information*
Training Learning Predict *ML: Set Algorithm Outcome
DISTRIBUTION A. Approved for public release; distribution unlimited. 32
Dictionary Learning for Classification in the Sparse Domain Kaitlin Fair 풙 = 푫 ∗ 풂
2.5
2
1.5
1
0.5 = 0 * -0.5 -1
-1.5 Test Image 0 50 100 150 200 250 300 350 400 Sparse Representation
Learned Dictionary Perform classification using only the relevant MNIST Training Dataset Sample sparse coefficients DISTRIBUTION A. Approved for public release; distribution unlimited. 33
Sensilla Identification/Characterization • Neuroptera Capt. Alex Hollenbeck – Owlfly Bridget Lyons • Very long antennae with bulb-shaped or spoon-shaped lobe • Distinct cone or stylus projects at or near apex • Stylus appears to have pits and/or grooves – Antlion • Antennae shorter and thicker than owlfly, with less pronounced bulb • Stylus almost cylindrical with abrupt taper and possibly pits at tip – Lacewing • Very long antennae
• Diptera, Family: Asilidae (Robberfly) – Stylus located in significant depression at antenna tip. – Feature has been noted in identification keys but no functional characterization can be found
Big picture: SEM and optical microscopy investigations show (previously uncharacterized?) cone/stylus at apex of antennae in robberflies and Neuroptera.
DISTRIBUTION A. Approved for public release; distribution unlimited. 34
Owlfly (mostly Ululodes)
bugguide.net Capt. Alex Hollenbeck; Bridget Lyons
DISTRIBUTION A. Approved for public release; distribution unlimited. 35
Antlion (Myrmeleontidae)
Capt. Alex Hollenbeck DISTRIBUTION A. Approved for public release; distribution unlimited. 36 Bridget Lyons Lacewing (Nodita spp.)
Capt. Alex Hollenbeck Bridget Lyons
DISTRIBUTION A. Approved for public release; distribution unlimited. 37
Robber Fly (family: Asilidae) Antennae Capt. Alex Hollenbeck Bridget Lyons Unidentified Laphria saffrana Diogmites spp. Unidentified Magnetoreception and Multimodal Sensing Dr. Brian Taylor • Goal: – Use Earth’s magnetic field in concert with other sensing modes to navigate without HAVING to rely on GPS • Why: – Several animals (e.g., insects to turtles) appear to use the magnetic field in concert with other sensor modes to navigate • Local homing to continental migration – Air Force Related Problems • Air Force Relevance and Benefits: – Animals do not have GPS, SIRI, or CORTANA, but are capable of navigating over large distances • What are they doing? • What can we learn? • What can we leverage? and what can we learn from them? – GPS Denied Navigation – Enhanced and Robust Autonomy
DISTRIBUTION A. Approved for public release; distribution unlimited. 39
Magnetoreception and Multimodal Sensing Dr. Brian Taylor • Behavioral Simulations – Based on a bicoordinate strategy – Inspired by turtle behavior • Putman et al. 2011 • Colors represent different field intensities • White lines represent different inclinations
Bicoordinate Strategy based on Lohmann 2007
Behavioral Simulations DISTRIBUTION A. Approved for public release; distribution unlimited. 40
Hardware Development
Dr. Brian Taylor • Hardware Facilities – Artificial Magnetic Environment (AME) • ~1m x 1m x 1m • Can run from an Earth- strength magnetic field to ~4x Earth’s magnetic field (~.4G to ~1.8G). – Distributed magnetometer awaiting construction
Hardware Development DISTRIBUTION A. Approved for public release; distribution unlimited. 41
Efficient Hexagonal Imaging
• Hexagonally sampled digital images have several Hexagonal Sampling is Optimal
advantages over rectangularly sampled digital ...... FFT Frequency images ...... Domain • Hexagonal sampling is optimal for isotropically ...... band-limited signals • An array representation of the hexagonal grid using Spatial Domain Ahex Arect ASA allows for efficient digital image processing of ...... hexagonally sampled imagery ...... FFT A 3 ...... gray • ...... Prototype hexagonal imagers have been developed ...... Agreen 2 ......
Array Set Addressing (ASA) Prototype Hexagonal Imagers x
y (0,0) (2/√3,0) (4/√3,0) (6/√3,0) (0,0,0) (0,0,1) (0,0,2) (0,0,3)
(0,1,0) (0,1,1) (0,1,2) (0,1,3) (1/√3,1) (3/√3,1) (5/√3,1) (7/√3,1)
(0,2) (2/√3,2) (4/√3,2) (6/√3,2) (1,0,0) (1,0,1) (1,0,2) (1,0,3)
(1/√3,3) (3/√3,3) (5/√3,3) (7/√3,3) (1,1,0) (1,1,1) (1,1,2) (1,1,3)
Developed by Centeye, Inc. 퐴푆퐴 ∈ {(0,1) × ℤ × ℤ}
DISTRIBUTION A. Approved for public release; distribution unlimited. 42
UV-Vis Camera Development
• Basic camera (QSI-640) is an astronomical camera with 2048 by 2048 pixels and no color filtering (no Bayer filter); configured with band-pass filters mounted on filter wheel; externally mounted Moxtek linear polarizers; controlled by software on a laptop. • Undergoing full field of view calibration • Will permit imaging objects as seen by insects (including other insects), providing spectral and linear polarization signatures
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 43
QSI-640 Filter Wheel
Populated 8-place filter wheel
DISTRIBUTION A. Approved for public release; distribution unlimited. 44
Angular Acuity Variation
Animal Eyes, Land and Nilsson, Oxford, 2002 DISTRIBUTION A. Approved for public release; distribution unlimited. 45
The “Automated Goniometer:” a computerized instrument for evaluating regional specializations in insect eyes [email protected] Specialized Eye Local eye parameters of Regions: Frontal interest include facet “acute zones” sizes and interommatidial improve spatial angles (from pseudopupil sampling resolution analyses). and photon catch. Problem: manual data Trade-offs: reduced collection is very slow resolution and/or (1000’s of visual sampling field of view in other units per eye). Robberfly (Asilidae) parts of the eye. Leaf-footed bug (Coreidae) Project Goals: digital 1. Develop a camera computerized, motorized microscope goniometer instrument insect for fast, highly automated analyses. 2. Study the scaling and Holcocephala fusca(Asilidae) diversity of regional eye insect specializations. 3. Understand acute zone custom motor PC design features and motorized controllers constraints. goniometer DISTRIBUTION A. Approved for public release; distribution unlimited. 46
“Automated Goniometer” Preliminary Results
John K. Douglass
Numerical values of facets sizes are raw data, not actual facet dimensions
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 47
Larger Scale Activities: AFRL Center of Excellence on Nature-Inspired Sciences
• A grant set up with a consortium of Universities in the US to focus on relevant basic research: U Washington, U Maryland, Case Western, Johns Hopkins U • Primary focus on insect sensing, flight path control; secondary focus on bat sensing, flight path control; interactions between bats and insects • Enhance collaborations across consortium members and with AFRL • Exchange personnel in laboratories
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 48
Larger Scale Activities: Bio-Inspired Unmanned Autonomous System Project Agreement
• Project Agreement between the US AFRL and the UK Dstl to explore utilization of bio- inspired technologies for flying platforms • Dual emphasis on basic research and demonstrating capabilities in prototypes • Examples: developing wide field of view sensors based on compound eye; developing GN&C techniques using distributed sensors in Wide Field Integration applications
7 November 2014 DISTRIBUTION A. Approved for public release; distribution unlimited. 49