Scaling a flying robot down to the size of a fly or bee requires advances in manufacturing, sensing and control, and will provide insights into mechanisms used by their biological counterparts. Controlled flight at this scale has previously required external cameras to provide the feedback to regulate the continuous corrective manoeuvres necessary to keep the unstable robot from tumbling. One stabilization mechanism used by flying insects may be to sense the horizon or Sun using the ocelli, a set of three light sensors distinct from the compound eyes. Here, we present an ocelli-inspired visual sensor and use it to stabilize a fly-sized robot. We propose a feedback controller that applies torque in proportion to the angular velocity of the source of light estimated by the ocelli. We demonstrate theoretically and empirically that this is sufficient to stabilize the robot's upright orientation. This constitutes the first known use of onboard sensors at this scale. Dipteran flies use halteres to provide gyroscopic velocity feedback, but it is unknown how other insects such as honeybees stabilize flight without these sensory organs. Our results, using a vehicle of similar size and dynamics to the honeybee, suggest how the ocelli could serve this role.
Flying robots on the scale of and inspired by flies may provide insights into the mechanisms used by their biological counterparts. These animals' flight apparatuses have evolved for millions of years to find robust and high-performance solutions that exceed the capabilities of current robotic vehicles. Dipteran flies, for example, are superlatively agile, performing millisecond turns during pursuit  or landing inverted on a ceiling . Moreover, these feats are performed using the resources of a relatively small nervous system, consisting of only 105–107 neurons processing information received from senses carried onboard. It is not well understood how they do this, from the unsteady aerodynamics of their wings interacting with the surrounding fluid to the sensorimotor transductions in their brain [3,4]. An effort to reverse-engineer their flight apparatus using a robot with similar characteristics could provide insights that would be difficult to obtain using other methods such as fluid mechanics models or experimentally probing animal behaviour. The result will be robot systems that will eventually rival the extraordinary capabilities of insects.
Creating a small flying autonomous vehicle the size of a fly such as that shown in figure 1 is a difficult undertaking. As vehicle size diminishes, many conventional approaches to lift, propulsion, sensing and control become impractical because of the physics of scaling. For example, propulsion based on rotating motors is inefficient, because heat dissipation per unit mass in the magnetic coils of a rotary electric motor increases as l−2 , where l is some characteristic length of the vehicle such as wingspan. Combined with increased friction losses owing to an increased surface-to-volume ratio, exacerbated by the need for significant gearing, this results in very low power densities in small electromagnetic motors . In addition, the lift-to-drag ratio for fixed aerofoils decreases at small scales because of the greater effect of viscous forces relative to lift-generating inertial forces at low Reynolds numbers .
The challenges imposed by the small scale of the vehicle shown in figure 1 extend to sensing and control. The rate of rotational acceleration increases with decreasing size, scaling as l−1 , requiring a feedback sensor with higher bandwidth . Silicon microfabricated gyroscopes and accelerometers have recently become small enough to be carried on such vehicles, primarily because of pressure from the consumer electronics industry. However, their signal fidelity may be significantly disrupted by the vibratory environment on vehicles with flapping wings , particularly as the flapping frequency increases at smaller scales. At a higher level of control, perceiving attitude and position relative to obstacles or targets will be essential to achieving autonomy. The global positioning system used by larger aircraft becomes impractical at small scale because of insufficient resolution (1–10 m) and signal disruption in cluttered and indoor environments . Emissive sensors such as a laser rangefinder are currently too heavy—the lightest commercially available scanning rangefinder has a mass of 160 g (Hokuyu URG-04LX-UG01), three orders of magnitude heavier than the vehicle in figure 1—and require too much power . Vision provides a promising solution: cameras can be made small and low power , but, because of the restricted onboard processing power available, computational requirements must be minimized. Although the rate of translational acceleration remains constant as scale decreases , obstacles may be nearer, requiring high frame rates and faster processing. Despite the difficulties imposed by small scale, demonstrations targeted at such vehicles, to name a few, include navigating confined spaces [13–15], a high-frame-rate (300 Hz) omnidirectional camera , obstacle avoidance using monocular vision [11,17] or stereo , altitude regulation [19,20], hovering  and an initial implementation on a robotic fly .
The first fly-sized robot to lift its own weight took inspiration from flying insects [6,22]. For the sake of mechanical simplicity, this vehicle has two hoverfly-inspired wings rather than four, driven by piezoelectric actuators performing a muscle-like reciprocating action . Piezoelectric actuators were chosen because they scale down more favourably than electromagnetic motors , and the flapping motion generates unsteady aerodynamics that enhance lift . This prototype is now being used to better understand and optimize the relevant fluid mechanics [24,25].
When first flown without guide wires, however, the robotic fly tumbled . This suggested that, to remain aloft, it requires constant corrective feedback as is required in unstable fighter jets . Simulations have suggested that insects have a similar instability [28–32]. Because of the fast rotational dynamics , control theory dictates that any stabilizing feedback controller must have a short time delay [9,32]. Controlled flight demonstrations have so far relied on an array of external cameras to precisely triangulate the position and orientation of the vehicle equipped with reflective markers , but this approach cannot extend beyond specific laboratory conditions. Stability has been achieved with aerofoils such as air dampers  or a tail , but this makes the vehicle susceptible to wind disturbances and sacrifices manoeuvrability.
Here, we address the challenge of achieving stability of a robotic fly by considering an approach inspired by flying insects. Most flying insects have three light sensors, distinct from the compound eyes, that point roughly upwards and sense light from the sky and/or the Sun . These three sensors, known as the ocelli, are relatively defocused and carry many neurons that sample the same light information [36,37]. With this optical apparatus, they are much more sensitive to light than the compound eyes , allowing them to quickly sense changes in the light levels to aid in rapid self-righting manoeuvres. However, the precise role they play in flight stability is not well understood [3,38,39]. While dipteran flies are thought to stabilize flight in part by using mechanosensory feedback from their elaborate gyroscopic halteres [40,41], other insect orders have no such apparatus. Hawkmoths use large vibrating antennae to sense rotations using a similar mechanism [42,43]. However, the process by which other species stabilize their flight, such as honeybees, remains unknown . It is possible that these animals' flight stabilizer relies heavily on the ocelli.
From the perspective of autonomous flight control, an ocelli-inspired light sensor is nearly the simplest possible visual sensor, minimizing component mass and computational requirements. A number of previous studies have considered ocelli-inspired sensors on flying robots, insect-sized [44–47] or larger [48,49]. In , it was shown that an adaptive classifier could be used to estimate the orientation of the horizon from omnidirectional camera images. In [44–48,50], the absolute direction of the light source or horizon relative to the vehicle was estimated. But whereas aligning to the absolute direction of a light source or horizon may be a valid approach for larger aircraft that fly above obstacles so there is a relatively clear view to the horizon, smaller vehicles may fly near buildings, under foliage or indoors. In these conditions, the horizon may be obstructed. This causes the direction of light sources to vary significantly . A control law that aligned the vehicle with a light source would most likely yield a tilted vehicle under these conditions, leading to significant lateral acceleration and dynamic instability.
In this work, we propose an alternative approach in which a feedback controller applies torque in proportion to angular velocity of the motion of the source of light. This has two benefits. First, it avoids the need for the light source to be directly above for the sensor to produce a useful result. Second, we show that an angular velocity estimate is all that is needed to stabilize the upright orientation of our flapping-wing robotic fly and many flying insects. The first was confirmed by previous work, inspired by observations of derivative-like responses in insect ocelli [36,37], that showed that ocelli simulated in a virtual environment can estimate angular velocity about the pitch and roll axes, regardless of initial orientation . The results also suggested that a linear ocelli response cannot estimate other vehicle motion parameters such as absolute attitude. A motor controller was described that computed a time integral of the ocelli angular velocity estimate. Although this did not require the light to arrive from a known direction, an estimate computed in this manner would slowly drift because of accumulated sensor noise. Here, we build on that work to suggest an alternative approach in which the angular velocity estimate is instead used directly in a feedback controller. By applying torque in proportion to angular velocity only, it is possible to harness this vehicle's flapping-wing dynamics to achieve a stable upright attitude that does not drift and that does not require absolute estimate of attitude.
In §2, we describe the design and fabrication of the ocelli-inspired sensor and flapping-wing robotic vehicle. In §3, we give a model of the flapping-wing vehicle that shows how its dynamics can be stabilized by angular velocity feedback, also known as ‘rate damping’, for both the pitch and roll axes. This is possible, because, like many flying insects [32,51], the centre of mass (CM) of the vehicle shown in figure 1 hangs below the wings . This gives rise to unstable pendulum-like dynamics that can be stabilized by damping. In §4, we build on previous analyses of four-sensor ocelli designs to show how to estimate angular velocity using an arbitrary number of light sensors. This could allow the number to reduce to three, as in most flying insects, to save weight. In §5, we demonstrate this feedback law in operation by stabilizing a fly-sized robot using only feedback from the ocelli-inspired light sensor. We conclude with a discussion about how this work suggests how animals without gyroscopic feedback, such as the honeybee, could use the fast, derivative-like reflex mediated by the ocelli [3,36,37,52] to perform an equivalent role to stabilize their flight dynamics.
2. Sensor and robot fly
2.1. Ocelli sensor design and fabrication
The design of the ocelli sensor (figure 2 and shown attached to the vehicle in figure 1) draws direct inspiration from insects' ocelli, as in previously described prototypes [48,49,53,54]. Each sensor is inclined roughly 30° above the horizon (as in flies ) and captures defocused light from an angular field spanning approximately 180° that is nearly circularly symmetric.
Our ocelli sensor is custom-fabricated to obtain small mass (25 mg) and small footprint (4 × 4 × 3.3 mm) compatible with the robotic fly's payload capacity. The basic structure of the device is formed by a lightweight folded circuit board. Traces were fabricated by laser-ablating the copper layer of copper-clad polyimide. Subsequently, the outline of the board was cut by laser, and a unidirectional carbon fibre layer was adhered to the back to increase structural stiffness. Surface-mount components were soldered on by hand, and the pyramid structure was then hand-folded into shape. Each of the four light detectors consists of a phototransistor (KDT00030 from Fairchild semiconductor) in a common-emitter configuration in series with a 27 kΩ surface-mount resistor. The phototransistor has an infrared cut-off filter, reducing its sensitivity to the bright infrared lights emitted by the motion capture system used to measure flight trajectories. The voltage reading is taken from the collector of each transistor and rises with increasing luminance.
2.2. Robotic fly mechanical characteristics
The flying vehicle used in this work (figure 1) is actuated by a pair of independently moving wings. By altering signals to the piezoactuators driving the wings, they can produce sufficient lift to take-off, as well as produce ‘pitch’ and ‘roll’ torques independently. Greater detail about its mechanical design is given in [33,56].
We define a right-handed coordinate system for the body in which, with the wings extending laterally along the y-axis and the body axis hanging downwards in the negative z-direction, the x-axis (roll) points forward, the y-axis (pitch) points to the left and the z-axis (yaw) points upwards (figure 3). Roll torque is induced by varying the relative stroke amplitudes of the left versus right wing. Pitch torque is induced by moving the ‘mean stroke angle’—the time-averaged angle of the forward–backward motion of the wings—in front (+x) or behind (−x) the CM . Yaw torque can also, in principle, be modulated , but we do not use this capability in this study.
In our current set-up, the robot is given power and controlled through a lightweight compliant tether wire. Power and control commands are transmitted over four thin and flexible 51-gauge (0.022 mm diameter) copper wires, each less than one-quarter the diameter of a human hair. When present, the ocelli sensor requires six additional wires to return sensor feedback: power, ground and four voltage levels from the phototransistors.
Although it is difficult to quantify the effect of the wire tether because of its variable conformation, we attempted to estimate its effects in a number of ways. We remark that by itself it cannot stabilize the vehicle: no flight in hundreds of attempts has been observed to stay upright without active feedback. Considered as a mass, the wires weigh approximately 1 mg for a 5 cm length of four (approximately the altitudes reported here). This is a small component of the 100 mg vehicle mass. Modelled as a point mass at the end of a moment arm extending laterally in the x-direction for a distance of 1 cm (the largest lateral displacement of the tether observed in this work), this would impart a fixed torque of approximately 0.1 μNm. This is smaller than the approximately 1 μNm corrective torques that can be produced by the wings . This is also smaller than typical trim, which are calibrated during initial flights so that the vehicle flies vertically at take-off. As a source of aerodynamic drag, the flapping wings have a larger area (approx. 0.75 cm2) and move nearly an order of magnitude faster (≈10 m s−1 at the tip) than the 5 cm wire relative to the downwash (0.04 cm2, ≈2 m s−1 on average at the wings, measured by particle image velocimetry), suggesting the tether has a negligible effect. Considered as a spring, we remark that the tether exhibits sufficient stiffness when the vehicle is on the ground that it sometimes caused the vehicle to tumble before becoming airborne. Data from such flights are not reported here. Once in the air, however, experiments using a precise torque sensor indicate that a four-wire tether acts as a gentle spring with a stiffness of approximately 0.25 µNm per 45° (electronic supplementary material, figure S1). When incorporated into the model as a spring, it has a small effect on its dynamics, which we consider in greater detail below. Lastly, feedback from the wings can consistently overcome any effects of the tether, because in hovering tests  the vehicle was able to move laterally large distances (20 cm) relative to its altitude (10 cm) without crashing, despite large conformational changes in the tether.
Difficulty of vehicle fabrication and limited lifetime constrained the number of flights to just a small number in this work. Each vehicle takes days of full-time labour to fabricate , and because of manufacturing or human error, some do not have the tolerances necessary to attain stable free-flight. Mechanical failure of the flexure joints in the wings limits flight times to minutes, after which the vehicle must be discarded. Although this is expected to improve through future technological developments, the results reported here are therefore concerned primarily with demonstrating feasibility.
3. Attitude stabilization using velocity feedback
We show that knowledge of absolute vehicle attitude is not required to attain stability. Instead, only angular velocity feedback or ‘rate damping’ is needed. More formally, the torque controller 3.1where ω is the angular velocity, is sufficient to stabilize the fly in the upright orientation.
This result holds under the following assumptions:
(1) Vehicle motions depend only on stroke-averaged forces, that is, forces and torques averaged over the time period of each wing stroke.
(2) Aerodynamic drag on the wings is proportional to airspeed in both the forward (x) and lateral (y) directions, with an equal proportionality constant for both directions.
(3) The vehicle is symmetric about its x–z plane.
3.1. Rigid body dynamics
We use the following quantities, all expressed in body-frame coordinates: is the velocity of the CM; is its angular velocity about the CM; is a torque applied to the body; are external forces acting on the CM; m is the mass; and is the matrix of the moment of inertia.
We use two different representations for the attitude: rotation matrices and Euler angles. Using Euler angles, the attitude is represented by an array of three angles . The attitude is obtained by first rotating by an angle θ3 (yaw) around the body z-axis, then by θ2 (pitch) around the new body y-axis, and then by θ1 (roll) around the new body x-axis. This representation has singularities at extreme attitudes but is convenient to prove local stability around a certain attitude.
Using this parametrization, the dynamics of the rigid body in body-attached coordinates are 3.2 3.3 3.4The quantity W(θ) is a matrix that relates the angular velocity ω to the rate of change in Euler angles. Rewriting equation (3.2) in terms of coordinates gives 3.5
We also represent the attitude as a rotation matrix . For any vector given in world coordinates, , where is the same vector expressed in body coordinates (figure 3). Using this representation, equation (3.2) is 3.6where is the skew-symmetric matrix that represents the cross-product of a vector with ω, so that .
3.2. Analysis of the planar model
Assuming that the z-torque is negligible, we can reduce the analysis of the six degrees-of-freedom (d.f.) system (equation (3.4)) to two independent planar systems. We first provide the planar analysis for the sake of simplicity. The avid reader can proceed to the subsequent section for the proof for full rigid body motion.
Consider a state defined by a scalar pitch angle θ, a scalar angular velocity , and scalar torque τ (figure 3). We show that the controller, 3.7that applies a torque proportional to the angular velocity stabilizes the fly in the upright orientation.
A key element of the model is how the aerodynamic drag on the wing acts on the rotational dynamics of the robotic fly. As the rotational dynamics are slow relative to the frequency of flapping at this scale , our analysis considers only stroke-averaged forces. A test of this vehicle flapping in a wind tunnel indicated that the stroke-averaged drag force on the wings is nearly linear with the incident airspeed for typical wing kinematics (figure 3). This is the case for wind in both the x- and y-directions. Accordingly, our model for aerodynamic drag in both cases is where vw is the lateral velocity of the point on the airframe at the midpoint between the two wings. If the vehicle is rotating at angular velocity ω, then the velocity of the wings, when linearized around θ = 0, is , where rw is the distance from the midpoint of the wings to the CM. Similarly, the force arising from aerodynamic drag is and the torque about the CM owing to this force is . With lift force owing to the flapping wings fl approximately balancing out the weight mg, the lateral force owing to the inclined gravity vector relative to the body frame is equal to for small θ. A simplified model of the wire tether found using a sensitive torque sensor suggests it could be incorporated into the model as a spring with constant ks (electronic supplementary material, figure S1), according to τs = −ksθ.
Next, we compute the dynamics under the influence of the torque controller in equation (3.7). In the planar motion case, we can neglect second-order cross-product terms in equation (3.4) and equate forces and torques to velocities according to and . The linearized equations of motion about zero pitch angle θ can be written as a state-space dynamical system with the state vector expressed in body coordinates, where 3.8If desired, the effect of the wire tether can be incorporated by setting A21 = ks/J, but we neglect its effects in the following analysis.
The dynamics are asymptotically stable if the eigenvalues of A have negative real part. The Routh–Hurwitz criterion can be used to determine the stability of this system . This allows us to determine the sign of the eigenvalues by looking at the characteristic equation , which gives a polynomial of the form . The dynamics are stable if and only if all ak > 0 and a2a1 > a3a0. If kd > 0, then all of the ak > 0, so the stability criterion reduces to a quadratic polynomial of the form , which has two solution domains. The negative solution for kd is not stable, because some ak are negative. The positive solution domain is where the bound is given by 3.9Therefore, we have found a lower bound on the gain for the controller in equation (3.7) that is necessary to achieve asymptotic stability at .
The rotational dynamics of the robotic fly given here can be compared with a pendulum, but are qualitatively different because of how lateral and rotary motion are coupled. In the course of flying, if the body is inclined, then the thrust vector takes on a lateral component, accelerating the vehicle laterally. As lateral velocity increases, drag from the wings increases. The lateral drag acts at a location above the CM to exert a torque that acts to right the vehicle (drag on the body is negligible). In our design, this torque is large enough that it causes the vehicle to swing with a larger amplitude in the opposite direction, leading to a growing oscillation that results in tumbling. The velocity-dependent action of the controller in equation (3.7) suppresses this through a damping action .
To compute the necessary minimum gain, we estimated parameters of the robotic fly (table 1). Mass was measured using a precision scale, and the wing drag factor was measured in a wind tunnel (figure 3). Moments of inertia were estimated using a detailed model in computer-aided design software. The quantity rw was estimated by measuring the distance from the leading edge of the wings to the approximate point at which the airframe of the robotic fly balanced on a sharp edge using a ruler under a microscope. Evaluating equation (3.9) using these parameters gives for both xz- and yz-dynamics. Experimental results (described later) confirm that this value is near the limiting value between stability and instability, validating our mathematical results and modelling assumptions.
3.3. Analysis of the full six degrees-of-freedom rigid body model
3.3.1. Forces and torques
We assume that aerodynamic drag, in addition to being proportional to velocity in the x- and y-directions (figure 3 and ), is also proportional to velocity in the z-direction, although this has not yet been tested. In vector form, the drag force is thus where is defined as the velocity of the point midway between the two wings in world coordinates. If the body is rotating with angular velocity ω, then the velocity at that point in world coordinates is Vw = Rv + (Rω) × (Rrw), where is the point where the two wings attach to the airframe and is approximated as being directly above the CM. Using that (Rω) × (Rrw) = R(ω × rw), and that RTR = I, where I is the identity matrix, we have that Gravity applies a force directly downwards through the CM and is given by in body coordinates. The lift force from the wings acts with a magnitude roughly equal to the force of gravity, fl = [0, 0, mg]T and in line with the CM. Thus, we have that the total external force applied to the vehicle, expressed in body coordinates, is
The torque applied by drag on the wings is and the control torque is τc = −kdω (equation (3.1)). Thus, the entire torque is
3.3.2. Linearized analysis around hover
We use a linearized analysis to assess stability of the hover configurations (ω = 0; v = 0; θ1 = 0, θ2 = 0, and any value for θ3). We use the local coordinates and derive the linearized dynamics valid around the point .
Consider the rigid body dynamics described by equations (3.2)–(3.4). Taking the derivative, the second-order cross-product terms ( and ) disappear. To fill out the matrix A, we compute the linearized dynamics and obtain that Similarly, we obtain , and . Because our vehicle is symmetrical, its moment of inertia is well approximated by Assembling the matrix A, we obtain
We can verify that for any (equation (3.9)) A has all eigenvalues with negative real parts, except for a zero eigenvalue corresponding to free state θ3, the heading angle of the vehicle, whose value has no effect on the dynamics. Because it is sufficient to show that the linearized dynamics are stable to prove that the full nonlinear system is stable in a neighbourhood of that equilibrium , we have proved that the angular velocity feedback control law in equation (3.1) stabilizes the robotic fly in the upright hovering position, provided that , with given by the lower bound in equation (3.9).
An inspection of the rows of A shows that the two planar systems and have no coupling, so that each planar system can be analysed independently. This allows for the analysis considered above. We must emphasize that this only applies for a linearization around ω3 = 0. However, if manufacturing asymmetry generates a small but constant disturbance torque around the z-axis (this is commonly the case), then the flapping wings generate a counter-torque roughly in proportion to the angular velocity ω3 , so that ω3 does not grow without bound. Therefore, the fly will stabilize in the upright orientation, and rotation around the vertical axis will remain small.
A similar analysis suggests that stability could also be achieved through a mechanical design change that reduces rw, the distance from the wings to the CM, assuming the wings themselves provide a small amount of rotational damping. For example, if the wings impart a torque τ = −cω with , then passive stability is achieved with 0 < rw < 0.7 mm.
4. Estimating angular velocity using the ocelli
Here, we show how to use the ocelli output to produce an estimate of the angular velocity for use in a feedback controller (equation (3.1)) to stabilize the robotic fly. We call each photoreceptor an ‘ocellus’ to highlight the biological inspiration. We will prove the following result: there exists a matrix L such that the least-squares estimate of the angular velocity is , where is the vector of the time derivatives of each ocellus signal. The result holds under the following assumptions:
(1) The sensitivity of each ocellus is circularly symmetric. This is a reasonable approximation for the phototransistors in our sensor prototype and fly ocelli .
(2) There is only one static circularly symmetric luminance source in the environment. This is typically the case outdoors: the sun is a localized luminance source; a cloudy sky is a very diffuse luminance source taking on a hemisphere.
(3) This light source is far away. With this assumption, translational motion of the vehicle does not induce a significant change in the direction of the light source relative to the vehicle.
(4) There are at least three ocelli, arranged in a non-singular configuration.
Previous work in the literature has considered the case in which the ocelli are configured in opposing pairs [45,49]. Here, we give a geometrical treatment that provides for an arbitrary number of ocelli in arbitrary directions, permitting either greater feedback precision or a reduced sensor mass as necessary.
4.1. Ocelli sensor model
Let the sensor be composed by n defocused photoreceptors. The output of the sensor is an array in which each component is the voltage signal from a single ocellus, all of which have identical response properties. This voltage is proportional to the luminance in the environment averaged over the visual sphere in the directions to which the ocellus is sensitive.
Let be the environment ‘map’, so that m(s) is the luminance in direction on the unit sphere. We assume this is unchanging according to assumption 3.
Let be the principal direction of the ith ocellus in body coordinates. If the current sensor attitude is , then the ocellus is pointing in the direction Using the assumptions outlined above, it can be shown1 that the output of the ith ocellus at time t is a function only of the angle between the centre of the light source d, 4.1
This assumption has been verified, in practice, for many lighting conditions when using radially symmetric and defocused light sensors such as those used in this work (assumption 2) .
4.2. The derivative of the ocellus signal is a linear function of angular velocity
Given the sensor model in equation (4.1), we can obtain an estimate of the angular velocity ω. We start by taking the derivative with respect to time of the sensor output. We assume the visual scene is static according to assumption 3 so that . We obtain
The last equality comes from the fact that is linear in Rt so .
We can linearize around the reference point R = I to obtain 4.2
This expression is a linear function of ωt, so we can find a matrix such that . Using the standard vector triple product and dot product identities, we obtain and rewrite equation (4.2) as 4.3
Therefore, we have concluded that the derivative of the ocellus signal is related linearly to the angular velocity.
4.3. Least-squares estimation
We have obtained the expression in equation (4.3) for the derivative of one ocellus signal. If we write it together for all ocelli, then we obtain that the derivative of the vector depends linearly on the angular velocity ωt through an n × 3 matrix M, 4.4
The least-squares estimate of ωt, which assumes equal noise on all measurements, can be written as 4.5
This estimate has a number of restrictions. First, the component of ωt parallel to d cannot be estimated because it is in the null space of M. This is because each row of M is the result of a cross-product with d (scaled by κ′), so each is orthogonal to d. Together, these rows constitute a plane that is the kernel of this matrix, and the pseudo-inverse provides only the projection of ωt onto this plane. This subspace is further restricted to a line if d and all si are coplanar, because, in that case, all rows of M are linear multiples of each other. Second, the row Mi is zero whenever d = ±si, because, at these points, d × si is zero. It is also zero when κ′ is zero for any ocellus, such as when the light source is entirely outside of its field of view, so that angle changes have no effect on its response. Third, the magnitude of the estimate varies linearly with κ′ and thus with the brightness of the light source function κ.
This formulation allows for an arbitrary arrangement of an arbitrary number of ocelli. As few as two can estimate the two essential components of to provide stability if the luminance function κ is known. If it is not, then redundant sensors can be used to estimate κ .
4.4. Specifics of the four-ocellus design
In our design, the directions of the four ocelli are closely approximated by and . This arrangement has the following characteristics:
(1) The ocelli are arranged in pairs that are redundant, so that the overall illumination level κ could be estimated and divided out, as in .
(2) They are tilted in a slight upwards direction, so that a light source above the vehicle will be detected by all of them in the course of normal manoeuvring so that . A tilt angle of approximately the same value, 40°, was found to be optimal for estimating angular rate in .
(3) None is tilted so far upwards that it will be pointed directly at the light source in the course of normal manoeuvring so that .
In the case of the light source directly overhead the M matrix (4.4) is given by
In this case, the quantity ω3 (the z-component of ω) is not observable, because d is directly above, but for hovering it is sufficient to leave it uncontrolled, so long as it remains small. For the other two components, the matrix pseudo-inverse has the form 4.6where is a constant. A new output defined by subtracting opposing pairs of ocelli according to  recovers a simplified relation similar to that given elsewhere for a pair of ocelli , 4.7If the light is not directly overhead, but at some unknown angle, then the quantity k may be different for that orientation. However, by combining equations (4.7) and (3.1), it can be seen that as long as it can be ensured, by making kd sufficiently large, that , where k = k0 when the light source is directly overhead, then the control law given by (3.1) is asymptotically stable.
5. Flight tests
5.1. Flight arena
We performed flights tests in a motion capture arena with an array of calibrated cameras (T040-series; Vicon, Oxford, UK). Each camera emits bright infrared that is reflected from a number of retroreflective markers mounted on the vehicle, so that its position and orientation can be reconstructed in real-time for later analysis. Rotation about yaw (z) was left uncontrolled in these experiments. In all flights, computations to generate signals for the piezoelectric actuators to drive wing motion, as well as to map desired control torques to these signals, were performed on an XPC Target, a desktop computer running a real-time operating system (MathWorks, Natick, MA, USA). Analogue voltage outputs from this computer were amplified by high-voltage amplifiers and transmitted to the robotic fly through the wire tether. The compliant wire tether has a small effect on vehicle dynamics, as detailed in §2.2.
In flights controlled by feedback from motion capture, position and orientation estimates were sent by serial cable to the control computer. In these tests, the estimate was calculated by smoothing the motion capture estimates with a third-order Butterworth filter with a 40 Hz cut-off frequency, taking the derivative, and multiplying by the matrix W (equation (3.5)).
For flights in which feedback from the ocelli was used, illumination was provided by a 150 W halogen bulb at 66% brightness mounted 1.5 m above the vehicle (Dolan-Jenner MI-150 fibre-optic illuminator; Edmund Optics, Barrington, NJ, USA). The control computer measured voltages from the four phototransistors using an analogue-to-digital data acquisition board (National Instruments). The infrared flashes from the motion capture cameras, at 500 Hz with a one-eighth duty cycle, were so bright that they saturated the phototransistors, regardless of orientation because of reflections from the ground surface. Accordingly, they did not provide any orientation information. After this initial transient, the voltage decayed to an attitude-dependent steady-state value (figure 4). A digital filter recovered these steady-state values by filtering with a third-order Butterworth filter with a 20 Hz cut-off frequency to smooth out the sudden flashes, as well as the 60 Hz AC ripple from the light source (figure 4). To calculate the estimate of k in equation (4.7), we rotated the robot by 45° in either direction and measured the resulting quantity y4 − y2 to compute the slope.
Experiments provide empirical evidence that the control law given by equation (3.1) gives stability. In flights without the ocelli attached in which the estimate was provided by motion capture using a gain of kd = 2 × 10−7, well into the region of stability, the robotic fly remained upright during the course of the flight before drifting out of the tracking volume of the motion capture arena. In these flights, only the rate damping feedback law was being tested: neither altitude nor lateral position was under feedback control so small asymmetries in flapping kinematics generated non-zero lateral thrust.
To further test the validity of the model given by equation (3.8), we observed the natural dynamics of the robotic fly in flight with a gain of kd = 1.0 × 10−7, just at the threshold of stability. In this flight, it exhibited sinusoidal oscillations similar to those of a simulation of the model in equation (3.8) (figure 5). When tested with gains of 0.75 × 10−7 or lower, the vehicle did not remain upright. This shows that the model and parameters given in table 1 are close to their true values and that un-modelled effects do not have a large impact on vehicle dynamics.
A demonstration that the robotic fly can be stabilized exclusively using feedback from the ocelli is shown in figure 6 and in the electronic supplementary material, video S2. In this experiment, the ocelli sensor was attached to the vehicle and provided the estimate . The added mass and inertia of the ocelli increased the minimum gain to . Using a gain of kd = 2 × 10−7 as above, the fly successfully remained upright during a 0.3 s climbing phase (approx. 40 wingstrokes) before the wiring reached its limit. The vehicle remained upright, but performed rotational oscillations about pitch and roll, the result of aggressive corrective manoeuvres. Without this feedback, the vehicle quickly tumbled (figure 6, inset). The oscillating instability in the ocelli-stabilized flight was likely to be due to the nonlinearity of torques exceeding the pre-programmed actuator limit, owing to an overestimate of near the upright orientation. In addition, if at take-off the vehicle was inclined at 30°, then this fell outside the range in which the estimate of was valid, causing tumbling. The maximum inclination during this flight was approximately 10°, suggesting that torques owing to the torsional stiffness of the wire tether (§3.2; approx. 0.16 µNm) should have a small effect relative to control torques.
To mitigate the effect of variable gain and consequent actuator saturation, and to extend the range of permissible inclination angles, we performed third-order least-squares fit of θ2 measured by motion capture to yb across a range of ±60° using the Matlab command polyfit, using this fit for both axes. During flight, evaluating this polynomial requires minimal additional computation. With this calibration, the region of operation expanded from roughly ±30° to ±60° (figure 4). A motion capture estimate is not necessary, in principle, to perform this calibration; it could easily be performed using a potentiometer, as in . In flights with this calibration, the orientation was more stable and the estimate matched the motion capture estimate more closely (figures 1 and 7). Although this calibration improved performance for only a specific lighting condition, it is possible this could be extended to improve performance in more general conditions. This could be achieved by performing a least-squares fit using data taken from a range of expected lighting scenarios. Our model of the wire tether indicates it could have imparted torques as high as 0.24 µNm when the vehicle reached an 18° inclination, about half the magnitude of the control torques in this trial.
To further demonstrate the utility of the proposed control law, we performed an additional flight in which this calibrated ocelli feedback was combined with the aerodynamic ‘ground effect’ to maintain altitude. The ‘ground effect’ is an increase in aerodynamic lift that occurs in close proximity to a horizontal surface . In this flight, both upright orientation and altitude were maintained for a prolonged period of nearly a half of a second, or 50 wing strokes (figure 8), limited only by tether length. This result suggests that, if not for the tether length limit, ocelli feedback alone could enable flights in proximity to the ground indefinitely. In practice, this would be limited by battery life and the presence of obstacles.
In this work, we showed how an ocelli-inspired vision sensor can be used to stabilize the upright orientation of a fly-sized flying robot. Although a number of sensors have been proposed that could, in principle, be integrated into such a vehicle [44,53,54], few have been operated on a fly-sized robot , and none have previously performed in free-flight. Hence, these results constitute the first demonstration of onboard sensors in free-flight at this scale. We remark that even though the vehicle uses a wire tether to provide control commands and returns feedback, it does not augment stability (see §2.2). Power autonomy is explored in separate, but parallel efforts [59,60]. Our key contribution is a demonstration that angular velocity feedback from the ocelli sensor can stabilize the pendulum-like dynamics of our vehicle. By responding only to the angular velocity of the light source rather than its position, the ocelli-output feedback controller proposed here can stabilize the vehicle even if the light source is not directly overhead.
The operating range for the light source of ±60° from vertical could be extended by choosing sensors that integrate luminance over a larger field of view and orienting them closer to the horizon, or by ‘triangulating’ the direction d of the light source. The analysis given by equations (4.1)–(4.5) shows how the number of sensors could be reduced to three to reduce weight, thereby mimicking the number found on insects. The computational requirements for estimating angular velocity on our robot were minimal and would represent a negligible component of a flight controller. Processing required approximately 1 million floating-point operations per second at 10 KHz for filtering and optional polynomial evaluation, but this was over-engineered to ensure fidelity. Simulations indicate that these requirements could be dramatically reduced without sacrificing performance by reducing filter order and update rate.
It may be possible for the ocelli to estimate and control motion around the body z-axis as well. One possible approach is to use approaches inspired by optic flow such as autocorrelation , which requires a nonlinearity , in contrast to present analysis which is linear in light input. Ambulatory flies use their ocelli to orient towards a light source , suggesting they may incorporate such a control strategy. In a higher-level flight scenario in which other states such as lateral position  are controlled, the high bandwidth and few inputs of the ocelli are well suited to a fast inner loop to stabilize attitude. For example, in , motion capture provides an estimate of ω used for stability; this could be replaced by the estimate derived from the ocelli sensor.
The results presented here also support a hypothesis for how the ocelli could function to stabilize many flying insects. Fluid mechanics models based on quasi-steady approximations have suggested that many flying insects that hover, including moths, locusts, beetles, flies and honeybees, also have an instability similar to our vehicle [28–31,64,65]. Like our vehicle, the basic structure of these animals consists of a pair of wings (or two pairs that are joined) and a body that hangs below them . In the case of the fruit fly Drosophila melanogaster, it has been suggested that they could achieve stability using feedback from the halteres [30,31], which measure angular velocity . This has extended to pitch stability in other species in . In the aforementioned studies [28–31,64,65], however, conclusions are based on simulations that approximate the unsteady flow phenomenon generated by flapping-wing flight with a ‘quasi-steady’ model . This could neglect important effects. It is therefore desirable to perform a more detailed characterization under the conditions of coupled translational and rotational motion that constitute this instability. To replicate this motion in a dynamically scaled fluid mechanics model would require a six degrees-of-freedom articulated device under a ‘captive trajectory’ to simulate its inertia in a large tow tank . This has not yet been achieved. Using a computer instead to perform computational fluid dynamics simulations would require a large fluid volume that could require a prohibitive amount of computation power [67–69]. Another approach is to carefully manipulate sensory feedback available to alert, behaving animals [70,71]. However, precisely controlling sensory input during flight is difficult because of the small size of the animal. In addition, multiple modes of sensory feedback are typically required to stay aloft , making it difficult to isolate the effects of a single sensory organ. Hence, our robotic fly constitutes a new method to probe insect fluid mechanics that avoids these difficulties.
Our results, particularly those in figure 5, support the view proposed in [30–32] and others that angular velocity feedback or ‘rate damping’ is sufficient for upright stability in Drosophila and similarly shaped species. While the robotic fly and these vehicles are not identical, the approximations made in the model given in §3 apply equally to both the robot and these animals. One difference, however, is that in these animals the body hangs diagonally [32,51] rather vertically as in our vehicle. In both cases, however, the CM is centred below the wings. For these animals, the inertia matrix has off-diagonal xz terms that couple torques in the x-direction to motions about the z-axis and vice versa, but it can be shown that this does not alter the stability of the system. To provide a particular example, we consider the honeybee. In table 2, we give the equivalent parameters for the honeybee Apis mellifera, which has similar size and weight to our vehicle. These were derived from measurements and calculations in [32,51]. Drag on the wings is higher probably because of the higher typical flapping frequency of bees (197 Hz) compared with our vehicle (120 Hz), but the moment arm rw is much shorter. In the case of the honeybee, the model suggests that the minimum necessary damping gain to achieve upright stability is similar to that of our robotic fly (table 2).
In insects, studies have suggested that the ocelli mediate a fast, derivative-like reflex that acts in corrective flight manoeuvres. Neural recordings of the large L-neurons in the locust show a phasic, rate-dependent response , and in flies the ocelli mediate a head rotation reflex that turns it towards sudden changes in light direction . This reaction happens with significantly less time delay than the compound eyes . Hence, the direction, speed of response and rate dependence of the ocelli reflex are consistent with a rate-damping flight stabilizer. Studies have shown that honeybees can nevertheless fly with their ocelli occluded by opaque paint, but flight precision and direction is significantly disrupted . Recent results in the blowfly Calliphora indicate that the eyes and ocelli act in concert, with the ocelli providing a faster but more crude estimation of rotation, whereas the eyes respond with a more precise estimate at a slightly later time . Hence, while, theoretically, the ocelli can provide the necessary damping for these animals to stay upright, in practice, the reflex may be mediated by a superposition of feedback from both the ocelli and the compound eyes. It is not currently known how the honeybee, which does not have the gyroscopic halteres that flies use to estimate their angular velocity, stays upright . Our results are therefore consistent with a view that they could stay upright using an angular velocity reflex mediated in part by feedback from the ocelli that is functionally equivalent to the controller described here.
This work was partially supported by the National Science Foundation (award nos. CCF-0926148 and CMMI-0746638) and the Wyss Institute for Biologically Inspired Engineering.
The authors thank Alexander Sands for assistance with sensor fabrication and characterization. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
↵1 More formally, if the luminance in direction s is m(s), then the output of the ith ocellus at time t is the average of this function over the visual sphere according to a sensitivity kernel. We assume the phototransistors give an output that is equal to input luminance power, but, in practice, the mapping between power input and voltage must be calibrated and inverted. Using assumption 1, let be the circularly symmetric kernel describing the luminance sensitivity of an ocellus, so that is the sensitivity to the luminance for the incidence angle α away from its direction . Using that arccos is equal to the angle between two unit vectors s and si, the output of the ocellus is obtained by averaging the environment luminance m(s) over the visual sphere and weighting using the kernel ,
This can be rewritten compactly as a circular convolution:
Assumption 2 states that there is only one circularly symmetric environment luminance source. Suppose that the luminance source is centred at direction . Using this assumption, the luminance at direction s only depends on the angle between d and s through some kernel : Using the circular convolution notation, m is the convolution of the impulse at direction s, indicated as δs, and the kernel : , the output of an ocellus is Using the properties of circular convolution, we can conclude that depends only on the angle between d and , giving the result in equation (4.1). In practice, the extensive image blurring caused by very defocused luminance sensors such as ours causes equation (4.1) to hold, even in environments with non-symmetric lighting such as outdoors beside a large building .
- Received March 17, 2014.
- Accepted May 20, 2014.
- © 2014 The Author(s) Published by the Royal Society. All rights reserved.