## Abstract

Natural selection favours phenotypes that match prevailing ecological conditions. A rapid process of adaptation is therefore required in changing environments. Maternal effects can facilitate such responses, but it is currently poorly understood under which circumstances maternal effects may accelerate or slow down the rate of phenotypic evolution. Here, we use a quantitative genetic model, including phenotypic plasticity and maternal effects, to suggest that the relationship between fitness and phenotypic variance plays an important role. Intuitive expectations that positive maternal effects are beneficial are supported following an extreme environmental shift, but, if too strong, that shift can also generate oscillatory dynamics that overshoot the optimal phenotype. In a stable environment, negative maternal effects that slow phenotypic evolution actually minimize variance around the optimum phenotype and thus maximize population mean fitness.

## 1. Introduction

Evolutionary mechanisms that enable individuals to adjust rapidly to novel environmental conditions are ubiquitously considered advantageous [1–3]. Phenotypic plasticity and maternal effects are two of many biological pathways that influence an individual's phenotype [1,4–7], change an individual's fitness [5,6,8] and facilitate adaptation to novel environments [5,7]. Less is known about how they interact to shape phenotypic evolution, however.

Maternal effects are the most commonly studied transgenerational effects [9] and they provide a flexible way of maximizing fitness in a changing environment [10]. Maternal effects have been defined as the effect of the maternal phenotype on offspring phenotype [11], owing to environmentally induced effects on maternal phenotype or to genetic variation in maternal phenotypes [12]. Kirkpatrick & Lande [13] defined ‘maternal inheritance’ as the particular impact of the maternal phenotype on the offspring phenotype independent of the inherited genes. There is much evidence that this non-genetic path is beneficial [6,9,12,14], but empirical studies also report results from statistical analyses that show it can slow phenotypic evolution [12]: maternal effects can be positive or negative. A positive maternal effect coefficient indicates accelerated rates of microevolution that can facilitate adaptation, whereas a negative maternal effect coefficient suggests that maternal effects slow (or even reverse) any response to selection in the offspring generation. A positive maternal effect coefficient means that (all other inheritance mechanisms being equal) larger mothers produce larger offspring, as has been reported in Darwin's finches and great tits *Parus major* [15]. A negative maternal effect coefficient generates fluctuating patterns of selection: large mothers produce small offspring, who in turn produce large offspring, and so on. A negative maternal effect coefficient therefore can reverse phenotypic evolution from one generation to the next. Empirical examples of a negative maternal effect coefficient include clutch size in collared flycatchers *Ficedula albicollis* [16], litter size in mice [17], age at maturity in springtails *Orchesella cincta* [18] or rosette size in the monocarpic herb *Campanulastrum americanum* [19]. In red squirrels *Tamiasciurus hudsonicus*, three estimates from different statistical and experimental approaches were remarkably congruent in their estimation of a negative maternal effect coefficient [20–22]. These studies show a negative maternal effect because inheritance via this non-genetic mechanism acts in the opposite direction to that of strict Mendelian inheritance. If a rapid response to environmental change is a critical coping mechanism in evolutionary biology, then why would these empirical estimates appear to suggest that maternal effects often act to slow adaptation to a changing environment?

To understand when maternal effects become more influential in determining the phenotype, we need to understand the consequences of the predictability of the environment between the point at which an environmental cue is processed and the point at which selection acts. The developmental lag before a juvenile reaches maturity is influenced, in part, by environmental conditions, but also by other biotic factors such as the presence and type of predators [23]. This juvenile development lag may therefore operate on a different timescale from any environmental stochasticity, and we explicitly decouple them in our model in order to capture their contributions to the phenotype under selection more accurately. Environmental stochasticity in ecological scenarios is frequently positively autocorrelated [24–26], although negative autocorrelation may be becoming more common [27]. If environmental stochasticity is positively autocorrelated, then deviations from mean conditions at successive times are likely to be in the same direction (e.g. hotter than average years typically follow hotter than average years), whereas if it is negatively autocorrelated they are likely to be in opposite directions (e.g. colder than average years typically follow hotter than average years). There is evidence from theoretical [28], laboratory [29] and empirical [27] studies that the predictability of environmental change propagates through to population mean fitness. Non-genetic inheritance is most likely to be beneficial when the parental phenotype contains useful information about the environment that is likely to be experienced by the offspring [30], i.e. if environmental change is predictable. Our focus is on comparing the impact of maternal effects on phenotypic evolution in novel and in stable environments. Experiments suggest that maternal effects affect adult traits most in benign environments [31], but ecological stimuli such as heat stress [32] or presence of predators [33] can provoke large maternal effects. In a random environment, Jablonka *et al.* [34] showed that transgenerational effects delivered higher fitness than either a plastic only or genetic only strategy; less is known about the benefits of transgenerational effects when environmental change is autocorrelated, however.

How do non-genetic inheritance and phenotypic plasticity interact to deliver the optimal phenotype? Here, we extend Lande's quantitative genetic framework for the evolution of phenotypic plasticity [8] to incorporate non-genetic inheritance via the maternal effect coefficient. We show how the optimal level of the maternal effect coefficient to maximize fitness depends on the extent of environmental shift and the lag between juvenile development and selection, but is consistently negative or zero for background levels of environmental change.

## 2. A quantitative genetic model of adaptation with fixed maternal effects

We start with the reaction norm approach of Lande [8] and extend it to include *m* as a fixed strength maternal effect coefficient [12,35] to represent maternal inheritance [13,36]. Furthermore, our extensions include decoupling environmental autocorrelation from juvenile development, and calculating expectations for population mean fitness and phenotypic variance. Our modified reaction norm is
where as in Lande [8], *z*_{t} is the adult phenotype of an individual subject to selection in generation *t*; *ɛ*_{t} is the environment at time *t*; *τ* is the lag between a critical period of juvenile development and the time when the adult is subject to selection; *a*_{t} gives the additive genetic effect in the reference environment *ɛ* = 0; *b*_{t} describes the plastic phenotypic response to the environment; is the phenotype of the parent after selection in generation *t* − 1; and *e*_{t} is the residual component of phenotypic variation, which is assumed to be normally distributed with a constant population mean of zero and variance *σ*_{e}^{2}. We are considering a sexual population, where mating is at random, and where generations are discrete and non-overlapping.

Averaging over the population distribution, for a given environment *ɛ*_{t−τ}, gives
2.1where the overbar denotes population mean.

The phenotypic variance, , of *z*_{t} is
2.2where *G*_{aa}, *G*_{bb} and *G*_{ab} are the variances of *a*_{t} and *b*_{t} and the covariance of *a*_{t} and *b*_{t} respectively, which we assume to be constant. , and are the covariances and variance of in the obvious way. The covariances and of *z*_{t} with *a*_{t} and *b*_{t} satisfy
2.3and
2.4

At equilibrium, in a constant reference environment *ε*, we have . Because offspring share on average half their genes with one parent, then in the case of weak selection, we also have and at equilibrium [13,37]. Hence, we can deduce
2.5

As in Lande [8], we assume that the reference environment, *ɛ* = 0, minimizes the phenotypic variance. The minimum phenotypic variance is achieved at *ε* = −*G*_{ab}/*G*_{bb} and so we must have *G*_{ab} = 0. Furthermore, the optimum phenotype, *θ*_{t}, is assumed to be a linear function of the environment at time *t*, and fitness, *W*, to be Gaussian:
and
where *A*, *B*, *W*_{max} and *ω* are constants. If *z*_{t} is normally distributed with variance *σ*_{zt}^{2}, then, as in Lande [8], we can average over the phenotype distribution *p*(*z*_{t}) to find the mean fitness
2.6
2.7where *γ* = 1/(*ω*^{2} + *σ*_{zt}^{2}).

Assuming that the additive genetic component (*a*_{t}) and plasticity (*b*_{t}) are bivariate normally distributed, the per generation change in their population means, and , is [38]:
where
and where we have used *G*_{ab} = *G*_{ba} = 0 to ensure that the phenotypic variance is minimized in the reference environment *ε* = 0 [8].

Equation (2.2) indicates that *σ*_{zt}^{2} does not depend directly on or and in this case, we have
2.8where we have used that the average phenotype after selection in the previous generation is given by
2.9

Thus, we have
2.10
2.11
2.12where equation (2.12) gives the mean value of the phenotype in generation *t* after selection.

Note that the value of *A* can be set to zero by the linear transformation {*θ* → *θ* − *A*, *z* → *z* − *A*, *a* → *a* − (1 − *m*)*A*}, which otherwise leaves the system unchanged, and that the transformed system is then invariant under the reflection {*ε* → −*ε*, *z* → −*z*, *a* → −*a*}. Thus, beyond these shifts of mean and changes of sign, there is no qualitative effect of the sign of environmental fluctuations on the behaviour of the system. In particular, whether a positive or negative value of the maternal effect coefficient, *m*, benefits fitness in a given environment is independent of whether that environment is shifted positively or negatively from the reference *ε* = 0.

### 2.1. Adaptation following an extreme environmental shift

To see the impact of maternal effects on the model of phenotypic plasticity when there is a sudden environmental shift, we will use the same environmental conditions as Lande [8], namely a noisy step change , where *U*_{t} is the unit step function that jumps from 0 to 1 at *t* = 0, *δ* is the size of the sudden change in average environment and *ξ*_{t} is a Gaussian stationary autocorrelated random process with mean zero, variance *σ*_{ξ}^{2} and autocorrelation *ρ*_{τ} over the interval *τ*. We let each time step equal one generation, so that the time lag *τ* is measured in fractions of a generation.

We now substitute for *ɛ*_{t} in equations (2.10)–(2.12) and take the mean over the distribution of environments. We therefore obtain the expected changes, and in and , respectively, and the expected value of the phenotype after selection, . We assume that the environment is uncorrelated over timescales of a generation or longer. We also regard , and as fixed when we average over the distribution of environments. This is equivalent to neglecting terms of that arise in and as a result of the dependence of , and on and earlier environmental states. This is a good approximation when *γ**G*_{aa} and *γ**G*_{bb} are small. The explicit dependence of on in equation (2.9) is retained. We therefore find:
2.13
2.14and
2.15where we have used the fact that third-order moments of stationary Gaussian processes vanish. We have also approximated *γ* by because *γ* depends on *σ*_{zt}^{2} and hence on *ε*_{t−τ} in a nonlinear manner. We use the expected phenotypic variance at equilibrium to approximate *E*(*σ*_{zt}^{2}) throughout the period of evolution: we use the expression calculated later in equation (3.1), but replace *δ* by *δ**U*_{t−τ}. We expect both these approximations to be good for .

Equilibrium solutions of equations (2.13)–(2.15) in a noisy equilibrium environment satisfy and . At leading order in *γ**G*_{aa} and *γ**G*_{bb}, this gives
and

The equilibrium state in the changed environment is thus found to be 2.16 2.17 2.18

Setting *δ* = 0 recovers the equilibrium state before the change.

Without maternal effects (i.e. *m* = 0), these results agree with those of Lande [8]. It is clear from this analysis that fixed maternal effects make no difference to the expected equilibrium phenotype, but that they reduce the expected equilibrium plasticity slightly and the expected equilibrium additive genetic effect to a greater extent, both before and after the change in environment. Plotting trajectories using equations (2.13)–(2.15) and equation (2.1) iteratively, starting from the equilibrium state in the original environment, shows how maternal effects change the dynamics (figure 1). In particular, the peak plasticity during the transient phase is lower with positive maternal effects than in their absence. Here, we approximate expected mean fitness as
which holds for . Again, we treat , and as fixed. For the parameters used in fig. 1 of Lande [8] and setting *m* = 0.45, it is clear that constant maternal effects speed up the adjustment to the new environment (figure 1). For *m* ∼ 0.48, however, we see oscillations during the transition to the new equilibrium (figure 1 and appendix C). In general, the onset of oscillatory behaviour depends on the maternal effect coefficient, *m* (appendix C); for the parameters of figure 1 and in the absence of environmental noise (*σ*_{ξ} = 0), this is at *m* = 0.48 (2 s.f.). Although we consider a positive environmental shift, *δ* > 0, the results would be the same for an equal and opposite negative shift as discussed at the end of the last subsection, except that the change in the phenotype, *z*, and additive genetic component, *a*, would be in the opposite direction: in particular, positive *m* would still speed up adjustment. While maternal effects make only a slight impact on the expected equilibrium plasticity, they have a clear impact on the transient dynamics: for *m* > 0, the peak plasticity is lower than without maternal effects (*m* = 0), and the oscillations in the phenotypic dynamics are driven by oscillations in the plastic component (figure 1). These overshoots increase the mismatch between optimal and observed phenotype, and therefore provide a natural restriction on unbounded increases in the maternal effect coefficient. It seems unlikely, however, that overshoots of a new optimum phenotype within the range of environmental change typically experienced in an ancestral environment are sufficient to keep the maternal effect coefficient at the modestly negative levels often reported empirically (see §4). The question therefore remains: what is the optimal maternal effect coefficient in a stochastic environment? We hypothesize that negative maternal effect coefficients are, in fact, favoured in relatively stable stochastic environments.

## 3. Maternal inheritance in relatively stable stochastic environments

### 3.1. Numerical simulations

To test whether negative values of the maternal effect coefficient are favoured in relatively stable stochastic environments, we generated stationary sequences of autocorrelated environmental stochasticity for *ρ*_{τ} set to , , and . *τ* was fixed at 0.25 of a generation. In this case, we considered environmental sequences with no step change, so that . We examined behaviour both in a noisy reference environment (*δ* = 0) and away from this (*δ* = 10).

The evolutionary response to these stochastic environments was modelled numerically using equations (2.10)–(2.12) to update , and , in each generation, starting from the expected equilibrium values. In order to calculate *γ* at each step, we first worked out the phenotypic variance in equation (2.2) by updating equations (2.3) and (2.4) and assuming that and (appendix B); errors from this approximation will be small when .

We calculated mean fitness over 10^{6} generations, taking the arithmetic mean over subsequent generations of a single realization to approximate the mean over realizations for a single generation, assuming that the system is ergodic. We might also be interested in the mean fitness of a population over a number of generations; this is measured by the geometric mean across generations. Because the trajectories show small fluctuations about an equilibrium value, the arithmetic and geometric means across generations of the population mean fitness are in fact equal to leading order in the fluctuations:
for , where *w*_{t} is the fluctuation of the population mean fitness in generation *t* relative to its deterministic equilibrium value. Thus, to leading order our fitness calculations capture both the expected value of population mean fitness for a single generation and also the geometric mean of the population mean fitness over a number of generations.

For all values of *ρ*_{τ} considered, fitness was maximized for negative or zero *m* (figure 2). Absolute fitness depended on the environmental autocorrelation, i.e. the predictability of change, hence we report relative differences from the mean value. As the predictability of environmental change increased (i.e. as *ρ*_{τ} increased), the value of *m* where fitness was maximized moved closer to zero and the relative fitness costs (i.e. curvature) of not expressing the optimal level of maternal effects increased (figure 2). The optimal value of the maternal effect coefficient was more strongly negative with larger fitness costs in the *δ* = 10 environment than in the noisy reference environment when *δ* = 0. In the noisy reference environment, an absence of maternal effects (*m* = 0) maximized fitness if the environment was sufficiently predictable, namely for values of *ρ*_{τ} of and . There was no qualitative impact of changing the lag between juvenile development and selection, or of considering negative environmental autocorrelation. We can shed further light on the benefits of negative values of the maternal effect coefficient by considering the expected mean fitness of the population as a function of *m*.

### 3.2. Analytical refinement

First, we take expectations over the distribution of environments in equations (2.2)–(2.4) to get
where we have set *G*_{ab} = 0, as before, and assumed that the environmental stochasticity is uncorrelated over timescales of a generation or longer, so as to neglect any covariance between and in equation (2.2). Now, we look for an equilibrium solution. Under weak selection, we have , and at leading order in *γ* and independent of time, and deduce that the expected equilibrium phenotypic variance is given by
3.1

An equilibrium state can develop only if is positive and finite: this restricts the range of possible maternal effect coefficients, *m*, as described in appendix A. Empirical evidence suggests that *m* is typically small (see §§1 and 4), so, in practice, we shall restrict our attention to the range , where is indeed always positive and finite (figure 3 and appendix A).

In the absence of environmental stochasticity, we can set and use equation (3.1) to get an expression for the equilibrium phenotypic variance. 3.2

Turning now to the population mean fitness, we can rewrite equation (2.7) as 3.3

When there is no stochasticity in the environment, we have , so that for a given value of *m* there is no deviation from optimal fitness. However, that optimal fitness itself varies with *m* according to

Because is continuous in the region − 1 < *m* < 1 and tends to positive infinity as there, we deduce that it must take a local minimum value somewhere between *m* =−1 and *m* = 1. Taking the derivative of equation (3.2) with respect to *m* and setting it to zero shows that has turning points when
where . In fact for all positive *x*, there is a local minimum of , and correspondingly a local maximum of at in the region . As *x* → 0, we have (2 s.f.) and as , we find , with intermediate values of for intermediate values of *x*. Thus, populations with a modest negative value of the maternal effect coefficient are expected to be optimally fit. For the parameters used in figure 1, we have *x* = 1 in the reference environment (*δ* = 0) and so (to 2 s.f.), whereas *x* = 0.1 when *δ* = 10, and thus the optimal *m* is (to 2 s.f.) after the environment shift.

If the environment is stochastic, then the expected phenotypic variance increases according to equation (3.1) and we expect the mean fitness to drop. If the noise is small enough compared with the width of the fitness function () then from equation (3.3), we find and hence

Using equation (3.1) and calculating at equilibrium, we have 3.4

So, environmental noise does indeed reduce the expected population mean fitness. For low enough noise levels, this expression provides a small correction to the optimal value of *m* calculated for the purely deterministic environment, as is confirmed by the observation that the maximum fitness occurs close to *m* = −0.2 in the noisy reference environment and close to *m* =−0.4 in the noisy *δ* = 10 environment (figure 2). Analysing equation (3.4) in more detail, we see that greater autocorrelation of the environmental noise (*ρ*_{τ} closer to 1) and greater distance from the reference environment (larger *δ*) both tend to increase the fitness costs of expressing suboptimal *m*. Finally, the absence of plasticity (*G*_{bb} = 0) would tend to lessen these fitness costs. Once again, these results are independent of the direction of the environmental shift.

The relative fitness of populations with negative values of the maternal effect coefficient suggests that in relatively stable environments, the benefit of lower phenotypic variance from *m* < 0 (equation (3.4) and figure 2) outweighs other factors. One might ask why the variance is minimized at negative *m* rather than at *m* = 0. From equation (3.1), it is clear that this favouring of negative *m* comes from the inclusion of in the numerator of the first term on the right-hand side. This arises from the fact that covaries with *a*_{t} and *b*_{t}. In other words, reacting in the opposite way to the maternal phenotype compared with the inherited genotype in some sense uses the information in the mother's phenotype to discount the effect of her genes and remain closer, on average, to the optimum phenotype.

## 4. Discussion

We used quantitative genetic models to show how positive maternal effects can speed up adaptation following an extreme environmental shift (figure 1), but, if sufficiently strong, cause oscillations in the phenotypic dynamics, and therefore increase the mismatch between observed and optimal phenotype. In relatively stable environments, however, the relative fitness of populations with a negative maternal effect coefficient *m* suggests that the lower phenotypic variance achieved when *m* is negative (equation (3.4) and figure 2) is beneficial. This means that selection should favour a small negative effect of the maternal phenotype on offspring phenotype. Direct empirical estimates of a negative *m* relate either to the case where a single trait affects itself maternally [16–18], which is the situation we have modelled, or to the case where a given maternal trait affects a different offspring trait, for example, litter size affecting juvenile growth rate in red squirrels [22]. In the latter case, a negative maternal effect may result from a negative direct-maternal covariance, because the two quantities are related to one another: statistical decompositions can be used to estimate the strengths of interactions among phenotypic traits [39] given a model of direct and indirect genetic effects [40]. There is some overlap between the two categories, in that the negative maternal effect of a single trait upon itself may be mediated by another trait: for example, large maternal litter size in mice leads on average to offspring of smaller body size who in turn have smaller litters [41]. In relatively stable environments, the expected local fitness maximum in our model occurs at modestly negative values of the maternal effect coefficient. Thus, however it arises, a negative maternal effect of a single trait on itself can lead to increased fitness in our model.

Following an extreme environmental shift, there was a clear benefit of a positive maternal effect coefficient *m*. Increasing *m* within the region of monotonic convergence (i) lowers the peak of plasticity during the transient phase, (ii) accelerates the approach to this peak (figure 1), and (iii) slightly reduces the equilibrium level of plasticity (equation (2.17)). The equilibrium level of the additive genetic component is reduced if *m* > 0 (equation (2.16) and figure 1), both before and after the step change *δ*. This lower contribution of the additive genetic component to the phenotype is consistent with conclusions from statistical decompositions on empirical populations that do not calculate total heritability [42,43] and so do not include maternal effects. For example, significantly more variance in *Collinsia verna* seed weight was explained despite a reduction in additive genetic variance when models included the maternal phenotype compared with those without it [44]. Variance in maternal phenotype represents an additional pool of raw variation that can amplify the response to selection, leading to oscillations as the mean phenotype overshoots its optimum level in the expected environment (figure 1). The amplification of the phenotypic dynamics induced by *m* > 0 can cause the population to be further from its optimum than would be the case in the absence of maternal effects (*m* = 0). This amplification operates via phenotypic plasticity (figure 1), emphasizing the importance of considering the interplay of phenotypic plasticity and maternal effects when studying adaptation in natural populations. An experimental unification of phenotypic plasticity and maternal effects in driving adaptation would be difficult (and we are unaware of any to date), but there are experiments that highlight the importance of both: Lind & Johannson [45] incorporated maternal effects into statistical analyses of their experiments into the role of phenotypic plasticity in adaptation on common frogs *Rana temporaria*, but not into their experimental design, whereas Plaistow & Benton [46] manipulated the strength of maternal effects to alter mean population fitness and transient population dynamics on experimental populations of soil mites *Sancassania berlesei*.

In relatively stable environments, a negative maternal effect coefficient minimizes phenotypic variance and hence maximizes mean fitness (equation (3.4) and figure 2). This minimum variance occurs at negative *m*, because this uses the information in the maternal phenotype to discount the effect of the inherited genes and express a phenotype that is closer to the average. The level of maternal effects that maximizes mean fitness in our simulations increased with increasing environmental autocorrelation (figure 2), but remained negative or zero to reach the optimal level of phenotypic variance (equation (3.4)). This impact of the predictability of environmental change is consistent with conclusions from the quantitative genetic models of Lande & Shannon [47], which showed how genetic variance impacts fitness negatively when environmental change is more predictable. The increased curvature of the trajectories in figure 2 indicates that, as *ρ*_{τ} increases, so too does the negative impact of suboptimal *m*. Increasing *ρ*_{τ} favours greater *m* because the changes in environment become more predictable, meaning that the parental phenotype carries more accurate information to prepare offspring to the environmental conditions they might experience during their lifetime [30].

Although we have not explicitly incorporated a cost of maternal effects in our model, the oscillatory dynamics (figure 1) are a constraint on unbounded increases in a positive maternal effect coefficient. Were we to do so, we expect that plasticity would compensate for cost-reduced *m* during the transient phase, whereas additive genetic variance would compensate for cost-reduced *m* once the population reaches its new equilibrium (or always in the stable environment; figure 1). Because the consequences of maternal effects depend on their demographic and environmental contexts [46] and parents frequently adjust their phenotype in response to changing conditions [48,49], our assumption of fixed maternal effects from one generation to the next is a strong one. It does facilitate comparison with many statistical studies [12], however. In reality, we expect the maternal effect coefficient, *m*, to be a continuous trait, varying across the population and subject to selection [50]. Nevertheless, we expect the results that positive maternal effect coefficients are beneficial in rapidly changing environments, whereas negative values of *m* are beneficial in stable ones, to be robust in a framework where *m* can vary. Positive *m* will still increase the rate of adaptation to a new environment, very likely to an enhanced degree as *m* evolves towards more positive values. In stable environments, the phenotypic variance will still involve covariance between the maternal phenotype and the offspring genotype and so a discounting of the former against the latter will still favour negative *m*. However, the simplified mathematical treatment that we have given here with *m* fixed makes these points much more transparently than would otherwise be possible. It seems logical that populations would benefit from the dampening effect of negative values of *m* in a stable environment, while retaining the possibility of evolving positive values to facilitate rapid adaptation following environmental upheaval. Relaxing the assumption of fixed maternal effects is the subject of future work.

## 5. Conclusion

Using quantitative genetic models of reaction norm evolution, we have shown that maternal effects can both facilitate rapid adaptation to environmental change and, in more stable environments, keep phenotypes close to the average and so maximize fitness. We suggest that one reason that the average level of maternal effects in stable environments is negative is because this minimizes the effect of genetic variance on fitness when the environment is predictable.

## Acknowledgments

This work was supported by the Engineering and Physical Sciences Research Council (grant no. EP/H031928/1). We thank Rufus Johnstone, Bram Kuijper, Louise Machell, Stuart Townley and Jonathan Wells for insightful comments and enjoyable discussions.

## Appendix A. Properties of the phenotypic variance

The expected phenotypic variance at equilibrium is given in equation (3.1) as

where . Note that and are both positive by definition.

Clearly, an equilibrium can occur only if the expected phenotypic variance would be positive and bounded. For , this restricts *m* to the ranges , and , whereas for , *m* must lie in the ranges −1 <*m* < 1 or (figure 3). If then and *m* > 2 are permitted. For values of *m* outside these permitted ranges, the system cannot reach an equilibrium.

In the absence of environmental stochasticity ), the parameter values in figure 1 give and thus , −1 < *m* < 1 and *m* > 2 for *δ* = 10, and and so −1< *m*< 1 and *m* > 2 for the reference environment *δ* = 0. When there is environmental noise as in Lande [8] with , the ranges of valid equilibria are *m* < −2.63 (3 s.f.), −1 < *m* < 1 and *m* > 2 for *δ* = 10 ( 3 s.f.), and −1 < *m* < 1 and for the reference environment ( 3 s.f.).

## Appendix B. Updating covariances

The evolutionary response to stochastic environments was modelled numerically using equations from (2.10) to (2.12) to update the population mean additive genetic effect, , plasticity, and phenotype, , in each generation, starting from the expected equilibrium values. In order to calculate *γ* at each step, we first worked out the phenotypic variance
and to do this we updated , and according to
where the subscripts and refer to the values after selection in generation . We expect the errors in the value of *σ*_{zt}^{2} from approximating , , and to be small when .

## Appendix C. Stability calculations

For *m* ∼ 0.48 with parameter values from Lande [8], we see oscillations during the transition to the new equilibrium (figure 1). This can be understood by analysing the stability of the equilibrium state. Note throughout that when is close to 1, we violate the assumption that (see equation (2.5)) and so our results are not strictly valid in those regions. However, in practice, this is not significant, as we show that only equilibria towards the middle of the region are of interest.

Consider an environment *ε* that is constant over time in the absence of environmental noise. If we set , then the corresponding equilibrium steady state is given in equations (2.16)–(2.18) and is a fixed point of the map
which is derived from equations (2.10)–(2.12) with .

We now make a linear change of variables

This step reduces the map to a simpler form, from which it is clear that is fixed:
where . The Jacobian of the map is given by
and has eigenvalues that satisfy the characteristic equation
where . Note that because and *γ* depend on *m* (see above and equation (2.5)), will also depend on *m*.

If for both eigenvalues , then the equilibrium state is stable, if for one or both eigenvalues then the state is neutrally stable and it is unstable otherwise. In the absence of maternal inheritance (*m* = 0), we have and . For the values of the parameters given in figure 1, (2 s.f.) before the change in environment and (2 s.f.) afterwards, so both these equilibrium states are stable. To see whether maternal effects can destabilize the equilibria, we will now let . Setting , where and are real, and separating the real and imaginary parts of the characteristic equation gives
C1and
C2

The equilibrium is unstable when , so and (neutrally) stable when .

From equation (C 2), we see that either or . Considering the case first, the stability boundary is given by . For and , equation (C 1) gives and so there is a stability boundary at , which is equivalent to . If and , equation (C 1) gives a stability boundary at .

If on the other hand we have , then if , we have complex eigenvalues and hence oscillatory dynamics. From equation (C 1) we get

In the range , we have for (2 s.f.) after the environmental shift. Therefore, once *m* ∼ 0.48 the step change in the environment *δ* triggers oscillations in the convergence to the new phenotypic optimum. However, in this region and so the new equilibrium state remains stable and the oscillations dampen and eventually die away. If we fix and require, as in Lande [8], that the relationship of the genetic variances to *δ* remains constant at , there is no dependence of on *δ*. Even when environmental noise is included, as below, varies very little with *δ*.

In the oscillatory case, the stability boundary is at . Calculating the locations of the stability boundaries for both real and complex eigenvalues reveals that in both the perturbed () and unperturbed () environments, the equilibria in the range are all stable. When , the remaining equilibria lie in the range , and these are unstable except in the region (3 s.f.). When , the other regions where equilibria exist are (i) , where they are unstable and (ii) where they are also unstable except in the region (3 s.f.). Thus, in both cases, stable equilibria are found only in the range and a region at significantly larger positive *m* that we consider to be implausible on biological grounds. This provides us additional rationale for restricting our attention to values of *m* in the range .

The analysis above assumes a fixed environment both before and after the step change; we can repeat it for expected mean quantities in the presence of environmental noise. In this case, the Jacobian matrix of the map is

where .

Analytical expressions for the stability boundaries are harder to obtain, but the eigenvalues can be determined numerically. For the parameter values used in figure 1, and at , the eigenvalues for the equilibrium after the change are and , compared with , and in the absence of noise, showing that the noise has a slight stabilizing effect. At , the eigenvalues are and (cf. 1.0 and 0.6589 ± 0.0713i) showing that oscillations will, as before, be triggered by the environmental shock.

- Received March 6, 2012.
- Accepted April 19, 2012.

- This journal is © 2012 The Royal Society