Did you know? A neural network does not always have to compute its representation through a fixed list of increasingly deep layers. A deep equilibrium model (DEQ) instead defines a transformation whose hidden state must satisfy a fixed-point equation, then uses a numerical solver to find that state. Rather than store every step as a separate layer activation, it can learn through implicit differentiation at the equilibrium. This is a real machine-learning method, not a claim about human consciousness or autonomous superintelligence.
To use DEQs responsibly, define the input-dependent state update, ensure a solver can reach an adequate fixed-point residual, compute gradients through the correct implicit system, check stability and conditioning, and compare the full cost and accuracy with an ordinary finite-depth network. A mathematically valid fixed point can still be hard to find; constant memory with respect to effective depth does not mean constant runtime, reliable convergence or good generalisation.
This article explains the mechanism from a solvable scalar case to practical high-dimensional training, including hand-checked gradients, diagnostic failures and a changed-context exercise. Its numeric examples are invented teaching calculations, not results from running a neural-network training experiment.
Contents
- A network defined by a fixed point rather than a layer count
- A hand-checked equilibrium with a converging update
- Why numerical root finding is not the same as unrolling a fixed depth
- Implicit differentiation without storing every intermediate state
- Solved derivatives for the same scalar equilibrium
- Convergence is a property to test, not a name to assume
- Why the implicit backward solve can be unstable
- What constant-memory claims mean in their proper scope
- Training workflow from input to a verified averaged loss
- DEQ, residual networks, recurrent networks and Neural ODEs
- Evaluating a solver-driven model beyond classification accuracy
- Ten failures that need specific diagnosis and repair
- Changed-context exercise: double the input and tighten the deadline
- Where deep equilibrium models are not the right engineering answer
- Frequently asked questions about DEQs
- Primary research and reader pathways
A network defined by a fixed point rather than a layer count
Most introductory neural networks apply a sequence of transformations: the first layer produces a hidden representation, the next modifies it and so on. Depth is visible in the number of transformations applied. Deep equilibrium models describe the desired hidden representation differently. The model defines an update function fθ(z,x), and asks for a state z* satisfying z*=fθ(z*,x), where x is the input and θ denotes trainable parameters.
One way to approximate z* is to repeatedly apply fθ to a current estimate: z_(k+1)=fθ(z_k,x). If this process converges, subsequent updates change the state by progressively smaller amounts. A numerical root-finding method can also solve F(z,x)=fθ(z,x)−z=0 directly, without treating every iteration as a separately stored layer. The choice of solver affects convergence, speed and accuracy.
The idea has a relationship to a weight-tied network, where the same transformation is applied again and again. If the repeated transformation converges, its limit can be viewed as an equilibrium representation. But ‘infinite depth’ is a mathematical analogy, not a declaration that infinitely many computations were executed or infinitely many parameter matrices were created.
Bai, Kolter and Koltun’s original paper Deep Equilibrium Models (NeurIPS 2019) introduced this approach to sequence modelling and described fixed-point solving and implicit differentiation. The paper reports advantages in studied architectures, not a proof that all future models should use the method.
The meaningful reader question is how a hidden state can represent a computation whose stopping condition is mathematical convergence rather than a preselected number of layers. Answering that requires both an equation and an honest numerical stopping rule, because a solver can return a number even when the required equilibrium was not reached.
A hand-checked equilibrium with a converging update
Take a simple scalar state z and input x with an update f(z,x)=0.5z+0.25x. Set x=4. The fixed-point equation is z*=0.5z*+1. Subtracting 0.5z* from both sides gives 0.5z*=1, hence z*=2. This solution follows from elementary algebra without any machine learning library.
Starting from z0=0 and repeatedly applying the update gives z1=1, z2=1.5, z3=1.75, z4=1.875 and z5=1.9375. Each step cuts the remaining distance to the fixed point by half. After the fifth update, the actual error is |2−1.9375|=0.0625. The iterative process did not magically become exact merely because several updates were made.
Compute the fixed-point residual at z5: r=f(z5,4)−z5=(0.5×1.9375+1)−1.9375=1.96875−1.9375=0.03125. Notice that the residual is half the state error in this particular scalar contraction. In more general problems, the relationship between residual and actual state error can depend on conditioning and need not match this convenient ratio.
For this example, the derivative of f with respect to z is 0.5 in every state. Because its magnitude is below one, the simple fixed-point iteration is a contraction in the scalar setting and converges toward the unique fixed point. That sufficient condition helps explain why the example behaves so cleanly.
A real DEQ layer includes high-dimensional nonlinear transformations, potentially attention or convolutional structure. A solver may require many iterations, fail to converge or settle near a point with unacceptable residual. Do not transfer the scalar example’s perfect convergence guarantee to every neural architecture.
Why numerical root finding is not the same as unrolling a fixed depth
A finite unrolled network computes a chosen number of updates, for example ten applications of the same transformation. A DEQ implementation aims to solve an equilibrium equation to a chosen numerical tolerance, often using more sophisticated algorithms than raw repeated substitution. Anderson acceleration and Broyden-style methods are examples of approaches used in nonlinear fixed-point and root-solving contexts.
The stopping rule matters. A typical absolute residual check examines a norm such as ||fθ(z,x)−z||. A relative check may scale the residual by the size of the state or another reference quantity, with appropriate protection against division by zero. A threshold of 10^-4 has no universal meaning without specifying units, norm, input scaling and tolerance convention.
High-dimensional root solvers can use past iterates or approximate Jacobian information to accelerate convergence. That can reduce iteration counts but introduces its own temporary memory, numerical conditioning and line-search decisions. The overall memory claim must include the actual solver implementation and relevant state buffers rather than casually repeating ‘constant memory’ without qualifications.
Iteration limits are necessary in production. A model that loops forever trying to satisfy an impossible tolerance is not a usable predictor. Set a maximum number of steps, collect residual and convergence telemetry, and define what happens when the budget is exceeded. The correct fallback may be to abstain, use a verified finite-depth alternative or flag the example for further review.
Numerical solvers are not interchangeable simply because they all produce a vector named z. Two algorithms may stop at different residuals, react differently to poorly scaled features and yield different gradients. The exact forward solver is part of the model’s reproducible computational contract.
Implicit differentiation without storing every intermediate state
Training requires the derivative of a loss L with respect to parameters θ. If z* satisfies z*=fθ(z*,x), changes to θ can change the equilibrium. Differentiating the fixed-point equation gives dz*/dθ=(∂f/∂z)(dz*/dθ)+∂f/∂θ. Rearranging yields (I−∂f/∂z)(dz*/dθ)=∂f/∂θ.
When the relevant linear operator is invertible, the formal expression is dz*/dθ=(I−∂f/∂z)^−1(∂f/∂θ). A practical implementation need not build an enormous dense matrix inverse explicitly. It can solve a linear system or a related adjoint equation using Jacobian-vector or vector-Jacobian products.
The key difference from ordinary backpropagation through an unrolled computation is that the backward pass can use derivatives evaluated at the equilibrium, rather than retaining a tape of every forward solver step. This is the basis for memory savings in studied DEQ implementations. The exact memory footprint still depends on how the solver and automatic differentiation are implemented.
Implicit differentiation assumes an adequate equilibrium and a suitably solvable derivative system. If the Jacobian has an eigenvalue near one in a relevant direction, I−∂f/∂z can become poorly conditioned, magnifying numerical errors in gradients. If the equilibrium solve is inaccurate, the gradient estimate may also be unreliable.
This mechanism is distinct from Neural ODEs, which describe a continuous-time derivative and integrate a differential equation. DEQs solve a fixed-point equation. Both involve numerical solvers and adjoint-like reasoning in some forms, but calling them the same architecture would conceal their different mathematical objects.
Solved derivatives for the same scalar equilibrium
Return to f(z,x)=az+bx and choose a=0.5, b=0.25 and x=4. The equilibrium equation is z*=az*+bx, so z*=bx/(1−a)=1/0.5=2. Its derivative with respect to b is dz*/db=x/(1−a)=4/0.5=8. Its derivative with respect to a is dz*/da=bx/(1−a)²=1/(0.5²)=4.
Check the result by the implicit formula. Here ∂f/∂z=a=0.5, so the scalar multiplier I−∂f/∂z equals 0.5. For parameter b, ∂f/∂b=x=4, and 0.5×dz*/db=4 implies dz*/db=8. For parameter a, ∂f/∂a=z*=2, and 0.5×dz*/da=2 implies dz*/da=4. The methods agree.
Now choose a toy squared-error loss L=0.5(z*−1)². At z*=2 the loss is 0.5×1²=0.5. Its derivative with respect to the equilibrium state is z*−1=1. By the chain rule, dL/db=1×8=8 and dL/da=1×4=4. These are exact symbolic derivatives of this constructed problem.
The example does not prove that an arbitrary high-dimensional DEQ will have stable gradients. It shows how the equilibrium constraint supplies a route from model parameters to output sensitivity without differentiating through each stored iteration. With an ill-conditioned operator, both exact formulas and numerical implementations can be sensitive to error.
Readers can verify the scalar computation using a tiny perturbation of b or a. For example, increasing b from 0.25 to 0.251 with a fixed gives z*=0.251×4/0.5=2.008, an increase of 0.008 for a parameter increase of 0.001, consistent with derivative eight. This is a local finite-difference check, not evidence of successful model training.
Convergence is a property to test, not a name to assume
The scalar example converged because the update had a derivative magnitude of 0.5. Replace it with f(z,x)=1.1z+0.25x and x=4. Solving the fixed-point equation gives z*=1.1z*+1, so −0.1z*=1 and z*=−10. The fixed point exists algebraically, but simple repeated substitution starting from zero gives 1, 2.1, 3.31 and continues away from −10 rather than approaching it.
The comparison shows why existence and convergence are different questions. A fixed-point equation may have a solution even when naive iteration fails. Conversely, a numerical solver might appear stable for a few iterations while approaching a different solution branch or violating the desired tolerance. The name equilibrium model does not guarantee that the actual forward pass found a trustworthy equilibrium.
A contraction mapping gives a useful sufficient condition for simple iteration to converge in a suitable complete metric space. Real neural layers can violate simple global contraction assumptions and still admit workable local solutions with appropriate solvers. The actual check should therefore include measured residuals, iteration behaviour, numerical stability and representative input cases.
Numerical precision matters. Float32 and lower precision can change residual estimates, particularly when activations become large or nearly cancel. A stopping tolerance smaller than the representable numerical variation may cause wasteful iterations or misleading success flags. Mixed-precision systems need an explicit plan for where critical solver calculations occur.
Convergence should be evaluated on difficult examples, not only average easy inputs. A few inputs that require hundreds of iterations can dominate latency and create unreliable deadlines. Report a distribution of iteration counts and residuals, with an explicit treatment of non-converged cases instead of excluding them from performance figures.
A model that returns quickly with a large residual may be operationally unacceptable even if its classification accuracy happens to look fine on a small validation sample. The residual checks the claimed mathematical state, while held-out prediction checks the task. Both are necessary evidence, and neither substitutes for the other.
Why the implicit backward solve can be unstable
The expression (I−J_f)dz*/dθ=∂f/∂θ describes a linear system. Its numerical difficulty depends on the spectrum and conditioning of the Jacobian J_f at the equilibrium. If J_f approaches the identity along some direction, then I−J_f approaches singularity there, making the solution sensitive to small errors.
In the scalar example with a=0.5, the factor 1−a is 0.5 and the sensitivities were moderate. Set a=0.99, b=0.25 and x=4. The equilibrium becomes bx/(1−a)=1/0.01=100. The derivative with respect to b is x/(1−a)=400, while the derivative with respect to a is bx/(1−a)²=1/0.0001=10,000. Small parameter changes can greatly affect the equilibrium.
This does not automatically mean all high sensitivities are errors; some underlying systems genuinely behave sharply near a boundary. But for training, the gradients can be difficult to use reliably. The forward solver may also converge slowly under simple fixed-point iteration because a close-to-one contraction reduces the error by only a small fraction per step.
An implementation should monitor solver convergence and gradient behaviour together. Stable-looking predictions from a loose forward tolerance can hide gradients that change dramatically when tolerance is tightened. Sensitivity checks across multiple tolerances, seeds and representative examples help reveal when the method’s numeric assumptions are fragile.
Gradient clipping, careful optimisation and architectural constraints may help in appropriate settings, but they cannot replace a correct implicit equation. The existing gradient clipping guide addresses one separate method of controlling update magnitudes. It does not prove that a DEQ equilibrium was found or that its backward linear solve was accurate.
The fundamental diagnostic is to ask whether a small residual implies a sufficiently accurate state and whether the derivative system is well conditioned for the specific prediction task. Neither answer follows merely from the fact that automatic differentiation returned finite numbers.
What constant-memory claims mean in their proper scope
The original DEQ research highlighted memory savings with respect to the effective depth represented by repeated hidden-state updates. If a conventional unrolled computation stores an activation for each of K steps, its activation memory can grow with K. Implicit differentiation can instead use a fixed-point state and auxiliary solver operations without retaining the full forward unroll.
This is a comparison about one important component of training memory. It does not mean the entire model uses the same number of bytes regardless of sequence length, batch size, width or solver design. Parameter tensors, input activations, backward vectors, Jacobian-vector computations, temporary states and optimiser storage all consume memory.
Consider a fictional hidden representation of 1,024 float32 values per example. A single such vector contains 1,024×4=4,096 bytes. In a simplistic accounting, storing forty separate copies for forty unrolled steps would require 40×4,096=163,840 bytes per example, while one state vector contains 4,096 bytes. The ratio of these raw stored-state counts is forty to one, but this says nothing about actual total framework allocations.
Real solvers may retain past iterates for acceleration, and batching multiplies state storage. The comparison also omits parameter gradients, layer computations and all other temporary buffers. The reader should not convert the toy state ratio into a claim of fortyfold end-to-end memory reduction.
The original paper reported substantial savings in certain studied sequence-model experiments. Those observations depend on architecture, implementation and workload. A current deployment should measure peak allocated device memory during both forward and backward passes and document numerical tolerances and achieved quality.
Memory and speed can trade against one another. A method that stores fewer intermediate states may require more function evaluations or a slower backward linear solve. The correct decision considers wall-clock latency, training throughput, energy, solver failures and prediction quality, not just the most attractive memory number.
Training workflow from input to a verified averaged loss
First construct a transformation fθ(z,x) whose input and state dimensions are understood, and specify an initial state for the forward solver. Validate that an ordinary forward call produces a finite state and a meaningful residual. If the operation can fail, make its non-convergence status observable before attaching a loss and optimiser.
Next calculate a task loss from the equilibrium representation or a downstream prediction head. The loss may be classification, regression, ranking or another legitimate objective. The equilibrium relation determines how gradients reach θ, while the task objective determines what the representation should learn. These are different choices and both require validation.
Then perform the implicit backward computation with the correct Jacobian-vector products and numerical solver, checking whether residual and gradient tolerances are adequate. Compare gradients against finite differences on a small test problem, as in the scalar example. An agreement on a toy function is not a proof for every architecture, but it detects many sign and orientation mistakes.
Optimiser updates change θ, which changes the equilibrium equation at the next training step. A cached equilibrium from an earlier parameter state is not automatically exact for the new model. Warm starting may be useful, but it requires clear semantics and checks to avoid relying on stale state as though it were verified.
Evaluate the model on independent held-out data with a declared performance metric. Record solver iterations, residuals, failures and latency for those examples too. If only successfully converged examples are included in task accuracy, the model’s reported quality may be artificially inflated. Non-convergence is part of the operating result.
Finally save the exact model parameters, solver configuration, preprocessing and version that produced the measured outcomes. A restored model using a different tolerance or solver can behave differently even when the learned weights are identical. Reproducibility includes computation, not just a tensor file.
DEQ, residual networks, recurrent networks and Neural ODEs
A residual network can be written as a sequence of state updates, often adding learned changes to the previous state. A recurrent network reuses parameters across time or layers and may be unrolled for a chosen number of steps. A deep equilibrium model focuses on the state that satisfies a fixed-point relation, with a solver deciding how the state is approximated.
The distinction is not just a programming style. A fixed-depth network has an explicit stopping point after a selected number of layers. A recurrent network may process sequence positions or iterations with its own training dynamics. A DEQ represents the converged state of a tied transformation, subject to numerical feasibility.
Neural ODEs define a time derivative of the hidden state and integrate it using an ODE solver. They describe continuous-depth dynamics rather than a fixed-point equation of the form z=fθ(z,x). Both may use numerical integration or implicit calculus ideas in broad mathematical terms, but their operational contracts and stopping conditions differ.
The existing neural tangent kernel guide deals with gradient geometry and the infinite-width perspective. That is another mathematical use of limits, not the same as the fixed-point limit in a DEQ. Distinguishing these meanings prevents word-level overlap from replacing real mechanisms.
No architecture wins by definition. A finite residual network may be easier to train and deploy under hard deadlines, while a DEQ can offer a different memory-computation balance in suitable problems. Compare them with consistent data splits, parameter budgets, numerical settings and task objectives.
Evaluating a solver-driven model beyond classification accuracy
A DEQ experiment should report prediction quality and numerical behaviour separately. Classification accuracy or held-out regression error answers whether the model is useful for a task. Fixed-point residual, convergence rate, iteration count and backward-solver diagnostics answer whether the specified equilibrium computation was carried out adequately. A model can score well on one measure and poorly on another.
Imagine a hypothetical test set of 1,000 independent examples. A DEQ solver reaches the declared residual tolerance for 920 examples and fails within the iteration budget for eighty. The convergence fraction is 920/1,000=92%, while the failure fraction is 80/1,000=8%. If its task score is calculated only on the 920 successful examples, that selection must be disclosed; it cannot be labelled full-population accuracy.
Suppose the model correctly classifies 828 of the 920 converged examples. Accuracy on converged examples is 828/920=90%, but 828/1,000=82.8% is the fraction of all test cases that both converged and were correct. Neither statistic necessarily describes the utility of a carefully designed fallback on the remaining eighty cases. The fallback must be separately specified and evaluated.
Latency also needs a distribution. If typical cases need ten iterations and difficult cases need ninety, an average iteration count can obscure a long-tail delay. Measure p95 or a suitable high percentile, hard deadline misses and memory peaks under the intended deployment constraints. Compare against a finite-depth baseline using the same hardware and batch settings.
Do not tune solver tolerances and stop rules on the final test set over and over. Select them on training and validation evidence, freeze the complete candidate, and then report independent performance and failure rates. A lower residual is useful only when its computational cost and actual task consequence make sense.
Ten failures that need specific diagnosis and repair
Failure one is assuming that a formal fixed point guarantees convergence of naive iteration. The a=1.1 scalar example provides a counterexample. Repair by checking the actual solver and its residuals. Failure two is accepting a small update difference as proof of a correct equilibrium without examining the defined residual or relevant numerical tolerance.
Failure three is constructing the implicit gradient with the wrong Jacobian orientation or sign. Repair by deriving the linear system explicitly and checking finite differences on a small known problem. Failure four is computing a dense inverse unnecessarily in a high-dimensional implementation instead of using a supported linear solve or Jacobian-product method.
Failure five is ignoring poor conditioning near an eigenvalue of one. Repair by monitoring solver behaviour and sensitivity to tolerances. Failure six is reporting constant total memory while neglecting solver history, sequence length, batch size and model state. State precisely which memory component avoids growth with effective depth.
Failure seven is discarding non-converged examples before computing an impressive test score. Repair by retaining all test cases in the declared denominator and reporting how unresolved predictions were handled. Failure eight is using a cached equilibrium from an old parameter set as though it were verified for current weights.
Failure nine is changing solver configuration between validation and deployment, perhaps using fewer iterations for latency without testing its effect. Repair by saving and verifying solver settings with the model artifact. Failure ten is calling every numerical fixed-point iteration a DEQ, even when no implicit backward method or coherent equilibrium computation is defined.
These repairs illustrate a discipline of verifying state, not just code execution. A returned vector, a finite loss and a completed optimiser step are insufficient evidence when the model’s central mathematical claim is an equilibrium.
Changed-context exercise: double the input and tighten the deadline
Keep the solvable update f(z,x)=0.5z+0.25x but change the input from x=4 to x=8. The new fixed point is z*=0.5z*+2, hence z*=4. Starting from zero yields z1=2, z2=3, z3=3.5, z4=3.75 and z5=3.875. After five iterations the state error is |4−3.875|=0.125, twice the 0.0625 error at x=4 under the same start and iteration count.
The residual after five steps is f(3.875,8)−3.875=(0.5×3.875+2)−3.875=3.9375−3.875=0.0625. It also doubles relative to the earlier input. A fixed five-step budget can therefore deliver different absolute residuals for inputs with different scales, even with exactly the same contraction factor.
Suppose a deployment requires absolute residual below 0.04 before a safety-sensitive output is accepted. At x=4, the five-step residual was 0.03125 and passes; at x=8, the five-step residual is 0.0625 and fails. One extra step at x=8 yields z6=3.9375 and residual f(3.9375,8)−3.9375=3.96875−3.9375=0.03125, which passes. The correct solver stopping point changes with the input.
Now add a hard request deadline that cannot accommodate the sixth update. A good production system does not pretend the input met tolerance because an answer was needed quickly. It may use an approved fallback, return an uncertainty or non-convergence status, or choose a different architecture. The mathematical rule and service deadline must be reconciled explicitly.
Next change a to 0.99 while leaving b=0.25 and x=8. The equilibrium becomes z*=2/0.01=200, and naive iteration converges slowly. A tolerance and iteration budget chosen for a=0.5 are not automatically adequate. This is a change in the model’s conditioning, not merely a bigger input.
That is the central transfer lesson: a DEQ’s behaviour depends on update dynamics, solver configuration and task context. A correct calculation on one toy problem does not prove stable behaviour across the intended data distribution.
Where deep equilibrium models are not the right engineering answer
DEQs can be attractive when an iterative representation has a useful equilibrium structure and activation-memory costs are significant. They are not a universal replacement for residual networks, transformers, recurrent networks or other architectures. Some tasks benefit from finite-depth computation with predictable latency and simpler debugging.
The method requires numerical solvers in forward and backward computations. Solvers add failure modes and hyperparameters that a straightforward finite stack may not have. In an application with strict deterministic latency, occasional long convergence times can outweigh a memory benefit. Benchmark that trade-off rather than relying on the phrase infinite depth.
An exact fixed point may be irrelevant to the real data task if the learned representation is poor. Likewise, a very small residual does not establish causal reasoning, fairness, security or calibrated uncertainty. Those outcomes require their own evidence. Mathematics defines what is being computed; independent evaluation establishes whether it is useful.
Claims about energy or carbon savings require measured workloads, hardware characteristics and electricity context. A smaller peak activation footprint can coexist with more function evaluations and higher energy use. The original 2019 paper offers evidence in studied settings, not a universal environmental guarantee.
Implicit differentiation also depends on regularity and solvability conditions. The inverse notation is formal unless the relevant operator is suitably nonsingular; numerical approximations introduce further error. Near-singular Jacobians and multiple solution branches can complicate training. Explain these conditions honestly rather than presenting automatic differentiation as magic.
The user-facing promise should be bounded: the system can attempt an equilibrium solve under an explicit tolerance, report when it met that condition, and present independently tested task outcomes. It should not imply that an unseen infinite computation took place.
Frequently asked questions about DEQs
What is a deep equilibrium model? It is a neural model whose hidden state is defined as a fixed point z*=fθ(z*,x), typically found by a numerical solver and trained through implicit differentiation.
Does infinite depth mean infinitely many computations occur? No. It is an analogy to the limit of a weight-tied iterative network when convergence holds. A finite numerical solver approximates the state.
Are DEQs the same as Neural ODEs? No. DEQs solve a fixed-point equation; Neural ODEs integrate a differential equation defining continuous-time hidden-state dynamics.
Does a small residual prove the prediction is correct? No. It checks the numerical equilibrium equation under a norm and tolerance, not the supervised task label or real-world consequence.
Why can training save activation memory? The implicit backward method need not store every forward iteration as a separate layer activation, though actual solver and model memory still matter.
Can the forward solver fail? Yes. It may not converge within the iteration budget, may oscillate or encounter numerical conditioning problems. Failure should remain visible in evaluation.
Why not use a matrix inverse directly for gradients? High-dimensional explicit inverses can be expensive and numerically fragile. Implementations often solve associated linear systems using Jacobian-vector operations.
What does the eigenvalue near one problem mean? The operator I−J_f can become poorly conditioned, amplifying sensitivity and making implicit gradients harder to calculate reliably.
Does a DEQ make an agent more intelligent automatically? No. It is a model architecture and optimisation method whose usefulness must be tested on the relevant task.
How should I compare a DEQ with a finite network? Match data and objectives, report held-out outcomes, solver failures, iteration counts, peak memory and latency under consistent constraints.
Primary research and reader pathways
If you are a student, solve z*=0.5z*+0.25x by hand at x=4 and x=8. Compare state error with fixed-point residual and explain why a five-step rule passes one absolute tolerance case but fails the other. Then verify the implicit derivatives with a small parameter perturbation.
If you are an ML engineer, instrument both forward and backward solvers, preserve the exact tolerance and iteration budget with model artifacts, and evaluate all held-out examples including non-convergence. Check numerical gradients on toy inputs before scaling to expensive experiments.
If you review model claims, ask whether constant memory means activation memory with respect to effective depth or an unsupported statement about the whole training system. Request independent task metrics, tail latency, failed solves, stability evidence and a fair finite-depth baseline.
Original source: Bai, Kolter and Koltun, Deep Equilibrium Models, NeurIPS 2019. The original authors provide additional background through the corresponding research manuscript. Related eduKateSG articles include Gradient Clipping, Neural Tangent Kernel, and the How AI Works hub.
All numerical examples are constructed for teaching and were checked algebraically. No actual DEQ training job, timing experiment or production deployment was performed while preparing this explanation.