Delegate variables to multiple optimizers

Wraps an optimizer_map() or a callable that selects an optimizer for a variable. Access the sub-optimizers through optimizer$optimizers, for example to inspect their learning rates, iterations, or loss-scale factors. A multi-optimizer does not expose a single learning_rate, because its sub-optimizers may have different learning rates. Optimizer-specific callbacks are not currently supported.

Usage

optimizer_multi(optimizer_map, loss_scale_factor = NULL, name = NULL)

Arguments

optimizer_map

An OptimizerMap or callable that accepts a variable and returns its optimizer.

loss_scale_factor

Optional loss scale overriding the value on each sub-optimizer.

name

Optional optimizer name.

Value

A MultiOptimizer instance.

Examples

optimizers <- optimizer_map(
  default_optimizer = optimizer_sgd(),
  optimizer_map = list("encoder/.*" = optimizer_adam())
)
optimizer <- optimizer_multi(optimizers)

# A callable can also select an optimizer for each variable.
optimizer <- optimizer_multi(function(variable) {
  if (grepl("encoder", variable$path)) optimizer_adam() else optimizer_sgd()
})

See also

Other optimizers:
optimizer_adadelta()
optimizer_adafactor()
optimizer_adagrad()
optimizer_adam()
optimizer_adam_w()
optimizer_adamax()
optimizer_ftrl()
optimizer_lamb()
optimizer_lion()
optimizer_loss_scale()
optimizer_map()
optimizer_muon()
optimizer_nadam()
optimizer_rmsprop()
optimizer_schedule_free_adam_w()
optimizer_sgd()