12 Optimizers for Training Neural Networks, clearly explained!

In today's newsletter:

• Your AI coding agents should get smarter every time you use them
• 12 optimizers for training neural networks, clearly explained.
• 6 steps to build an ML model.

AI AGENTS

Your AI coding agents should get smarter every time you use them

AI coding agents write good code now, but they still don’t learn from experience.

Say you spend 30 minutes helping Claude Code debug a tricky issue. It explores your codebase, tries several approaches, hits dead ends, and eventually finds the fix.

Tomorrow you start a new session, and the agent makes the exact same mistakes.

Switch to Codex or Cursor and it gets worse. They have no idea what your previous agent figured out.

Every session produces useful knowledge: debugging breakthroughs, failed approaches, architectural decisions, and patterns specific to your codebase.

But most of it disappears when the session ends.

Beacon, built by Asymptote Labs, is an open-source memory layer that lets your coding agents learn from past sessions.

It works across Claude Code, Cursor, Codex, OpenCode, Cline, and 20+ other agent harnesses.

Beacon launched on Product Hunt today.

Check out the launch and show your support with an upvote →

Instead of treating every session as an isolated conversation, Beacon captures what your agents actually do and turns that experience into reusable knowledge.

Here’s how it works:

↳ Capture every session. Beacon records execution traces across your coding agents, including commands, file changes, errors, and debugging paths.

↳ Extract reusable knowledge. It picks out successful fixes, recurring workflows, and lessons worth remembering.

↳ Improve future sessions. Those lessons become skills and context your agents can use, even in a different coding tool.

↳ Close the feedback loop. Later traces show when saved knowledge stops working, so it can be updated or retired.

Beacon Cloud continuously finds ways to improve your agents

Capturing sessions is only the first step.

Every hour, Beacon Cloud reviews your agents’ new activity and turns it into findings in three categories:

↳ Security: risky things your agents did. Secrets pasted into prompts, destructive commands, permission bypasses, or connections to unknown MCP servers. Beacon flags the action, shows which session it happened in, and suggests a fix.

↳ Skill: workflows worth reusing. When you keep walking agents through the same debugging steps, Beacon notices and drafts a reusable skill. You download it and make it available to all your coding agents.

↳ Behavior: patterns in how you work with agents. Retry loops, abandoned sessions, unnecessary approvals, or expensive models used for simple tasks. Beacon points out the waste and recommends faster, cheaper ways to work.

The findings keep updating as your agents generate new traces.

One memory layer for every coding agent

Beacon sits across the agent harness layer.

So your Claude Code debugging lessons can help your next Codex session, and your Cursor discoveries can become skills in OpenCode.

With beacon handoff, you can also resume work in a different agent when you hit a rate limit or need to switch tools. You don’t re-explain the repo or retrace failed approaches.

It matters even more across a team. One engineer spends an hour debugging an infrastructure issue, Beacon captures the approach that worked, and the next engineer can reuse it as a skill in whichever coding agent they use.

The loop looks like this:

Run agents → Capture traces → Generate findings → Reuse skills → Improve → Repeat

Beacon runs locally without a hosted backend. Beacon Cloud adds hosted trace storage, cross-harness search, session analytics, and the findings above.

A smarter model only gets you so far. The coding agent that keeps getting better is the one that learns from experience.

Support Beacon on Product Hunt →

Star Beacon on GitHub →

Stop teaching your agents the same lessons twice.

Thanks to Asymptote Labs for partnering today!

DEEP LEARNING

12 optimizers for training neural networks

An optimizer can reduce the number of training steps and still make your run slower. Each update has its own compute and memory cost, which becomes part of the training budget.

That tradeoff gets easier to reason about once you look at what each optimizer stores and how it transforms the gradient.

The 12 optimizers in the visual approach this differently. Let’s break them down through the engineering decisions they change, starting with how much gradient history survives each update.

Smoothing gradients and adapting step sizes

SGD with momentum keeps a running accumulation of gradients. Directions that persist across batches build up, while inconsistent directions tend to cancel.

This helps reduce oscillation, but momentum alone doesn’t give each parameter its own adaptive step size.

AdaGrad adds that adaptation by accumulating squared gradients for each parameter. Frequently updated parameters receive progressively smaller effective steps, which can help with sparse features.

Its limitation follows directly from that accumulation: the denominator keeps growing, so effective learning rates can become too small.

RMSprop replaces the full accumulation with an exponential moving average of squared gradients. Older observations fade, allowing the scaling to respond to more recent gradients.

Adam combines momentum with this second-moment scaling. One moving average tracks gradient direction; another tracks squared gradient magnitude.

That second quantity isn’t a direct curvature measurement. Adam adapts to gradient statistics without estimating the Hessian.

Changing regularization, memory, and gradient history

AdamW separates weight decay from the adaptive gradient update. Adding an L2 penalty to Adam’s loss lets its adaptive scaling affect that penalty; decoupled weight decay applies parameter shrinkage separately.

That distinction matters when you’re tuning regularization. Weight decay becomes easier to reason about independently of the gradient scaling.

Memory introduces another constraint. Standard Adam maintains two parameter-sized state arrays, in addition to the model weights and gradients.

With one billion parameters and FP32 moment buffers, those two arrays alone require about 8 GB before sharding or compression.

Adafactor reduces the second-moment storage for matrices using row and column statistics. For an n × m matrix, that replaces nm stored second-moment values with roughly n+m.

You’re trading a full per-parameter estimate for a factored approximation. That’s useful when optimizer state limits the model or batch size you can fit.

Lion keeps momentum without a second-moment buffer. Its gradient update uses the sign of a momentum-gradient blend, giving coordinates the same update magnitude within a parameter group.

That changes the update scale, so copying AdamW’s learning rate into Lion isn’t a fair comparison. The Lion paper specifically calls for smaller learning rates.

AdEMAMix takes another approach: it combines fast and slow moving averages of gradients alongside second-moment scaling. Recent gradients remain influential while older directional information survives longer, at the cost of additional state.

Using matrix structure and curvature

Adam scales coordinates individually. Shampoo uses matrix-valued statistics to transform updates across parameter dimensions, capturing relationships that elementwise scaling misses.

Those richer statistics bring additional storage and matrix computation.

SOAP connects this approach back to Adam. It rotates gradients into the eigenbasis of Shampoo’s preconditioners, applies Adam there, and rotates the update back.

The coordinate system changes how Adam’s elementwise scaling acts on the update. Computing and refreshing that basis also adds overhead.

Muon approximately orthogonalizes momentum updates for matrix parameters using Newton–Schulz iterations. It changes the matrix update’s singular-value structure rather than scaling every coordinate independently.

Parameter selection matters here. Muon is typically applied to hidden weight matrices, with another optimizer handling embeddings, output layers, and vector parameters.

Sophia uses a diagonal curvature estimate to scale momentum, then clips updates elementwise. Its curvature estimate is refreshed periodically, limiting the average overhead of obtaining that information.

Comparing optimizers on your workload

I’d use AdamW as the reference run for a transformer experiment, then choose alternatives against a specific constraint.

For memory pressure, investigate Adafactor or Lion. For matrix updates, evaluate Muon, Shampoo, or SOAP while accounting for their parameter coverage and implementation costs.

Give each optimizer a learning-rate search. Keep the data, model, and evaluation protocol consistent, and report both validation quality at a fixed compute budget and time to a target quality.

Also measure peak memory and step time. An optimizer that reaches the target in 20% fewer steps but makes each step 30% slower takes about 4% longer overall.

The useful result is a reproducible improvement under your training constraints, including the cost of tuning it.

If you want to build these foundations properly, our Classical ML and Deep Learning roadmap covers the full stack in order, from the assumptions behind linear models to neural network training and the architectures built on top:

Start the Classical ML and Deep Learning roadmap →

👉 Over to you: Which optimizer has improved your training runs enough to justify changing your default?

MLOps

6 steps to build an ML model

Building an ML model isn’t just about picking an algorithm and hitting train.

Getting it to production requires 6 steps, and the algorithm selection is just one of them.

Here’s the full breakdown:

Step 1: Setting objectives

Before writing a single line of code, you need clarity.

What problem are you solving? Is ML even the right approach? What does success look like?

This means identifying the use case, running a feasibility study, and defining your KPIs upfront.

Step 2: Data preparation

This is where you’ll spend most of your time since no fancy algorithm fixes bad data.

Here, you collect your data, clean it (handle missing values, outliers, inconsistencies), engineer meaningful features, and split it properly into train/validation/test sets.

Step 3: Choose the algorithm

Now you pick your approach, like Random Forest, XGBoost, Neural network, etc.

The choice depends on your problem type, data size, interpretability needs, and latency requirements.

Also, decide on your framework: scikit-learn for classical ML, TensorFlow, or PyTorch for deep learning.

Step 4: Train the model

Feed your prepared data to the model and let it learn.

But training isn’t a one-shot thing. Here, you iterate, adjust hyperparameters and experiment with different configurations.

This loop continues until performance plateaus.

Step 5: Evaluate and test

Now, you test how good your model really is.

Run it on your held-out test set. Analyze metrics relevant to your problem (accuracy, precision, recall, F1, AUC).

And don’t forget bias testing. Your model should work fairly across different segments.

Step 6: Deploy and monitor

Finally, you containerize it, deploy it to the cloud (AWS, GCP, Azure), and set up monitoring.

Moreover, since models degrade over time, you need to catch data drifts and other issues before your users do.

That’s the full picture.

The algorithm typically gets all the attention, but it’s maybe 15% of the work. The rest is Engineering, infra, and careful thinking.

If you want to see this in practice, we have already covered MLOps from a fully beginner-friendly perspective in our 18-part course.

It covers foundations, ML system lifecycle, reproducibility, versioning, data and pipeline engineering, Spark, model compression, deployment phase, Kubernetes, cloud infra, virtualisation, deep dive into AWS, and monitoring in production.

Start with MLOps Part 1 here →

👉 Over to you: Have we missed any step in the model-building process?

NO-FLUFF RESOURCES TO...

Succeed in AI Engineering roles

All businesses care about impact. That’s it!

• Can you reduce costs?
• Drive revenue?
• Can you scale ML models?
• Predict trends before they happen?

We have discussed several other topics (with implementations) in the past that align with such topics.

Here are some of them:

• Learn MLOps from first principles to production in this course with 18 parts →
• Learn everything about MCPs in this course with 9 parts →
• Learn how to build Agentic systems in this course with 14 parts.
• Learn how to build real-world RAG apps, evaluate, and scale them in this course.
• Learn sophisticated graph architectures and how to train them on graph data in this course.
• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
• Learn how to run large models on small devices using Quantization techniques.
• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
• Learn how to identify causal relationships and answer business questions using causal inference in this course.
• Learn how to scale and implement ML model training in this practical guide.
• Learn techniques to reliably test new models in production.
• Learn how to build privacy-first ML systems using Federated Learning.
• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

PARTNER WITH US

ADVERTISE TO 950k+ AI PROFESSIONALS

Our newsletter puts your products and services directly in front of an audience that matters, including thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.

Get in touch today by replying to this email.