Picture for Shiwei Liu

Shiwei Liu

SoS1: O1 and R1-Like Reasoning LLMs are Sum-of-Square Solvers

Add code
Feb 27, 2025
Viaarxiv icon

Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam

Add code
Feb 24, 2025
Figure 1 for Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
Figure 2 for Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
Figure 3 for Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
Figure 4 for Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
Viaarxiv icon

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More

Add code
Feb 11, 2025
Figure 1 for Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
Figure 2 for Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
Figure 3 for Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
Figure 4 for Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
Viaarxiv icon

The Curse of Depth in Large Language Models

Add code
Feb 09, 2025
Viaarxiv icon

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training

Add code
Jan 12, 2025
Figure 1 for SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
Figure 2 for SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
Figure 3 for SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
Figure 4 for SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
Viaarxiv icon

Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN

Add code
Dec 18, 2024
Viaarxiv icon

SIDE: Socially Informed Drought Estimation Toward Understanding Societal Impact Dynamics of Environmental Crisis

Add code
Dec 17, 2024
Figure 1 for SIDE: Socially Informed Drought Estimation Toward Understanding Societal Impact Dynamics of Environmental Crisis
Figure 2 for SIDE: Socially Informed Drought Estimation Toward Understanding Societal Impact Dynamics of Environmental Crisis
Figure 3 for SIDE: Socially Informed Drought Estimation Toward Understanding Societal Impact Dynamics of Environmental Crisis
Figure 4 for SIDE: Socially Informed Drought Estimation Toward Understanding Societal Impact Dynamics of Environmental Crisis
Viaarxiv icon

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning

Add code
Nov 26, 2024
Figure 1 for Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
Figure 2 for Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
Figure 3 for Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
Figure 4 for Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
Viaarxiv icon

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

Add code
Oct 14, 2024
Figure 1 for AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Figure 2 for AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Figure 3 for AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Figure 4 for AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Viaarxiv icon

Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models

Add code
Oct 10, 2024
Figure 1 for Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
Figure 2 for Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
Figure 3 for Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
Figure 4 for Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
Viaarxiv icon