Frequency Principle
2021年4月17日在机器学习联合研讨计划的报告PPT，报告见B站。
A PPT and a summary for F-Principle are also provided.
科普
从频率角度理解为什么深度可以加速神经网络的训练 文献[11]
线性Frequency Principle动力学：定量理解深度学习的一种有效模型 文献[13,17]
F-Principle：初探深度学习在计算数学的应用 文献[4,9,12]
从傅里叶分析角度解读深度学习的泛化能力 [1,2,4,17]
code
1d F-Principle code at github.
Useful technique：A note of using Tensorflow to code Laplacian operator in high dimension
1d example of F-Principle
 |
Fourier Domain F-Principle: DNNs often fit target functions from low to high frequencies. Each frame is several training steps. Red: FFT of the target function; Blue: FFT of DNN output. Abscissa: frequency; Ordinate: amplitude. |
Paper list
* indicates the corresponding author
#: Equal contribution
[27] Zhi-Qin John Xu*, Yaoyu Zhang, Tao Luo, Overview frequency principle/spectral bias in deep learning. arxiv 2201.07395 (2022) pdf, and in arxiv.
[17] Yaoyu Zhang, Tao Luo, Zheng Ma, Zhi-Qin John Xu*, Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks. Chinese Physics Letters, 2021. pdf, and in arxiv, see CPL web
[16] (Alphabetic order) Yuheng Ma, Zhi-Qin John Xu*, Jiwei Zhang*, Frequency Principle in Deep Learning Beyond Gradient-descent-based Training, arxiv 2101.00747 (2021). pdf, and in arxiv
[15] (Alphabetic order) Jihong Wang, Zhi-Qin John Xu*, Jiwei Zhang*, Yaoyu Zhang, Implicit bias in understanding deep learning for solving PDEs beyond Ritz-Galerkin method, CSIAM Trans. Appl. Math. web, arxiv 2002.07989 (2020). pdf, and in arxiv
[14] (Alphabetic order) Tao Luo*, Zheng Ma, Zhiwei Wang, Zhi-Qin John Xu, Yaoyu Zhang, An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network, To appear in Mathematical and Scientific Machine Learning 2022 (MSML22), arxiv 2105.11675 (previous version: 2012.03238) (2020). pdf, and in arxiv
Note: [13] is a comprehensive version of [6].
[13] (Alphabetic order) Tao Luo*, Zheng Ma, Zhi-Qin John Xu, Yaoyu Zhang, On the exact computation of linear frequency principle dynamics and its generalization, SIAM Journal on Mathematics of Data Science (SIMODS), arxiv 2010.08153 (2020). in web, and in pdf, and in arxiv, some code is in github. Supplemental Material
[11] Zhi-Qin John Xu* , Hanxu Zhou, Deep frequency principle towards understanding why deeper learning is faster, Proceedings of the AAAI Conference on Artificial Intelligence 2021, arxiv 2007.14313 (2020) pdf, and in arxiv, and AAAI web, and slides, and AAAI speech script slides
[9] Ziqi Liu, Wei Cai, Zhi-Qin John Xu* , Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains, arxiv 2007.11207 (2020) Communications in Computational Physics (CiCP). pdf, and in web, some code is in github.
[7] (Alphabetic order) Tao Luo, Zheng Ma, Zhi-Qin John Xu, Yaoyu Zhang, Theory of the frequency principle for general deep neural networks, CSIAM Trans. Appl. Math., arXiv preprint, 1906.09235 (2019). arxiv, in web, see pdf
[6] Yaoyu Zhang, Zhi-Qin John Xu* , Tao Luo, Zheng Ma, Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks. arXiv preprint, 1905.10264 (2019) pdf, and in arxiv
[4] Zhi-Qin John Xu* , Yaoyu Zhang, Tao Luo, Yanyang Xiao, Zheng Ma, Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks, arXiv preprint: 1901.06523, Communications in Computational Physics (CiCP). pdf, and in web, some code is in github (2021世界人工智能大会青年优秀论文提名奖).
Note: Most of [2] and [3] are combined into paper [4].
[3] Zhi-Qin John Xu* , Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application, arXiv preprint: 1811.10146 (2018). pdf, and in arxiv
[2] Zhi-Qin John Xu* , Understanding training and generalization in deep learning by Fourier analysis, arXiv preprint: 1808.04295, (2018). pdf, and in arxiv
[1] Zhi-Qin John Xu* , Yaoyu Zhang, and Yanyang Xiao, Training behavior of deep neural network in frequency domain, arXiv preprint: 1807.01251, (2018), 26th International Conference on Neural Information Processing (ICONIP 2019). pdf, and in web