Physics-informed neural networks (PINNs) have been widely applied in solving various physical models, including partial differential equations (PDEs). However, they have a considerable probability of failure when simulating dynamical systems with multiscale, high-frequency, or chaotic behaviors. The possible reason might be that the majority of PINNs methodologies regard space and time as a unified entity, thereby neglecting temporal dependencies across previous or subsequent time steps. In this paper, an advanced network structure, referred to as AttnPINNs, is proposed by stacking self-attention blocks behind a pretrained PINN. We introduce a sequence operator to transform the inputs from spatio-temp oral points into sequential formats and a self-attention layer to capture the correlation of solution sequences. Furthermore, we provide a rigorous proof of convergence, indicating that merely the introduction of self-attention blocks is sufficient to bring about significant performance improvements by comparing the AttnPINNs with other advanced architectures (e.g., PINNs, QRes, first-layer sine, and PINNsFormer). Meanwhile, the numerical experiment results show that AttnPINNs method demonstrates superior performance and outperforms most of the other strategies on a wide range of PDEs, the solutions of which tend to have abrupt changes or exhibit multiscale, high-frequency, and chaotic properties.

Publication:

SIAM JOURNAL ON SCIENTIFIC COMPUTING

http://dx.doi.org/10.1137/24M1706670

Author:

JIN SONG

School of Advanced Interdisciplinary Science, University of Chinese Academy of Sciences, Beijing, 100190 China

State Key Laboratory of Mathematical Sciences, Academy of Mathematics andSystems Science, Chinese Academy of Sciences, Beijing, 100190, China

e-mail:songjin21@mails.ucas.ac.cn

ZHENYA YAN

Corresponding author

School of Mathematics and Information Science, Zhongyuan Universityof Technology, Zhengzhou, 450007 China

State Key Laboratory of Mathematical Sciences, Academyof Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China

School ofMathematical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China

e-mail:zyyan@mmrc.iss.ac.cn