The polynomial chaos expansion is widely used as a surrogate model in the Bayesian inference to speed up the Markov chain Monte Carlo calculations. However, the use of such a surrogate introduces modeling errors that may severely distort the estimate of the posterior distribution. In this talk, we present an adaptive procedure to construct a multi-fidelity polynomial surrogate. More precisely, the new strategy starts with a low-fidelity surrogate model, and this surrogate will be adaptively corrected using online high-fidelity data. The key idea is to construct and refine the multi-fidelity surrogate over a sequence of samples adaptively determined from data so that the approximation can eventually concentrate to the posterior distribution. We also introduce a multi-fidelity surrogate based on the deep neural networks to deal with problems with high dimensional parameters. The performance of the proposed strategy is illustrated through two nonlinear inverse problems.