Multi-site recordings are a common practice in modern neuroscience. Inferring how the different structures interact allows us to build models of representation in the brain. Nevertheless, multi-site recordings are often confronted with challenges: (1) the data comprise a large number of time series, (2) not all sites can be simultaneously observed, and (3) it is sometimes impossible to align signals to a common time reference. In these cases, recording the relevant sites in separate sessions or using different animals is the only option. We ask whether it is possible to infer the information flow between these sites even when some observations are statistically independent and a subset of the sites are never observed simultaneously. Moreover, given that parametric models may not always be suitable, there is a demand for a non-parametric inference procedure. We demonstrate that a regularized maximal prediction error algorithm combined with canonical spectral factorization can effectively reconstruct the information flow network under realistic conditions. We validate this approach through applications on simulated and empirical neural data.