Title: One never walks alone: the effect of the perturber population on subhalo measurements in strong gravitational lenses

Abstract: Analyses of extended arcs in strong gravitational lensing images to date have constrained the properties of dark matter by measuring the parameters of one or two individual subhalos. However, since such analyses are reliant on likelihood-based methods like Markov-chain Monte Carlo or nested sampling, they require various compromises to the realism of lensing models for the sake of computational tractability, such as ignoring the numerous other subhalos and line-of-sight halos in the system, assuming a particular form for the source model and requiring the noise to have a known likelihood function. Here we show that a simulation-based inference method called truncated marginal neural ratio estimation (TMNRE) makes it possible to relax these requirements by training neural networks to directly compute marginal posteriors for subhalo parameters from lensing images. By performing a set of inference tasks on mock data, we verify the accuracy of TMNRE and show it can compute posteriors for subhalo parameters marginalized over populations of hundreds of subhalos and line-of-sight halos, as well as lens and source uncertainties. We also find the MLP Mixer network works far better for such tasks than the convolutional architectures explored in other lensing analyses. Furthermore, we show that since TMNRE learns a posterior function it enables direct statistical checks that would be extremely expensive with likelihood-based methods. Our results show that TMNRE is well-suited for analyzing complex lensing data, and that the full subhalo and line-of-sight halo population must be included when measuring the properties of individual dark matter substructures.