We present the methodology for a joint cosmological analysis of weak gravitational lensing from the fourth data release of the ESO Kilo-Degree Survey (KiDS-1000) and galaxy clustering from the partially overlapping BOSS and 2dFLenS surveys. Cross-correlations between galaxy positions and ellipticities are incorporated into the analysis, developing a hybrid model of non-linear scales that blends perturbative and non-perturbative approaches, and assessing signal contributions by astrophysical effects. All weak lensing signals are measured consistently via Fourier-space statistics that are insensitive to the survey mask and display low levels of mode mixing. The calibration of photometric redshift distributions and multiplicative gravitational shear bias is updated, and a more complete tally of residual calibration uncertainties is propagated into the likelihood. A dedicated suite of more than 20000 mocks is used to assess the performance of covariance models and to quantify the impact of survey geometry and spatial variations of survey depth on signals and their errors. The sampling distributions for the likelihood and the $\chi^2$ goodness-of-fit statistic are validated, with proposed changes to calculating the effective number of degrees of freedom. Standard weak lensing point estimates on $S_8=\sigma_8\,(\Omega_{\rm m}/0.3)^{1/2}$ derived from its marginal posterior are easily misinterpreted to be biased low, and an alternative estimator and associated credible interval are proposed. Known systematic effects pertaining to weak lensing modelling and inference are shown to bias $S_8$ by no more than 0.1 standard deviations, with the caveat that no conclusive validation data exist for models of intrinsic galaxy alignments. Compared to the previous KiDS analyses, $S_8$ constraints are expected to improve by 20% for weak lensing alone and by 29% for the joint analysis. [abridged]

We present weak lensing shear catalogues from the fourth data release of the Kilo-Degree Survey, KiDS-1000, spanning 1006 square degrees of deep and high-resolution imaging. Our `gold-sample' of galaxies, with well calibrated photometric redshift distributions, consists of 21 million galaxies with an effective number density of $\sim 6.2$ galaxies per square arcminute. We quantify the accuracy of the spatial, temporal and flux-dependent point-spread function (PSF) model, verifying that the model meets our requirements to induce less than a $0.1\sigma$ change in the inferred cosmic shear constraints on the clustering cosmological parameter $S_8 = \sigma_8\sqrt{\Omega_{\rm m}/0.3}$. Through a series of two-point null-tests we validate the shear estimates, finding no evidence for significant non-lensing B-mode distortions in the data. PSF residuals are detected in the highest-redshift bins, originating from object selection and/or weight bias. The amplitude is however shown to be sufficiently low and within our stringent requirements. With a shear-ratio null-test we verify the expected redshift scaling of the galaxy-galaxy lensing signal around luminous red galaxies. We conclude that the joint KiDS-1000 shear and photometric redshift calibration is sufficiently robust for combined-probe gravitational lensing and spectroscopic clustering analyses.

We simultaneously present constraints on the stellar-to-halo mass relation for central and satellite galaxies through a weak lensing analysis of spectroscopically classified galaxies. Using overlapping data from the 4th data release of the Kilo-Degree Survey (KiDS), and the Galaxy And Mass Assembly survey (GAMA), we find that satellite galaxies are hosted by halo masses that are $0.53 \pm 0.39$ dex (68% confidence, $3\sigma$ detection) smaller than those of central galaxies of the same stellar mass (for a stellar mass of $\log(M_{\star}/M_{\odot}) = 10.6$). This is consistent with galaxy formation models, whereby infalling satellite galaxies are preferentially stripped of their dark matter. We find consistent results with similar uncertainties when comparing constraints from a standard azimuthally averaged galaxy-galaxy lensing analysis and a two-dimensional likelihood analysis of the full shear field. As the latter approach is somewhat biased due to the lens incompleteness, and does not provide any improvement in precision when applied to actual data, we conclude that stacked tangential shear measurements are best-suited for studies of the galaxy-halo connection.

We report new high-quality galaxy scale strong lens candidates found in the Kilo Degree Survey data release 4 using Machine Learning. We have developed a new Convolutional Neural Network (CNN) classifier to search for gravitational arcs, following the prescription by \cite{2019MNRAS.484.3879P} and using only $r-$band images. We have applied the CNN to two "predictive samples": a Luminous red galaxy (LRG) and a "bright galaxy" (BG) sample ($r<21$). We have found 286 new high probability candidates, 133 from the LRG sample and 153 from the BG sample. We have then ranked these candidates based on a value that combines the CNN likelihood to be a lens and the human score resulting from visual inspection (P-value) and we present here the highest 82 ranked candidates with P-values $\ge 0.5$. All these high-quality candidates have obvious arc or point-like features around the central red defector. Moreover, we define the best 26 objects, all with scores P-values $\ge 0.7$ as a "golden sample" of candidates. This sample is expected to contain very few false positives and thus it is suitable for follow-up observations. The new lens candidates come partially from the the more extended footprint adopted here with respect to the previous analyses, partially from a larger predictive sample (also including the BG sample). These results show that machine learning tools are very promising to find strong lenses in large surveys and more candidates that can be found by enlarging the predictive samples beyond the standard assumption of LRGs. In the future, we plan to apply our CNN to the data from next-generation surveys such as the Large Synoptic Survey Telescope, Euclid, and the Chinese Space Station Optical Survey.

The physics of gravity on cosmological scales affects both the rate of assembly of large-scale structure, and the gravitational lensing of background light through this cosmic web. By comparing the amplitude of these different observational signatures, we can construct tests that can distinguish General Relativity from its potential modifications. We use the latest weak gravitational lensing dataset from the Kilo-Degree Survey, KiDS-1000, in conjunction with overlapping galaxy spectroscopic redshift surveys BOSS and 2dFLenS, to perform the most precise existing amplitude-ratio test. We measure the associated E_G statistic with 15-20% errors, in five dz = 0.1 tomographic redshift bins in the range 0.2 < z < 0.7, on projected scales up to 100 Mpc/h. The scale-independence and redshift-dependence of these measurements are consistent with the theoretical expectation of General Relativity in a Universe with matter density Omega_m = 0.27 +/- 0.04. We demonstrate that our results are robust against different analysis choices, including schemes for correcting the effects of source photometric redshift errors, and compare the performance of angular and projected galaxy-galaxy lensing statistics.

We present a tomographic cosmic shear analysis of the Kilo-Degree Survey (KiDS) combined with the VISTA Kilo-Degree Infrared Galaxy Survey (VIKING). This is the first time that a full optical to near-infrared data set has been used for a wide-field cosmological weak lensing experiment. This unprecedented data, spanning $450~$deg$^2$, allows us to improve significantly the estimation of photometric redshifts, such that we are able to include robustly higher-redshift sources for the lensing measurement, and - most importantly - solidify our knowledge of the redshift distributions of the sources. Based on a flat $\Lambda$CDM model we find $S_8\equiv\sigma_8\sqrt{\Omega_{\rm m}/0.3}=0.737_{-0.036}^{+0.040}$ in a blind analysis from cosmic shear alone. The tension between KiDS cosmic shear and the Planck-Legacy CMB measurements remains in this systematically more robust analysis, with $S_8$ differing by $2.3\sigma$. This result is insensitive to changes in the priors on nuisance parameters for intrinsic alignment, baryon feedback, and neutrino mass. KiDS shear measurements are calibrated with a new, more realistic set of image simulations and no significant B-modes are detected in the survey, indicating that systematic errors are under control. When calibrating our redshift distributions by assuming the 30-band COSMOS-2015 photometric redshifts are correct (following the Dark Energy Survey and the Hyper Suprime-Cam Survey), we find the tension with Planck is alleviated. The robust determination of source redshift distributions remains one of the most challenging aspects for future cosmic shear surveys.

We present here the cosmo-SLICS, a new suite of simulations specially designed for the analysis of current and upcoming weak lensing data beyond the standard two-point cosmic shear. We sample the $[\Omega_{\rm m}, \sigma_8, h, w_0]$ parameter space at 25 points organised in a Latin hyper-cube, spanning a range that contains most of the $2\sigma$ posterior distribution from ongoing lensing surveys. At each of these nodes we evolve a pair of $N$-body simulations in which the sampling variance is highly suppressed, and ray-trace the volumes 800 times to further increase the effective sky coverage. We extract a lensing covariance matrix from these pseudo-independent light-cones and show that it closely matches a brute-force construction based on an ensemble of 800 truly independent $N$-body runs. More precisely, a Fisher analysis reveals that both methods yield marginalized two-dimensional constraints that vary by less than 6% in area, a result that holds under different survey specifications and that matches to within 15% the area obtained from an analytical covariance calculation. Extending this comparison with our 25 $w$CDM models, we probe the cosmology dependence of the lensing covariance directly from numerical simulations, reproducing remarkably well the Fisher results from the analytical models at most cosmologies. We demonstrate that varying the cosmology at which the covariance matrix is evaluated in the first place might have an order of magnitude greater impact on the parameter constraints than varying the choice of covariance estimation technique. We present a test case in which we generate fast predictions for both the lensing signal and its associated variance with a flexible Gaussian process regression emulator, achieving an accuracy of a few percent on the former and 10% on the latter.

We introduce an emulator approach to predict the non-linear matter power spectrum for broad classes of beyond-$\Lambda$CDM cosmologies, using only a suite of $\Lambda$CDM $N$-body simulations. By including a range of suitably modified initial conditions in the simulations, and rescaling the resulting emulator predictions with analytical `halo model reactions', accurate non-linear matter power spectra for general extensions to the standard $\Lambda$CDM model can be calculated. We optimise the emulator design by substituting the simulation suite with non-linear predictions from the standard {\sc halofit} tool. We review the performance of the emulator for artificially generated departures from the standard cosmology as well as for theoretically motivated models, such as $f (R)$ gravity and massive neutrinos. For the majority of cosmologies we have tested, the emulator can reproduce the matter power spectrum with errors $\lesssim 1\%$ deep into the highly non-linear regime. This work demonstrates that with a well-designed suite of $\Lambda$CDM simulations, extensions to the standard cosmological model can be tested in the non-linear regime without any reliance on expensive beyond-$\Lambda$CDM simulations.

Intrinsic galaxy alignments are a source of bias for weak lensing measurements as well as a tool for understanding galaxy formation and evolution. In this work, we measure the alignment of shapes of satellite galaxies, in galaxy groups, with respect to the brightest group galaxy (BGG), as well as alignments of the BGG shape with the satellite positions, using the highly complete Galaxy And Mass Assembly (GAMA) spectroscopic survey and deep imaging from the Kilo Degree Survey. We control systematic errors with dedicated image simulations and measure accurate shapes using the DEIMOS shape measurement method. We find a significant satellite radial alignment signal, which vanishes at large separations from the BGG. We do not identify any strong trends of the signal with galaxy absolute magnitude or group mass. The alignment signal is dominated by red satellites. We also find that the outer regions of galaxies are aligned more strongly than their inner regions, by varying the radial weight employed during the shape measurement process. This behaviour is evident for both red and blue satellites. BGGs are also found to be aligned with satellite positions, with this alignment being stronger when considering the innermost satellites, using red BGGs and the shape of the outer region of the BGG. Lastly, we measure the global intrinsic alignment signal in the GAMA sample for two different radial weight functions and find no significant difference.

The Kilo-Degree Survey (KiDS) is an ongoing optical wide-field imaging survey with the OmegaCAM camera at the VLT Survey Telescope, specifically designed for measuring weak gravitational lensing by galaxies and large-scale structure. When completed it will consist of 1350 square degrees imaged in four filters (ugri). Here we present the fourth public data release which more than doubles the area of sky covered by data release 3. We also include aperture-matched ZYJHKs photometry from our partner VIKING survey on the VISTA telescope in the photometry catalogue. We illustrate the data quality and describe the catalogue content. Two dedicated pipelines are used for the production of the optical data. The Astro-WISE information system is used for the production of co-added images in the four survey bands, while a separate reduction of the r-band images using the theli pipeline is used to provide a source catalogue suitable for the core weak lensing science case. All data have been re-reduced for this data release using the latest versions of the pipelines. The VIKING photometry is obtained as forced photometry on the theli sources, using a re-reduction of the VIKING data that starts from the VISTA pawprints. Modifications to the pipelines with respect to earlier releases are described in detail. The photometry is calibrated to the Gaia DR2 G band using stellar locus regression. In this data release a total of 1006 square-degree survey tiles with stacked ugri images are made available, accompanied by weight maps, masks, and single-band source lists. We also provide a multi-band catalogue based on r-band detections, including homogenized photometry and photometric redshifts, for the whole dataset. Mean limiting magnitudes (5 sigma in a 2" aperture) are 24.23, 25.12, 25.02, 23.68 in ugri, respectively, and the mean r-band seeing is 0.70".

We present a new sample of galaxy-scale strong gravitational-lens candidates, selected from 904 square degrees of Data Release 4 of the Kilo-Degree Survey (KiDS), i.e., the "Lenses in the Kilo-Degree Survey" (LinKS) sample. We apply two Convolutional Neural Networks (ConvNets) to $\sim88\,000$ colour-magnitude selected luminous red galaxies yielding a list of 3500 strong-lens candidates. This list is further down-selected via human inspection. The resulting LinKS sample is composed of 1983 rank-ordered targets classified as "potential lens candidates" by at least one inspector. Of these, a high-grade subsample of 89 targets is identified with potential strong lenses by all inspectors. Additionally, we present a collection of another 200 strong lens candidates discovered serendipitously from various previous ConvNet runs. A straightforward application of our procedure to future Euclid or LSST data can select a sample of $\sim3000$ lens candidates with less than 10 per cent expected false positives and requiring minimal human intervention.

We present a public suite of weak lensing mock data, extending the Scinet Light Cone Simulations (SLICS) to simulate cross-correlation analyses with different cosmological probes. These mocks include KiDS-450- and LSST-like lensing data, cosmic microwave background lensing maps and simulated spectroscopic surveys that emulate the GAMA, BOSS and 2dFLenS galaxy surveys. With 844 independent realisations, our mocks are optimised for combined-probe covariance estimation, which we illustrate for the case of a joint measurement involving cosmic shear, galaxy-galaxy lensing and galaxy clustering from KiDS-450 and BOSS data. With their high spatial resolution, the SLICS are also optimal for predicting the signal for novel lensing estimators, for the validation of analysis pipelines, and for testing a range of systematic effects such as the impact of neighbour-exclusion bias on the measured tomographic cosmic shear signal. For surveys like KiDS and DES, where the rejection of neighbouring galaxies occurs within ~2 arcseconds, we show that the measured cosmic shear signal will be biased low, but by less than a percent on the angular scales that are typically used in cosmic shear analyses. The amplitude of the neighbour-exclusion bias doubles in deeper, LSST-like data. The simulation products described in this paper are made available at http://slics.roe.ac.uk/.

We present the first "clipped" cosmic shear measurement using data from the Kilo-Degree Survey (KiDS-450). "Clipping" transformations suppress the signal from the highest density, non-linear regions of cosmological fields. We demonstrate that these transformations improve constraints on $S_8=\sigma_8(\Omega_{\rm{m}}/0.3)^{0.5}$ when used in combination with conventional two-point statistics. For the KiDS-450 data, we find that the combined measurements improve the constraints on $S_8$ by 17%, compared to shear correlation functions alone. We determine the expectation value of the clipped shear correlation function using a suite of numerical simulations, and develop methodology to mitigate the impact of masking and shot noise. Future improvements in numerical simulations and mass reconstruction methodology will permit the precise calibration of clipped cosmic shear statistics such that clipping can become a standard tool in weak lensing analyses.

We present an intrinsic AGN SED extending from the optical to the submm, derived with a sample of unobscured, optically luminous (vLv(5100)>10^43.5 erg/s) QSOs at z<0.18 from the Palomar Green survey. The intrinsic AGN SED was computed by removing the contribution from stars using the 11.3um polycyclic aromatic hydrocarbon (PAH) feature in the QSOs' mid-IR spectra; the 1sigma uncertainty on the SED ranges between 12 and 45 per cent as a function of wavelength and is a combination of PAH flux measurement errors and the uncertainties related to the conversion between PAH luminosity and star-forming luminosity. Longwards of 20um the shape of the intrinsic AGN SED is independent of the AGN power indicating that our template should be applicable to all systems hosting luminous AGN (vLv(5100) or L_X(2-10keV) > 10^43.5 erg/s). We note that for our sample of luminous QSOs, the average AGN emission is at least as high as, and mostly higher than, the total stellar powered emission at all wavelengths from the optical to the submm. This implies that in many galaxies hosting powerful AGN, there is no `safe' broadband photometric observation (at lambda<1000um) which can be used in calculating star-formation rates without subtracting the AGN contribution. Roughly, the AGN contribution may be ignored only if the intrinsic AGN luminosity at 5100 Ang is at least a factor of 4 smaller than the total infrared luminosity (L_IR; 8-1000um) of the galaxy. Finally, we examine the implication of our work in statistical studies of star-formation in AGN host galaxies.