HARVESTER
Contents
Data products
Data flow
Sky subtraction
The ARRAKIHS Harvester is the dedicated data reduction pipeline of the ARRAKIHS mission, designed specifically to preserve and accurately recover extremely low surface brightness (LSB) signals that are typically lost in standard astronomical processing. LSB science requires exceptional control of sky modelling, background subtraction, and instrumental systematics. Even small residual errors can remove or distort the faint structures that trace galaxy formation, accretion processes, and diffuse stellar halos. The Harvester is therefore optimised to maintain these faint signals throughout the full processing chain.
Data Products
The Harvester processes the following steps of the ARRAKIHS data flow, from single exposures to final science products:
- Level 1 (L1): FITS formatted single exposures of science and calibration targets (input, similar format to ATREIDS outputs).
- Level 2 (L2): Cleaned and calibrated science exposures.
- Level 3 (L3): Deep co-added images per target.
- Level 4 (L4): Scientific catalogs and derived products.
- In-flight reference files: Processed calibration exposures used in the creation of L2.
Starting from L1, the pipeline generates in-flight reference files and calibrated science exposures (L2) products through a full calibration sequence including bad pixel masking, bias and dark subtraction, flat-fielding, non-linearity corrections, cosmic ray and satellite trail removal, astrometric refinement, and photometric calibration. A key aspect of the processing is the treatment of extended point spread functions (PSF) and optical artefacts such as internal reflections (ghosts), which are modelled and corrected to avoid contamination of faint diffuse emission. Deep coadds (L3) are built for every filter and target by co-adding all
L2 exposures of the same filter and target, with different warping and coadding strategies available (e.g. PSF-matched or optimised for low surface brightness recovery). Finally, L4 products consist of scientific catalogues and derived measurements.
Besides the data products derived by Harvester,the AMC science team will produce the high level data product known as (Level 5), which will remove unresolved background sources from the coadds of Level 3.
Sky subtraction for LSB science
A central goal of the Harvester is the robust recovery of extremely faint, extended emission. This requires a careful separation between instrumental background and true astrophysical signal.
Sky modelling and subtraction are therefore implemented with particular care to avoid over-subtraction, which can artificially remove diffuse structures such as galaxy halos or tidal streams. The pipeline is designed to minimise these effects by combining instrument modelling, masking strategies, and large-scale background estimation techniques.
The resulting processing strategy is continuously validated using simulations and representative observational datasets, ensuring consistency and stability across different observing conditions.
Software framework
The ARRAKIHS Harvester is developed by a team at CEFCA, under the supervision of M.Akhlaghi. It follows a modular and reproducible design, allowing individual stages to be executed independently or as part of full end-to-end processing. The Harvester’s public coding standards are listed below.
- Upon each data release, the ARRAKIHS Harvester code will also be released as free and open source software under the GNU General Public License (GNU GPL) v3 license.
- We need to impose minimal technical debt on the mission. In other words, if an operation is possible with existing software dependencies, new dependencies will not be added just because of a different interface. The data reduction team will help in using the existing dependencies for new contributors. If an operation/function does not yet exist in existing dependencies they will be added only if the software has a free and open source license compatible with the Harvester (GNU GPL v3), is stable enough and does not have any further dependencies (dependencies of the dependency). Otherwise, the feature will be added to the core Harvester dependencies which we maintain (listed below). The Harvester primarily uses the following two software packages (both maintained by the data reduction team and are in active development for more than 10 years).
- The Harvester’s software environment and job manager is Maneage (see Akhlaghi et al. 2021). Maneage is able to build itself on any Unix-like operating system (mostly tested on GNU/Linux and macOS) that only has a C compiler. Maneage’d projects like the Harvester can also run inside a container (Apptainer or Docker) to avoid the need for building from source. It has also been customized for High Performance Computing (HPC) systems through Slurm and High Throughput Computing (HTC) systems through HTCondor. Maneage is written in GNU Make (a job organizer managing dependencies between steps of a pipeline) and GNU Bash (for high-level interface as well as the Make recipes).
- The Harvester’s number-cruncher, in charge of atomic operations in each step with one/some input file(s) and one output is the GNU Astronomy Utilities (Gnuastro; see Akhlaghi et al. 2025). Gnuastro is written in C and only has three dependencies which have no dependencies themselves. C is critical for us due to its efficiency, longevity, performance and portability (see Section 13.1 of Akhlaghi et al. 2025). Gnuastro has two interfaces for general usability: a collection of command-line programs (Section 1.2 of Akhlaghi et al. 2025) and a C library (Section 12.3 of Akhlaghi et al. 2025), all accompanied by a very complete documentation. The Harvester will primarily be using the command-line programs of Gnuastro.
- Low-level coding of the Harvester will follow the Gnuastro coding conventions.