Accessing L1 (raw), L2 (observations and mosaics), and LE3 products through the command line, with the Data Download (sas-dd) service

Once authenticated, this service allow the user to download data products from different levels and data releases. The Search - Observations option allows to download L1 (raw) and L2 (observations and mosaics) using the Euclid web interface, but LE3 products can only be downloaded from the command line. As reference here is the list of possible parameters to ask for the different products, followed below by some examples.

  • To retrieve Observations use RETRIEVAL_TYPE=OBSERVATION, followed by the next parameters.

Parameter

Description

Mandatory

Possible values

OBS_ID

Describes the observation identifier or a list of them (column observation_id)

YES

NA

TYPE

Describes the type of data products within the following valid values or a list of them

NO

  • STK for stacked images

  • DET for single detections (calibrated frames)

  • PSF for auxiliary PSF files

  • BKG for auxiliary BKG files

  • WGT for auxiliary WGT files

  • ALL (will return all, default value)

INSTRUMENT

Describes the instrument. By default all instruments are considered

NO

  • VIS

  • NISP

FILTER

Describes the filter within the following valid values or a list of them.

By default the products for all filters will be downloaded

NO

  • VIS

  • NIR_J

  • NIR_Y

  • NIR_H

RELEASE

Identifier of the Data Release

NO

  • sedm_pvpr01 (default value)

  • sedm_sc8

  • To retrieve Mosaics use RETRIEVAL_TYPE=MOSAIC, followed by the TYPE parameter and those listed in any of the three options.

    • TYPE: describes the type of data products within the following valid values or a list of them: STK, BKG, FLAG, PSF, GRID_PSF, RMS, and ALL.

    Option 1: To retrieve Mosaic data products by Tile Index and Instrument use the following parameters.

    • TILE_INDEX:: describes the Tile identifier or a list of them.

    • INSTRUMENT and FILTER: describes the instrument and filter/s.

    Option 2: To retrieve Mosaic data products by unique identifier use the following parameter:

    • MSC_ID: describes the mosaic identifier or a list of them.

Parameter

Description

Mandatory

Possible values

TYPE

Describes the type of data products within the following valid values or a list of them

YES

  • STK for stacked frames

  • PSF for aux PSF files

  • GRID_PSF for aux GRID PSF files

  • BKG for BKG files

  • FLAG

  • RMS

  • ALL

RELEASE

Identifier of the Data Release

NO

  • sedm_pvpr01 (default)

  • sedm_sc8

TILE_INDEX

Describes the Tile identifier or a list of them

YES (Option 1)

NA

INSTRUMENT

Describes the instruments within the following valid values or a list of them

YES (1)

  • VIS

  • NISP

FILTER

Describes the filter within the following valid values

YES (1)

  • VIS

  • NIR_J

  • NIR_Y

  • NIR_H

MSC_ID

Describes the mosaic identifier or a list of them (column mosaic_oid)

YES (2)

NA

  • To retrieve LE3 use RETRIEVAL_TYPE=LE3, followed by the next parameter.

Parameter

Description

Mandatory

Possible values

NAME

Product name to identify the .tar.gz file to download.

Both parameter name and values are not case sensitive.

NO

without it all 4 products will be

downloaded at once (several GB)

  • InternalDP (internal data products)

  • ClusterDP (cluster of galaxies)

  • WLDP (weak lensing)

  • GCDP (galaxy clustering)

  • It is possible to download products of any kind knowing their file names or product Ids. See section 4 below for examples.

Parameter

Description

Mandatory

Possible values

FILE_NAME

File name/s of the fits file/s to download.

If more than one, list them between a comma.

NO

NA

PRODUCT_ID

Product Id/s of the fits file/s to download.

If more than one, list them between a comma.

NO

NA

  • To retrieve Frame Catalogs use RETRIEVAL_TYPE=FRAME_CATALOG, followed by the next parameters. Note in this case the release parameter is mandatory. See section 5 below for examples.

Parameter

Description

Mandatory

Possible values

OBS_ID

Describes the observation identifier or a list of them (column observation_id)

YES

NA

FILTER

Describes the filter used within the following valid values or a list of them

NO

  • VIS

  • NIR_J

  • NIR_Y

  • NIR_H

RELEASE

Identifier of the Data Release

YES

  • sedm_pvpr01

  • sedm_sc8

Login to sas-dd

The sas-dd service requires authentication from the user.

Login

curl -k -c cookies.txt -X POST -d username=USERNAME -d password=PASSWORD -L "https://eas.esac.esa.int/sas-dd/login"

Logout

curl -k -b cookies.txt -X POST -d -L "https://eas.esac.esa.int/sas-dd/logout"

Observations and Mosaics retrieval examples

The following examples will show the different options to retrieve Euclid data products through the command line. The links can also be copied to a browser once the authentication has been done on it.

Retrieve data products by Observation Id "13", Instrument "VIS" and Type "STK" from release sedm_pvpr01

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=OBSERVATION&obs_id=13&INSTRUMENT=VIS&type=STK&RELEASE=sedm_pvpr01"

Retrieve data products by Observation Id {"10351", "10354", "10355"}, Instrument "VIS" and Type "STK from release SC_8"

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?RETRIEVAL_TYPE=OBSERVATION&OBS_ID=10351%2C10354%2C10355&INSTRUMENT=VIS&TYPE=STK&RELEASE=SC_8"

The %2C is a way to encode the comma character, but generally the comma can be used directly as well.

Retrieve data products by Observation Id {"10351", "10354", "10355"}, Instruments {"VIR","NIR_H","NIR_J"} and Type "STK" from SC_8

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?RETRIEVAL_TYPE=OBSERVATION&RETRIEVAL_ACCESS=DIRECT&OBS_ID=10351%2C10354%2C10355&INSTRUMENT=VIS%2CNIR_H%2CNIR_J&TYPE=STK&RELEASE=SC_8"

Retrieve aux data products by Observation Id "13", Instrument "VIS" and Type "PSF" from sedm_pvpr01

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=OBSERVATION&RETRIEVAL_ACCESS=DIRECT&obs_id=13&INSTRUMENT=VIS&type=PSF&RELEASE=sedm_pvpr01"

Retrieve mosaics Mosaic ID "2", Instrument "VIS" from sedm_pvpr01

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?&MSC_ID=2&RELEASE=sedm_pvpr01&RETRIEVAL_TYPE=MOSAIC&INSTRUMENT=VIS"

Retrieve mosaics by: list of Tile Index list {"40002", "40003"} and Instrument "NISP" from SC_8

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?&TILE_INDEX=40003,40002&RELEASE=SC_8&RETRIEVAL_TYPE=MOSAIC&INSTRUMENT=NISP"

Retrieve mosaics by: list of Tile Index "40003", Instrument "NISP" and TYPE "RMS" from SC_8

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?&TILE_INDEX=40003&RELEASE=SC_8&RETRIEVAL_TYPE=MOSAIC&INSTRUMENT=NISP&TYPE=RMS"

LE3 data products retrieval examples

The following examples will show the different options to retrieve LE3 data products. In each subsection you can clic on the product names to access some external documentation.

Retrieve data products for internal products

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=LE3&name=InternalDP"

Retrieve data products for galaxy clusters products

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=LE3&name=ClusterDP"

Retrieve data products for weak lensing products

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=LE3&name=WLDP"

Retrieve data products for galaxy clustering products

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=LE3&name=GCDP"

Retrieve all data products at once in a zip file

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=LE3"

Retrieving data products based on file name of product Id

The following examples will show the different options to retrieve any kind of data products if the file name or product Id values are known. Note the parameter RETRIEVAL_TYPE is still required, although the value provided is ignored in case FILE_NAME or PRODUCT_ID are provided. Multiple files can be downloaded at once, as a zip file, by listing them between comma. The parameter RELEASE is mandatory to identify in which data release the file name or product should be searched.

Retrieve one mosaic knowing its file name

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?FILE_NAME=EUC_MER_BGSUB-MOSAIC-NIR-H_TILE100159190-7F8E05_20220919T214307.064518Z_00.00.fits&RETRIEVAL_TYPE=FILE&RELEASE=sedm_pvpr01"

Retrieve two or more data products (listed between comma) of different types, knowing their product Ids

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?PRODUCT_ID=NIR_ProcessField_SelfCal-2.1.1-9-ON_THE_FLY-pcasenov-PLAN-000001-4Q5NYURF-20220915-080534-001-stk_image_xml-0,SIMPV008221026311-9GU5X5Z8-20221026-103547-001-outputSIM-9&RETRIEVAL_TYPE=PRODUCT_ID&RELEASE=sedm_pvpr01"

Retrieve products by file name and product Id at once

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?FILE_NAME=EUC_SIM_NISRGS180-8-1_20220722T094150.427Z_PV023_NISP-S_8_18_0.fits&PRODUCT_ID=MER_ProcessTile_SelfCal_9.0.1-4-ON_THE_FLY-pcasenov-PLAN-000000-G4UOE2QX-20220919-174958-001-all_mosaics-2&RETRIEVAL_TYPE=ID&RELEASE=sedm_pvpr01"

Frame catalog retrieval examples

curl -k -b cookies.txt -X GET "https://eas.esac.esa.int/sas-dd/data?retrieval_type=FRAME_CATALOG&obs_id=13&FILTER=VIS&RELEASE=sedm_pvpr01"