make printable	New:	see also:
	v2.2: - CERTIF_PGI (for optional certification pgi, for MUSE) - new option -n for step -M (no cleanup) v2.3: - supports dual-instance processing (MASTER/SLAVE)	databases none dfos tools calling createAB, createJob, processAB, processQC, moveProducts, ingestProducts output pipeline products, QC plots upload/download down: ABs from qc@qcweb up: logs and plots to qc@qcweb
topics: description | workflow | technical implementation | log | output | usage | OCA and data pool | release | QC procedures | history scores, qualityChecker, autoCertifier | Job execution | QC job concentration | ingestion | installation | config | PGI's | workflow | data directories | qcweb | notification | DON'T

Description and process

While the supervised, quality-checked, PI-package oriented processing of science data was terminated in October 2011, an unsupervised (automatic) processing, without quality statement and aiming at the general scientific community, was proposed by R. Hanuschik in 2012. It was implemented in May 2013 on muc08, for the UVES IDP processing being the pathfinder in a series of similar projects. IDP stands for 'Internal Data Products' and is the acronym for science data products produced in-house (i.e. by QC), in contrast to the science data products produced externally by PIs (EDPs).

The phoenix tool is the workflow tool to support the automatic processing of large homogeneous science datasets.

With version 2.x, phoenix can also be used to create huge homogeneous batches of master calibrations. This could be necessary e.g. for historical periods of an instruments when the pipeline was not yet mature, or when no historical master calibrations were stored in the archive, or when the existing master calibrations are not compatible with the current pipeline. The main goal remains - creation of science-grade data products, hence a master calibration phoenix project is normally always followed by an IDP project.

The goal of the PHOENIX process is:

• to provide science-grade products to the general archive community;
• to offer them as internal data products (IDPs) via dedicated archive interfaces.

The processing is based on

• certified and ingested master calibrations which went through the classical QC processing, scoring and certification workflow
• existing associations (stored on the qcweb QC server and as calSelector static associations)
• carefully reviewed and certified pipelines (by SDP and SDD).

This means in turn that science processing "the phoenix way" is only possible if

• a mature and stable pipeline exists, certified by SDG to deliver science-grade products
• all required master calibrations are available in the archive
• all required ABs are available from a central resource.

The benefits of phoenix processing, in comparison to the existing science products from the PI packages provided by QC in the past (the "S. products"), are:

• the homogeneous processing with a mature and reviewed pipeline (while all historical attempts were always done with the latest available pipeline and therefore have been naturally of inhomogeneous quality)
• the earlier processing was in many cases less complete (e.g. SM only, standard modes or setups only)
• now there are probably better processing parameters known, or available.

phoenix and autoDaily in comparison:

Workflow description (see also 'Operational workflow' here)

The IDP process has two (for MUSE: three) components:

• pre-processing: scan the ABs, redefine input datasets (MUSE only);
• the processing component, covered by phoenix;
• the post-processing ingestion component which in the case of IDPs is covered by DFS tools like idp2sdp (the conversion tool for UVES) and the ingestionTool (both plugged in the standard DFOS ingestProducts), or, in the case of MCALIBs, by the DFOS tool ingestProducts directly (see below).

The phoenix (re)processing workflow has many similarities with the autoDaily workflow.

A) For IDPs it has the following components:

While this is the basic processing step, the typical processing pattern is by month (looping over all dates).

B) For MCALIBs, it has the following components:

A flow diagram may also help to understand the processing workflow. Here is a sketch that combines the IDP ("SCIENCE") branch (supported by phoenix v1.x)) and the MCALIB branch, available with phoenix v2.x [click for full-size]:

Technical implementation

A) Workspace.

B) Data directories.

C) Tools.

For the pipeline, you need to check $HOME/.esorex/esorex.rc to contain the proper pipeline version in the key esorex.caller.recipe-dir.

The PHOENIX process requires a tool set which consists of a stripped-off version of the DFOS tools, and of some tools dedicated to the PHOENIX workflow. The most important key in the .dfosrc file of a PHOENIX installation is the key THIS_IS_PHOENIX which is set to YES for a PHOENIX installation, and to NO (or does not exist) for a DFOS installation. For a PHOENIX MCALIB installation, you also need the key THIS_IS_MCAL set to YES (in order to tell the cleanupProducts tool to behave like in a normal DFOS installation).

The installation is monitored by the standard tool dfosExplorer which is reading THIS_IS_PHOENIX and, if YES, downloads a reference file appropriate for PHOENIX. The DFOS tools required for a PHOENIX installation are all evaluating the key THIS_IS_PHOENIX.

Only a subset of DFOS tools (blue in the following table) is needed for PHOENIX, plus some special tools (red):

D) Dual-instance processing (MASTER/SLAVE model).

To speed up the processing of historical data, and to have some contingency for nights with large amounts of data, the MUSE phoenix process is set up on two instances, muse_ph@muc09 and muse_ph2@muc10. Both can run independently. After moveProducts it is important to join bookkeeping information on one account, called the MASTER. The other account is called the SLAVE.

To define the SLAVE, use the keys IS_SLAVE, MASTER_ACCOUNT, SLV_ROOT and MST_ROOT in config.phoenix for the SLAVE account. For the MASTER account, no additional phoenix configuration is necessary (but this is needed then for config.phoenixMonitor, see there).

Logs

The process log is created per standard execution (-d <date>). It goes to the command line and is stored under $DFO_MON_DIR/AUTO_DAILY/phoenix_<date>.log (in close analogy to autoDaily logs).

Output

How to use

Type phoenix -h for on-line help, and phoenix -v for the version number. Call

IDPs:
phoenix -d <date>	to process all SCIENCE ABs for a specified date
phoenix -m <month>	to process all SCIENCE ABs for a specified month (in a loop over all dates), steps 1-3 (not available for MUSE)
phoenix -d <date> -C	to download & filter all ABs (step 1 only)
phoenix -d <date> -P	to download & filter & process all ABs (steps 1 and 2)
phoenix -d <date> -M	to call moveProducts (step 3 only, requires steps 1 and 2 executed before)
	Note: all three options C|P|M also work with -m <month>
phoenix -d <date> -M -n	same, but no cleanup of mcalibs done (useful in dry runs)
phoenix -d <date> -p	process all ABs as filtered with config.phoenix; no update of processing statistics (aiming at partial reprocessing of data after an issue has been discovered and fixed)
finishNight -d <date> -c	cleanup all remnants of date (only needed in manual use, if options -C or -P have been called before)
MCALIBs:
phoenix -X -m <month>	to create and process all CALIB ABs for a specified month (for a given pool), steps 1-3
phoenix -X -m <month> -C	to create all CALIB ABs (step 1 only)
phoenix -X -m <month> -P	to create & process all CALIB ABs (steps 1 and 2)
phoenix -X -m <month> -M	to call moveProducts (step 3 only, requires steps 1 and 2 executed before)
finishNight -d <date> -c	same as above
phoenix -X -d <date> [-C|-P|-M]	exceptionally you can also call the tool in X mode by date, but then you might end up with incomplete ABs because, in mode -X, the VCAL directory always needs to be erased before a new call

Use the various options on the command-line. In mass production, when you use phoenix for reprocessing of historical data, you may want to use it typically like that:

OCA rules and data pool (MCALIBs only)

RELEASE definition

The configuration key RELEASE is used to define the name of the release. While the concept of $DFO_INSTRUMENT and $PROC_INSTRUMENT relates to the instrument-specific aspects of the processing (like definition of RAW_TYPEs), the RELEASE defines the logics of the (re)processing. It describes aspects like "this is the reprocessing version 2 of all UVES ECHELLE data" where this current reprocessing scheme includes certain pipeline parameter choices, or improved reduction strategies as opposed to e.g. version 1.

The RELEASE tag could be e.g. 'UVESR_2' where 'UVESR' stands for 'UVES reprocessing' and '2' for the current version.

For MCALIB projects, a tag like 'GIRAF_M' would be reasonable where the M stands for 'master calibrations'. This tag is of course not a real release but only used to define the workspace uniquely and distinguish it from the IDP releases. (The corresponding IDP project has the release name 'GIRAF_R'.)

QC procedures

While the original concept of phoenix did not include certification and scoring, it turned out that it is extremely useful, if not undispensible, to create QC reports, mainly for the purpose of quick look. This is true both for IDP and for MCALIB projects. QC reports could also be used to e.g. quickly check that the reduction strategy is the correct one, that the flux calibration looks reasonable etc. These QC reports can likely be easily derived from existing, historical QC procedures for science data.

It is also extremely useful to maintain a set of QC1 parameters and store them in a QC1 database table. This is needed for process control (for instance for proving statistically that the SNR is correctly computed) and also for the scoring. To run a PHOENIX project without QC1 parameters, reports and scores means essentially flying blind.

The existing UVES, XSHOOTER and GIRAFFE IDP streams all have scoring, refer to these examples for gathering more information. Generally the scoring for IDPs, and also for MCALIBs, should focus on key parameters relevant for science reduction. They should check for saturation, negative fluxes etc. It is also a good idea to measure and score association quality, i.e. proximity in time of an IDP to a key calibration.

Technical hints:

• If you use scoring, don't forget to set the TMODE configuration key to HISTORY in your config.scoreQC, to have the red and green buttons returning useful database queries.
• Note that it is extremely useful to make the QC procedures executable in parallel, then handled by condor in the same way as the AB pipeline jobs. Otherwise phoenix execution times will be dominated by the QC procedures (running one after the other in an extremely inefficient way). Actually it is unclear whether phoenix will run at all with QC procedures not enabled for parallel execution.

History scores, qualityChecker, autoCertifier (MCALIBs; for IDPs see below)

These are the possible combinations:

* in order to be auto-certified, otherwise one-by-one assessment needed

The auto-certifier is another embedded procedure of phoenix v2 that uses the history score, plus the current score if available, to automatically decide about certification.

These are the concepts behind the auto-certifier:

For IDPs, there is an optional step for certification, just before calling 'moveProducts'.

Job execution patterns

Normally the DRS_TYPE to be chosen in CON (for condor) which can use up to 30 cores in parallel on muc08 (48 on muc10), both for AB processing and for QC reports. The execution pattern is then identical to the one known from the autoDaily DFOS processing.

The following special cases exist (because standard condor processing would be too inefficient):

1. QCJOB_CONCENTRATION (for QC jobs and IDP processing): this is a special queue optimization to collect QC jobs for a full month (instead of daily QC jobs). It is used for efficiency if an individual QC job takes long and only few exist in daily processing: then the standard queue mechanism would be inefficient since the processing machine cannot be saturated. Efficiency gains can be as high as 30 (running 30 jobs in parallel instead of just 1).

2. 2-stream processing (for MUSE AB jobs): this is special for MUSE. The MUSE pipeline uses an internal parallelization using 24 cores at certain phases of a recipe. With a custom JOB_PGI a 2 stream processing is set up, meaning that 2 ABs can run in parallel, using (up to) 48 cores in parallel (available on muc09 and muc10), for efficiency gains by up to a factor 2.

3. Dual-instances processing (for MUSE only): a phoenix process can be implemented on two different accounts, for gaining efficiency. While both instances can execute independently, at the end one wants to bring all relevant information to a central account, which is then the MASTER, while the other account is called the SLAVE. For more see here. A SLAVE accounts needs to be marked as such in the config.phoenix file, a MASTER account not.

QCJOB_CONCENTRATION (IDPs only)

[This section is currently only relevant for IDPs on the giraffe_ph@muc08 account.] In normal cases the execution time for pipeline jobs (ABs) is longer than for the QC jobs, and the number of daily jobs is high enough to saturate the multi-core machine:

In that regime all phoenix jobs are executed day after day, and within a day batch there is first the AB call and then the QC call. Let's call this regime the "normal condor regime".

For giraffe_ph, the situation is different: typically there are only a few jobs (let's say 2) per day, meaning the muc machine cannot saturate, most cores are idle and the processing is not optimal. Furthermore, the QC jobs take much longer than the AB jobs (because a single QC jobs actually loops over up to 130 MOS products). For this regime (let's call it the "MOS condor regime") there is a special operational setup needed to optimize the processing. It is possible only for monthly processing (because then a sufficient number of AB/QC jobs exist to saturate the muc machine). In a first step all AB jobs are processed in the "normal" way (day by day, without saturation). This is suboptimal but because of the short execution time not critical for optimization. The daily QC jobs are collected but not executed. Only after all AB jobs of the month are done, the QC jobs are executed in one big call, thereby saturating the muc machine and optimizing the processing scheme. Let's call this regime the "QC job concentration regime".

In some test runs, the effective processing time for a month worth of GIRAFFE IDP processing with phoenix went down from more than 6 hours for "normal condor processing" to about 2 hours for the "QC job concentration processing".

The switch to this processing scheme is made via configuration (key QCJOB_CONCENTRATION). It can be used for any PHOENIX account, but currently makes sense for giraffe_ph only.

Ingestion

Each phoenix call results in an entry in the $DFO_JOB_DIR/JOBS_INGEST file, in the traditional DFOS way. The standard dfos tool ingestProducts is aware of running in the phoenix environment if

• INGEST_ENABLED is set to YES in config.phoenix
• PATH_TO_IT and CONVERTER are properly filled in config.ingestProducts.

If INGEST_ENABLED=YES, the tool is enabled to call the ingestionTool and the converter (optional).

Find more about the IDP ingestion process here.

After ingestion, don't forget to call the tool cleanupProducts to replace the fits files with their extracted headers, in the standard DFOS way. For SLAVE accounts, the headers are transfered to the MASTER account, for central bookkeeping.

Installation

phoenix, not being a dfos tool for standard installations, is required as the central tool for the PHOENIX process.

The PHOENIX process requires a customized file .dfosrc and a standard file .qcrc. The following keys are needed in .dfosrc in addition to the one used in DFOS:

The PHOENIX process requires the dfos tools listed above. That list might evolve. Please always use dfosExplorer to control your current PHOENIX installation.

The workspace as defined in the .dfosrc file is defined by $PROC_INSTRUMENT.

Configuration file

phoenix v2 can run two configuration files: the standard one for IDPs (always needed), config.phoenix, and a second one (config.phoenix_mcal, only needed for MCALIBs) to replace the standard one. Both have the same structure:

The name of the MCALIB config file is not hard-coded, you might use another name but it needs to be registered in the main config file, under MCAL_CONFIG. In MCALIB mode, the 'main' config file (let's call it IDP config file) is only read to find the MCALIB config file, no other information is used. The MCALIB config file has the same structure as the IDP config file. This has the effect that it is very easy to use the same account for either MCALIB or IDP production, by simply switching on or off the MCAL_CONFIG key in the IDP config file.

Both configuration files go to the standard $DFO_CONFIG_DIR. The following configuration keys exist (a few keys are specific for either the IDP or the MCALIB mode, they are marked below in the corresponding colours):

Operational plug-ins (PGIs)

phoenix is a standard workflow tool. Usually instruments have a certain range of peculiarities which need to be handled via plug-ins (PGI's). This concept has proven useful for the daily workflow with autoDaily, and it is also used here. Some PGIs are configured directly in config.phoenix (or config.phoenix_mcal), others are PGIs for a specific dfos tool and are configured in their respective config files. All PGI's are expected under $DFO_BIN_DIR. Here is a list of PGIs directly related to phoenix:

Operational workflow description

Self-management of $DFO_CAL_DIR and $DFO_RAW_DIR, and the other DFO directories with permanent data

Although the data disk of muc08 is big (7 TB), it is reasonable to constrain the size of these data directories. Here is a description of the applied strategies (same strategy for IDPs and MCALIBs unless otherwise noted):

$DFO_RAW_DIR:
The <date> directory is always deleted after processing.

$DFO_LOG_DIR and $DFO_PLT_DIR:
The usual content of these directories is preserved and copied to qc@qcweb. All logs and plots are accessible from the exported phoenixMonitor. On MASTER accounts, also logs and plots from the SLAVE account are collected.

Data structure on qcweb

The phoenixMonitor tool creates an overview of the phoenix processing history, in the same way as histoMonitor does for DFOS operations.

Special notes

Don't do

• Never call 'moveProducts -m SCIENCE -d <date>' on the command-line in a PHOENIX environment. The tool will export all log and plot information to qc@qcweb:public_html/${DFO_INSTRUMENT}/logs/<date> and plots/<date> where it will a) get mixed up with dfos information (calibrations, historical science processing), and b) get used as AB source the next time you call the phoenix tool again on that date (if ever). There is no way to prevent this safely (setting DFO_INSTRUMENT to the value of RELEASE would also cause issues). Only if wrapped in 'phoenix -d <date> -M' (or 'phoenix -d <date>', the tool sets the target directory properly to qc@qcweb:public_html/${RELEASE}/logs/<date>.

Notification about phoenix on operational account

In the following we assume that each PHOENIX account has an operational counterpart on muc01...muc07. The case '(re)processing of data of a non-operational (retired) instrument' could be supported but currently is not. For the stream part of a PHOENIX process (after the bulk processing has been done), it is desirable to get a signal from the operational account that a certain set of master calibrations has been finished and is available for phoenix. This 'batch unit' for phoenix is one month, for no other than pragmatic reasons. For 'PHOENIX-enabled' instruments, the dfos tools dfoMonitor and histoMonitor on the operational account can be configured to become aware of the respective PHOENIX account. If the configuration keys PHOENIX_ENABLED and PHOENIX_ACCOUNT are set in $DFO_CONFIG_DIR/config.dfoMonitor, then the following happens: a) histoMonitor, when encountering a new month upon being called from 'finishNight', sends a signal (email) to the QC scientist that a new month has started, meaning that a set of certified master calibrations is available for the previous month. At the same time, a new status flag 'phoenix_Ready' is written into DFO_STATUS, along with the previous month (format YYYY-MM). b) This flag is catched by dfoMonitor and used to flag that month on the main output page: PHO ENIX: 2013-06   If no new PHOENIX job is on the ToDo list, this field is empty: PHO ENIX: none   c) It is left to the QC scientist, when and how this new PHOENIX task is launched. There is no automatic job launch, and there is no communication between the operational machine and the PHOENIX machine (muc08/09/10). The standard way is to launch 'phoenix -m 2013-06' as sciproc@muc08 (or xshooter_ph@muc08 etc.). Depending on the data themselves, and on the data volume, this might take a few hours. Since in principle it might be that in the future there are several concurrent phoenix jobs, or an operational account in addition to sciproc, there is no automatic mechanism to launch PHOENIX jobs. d) When this step has been done, the QC scientist can confirm this PHOENIX job to be done, by pushing the button 'done'. This triggers a dialogue, where the user is asked to confirm the execution, and then this month is removed from the DFO_STATUS file. If there is more than one month, all months will offered for confirmation, one after the other.