vdp

This is the main script for the VLITE Database Pipeline (vdp). It is responsible for reading in the configuration file, connecting to the the PostgreSQL database, and calling the processing stages.

vdp.catmatch(conn, imobj, sources, catalogs, save)

Performs positional cross-matching of VLITE detected sources to other radio sky survey catalogs. This function represents the fourth and final stage in the VLITE Database Pipeline.

Parameters:

• conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
• imobj (database.dbclasses.Image instance) – Initialized Image object with attribute values set from header info.
• sources (list) – VLITE detected sources to be matched to other radio catalog sources.
• catalogs (list) – Names of the radio catalogs to use.
• save (bool) – If True, match results are recorded in the catalog_match table and the assoc_source table is updated. VLITE unique sources with no sky catalog match are inserted into the vlite_unique table. If False, results are printed to the terminal and no changes are made to the database.

vdp.cfgparse(cfgfile)

This function reads the YAML configuration file provided as a command line argument when vdp.py is called and parses each section of the file into dictionaries.

Parameters:cfgfile (str) – Name of the configuration file. Returns:

• stages (dict) – Keys are the processing stages (source finding, source assocation, and catalog matching) and values are boolean True or False.
• opts (dict) – Keys are the processing options (save to database, quality checks, overwrite, reprocess, redo match, and update match) and values are boolean True or False.
• setup (dict) – Keys are the setup parameters (root directory, year, month, day, files, database name, database user, and catalogs) and values are the user-supplied inputs.
• sfparams (dict) – Keys are the required source finding parameters mode and scale and other optional PyBDSF parameters and values are the user inputs.
• qaparams (dict) – Keys are the image quality parameters (min time on source (s), max noise (mJy/beam), max beam axis ratio, min problem source separation (deg), and max source metric) and values are the user inputs. Defaults are defined if none are specified.
• dirs (list) – List of strings specifying paths to daily image directories to be processed during the run.

vdp.dbinit(dbname, user, overwrite, qaparams, safe_override=False)

Creates a psycopg2 connection object to communicate with the PostgreSQL database. If no database with the provided name exists, the user is prompted to create a new one. The “radcat” schema which holds all the radio catalogs in tables is created at this stage if it does not already exist. The user will be prompted to verify deletion of all current tables if the database exist and the overwrite option in the configuration file is True. All necessary tables, functions, and triggers are created through a call to database.createdb.create().

Parameters:

• dbname (str) – Name of the PostgreSQL database.
• user (str) – Username for the PostgreSQL database connection.
• overwrite (bool) – If True, tables and data will be deleted and re-created, assuming there is a pre-existing database of name “dbname”. If False, the existing database with “dbname” is used.
• qaparams (dict) – The dictionary of image quality requirements specified by the user in the configuration file. The values are written to the database error table.
• safe_override (bool, optional) – If True, this overrides the ‘safe’ parameter in database.createdb.create(). Default value is False.

Returns:

conn – The PostgreSQL database connection object.

Return type:

psycopg2.extensions.connect instance

vdp.iminit(conn, imobj, save, qa, qaparams, reproc, stages, scale)

This function handles ingestion of the image metadata into the database image table and represents the first stage in the VLITE Database Pipeline.

Parameters:

• conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
• imobj (database.dbclasses.Image instance) – Initialized Image object with attribute values set from the header info.
• save (bool) – If True, the image info is written and saved to the database image table.
• qa (bool) – If True, quality checks are run on the image data.
• qaparams (dict) – Dictionary of image quality requirements and their user-specified values from the configuration file.
• reproc (bool) – If True, any existing entry for this image in the database image table is updated. If False and there is an existing entry, no further processing is done and the code moves on to the next image in the list.
• stages (dict) – Dictionary specifying which processing stages are to be run.
• scale (float) – Fraction between 0 and 1 of the image radius to use. The full size of the image field-of-view is multiplied by this number.

Returns:

imobj – Image object with id attribute updated after insertion into the database image table, or None if the image has already been processed and will not be reprocessed.

Return type:

database.dbclasses.Image instance

vdp.loggerinit(logfile=None, quiet=False)

Initializes handlers for logging to both the console and a text file.

Parameters:

• logfile (str, optional) – Name of the log file. If None, messages will only be printed to the console. Default is None.
• quiet (bool, optional) – If True, no messages will be printed to the console. Default is False.

vdp.main()

One function to rule them all.

vdp.print_run_stats(start_time)

Prints general information about the just completed run of the pipeline, including the number of images processed and the total execution time.

Parameters:start_time (datetime.datetime instance) – Time when the run was started. Returns:

• nimgs (int) – Number of image files initialized as Image objects.
• runtime (datetime.timedelta instance) – Total execution time of the just completed pipeline run.

vdp.process(conn, stages, opts, dirs, files, catalogs, sfparams, qaparams)

This function handles the logic and transitions between processing stages.

Parameters:

• conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
• stages (dict) – Keys are the processing stages (source finding, source assocation, and catalog matching) and values are boolean True or False.
• opts (dict) – Keys are the processing options (save to database, quality checks, overwrite, reprocess, redo match, and update match) and values are boolean True or False.
• dirs (list) – List of strings specifying paths to daily image directories to be processed during the run.
• files (list) – List of files to process in each daily directory.
• catalogs (list) – Names of radio sky survey catalogs to use when running catalog matching.
• sfparams (dict) – Specifies any non-default PyBDSF parameters to be used in source finding.
• qaparams (dict) – User-specified quality requirements or default values defined and set in cfgparse.

vdp.srcassoc(conn, imobj, sources, save)

Associates through positional cross-matching sources extracted from the current image with previously detected VLITE sources stored in the assoc_source database table. This function represents the third stage of the VLITE Database Pipeline.

Parameters:

• conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
• imobj (database.dbclasses.Image instance) – Initialized Image object with attribute values set from header info & updated with source finding results.
• sources (list) – List of database.dbclasses.DetectedSource objects. Attributes of each object are from the PyBDSF fit results.
• save (bool) – If True, the assoc_source table is updated with the association results and the ‘assoc_id’ is updated in the detected_source table. If False, no results are saved to the database.

Returns:

• detected_unmatched (list) – List of new VLITE detected sources.
• imobj (database.dbclasses.Image instance) – Initialized Image object with updated stage attribute.

vdp.srcfind(conn, imobj, sfparams, save, qa, qaparams)

Runs PyBDSF source finding and inserts source fit parameters into database detected_source, detected_island, and corrected_flux tables, if the save to database option is set to True. This function represents the second stage of the VLITE Database Pipeline.

Parameters:

• conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
• imobj (database.dbclasses.Image instance) – Initialized Image object with attribute values set from header info.
• sfparams (dict) – Specifies any non-default PyBDSF parameters to be used in source finding.
• save (bool) – If True, the source fit parameters are written and saved to the database detected_island, detected_source, and corrected_flux tables. The PyBDSF files are always written out.
• qa (bool) – If True, quality checks are run on the source finding results.
• qaparams (dict) – User-specified requirements from the configuration file for the source finding quality checks.

Returns:

• imobj (database.dbclasses.Image instance) – Initialized Image object with updated attributes from the source finding results.
• sources (list) – List of database.dbclasses.DetectedSource objects. Attributes of each object are set from the PyBDSF output object.

vdp.vlite_unique(conn, src, image_id, radius)

This function adds a VLITE unique (VU) source, or a source with no other radio catalog matches, to the vlite_unique table. After adding a new VU source, the image table is queried to find all previously processed images in which the VU source could have been detected based on field-of-view size. The vlite_unique table, therefore, keeps a record of every image in which a VU source was in the field-of-view and whether it was detected.

Parameters:

• conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
• src (database.dbclasses.DetectedSource instance) – The VU source from the assoc_source table.
• image_id (int) – Id number of the image in which the VU source was detected.
• radius (float) – Radius (in degrees) of the image’s field-of-view. Used in querying the image table.

Returns:

src – The same VU src with updated ‘detected’ attribute.

Return type:

database.dbclasses.DetectedSource instance