Database Input/Output

Functions for specific I/O to the PostgreSQL database tables.

database.dbio.add_assoc(conn, sources)[source]

Adds a newly detected VLITE source to the assoc_source table and updates the ‘assoc_id’ for that source in the detected_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – Newly detected VLITE sources stored as DetectedSource objects.
Returns:

sources – DetectedSource objects with updated assoc_id attribute.
Return type:

list
database.dbio.add_catalog_match(conn, sources)[source]

Adds an entry to the catalog_match table for every sky catalog source matched to a VLITE source in the assoc_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – List of CatalogSource objects matched to VLITE DetectedSource objects, or list of tuples.
database.dbio.add_corrected(conn, src)[source]

Inserts primary beam corrected flux values into the database corrected_flux table.

database.dbio.add_image(conn, img, status, delete=False)[source]

Inserts or updates rows in the database image table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • img (database.dbclasses.Image instance) – Initialized Image object with attributes set from image header.
  • status (tuple) – If None, the image is new and is added to the database image table. If not None, the image id stored in the tuple is used to update the correct row in the image table.
  • delete (bool, optional) – If True, rows in the detected_island database table with the appropriate ‘image_id’ will be deleted, cascading to the detected_source and corrected_flux tables and triggering updates on the assoc_source, catalog_match, and vlite_unique tables. Default value is False.
Returns:

img – Initialized Image object with updated id attribute, if newly added to the database.
Return type:

database.dbclasses.Image instance
database.dbio.add_sources(conn, img, sources)[source]

Inserts source finding and measurement results from PyBDSF stored as DetectedSource object attributes into the database detected_island and detected_source tables. The image table is also updated with some results from the source finding.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • img (database.dbclasses.Image instance) – Image object with attributes updated with source finding results.
  • sources (list) – DetectedSource objects whose attributes are the elliptical Gaussian fitting results.
database.dbio.add_vlite_unique(conn, src, image_id, update=False)[source]

Adds or updates an entry in the vlite_unique table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • src (database.dbclasses.DetectedSource instance) – DetectedSource object with attribute nmatches = 0.
  • image_id (int) – Id number of the image from which the source came.
  • update (bool, optional) – If True, the ‘detected’ column is updated for the existing row with the specified ‘image_id’ and ‘assoc_id’. Otherwise, a new row is added. Default is False.
database.dbio.check_catalog_match(conn, asrc_id, catalog)[source]

Checks if a source from the assoc_source table already has a match to a sky catalog source in the catalog_match table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • asrc_id (int) – Row id of the source in the assoc_soure table to match to the ‘assoc_id’ in the catalog_match table.
  • catalog (str) – Name of the sky catalog.
Returns:

rowid – The row id of the entry if it exists. Returns None otherwise.
Return type:

int
database.dbio.check_vlite_unique(conn, asrc_id)[source]

Checks if a given source from the assoc_source table is already in the vlite_unique table. This is so that sources don’t get added twice to the vlite_unique table (once when the nmatches = 0 source is pulled from assoc_source table and again if no sky catalog match is found) when updating the catalog matching results by adding new sky catalogs without re-doing the previous catalog matching results.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • asrc_id (int) – Row id of the source in the assoc_soure table to match to the ‘assoc_id’ in the vlite_unique table.
Returns:

existing – Returns the ‘id’, ‘image_id’, and ‘detected’ columns of the entry with the given ‘assoc_id’. Otherwise, returns None.
Return type:

list
database.dbio.delete_matches(conn, sources, image_id)[source]

Deletes all previous sky catalog cross-matching results for a given set of sources. This function is called when the configuration file option redo match is True.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – DetectedSource objects belonging to a particular image pulled from the assoc_source table based on matching ‘assoc_id’.
  • image_id (int) – Id number of the image.
Returns:

sources – DetectedSource objects with their nmatches attribute re-initialized to 0.
Return type:

list
database.dbio.get_associated(conn, sources)[source]

Returns a list of sources belonging to a particular image from the assoc_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – DetectedSource objects pulled from the detected_source table based on ‘image_id’.
Returns:

assoc_sources – Sources pulled from the assoc_source table based on matching ‘assoc_id’ and translated from row dictionary objects to DetectedSource objects.
Return type:

list
database.dbio.get_image_sources(conn, image_id)[source]

Returns a list of sources belonging to a particular image from the detected_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • image_id (int) – Id number of the image.
Returns:

detected_sources – Sources pulled from the detected_source table translated from row dictionary objects to DetectedSource objects.
Return type:

list
database.dbio.get_new_vu(conn)[source]

Returns list of sources from the assoc_source table for which the ‘nmatches’ dropped to 0 after removing catalog matching results. The new_vu table is created to record the row id number of sources in the assoc_source table whose ‘nmatches’ column is 0 after subtracting 1 as triggered by deletion of rows in the catalog_match table.

Parameters:conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
Returns:assoc_objs – List of DetectedSource objects pulled from the assoc_source table whose ‘nmatches’ now equals 0.
Return type:list
database.dbio.get_vu_image(conn, assoc_id)[source]

Retrieves the id numbers and field-of-view radii for any image which contains the specified the VLITE source. This information is needed to update the vlite_unique table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • assoc_id (int) – The id number of the VLITE source from the assoc_source table.
Returns:

vu_image_list – List of tuples containing the id number and radius of any image in which the specified VLITE source was detected.
Return type:

list
database.dbio.record_config(conn, cfgfile, logfile, start_time, exec_time, nimages, stages, opts, setup, sfparams, qaparams)[source]

Inputs information about the current run of the pipeline into the database run_config table. All contents of the configuration file are stored as dictionaries/json.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • cfgfile (str) – Name of the YAML run configuration file.
  • logfile (str) – Name of the log file.
  • start_time (datetime.datetime instance) – Date & time at the start of the pipeline run.
  • exec_time (datetime.timedelta instance) – Execution time of the pipeline run.
  • nimages (int) – Number of images processed. (Technically, number of Image objects initialized.)
  • stages (dict) – Dictionary of the stages section of the configuration file.
  • opts (dict) – Dictionary of the options section of the configuration file.
  • setup (dict) – Dictionary of the setup section of the configuration file.
  • sfparams (dict) – Dictionary of the pybdsf_params section of the configuration file.
  • qaparams (dict) – Dictionary of the image_qa_params section of the configuration file.
database.dbio.remove_catalog(conn, catalogs)[source]

Deletes rows with results from specified catalogs from the catalog_match table.

database.dbio.remove_images(conn, images)[source]

Deletes the specified images from the database. Removal propagates to all affected tables.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • images (list) – List of image filenames to be removed from the database image table.
database.dbio.remove_sources(conn, assoc_ids)[source]

Deletes the specified sources from the database assoc_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • assoc_ids (list) – List of id numbers corresponding to the VLITE sources to be removed from the assoc_source table.
database.dbio.status_check(conn, impath)[source]

Returns the id, highest completed stage, and radius used for source finding from the database image table or None if the image filename is not in the database.

database.dbio.update_assoc_nmatches(conn, sources)[source]

Updates the number of sky catalog matches to a given source in the assoc_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – DetectedSource objects with updated nmatches attribute after running sky catalog matching, or list of assoc_ids.
database.dbio.update_checked_catalogs(conn, image_id, catalogs)[source]

Updates the list of sky catalogs that have been checked for matches to VLITE sources detected in the image. A list of new catalogs to check is additionally defined by first querying the ‘catalogs_checked’ column in the image table and comparing to the list of catalogs specified from the configuration file.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • image_id (int) – Id number of the image in consideration.
  • catalogs (list) – List of sky catalogs to use in matching as specified in the run configuration file.
Returns:

new_catalogs – Filtered list of only the sky catalogs for which cross-matching has not yet been carried out for the VLITE sources in the image.
Return type:

list
database.dbio.update_detected_associd(conn, sources)[source]

Updates the ‘assoc_id’ for sources in the detected_source table which have been successfully associated with existing VLITE sources in the assoc_source table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – DetectedSource objects detected in the current image that have been associated with previous VLITE sources in the assoc_source table.
database.dbio.update_matched_assoc(conn, sources)[source]

Updates the RA & Dec of existing sources in the assoc_source table to the weighted average of all detections.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • sources (list) – DetectedSource objects extracted from the assoc_source table which have been matched to sources detected in the current image.
database.dbio.update_stage(conn, imobj)[source]

Updates the stage column in the image table.

Parameters:
  • conn (psycopg2.extensions.connect instance) – The PostgreSQL database connection object.
  • imobj (database.dbclasses.Image instance) – Image object used to set table column values.