Raw Archive Ingest

Level-0 data is delivered from the MOC in a "transfer package" consisting of a tarball containing one or more level-0 files, and a manifest file which provides MD5 checksums of the tarball contents. These transfer packages are sent using commercial FASTCopy file-transfer software. When the MOC sends us a file, it also arranges to have a "post-transfer" command that we've defined executed once the physical file transfer is complete. This post-transfer command performs the following actions for all incoming transfer packages, regardless of content:

• Move the tarball from the incoming directory to a time-of-receipt-based archive directory.

• Unpack the tarball contents.

• Verify the tarball contents against the supplied manifest file.

• Identify the type(s) of product files delivered, and post their names and types to a database table with state "NEW"

Cron job. A cron job running on glastlnx11 scans the "incoming product file" database table for new entries. Each new level-0 data file undergoes the following operations:

• Decompression with gunzip.

• Reading and indexing by the ISOC raw-archive software to build an in-memory collection of STL sets<> of CCSDS packets, grouped by spacecraft ID, APID, and UTC hour.

• Reading and indexing of any existing archive files for the corresponding UTC hours into the same collection of sets<>. This has the effect of merging overlapping timespans as well as eliminating multiple copies of the same packets.

• Re-writing the archive files for each (SCID, APIC, UTC) group to include the new data.

• Packets are collected into datagrams by secondary-header timestamp.

• "Orphaned" packets from previous downlinks are retrieved from a database table and added to the datagrams.

• Complete datagrams are identified by examining the CCSDS grouping flags and the CCSDS sequence counter. A summary record for each new, complete datagram is posted to a complete-datagram database table.

• Summary records for each packet in each incomplete datagram are posted to the orphan-packet table.

After all these operations have completed, the database record for the level-0 product file is marked 'INGESTCOMPLETE".