GitLab

Some servers provide an etag which is a unique identifier for the
content being served (not always a hash).  This adds two new column to
the archival, an etag and a last-modified field.  If the etag is
available in the archival, and we get the same etag from the request,
then we will not download the file.  The last-modified field is saved
for sanity, it isn't yet used with If-Modified-Since in the request.

This requires a migration:

    alter table archival add column etag character varying;
    alter table archival add column last-modified character varying;

Notification of package updates is only done if at least 1 resource was
downloaded, to avoid unnecessarily running QA and Packagezip.

Expose archival info much more. Now it appears in the dataset dict, displayed basically on a resource page and has action functions (not used, but may be useful?).

[1418] Cached report refactor - Add DataPipe concept for ckanext-qa to trigger off ckanext-archiver - added interface for it and connected up to ckanext-qa. Stop archiving deleted resources... Merged broken links reports into one registered report - an index of orgs and per org.

[1418] Cached report refactor - Moved broken link reports here from ckanext-dgu. Register reports using new 1418 infrastructure.

[1403] Remove all use of task_status - Archiver table now used exclusively. Works from command-line and notifications.

Now stores relevant information in the ArchiveTask model (archive_task table)
which removes the need for updating the task_status API