GitLab

Some servers provide an etag which is a unique identifier for the
content being served (not always a hash).  This adds two new column to
the archival, an etag and a last-modified field.  If the etag is
available in the archival, and we get the same etag from the request,
then we will not download the file.  The last-modified field is saved
for sanity, it isn't yet used with If-Modified-Since in the request.

This requires a migration:

    alter table archival add column etag character varying;
    alter table archival add column last-modified character varying;

Notification of package updates is only done if at least 1 resource was
downloaded, to avoid unnecessarily running QA and Packagezip.

Put Archiver data into the package_dict using after_show instead of IDatasetForm because you can only have one IDatasetForm - cannot have one for QA as well. Harmonize naming for config options while this is a major version change.

Expose archival info much more. Now it appears in the dataset dict, displayed basically on a resource page and has action functions (not used, but may be useful?).

[#11] Requests can now pass a User-Agent header. i.e. https://github.com/ckan/ckanext-archiver/commit/c6abff2d7539657f2bb3d050c93c14db5b0e42d7 merged to this branch

Copy CKAN extension boiler-plate from core ckan template: paster --plugin=ckan create -t ckanext ckanext-archiver

Compatibility with requests==2.7.0 - response.headers is a funny dict which doesnt JSON dump unless you cast it.

If an exception happens, during the update of a resource of a task, the attempt to log context
information (status of failed request) will lookup for a non-existent `error` attribute of a
`requests.Response` object . To fix this, we must replace with the `reason` attribute (provides
status text of HTTP response).

If the USER_AGENT_STRING setting, or ckanext.archiver.user_agent_string
setting in ckan config, is set, its value is passed in all request
headers. This may be used to help services identify archiver requests.

[noticket] Fix for saving filenames with unicode chars in. e.g. audit-commission-expenses-october-2013/5b1d00ae-2d6c-4016-8879-50b4ba19a114.

[1418] Cached report refactor - Pass queue name so that QA can use the same queue as the original request - i.e. know whether it is priority or not.

[1418] Cached report refactor - Add DataPipe concept for ckanext-qa to trigger off ckanext-archiver - added interface for it and connected up to ckanext-qa. Stop archiving deleted resources... Merged broken links reports into one registered report - an index of orgs and per org.

[1403] Remove all use of task_status - Archiver table now used exclusively. Works from command-line and notifications.

Now stores relevant information in the ArchiveTask model (archive_task table)
which removes the need for updating the task_status API

[1344] WMS links detected as broken - Disable this feature while the changing of URL is investigated.

[1344] WMS links detected as broken - fix the setting of actual resources url to the version with WMS params

Sort out L441 in 2a756807 - res.error included all non-OK status codes, as well as the RequestExceptions, so for requests 1.1+ it needs to be replaced by "not res.ok". In get_task_status catch the precise exception, rather than the catch-all, and deal with it straight away since the response object does not exist in this circumstance. HTTP_ERROR_CODES can go, now that requests has res.reason.