GitLab

Put Archiver data into the package_dict using after_show instead of IDatasetForm because you can only have one IDatasetForm - cannot have one for QA as well. Harmonize naming for config options while this is a major version change.

Fix for difficulty archiving "private" datasets in an group/organization. Still need to solve https://github.com/ckan/ckan/issues/1975 for groups.

[1418] Cached report refactor - Move imports to allow celery_app to init ok when running "paster archiver init".

[1418] Cached report refactor - Add DataPipe concept for ckanext-qa to trigger off ckanext-archiver - added interface for it and connected up to ckanext-qa. Stop archiving deleted resources... Merged broken links reports into one registered report - an index of orgs and per org.

[1418] Cached report refactor - Moved broken link reports here from ckanext-dgu. Register reports using new 1418 infrastructure.

[1403] Remove all use of task_status - Archiver table now used exclusively. Works from command-line and notifications.

Now stores relevant information in the ArchiveTask model (archive_task table)
which removes the need for updating the task_status API

report will generate a CSV containing information on resources whose cached_filepath does not exist, and files that don't have a corresponding resource.

The delete-orphans command uses report to find those files without a resource and delete them (and the containing dir - if it contains a UUID) from disk.

Added migration for the cache url directories. Moved get_cached_resource_filepath into lib.py from ckanext-qa where it was user - cos it looked useful to archiver for a moment and seems more appropriate here.

Option to update a particular group. Use HTTP instead of WSGI to get packages - timeout problem is solved, and requests is easier to use.

Fix bug writing task status - needed site_user_apikey. Fix bug when /tmp and eventual place to store resource files are on different volumes - need to move rather than rename.

Getting all packages at once is inefficient. On DGU it takes 1.2Gb RAM. So I make it do it dataset by dataset. This is a better now we are using WSGI rather than HTTP, which would have been unreliable.

Use WSGI instead of HTTP requests for info about resources - see note about DGU. Improve logging generally.