Commit 5d479741 authored by Marko Kuder's avatar Marko Kuder
Browse files

initial renaming of dgu sources to pops, unfinished

- changed my mind due to too many dependencies and possible troubles if
we decide to give back to upstream
parent d724e263
......@@ -3,8 +3,8 @@ omit =
*/site-packages/*
*/python?.?/*
ckan/*
ckanext/dgu/tests/*
ckanext/dgu/testtools/*
ckanext/dgu/bin/*
ckanext/pops/tests/*
ckanext/pops/testtools/*
ckanext/pops/bin/*
include =
ckanext/dgu/*
ckanext/pops/*
......@@ -9,7 +9,7 @@ module.exports = function(grunt) {
grunt.loadNpmTasks('grunt-contrib-coffee');
// Change relative directory
grunt.file.setBase('ckanext/dgu/theme/');
grunt.file.setBase('ckanext/pops/theme/');
// Project configuration.
grunt.initConfig({
......
[![Build Status](https://travis-ci.org/datagovuk/ckanext-dgu.svg?branch=master)](https://travis-ci.org/datagovuk/ckanext-dgu)
[![Coverage Status](https://coveralls.io/repos/datagovuk/ckanext-dgu/badge.svg?branch=master&service=github)](https://coveralls.io/github/datagovuk/ckanext-dgu?branch=master)
ckanext-dgu - data.gov.uk extension
ckanext-pops - data.gov.si extension
===================================
This is an extension to CKAN that provides customisations specifically for the data.gov.uk project.
This is an extension to CKAN that provides customisations specifically for the data.gov.si project.
The official version is available at: https://github.com/datagovuk/ckanext-dgu
The official version is available at: https://gitlab.xlab.si/MJU-POPS/ckanext-pops
Contributions from 1st March 2012 are by the Cabinet Office. It is Crown Copyright and opened up under both the Open Government Licence (OGL) (which is compatible with Creative Commons Attibution License) and the GNU Affero General Public License (AGPL) v3.0. Before 1st March 2012, contributions are Copyright (c) 2010-2012 Open Knowledge Foundation. This material is open and licensed under the GNU Affero General Public License (AGPL) v3.0.
OGL terms: http://www.nationalarchives.gov.uk/doc/open-government-licence/
AGPL terms: http://www.fsf.org/licensing/licenses/agpl-3.0.html
Related extensions that data.gov.uk use in addition are: ckanext-archiver, ckanext-harvest, ckanext-spatial, ckanext-os, ckanext-qa, ckanext-social
Related extensions that data.gov.si use in addition are: ckanext-archiver, ckanext-harvest, ckanext-spatial, ckanext-os, ckanext-qa, ckanext-social
Plugins
=======
......@@ -84,8 +84,8 @@ dgu_theme
ckanext.dgu.controllers.tag:TagController
ckanext.dgu.controllers.reports:ReportsController
from ckanext.dgu.lib import helpers
ckanext/dgu/theme/templates
ckanext/dgu/theme/public
ckanext/pops/theme/templates
ckanext/pops/theme/public
Data, Tag and Reports Controllers. Templates, helper functions.
Random extras:
......@@ -138,21 +138,21 @@ When a resource is created/deleted/URL-changed, this updates the last_major_modi
Non-plugin code
===============
`ckanext/dgu/schema.py` - mostly not used now
`ckanext/dgu/drupalclient.py` - for getting user info from Drupal
`dgu/ckanext/dgu/bin/` - scripts used at one time or another
`dgu/ckanext/dgu/commands/` - scripts used at one time or another
`ckanext/pops/schema.py` - mostly not used now
`ckanext/pops/drupalclient.py` - for getting user info from Drupal
`dgu/ckanext/pops/bin/` - scripts used at one time or another
`dgu/ckanext/pops/commands/` - scripts used at one time or another
Install
=======
This is how to install ckanext-dgu, ckan and their dependencies into a python virtual environment::
This is how to install ckanext-pops, ckan and their dependencies into a python virtual environment::
virtualenv pyenv
pip -E pyenv install -e git+https://github.com:okfn/ckanext-dgu.git#egg=ckanext-dgu
pip -E pyenv install -e git+https://gitlab.xlab.si/MJU-POPS/ckanext-pops.git#egg=ckanext-pops
pip -E pyenv install -e git+https://github.com/okfn/ckan.git#egg=ckan
pip -E pyenv install -r pyenv/src/ckan/pip-requirements.txt
pip -E pyenv install -r pyenv/src/ckanext-dgu/pip-requirements.txt
pip -E pyenv install -r pyenv/src/ckanext-pops/pip-requirements.txt
There are plenty of other little details about getting CKAN running under Apache, SOLR config etc in the CKAN README.
......@@ -179,7 +179,7 @@ and credentials for HTTP Basic Authentication (if necessary)::
The DGU-version of the SOLR schema is required instead of the CKAN SOLR schema. Whether you use a single or mult-core SOLR setup, you'll need a link to the DGU SOLR schema like this::
sudo ln -s /home/okfn/pyenv/src/ckanext-dgu/config/solr/schema-1.4-dgu.xml /etc/solr/conf/schema.xml
sudo ln -s /home/okfn/pyenv/src/ckanext-pops/config/solr/schema-1.4-dgu.xml /etc/solr/conf/schema.xml
Initialise database
......@@ -192,7 +192,7 @@ Creating the database is as usual for CKAN::
sudo -u postgres createuser -S -D -R -P $db_user
sudo -u postgres createdb -O $db_user $db
And because we usually use ckanext-dgu also with ckanext-spatial, then PostGIS needs setting up too::
And because we usually use ckanext-pops also with ckanext-spatial, then PostGIS needs setting up too::
sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql -v ON_ERROR_ROLLBACK=on
sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql
......@@ -216,51 +216,51 @@ Scripts
There are a number of command-line scripts for processing data. Over the years they have been created in three different ways, but more recently we've been focussing on just creating stand-alone python scripts.
To run a *python script* (in the ckanext/dgu/bin directory) you activate the environment and then call it with python::
To run a *python script* (in the ckanext/pops/bin directory) you activate the environment and then call it with python::
. pyenv/bin/activate
python ckanext/dgu/bin/dataset_change_publisher.py --help
python ckanext/pops/bin/dataset_change_publisher.py --help
To run a *console script* (listed in setup.py) you should activate the environment first. For example::
. pyenv/bin/activate
bulk_delete --help
To run a *paster command* (listed in setup.py) you activate the environment and if your CWD is not ckanext-dgu, specify it as the plugin::
To run a *paster command* (listed in setup.py) you activate the environment and if your CWD is not ckanext-pops, specify it as the plugin::
. pyenv/bin/activate
paster --plugin=ckanext-dgu prod --help
paster --plugin=ckanext-pops prod --help
Assets
======
The DGU theme uses assets (images, javascript, css) from this repo and the shared assets repo::
The POPS theme uses assets (images, javascript, css) from this repo and the shared assets repo::
https://github.com/datagovuk/shared_dguk_assets
https://gitlab.xlab.si/MJU-POPS/shared_pops_assets.git
Both repos should be cloned next to each other on developer and server machines. If this is not possible then you need to set the dgu.shared_assets_timestamp_path config option to tell CKAN where the shared assets timestamp file is. e.g.::
dgu.shared_assets_timestamp_path = /vagrant/src/shared_dguk_assets/assets/timestamp
dgu.shared_assets_timestamp_path = /vagrant/src/shared_pops_assets/assets/timestamp
Assets are stored in the repo in 'source' form - the form easiest for developers to edit them in. Before they can be served, Grunt must be run on both repos to create the 'public' versions of these files. This does concatenation, minification, compilation of the less, and recording a timestamp (see Gruntfile.js for details).
Source:
images: ckanext/dgu/theme/src/images
javascript: ckanext/dgu/theme/src/scripts
css (less): ckanext/dgu/theme/src/css
images: ckanext/pops/theme/src/images
javascript: ckanext/pops/theme/src/scripts
css (less): ckanext/pops/theme/src/css
And when you run grunt, you get:
images: ckanext/dgu/theme/public/images
javascript: ckanext/dgu/theme/public/scripts
css: ckanext/dgu/theme/public/css
timestamp: ckanext/dgu/theme/timestamp.py
images: ckanext/pops/theme/public/images
javascript: ckanext/pops/theme/public/scripts
css: ckanext/pops/theme/public/css
timestamp: ckanext/pops/theme/timestamp.py
Read more about Grunt installation and running it: https://github.com/datagovuk/shared_dguk_assets/blob/master/README.md
The shared assets need to be served at /assets. On a deployment server, setup nginx or apache to do this. A developer running under paster will find that the shared assets are served autoimatically, as long as the repo is cloned alongside ckanext-dgu and that this config option is not set: dgu.shared_assets_timestamp_path.
The shared assets need to be served at /assets. On a deployment server, setup nginx or apache to do this. A developer running under paster will find that the shared assets are served autoimatically, as long as the repo is cloned alongside ckanext-pops and that this config option is not set: dgu.shared_assets_timestamp_path.
Tests
......@@ -274,14 +274,14 @@ To test the DGU extension you need the setup with CKAN (see above) and creation
To run the tests::
{pyenv}/bin/activate
cd {pyenv}/ckanext-dgu
nosetests --ckan ckanext/dgu/tests/
cd {pyenv}/ckanext-pops
nosetests --ckan ckanext/pops/tests/
or run them from another directory by specifying the test.ini::
nosetests --ckan --with-pylons={pyenv}/src/ckanext-dgu/test.ini {pyenv}/src/ckanext-dgu/ckanext/dgu/tests/
nosetests --ckan --with-pylons={pyenv}/src/ckanext-pops/test.ini {pyenv}/src/ckanext-pops/ckanext/pops/tests/
You can either run the 'quick and dirty' tests with SQLite or more comprehensively with PostgreSQL. Set ``--with-pylons`` to point to the relevant configuration - either ``test.ini`` or ``test-core.ini`` (both from the ckanext-dgu repo, not the ckan one). For more information, see http://docs.ckan.org/en/latest/install-from-source.html .
You can either run the 'quick and dirty' tests with SQLite or more comprehensively with PostgreSQL. Set ``--with-pylons`` to point to the relevant configuration - either ``test.ini`` or ``test-core.ini`` (both from the ckanext-pops repo, not the ckan one). For more information, see http://docs.ckan.org/en/latest/install-from-source.html .
#####Browser tests
......@@ -301,7 +301,7 @@ Troubleshooting
These errors usually means a previous run of the tests has not cleaned up the Mock Drupal process. You can verify that::
$ ps a | grep mock_drupal
4748 pts/8 S 0:00 /home/dread/hgroot/pyenv-dgu/bin/python /home/dread/hgroot/pyenv-dgu/bin/paster --plugin=ckanext-dgu mock_drupal run -q
4748 pts/8 S 0:00 /home/dread/hgroot/pyenv-dgu/bin/python /home/dread/hgroot/pyenv-dgu/bin/paster --plugin=ckanext-pops mock_drupal run -q
Now kill it before running the tests again::
......@@ -312,7 +312,7 @@ Now kill it before running the tests again::
* ``DrupalXmlRpcSetupError: Drupal XMLRPC not configured.``
The missing settings that result in this error are to be found in {pyenv}/src/ckanext-dgu/test-core.ini which is also imported into {pyenv}/src/ckanext-dgu/test.ini, so make sure you are specifying either of these config files in your nosetests ``--with-pylons`` parameter.
The missing settings that result in this error are to be found in {pyenv}/src/ckanext-pops/test-core.ini which is also imported into {pyenv}/src/ckanext-pops/test.ini, so make sure you are specifying either of these config files in your nosetests ``--with-pylons`` parameter.
* ``ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ERROR: [f752f33380e3eec1379cfb89e0fdded8] multiple values encountered for non multiValued field parent_publishers: [london-borough-of-barnet, local-authorities]``
......@@ -325,7 +325,7 @@ Documentation
DGU is an extension for CKAN: http://ckan.org
This README file is part of the DGU Developer Documentation, stored in the
ckanext-dgu repo at ``ckanext-dgu/doc``.
ckanext-pops repo at ``ckanext-pops/doc``.
The Developer Docs can be built using `Sphinx <http://sphinx.pocoo.org/>`_::
......
......@@ -35,7 +35,7 @@ echo "SOLR config..."
# see https://github.com/ckan/ckan/issues/2972
sed -i -e 's/solr_url.*/solr_url = http:\/\/127.0.0.1:8983\/solr/' ckan/test-core.ini
echo "Installing ckanext-dgu and its requirements..."
echo "Installing ckanext-pops and its requirements..."
python setup.py develop
# Seems like ckanext-taxonomy dependency 'python-skos' won't even start
# installing without rdflib already being installed previously!
......@@ -44,7 +44,7 @@ pip install -r pip-requirements.txt
pip install -r pip-requirements-dev.txt
pip install -r pip-requirements-local.txt
# shared assets is not python, so we can't pip install it
git clone https://github.com/datagovuk/shared_dguk_assets.git
git clone https://gitlab.xlab.si/MJU-POPS/shared_pops_assets.git
echo "Node install..."
sudo apt-get install python-software-properties python g++ make
......@@ -53,7 +53,7 @@ sudo apt-get update
sudo apt-get install nodejs
sudo npm install -g grunt-cli
npm install
cd shared_dguk_assets
cd shared_pops_assets
npm install
cd -
......@@ -62,13 +62,13 @@ python -m nltk.downloader stopwords
echo "Asset compilation (grunt)..."
grunt
cd shared_dguk_assets
cd shared_pops_assets
grunt
cd -
echo "Moving test-core.ini into a subdir..."
# because it expects to find ckan's test-core.ini in a sister directory of
# ckanext-dgu but on travis it is in a sub-directory
# ckanext-pops but on travis it is in a sub-directory
mkdir subdir
mv test-core.ini subdir
# put resource_formats.json where the newly moved test-core.ini can find it
......@@ -76,7 +76,7 @@ mkdir subdir/config
mv config/resource_formats.json subdir/config/resource_formats.json
# Copy who.ini into the subdir
mkdir -p subdir/ckanext/dgu
cp ckanext/dgu/who.ini subdir/ckanext/dgu/
mkdir -p subdir/ckanext/pops
cp ckanext/pops/who.ini subdir/ckanext/pops/
echo "travis-build.bash is done."
......@@ -6,4 +6,4 @@ echo "NO_START=0\nJETTY_HOST=127.0.0.1\nJETTY_PORT=8983\nJAVA_HOME=$JAVA_HOME" |
#sudo cp ckan/ckan/config/solr/schema.xml /etc/solr/conf/schema.xml
sudo cp config/solr/schema-2.0-dgu.xml /etc/solr/conf/schema.xml
sudo service jetty restart
nosetests --nologcapture --with-pylons=subdir/test-core.ini --with-coverage --cover-package=ckanext.dgu --cover-inclusive --cover-erase --cover-tests -v ckanext/dgu/tests
nosetests --nologcapture --with-pylons=subdir/test-core.ini --with-coverage --cover-package=ckanext.dgu --cover-inclusive --cover-erase --cover-tests -v ckanext/pops/tests
......@@ -92,7 +92,7 @@ install_dgu () {
echo "Installing DGU plugin dependencies for instance \"$instance\""
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install -M --ignore-installed -e "git+https://github.com/okfn/ckanext-dgu.git@$branch#egg=ckanext-dgu"
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install -M --ignore-installed -e "git+https://gitlab.xlab.si/MJU-POPS/ckanext-pops.git@$branch#egg=ckanext-pops"
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install -M --ignore-installed -e "git+https://github.com/okfn/ckanext-csw.git#egg=ckanext-csw"
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install -M --ignore-installed -e "git+https://github.com/okfn/ckanext-harvest.git#egg=ckanext-harvest"
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install -M --ignore-installed -e "git+https://github.com/okfn/ckanext-inspire.git#egg=ckanext-inspire"
......@@ -140,7 +140,7 @@ install_dgu () {
# sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.xml.ckan
#fi
#sudo cp "/var/lib/ckan/$instance/pyenv/src/ckanext-dgu/config/solr/schema-1.4-dgu.xml" /etc/solr/conf/schema.xml
#sudo cp "/var/lib/ckan/$instance/pyenv/src/ckanext-pops/config/solr/schema-1.4-dgu.xml" /etc/solr/conf/schema.xml
#sudo /etc/init.d/jetty start
}
......@@ -158,7 +158,7 @@ configure () {
local ini_file="/etc/ckan/$instance/$instance.ini"
sudo mv "/etc/ckan/$instance/who.ini" "/etc/ckan/$instance/who.ini.bak"
sudo ln -s "/var/lib/ckan/$instance/pyenv/src/ckanext-dgu/ckanext/dgu/who.ini" "/etc/ckan/$instance/who.ini"
sudo ln -s "/var/lib/ckan/$instance/pyenv/src/ckanext-pops/ckanext/pops/who.ini" "/etc/ckan/$instance/who.ini"
# Some entries in who.ini (which lives in /etc/ckan), expect to be in /var/lib/ckan/$instance
sudo sed -e "s,%(here)s,/var/lib/ckan/$instance,g" -i "/etc/ckan/$instance/who.ini"
......@@ -298,13 +298,13 @@ clean_and_load_database () {
sudo -u postgres psql -c "UPDATE resource_revision SET format = '' where format = 'Unverified';" "$instance"
local pyenv_root="/var/lib/ckan/$instance/pyenv"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-dgu/ckanext/dgu/bin/import_publishers.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-dgu/buildbot/fixtures/dgupub.csv"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-dgu/ckanext/dgu/bin/import_publisher_contacts.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-dgu/buildbot/fixtures/dgu_pub_contacts.csv"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-dgu/ckanext/dgu/bin/publisher_datasets_assoc.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-dgu/buildbot/fixtures/nodepublishermap.csv" | sudo -u postgres psql "$instance"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-dgu/ckanext/dgu/bin/user_import.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-dgu/buildbot/fixtures/nodepublishermap.csv" "$users_file"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-pops/ckanext/pops/bin/import_publishers.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-pops/buildbot/fixtures/dgupub.csv"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-pops/ckanext/pops/bin/import_publisher_contacts.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-pops/buildbot/fixtures/dgu_pub_contacts.csv"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-pops/ckanext/pops/bin/publisher_datasets_assoc.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-pops/buildbot/fixtures/nodepublishermap.csv" | sudo -u postgres psql "$instance"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-pops/ckanext/pops/bin/user_import.py" "/etc/ckan/$instance/$instance.ini" "$pyenv_root/src/ckanext-pops/buildbot/fixtures/nodepublishermap.csv" "$users_file"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-dgu/ckanext/dgu/bin/tidy_resource_formats.py" --config "/etc/ckan/$instance/$instance.ini"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-dgu/ckanext/dgu/bin/tidy_package_fields.py" --config "/etc/ckan/$instance/$instance.ini"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-pops/ckanext/pops/bin/tidy_resource_formats.py" --config "/etc/ckan/$instance/$instance.ini"
"$pyenv_root/bin/python" "$pyenv_root/src/ckanext-pops/ckanext/pops/bin/tidy_package_fields.py" --config "/etc/ckan/$instance/$instance.ini"
}
create_test_admin_user () {
......@@ -402,7 +402,7 @@ EOF
## create_test_admin_user $INSTANCE
##
## # source the particular checkout's dependency post-installation function, post_install()
## source "/var/lib/ckan/$INSTANCE/pyenv/src/ckanext-dgu/buildbot/install_dgu_instance.sh"
## source "/var/lib/ckan/$INSTANCE/pyenv/src/ckanext-pops/buildbot/install_dgu_instance.sh"
## post_install $INSTANCE
##
## restart_apache
......
......@@ -56,7 +56,7 @@ install_ckan $REPO
pause Creating CKAN instance
ckan-create-instance $INSTANCE $DOMAIN yes
pause Installing ckanext-dgu
pause Installing ckanext-pops
install_dgu $INSTANCE $BRANCH
pause Loading database
......
......@@ -57,10 +57,10 @@ install_dgu () {
echo "Installing DGU plugin dependencies for instance \"$instance\""
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install --ignore-installed -e "git+https://github.com/okfn/ckanext-dgu.git@$branch#egg=ckanext-dgu"
sudo -u "$user" "/var/lib/ckan/$instance/pyenv/bin/pip" install --ignore-installed -e "git+https://gitlab.xlab.si/MJU-POPS/ckanext-pops.git@$branch#egg=ckanext-pops"
# source the particular checkout's dependency installation function, install_dependencies()
source "/var/lib/ckan/$instance/pyenv/src/ckanext-dgu/buildbot/install_dgu_instance.sh"
source "/var/lib/ckan/$instance/pyenv/src/ckanext-pops/buildbot/install_dgu_instance.sh"
# ... and call it
install_dependencies $instance
......@@ -92,7 +92,7 @@ configure_ini_file () {
echo "Configuring ini file"
# source the checkout's installation file for the configure() function
source "/var/lib/ckan/$instance/pyenv/src/ckanext-dgu/buildbot/install_dgu_instance.sh"
source "/var/lib/ckan/$instance/pyenv/src/ckanext-pops/buildbot/install_dgu_instance.sh"
configure $instance $domain
......@@ -203,7 +203,7 @@ configure_ini_file $INSTANCE $DOMAIN
create_test_admin_user $INSTANCE
# source the particular checkout's dependency post-installation function, post_install()
source "/var/lib/ckan/$INSTANCE/pyenv/src/ckanext-dgu/buildbot/install_dgu_instance.sh"
source "/var/lib/ckan/$INSTANCE/pyenv/src/ckanext-pops/buildbot/install_dgu_instance.sh"
post_install $INSTANCE
restart_apache
......
......@@ -110,7 +110,7 @@ def usage():
print """
Removes dead resources by deleting them (and the package if then empty)
python ckanext/dgu/bin/delete_dead_resources.py <CSV_FILE> <CONFIG_FILE>
python ckanext/pops/bin/delete_dead_resources.py <CSV_FILE> <CONFIG_FILE>
"""
if __name__ == '__main__':
......
......@@ -3,7 +3,7 @@ Analyses the data.gov.uk JSON dataset dumps.
Gets run by gov_daily.py or from the command-line:
$ dump_analysis
(in an activated pyenv with ckanext-dgu installed)
(in an activated pyenv with ckanext-pops installed)
'''
import zipfile
import gzip
......@@ -83,7 +83,7 @@ def get_run_info():
run_info = 'This analysis is produced by a script\n'
run_info += 'Date last updated: %r\n' % format_date(datetime.date.today())
run_info += 'Script filename: %r\n' % os.path.basename(__file__)
run_info += 'Script repository: http://github.com/datagovuk/ckanext-dgu\n'
run_info += 'Script repository: https://gitlab.xlab.si/MJU-POPS/ckanext-pops\n'
run_info += 'Dump files for analysis: http://data.gov.uk/data/dumps\n'
return run_info
......
......@@ -4,7 +4,7 @@ https://github.com/datagovuk/ckanext-dgu/issues/263
Usage:
python ckanext/dgu/bin/duplicates_from_harvest.py $CKAN_INI
python ckanext/pops/bin/duplicates_from_harvest.py $CKAN_INI
'''
import sys
......
......@@ -16,7 +16,7 @@ To play about with it, get setup with IPython notebook as well::
sudo apt-get install libzmq-dev libpng-dev libfreetype6-dev g++ libhdf5-serial-dev
pip install numpy numexpr ipython pyzmq tornado matplotlib pandas cython
pip install tables
ipython notebook --pylab inline --ip=0.0.0.0 --notebook-dir=/vagrant/src/ckanext-dgu/ckanext/dgu/notebooks
ipython notebook --pylab inline --ip=0.0.0.0 --notebook-dir=/vagrant/src/ckanext-pops/ckanext/pops/notebooks
Browse the notebook on the host machine at: http://localhost:2888/
......@@ -26,11 +26,11 @@ Get hold of a log like this:
Parse and save as HDF5:
python /vagrant/src/ckanext-dgu/ckanext/dgu/bin/server_log_tool.py /vagrant/access.log --save /vagrant/access.log.h5
python /vagrant/src/ckanext-pops/ckanext/pops/bin/server_log_tool.py /vagrant/access.log --save /vagrant/access.log.h5
Have a shell to play about with data:
python /vagrant/src/ckanext-dgu/ckanext/dgu/bin/server_log_tool.py /vagrant/access.log.h5 --shell
python /vagrant/src/ckanext-pops/ckanext/pops/bin/server_log_tool.py /vagrant/access.log.h5 --shell
Work on it in python like this:
......
......@@ -3,7 +3,7 @@ Goes through a CKAN database and move old and incorrect fields into
the new ones for June 2012 relaunch.
Usage:
$ python ../dgu/ckanext/dgu/bin/tidy_package_fields.py --config=ckan-demo.ini
$ python ../dgu/ckanext/pops/bin/tidy_package_fields.py --config=ckan-demo.ini
'''
......
......@@ -3,7 +3,7 @@ Goes through a CKAN database and normalises the value in the format
field for all resources. It makes them capitalised and not .xls etc.
Usage:
$ python ../dgu/ckanext/dgu/bin/tidy_resource_types.py --config=ckan-demo.ini
$ python ../dgu/ckanext/pops/bin/tidy_resource_types.py --config=ckan-demo.ini
'''
......
......@@ -4,7 +4,7 @@ associate them with the appropriate publishers.
To create the users.csv:
python ../dgu/ckanext/dgu/bin/user_list.py dgutest.ini
python ../dgu/ckanext/pops/bin/user_list.py dgutest.ini
(was done using an SQL query, but that missed off some people)
To create nodepublishermap.csv see publisher_datasets_assoc.py
......
Write one of these paster commands only if it's going to be useful to run by DGU in the future.
Too many of these are scripts only needed once, which clog up the list of paster commands and would be a little easier to write as a non-paster script. Non-paster scripts are in ckanext/dgu/bin and access the ORM using common.py.
Too many of these are scripts only needed once, which clog up the list of paster commands and would be a little easier to write as a non-paster script. Non-paster scripts are in ckanext/pops/bin and access the ORM using common.py.
......@@ -48,7 +48,7 @@ class DataController(BaseController):
rvm use 1.9.3
rvm gemset create ukgovld
rvm alias create ukgovldwg 1.9.3@ukgovld
cd /vagrant/src/ckanext-dgu/
cd /vagrant/src/ckanext-pops/
~/.rvm/wrappers/ukgovldwg/bundle
rvm wrapper ukgovldwg jekyll
......
......@@ -10,7 +10,7 @@ def get_icon(format_):
# Format, icon
# Where:
# * format is canonical format from resource_formats.json
# * icon is in ckanext/dgu/theme/src/images/fugue/
# * icon is in ckanext/pops/theme/src/images/fugue/
('HTML', 'globe--arrow'),
('JPEG', 'image'),
('TIFF', 'image'),
......
......@@ -908,7 +908,7 @@ def render_mandates(pkg_extras):
def results_sort_by():
# Default to location if there is a bbox and no other parameters. Otherwise
# relevancy if there is a keyword, otherwise popularity.
# NB This ties in with the default sort set in ckanext/dgu/plugin.py
# NB This ties in with the default sort set in ckanext/pops/plugin.py
bbox = t.request.params.get('ext_bbox')
search_params_apart_from_bbox_or_sort = [key for key, value in t.request.params.items()
if key not in ('ext_bbox', 'sort') and value != '']
......@@ -1752,11 +1752,11 @@ def get_shared_assets_timestamp():
# Deployments should set this config
timestamp_filepath = config.get('dgu.shared_assets_timestamp_path')
if not timestamp_filepath:
# Default place to find shared_dguk_assets repo is next to this
# Default place to find shared_pops_assets repo is next to this
# repo - perfect for developers
import os
this_file = os.path.dirname(os.path.realpath(__file__))
timestamp_filepath = os.path.join(this_file, '..', '..', '..', '..', 'shared_dguk_assets', 'assets', 'timestamp')
timestamp_filepath = os.path.join(this_file, '..', '..', '..', '..', 'shared_pops_assets', 'assets', 'timestamp')
try:
with open(timestamp_filepath) as f:
shared_assets_timestamp = f.read()
......
......@@ -144,7 +144,7 @@ class ThemePlugin(p.SingletonPlugin):
# simply next door to this repo, dgu.shared_assets_timestamp_path
# won't be set, so as a convenience we get paster to serve them.
if not config.get('dgu.shared_assets_timestamp_path'):
toolkit.add_public_directory(config, '../../../shared_dguk_assets')
toolkit.add_public_directory(config, '../../../shared_pops_assets')
def get_helpers(self):
"""
......@@ -476,7 +476,7 @@ class SearchPlugin(p.SingletonPlugin):
# If the user does not specify a "sort by" method manually,
# then it defaults here (and the UI has to have the same logic)
# NB The UI has to be kept in step with this logic:
# ckanext/dgu/theme/templates/package/search.html
# ckanext/pops/theme/templates/package/search.html
order_by = search_params.get('sort')
bbox = search_params.get('extras', {}).get('ext_bbox')
search_params_apart_from_bbox = search_params.get('q', '') + search_params.get('fq', '')
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment