Django Ditto Documentation Release 0.9.0

Phil Gyford

Nov 03, 2018

Contents

1 Introduction 3 1.1 Services covered...... 3 1.2 What Ditto includes...... 4

2 Installation 5 2.1 Pillow...... 5 2.2 Install django-ditto...... 5 2.3 Add to INSTALLED_APPS...... 5 2.4 Add to urls.py...... 6 2.5 Settings...... 7 2.6 Set up each service...... 8

3 Flickr 9 3.1 Set-up...... 9 3.2 Models...... 10 3.3 Managers...... 10 3.4 Template tags...... 11 3.5 Management commands...... 13

4 Last.fm 17 4.1 Set-up...... 17 4.2 Models...... 17 4.3 Managers...... 18 4.4 Template tags...... 19 4.5 Management commands...... 22

5 Pinboard 23 5.1 Set-up...... 23 5.2 Models...... 23 5.3 Managers...... 23 5.4 Template tags...... 24 5.5 Management commands...... 25

6 Twitter 27 6.1 Set-up...... 27 6.2 Models...... 27 6.3 Managers...... 28

i 6.4 Template tags...... 29 6.5 Management commands...... 31

7 Development 35 7.1 Tests...... 35 7.2 Other notes for development...... 36

ii Django Ditto Documentation, Release 0.9.0

A collection of Django apps for copying things from Flickr, Twitter and Pinboard, and displaying them on your Django site.

Contents 1 Django Ditto Documentation, Release 0.9.0

2 Contents CHAPTER 1

Introduction

A collection of Django apps for copying things from third-party sites and services. This is still in-progress and things may change. If something doesn’t make sense, email Phil Gyford and I’ll try and clarify it. Requires Python 3.4, 3.5 or 3.6, and Django 1.11, 2.0 or 2.1. See screenshots of a site using the supplied templates.

1.1 Services covered

Currently, Ditto can copy these things from these services: • Flickr – Photos – Photosets – Original image and video ﬁles – Users • Last.fm – Scrobbles (Artist, Track and Album) • Pinboard – Bookmarks • Twitter – Tweets – Favorites/Likes – Images and Animated GIFs (but not videos) – Users

3 Django Ditto Documentation, Release 0.9.0

It can save these things for one or more account on each service. See possible future services, and overall progress, in this issue. Public and private Photos, Bookmarks and Tweets are saved, but only public ones are used in the included Views, Templates and Template tags; non-public data are only visible in the Django admin. Ditto does not sync data – it’s a one-way fetch of data from the service to Ditto. You can repeatedly fetch the same Photos, Tweets, etc and their data will be overwritten in Ditto. You could do a single fetch of all your data as a snapshot/archive. And/or, after that, keep fetching the latest items to keep it up-to-date. For each item fetched, the original JSON data is saved on its object.

1.2 What Ditto includes

The Ditto apps provide: • Models • Admin • Management commands to fetch the data/ﬁles • Views and URLs • Templates (that use Bootstrap 4.1) • Template tags for common things (eg, most recent Tweets, or Flickr photos uploaded on a particular day) You could use the whole lot to create a minimal site that displays your Tweets, photos, etc – see the devproject/ for a bare-bones example. Or you might want to use the management commands, Models and Admin to fetch and store your data, but use that data in your own Views and Templates, maybe using the Template tags. Or some other combination.

4 Chapter 1. Introduction CHAPTER 2

Installation

2.1 Pillow

Ditto uses Pillow which has some prerequisites of its own. You may need to install libjpeg and zlib. (On a Mac, zlib was installed for me by XCode, and I used Homebrew to install libjpeg.)

2.2 Install django-ditto

Ditto can be installed using pip:

$ pip install django-ditto

2.3 Add to INSTALLED_APPS

To use Ditto in your own project (untested as yet), add the core ditto.core application to your project’s INSTALLED_APPS in your settings.py, and add the applications for the services you need. This example includes Flickr, Last.fm, Pinboard and Twitter:

INSTALLED_APPS=( # other apps listed here. # ... 'imagekit', # Required only to use downloaded images and videos 'sortedm2m', # Required only for ditto.flickr 'taggit', # Required only for ditto.flickr and ditto.pinboard 'ditto.core', 'ditto.flickr', 'ditto.lastfm', 'ditto.pinboard', 'ditto.twitter', )

5 Django Ditto Documentation, Release 0.9.0

If you only wanted to use the Flickr part, including displaying downloaded photos, you would do this:

Or, to use only the Twitter part, and not worry about using local versions of images:

INSTALLED_APPS=( # other apps listed here. # ... 'ditto.core', 'ditto.twitter', )

2.4 Add to urls.py

To use Ditto’s supplied views you can include each app’s URLs in your project’s own urls.py. Note that each app requires the correct namespace (flickr, lastfm, pinboard or twitter), eg: from django.conf.urls import include, url from django.contrib import admin urlpatterns=[ url(r'^admin/', include(admin.site.urls)),

url(r'^flickr/', include('ditto.flickr.urls')), url(r'^lastfm/', include('ditto.lastfm.urls')), url(r'^pinboard/', include('ditto.pinboard.urls')), url(r'^twitter/', include('ditto.twitter.urls')),

# To include the overall, aggregated views: url(r'ditto/', include('ditto.core.urls')), ]

Change the URL include paths (eg, r'^ditto/pinboard/' as appropriate) to suit your project. See the urls.py in the devproject/ project for a full example. Each app’s URL conf is included under an appropriate app_name: • flickr • lastfm • pinboard • twitter • ditto (The Ditto Core URLs)

6 Chapter 2. Installation Django Ditto Documentation, Release 0.9.0

2.5 Settings

There are some optional settings that can be placed in your project’s settings.py.

2.5.1 Core settings

The ditto.core app has some optional settings for customing the formats used to display dates and times in the default templates (and the ditto_core.display_time() template tag). The formats are those used for strftime. Here they are, with their default values:

# e.g. "07:34" DITTO_CORE_TIME_FORMAT='%H:%M'

# e.g. "8 Apr 2018" DITTO_CORE_DATE_FORMAT=' %-d %b%Y'

# Used when both a time and a date are displayed. # The [time] and [date] tokens are replaced with the formats from the # two settings above. # e.g. "07:34 on 8 Apr 2018" DITTO_CORE_DATETIME_FORMAT='[time] on [date]'

# Used when only a year is displayed. # e.g. "2018" DITTO_CORE_DATE_YEAR_FORMAT='%Y'

# Used when only a month and year are displayed. # e.g. "Apr 2018" DITTO_CORE_DATE_YEAR_MONTH_FORMAT='%b%Y'

2.5.2 Service-speciﬁc settings

In addition, some of the other apps have their own optional settings. They’re described in detail in each service’s documentation. This is the complete list of service-speciﬁc settings with their default values:

DITTO_FLICKR_DIR_BASE='flickr' DITTO_FLICKR_DIR_PHOTOS_FORMAT='%Y/%m/ %d' DITTO_FLICKR_USE_LOCAL_MEDIA= False

DITTO_TWITTER_DIR_BASE='twitter' DITTO_TWITTER_USE_LOCAL_MEDIA= False

2.5.3 Other optional settings

To have large numbers formatted nicely in the included templates, ensure these are in your settings.py:

USE_L10N= True USE_THOUSAND_SEPARATOR= True

2.5. Settings 7 Django Ditto Documentation, Release 0.9.0

2.6 Set up each service

Each service (such as Flickr or Twitter) you want to use will require some set-up in order to link your account(s) on the service with Django Ditto. See the documentation for each service for how to do this.

8 Chapter 2. Installation CHAPTER 3

Flickr

You can fetch, store and display data about all your Photos and Photosets (Albums) for one or more Flickr Accounts. By default the included models and templates will link to the photo files on flickr.com, but you can also download the original files to store locally. These can then be used to generate images in all other sizes, to be served locally. See the Fetch Files management command below.

3.1 Set-up

In the Django admin, create a new Account in the Flickr app, and add your Flickr API key and secret from https: //www.flickr.com/services/apps/create/apply/ (you can ignore the User for the moment). By default this will only allow the fetching of fully public photos. To fetch all photos your Flickr account can access, you’ll need to do this: 1. Enter your API key and secret in the indicated place in the file ditto/scripts/flickr_authorize.py. (If you’ve installed Ditto using pip, it might be easier to download the script from GitHub.) 2. Run the script on the command line (if you downloaded the file, change the path to wherever the script is):

$ python ditto/scripts/flickr_authorize.py

3. Follow the instructions. You should get a URL to paste into a web browser, in order to authorize your Flickr account. You’ll then get a code to paste into your Terminal. Finally, back in the Django Admin note the ID of the Account you created. e.g. if the Django Admin’s URL is something like /admin/flickr/account/1/change/ then the ID is 1. Run the following management command to fetch information about the Account’s associated Flickr User (replacing 1 with your Account’s Django ID, if different):

$ ./manage.py fetch_flickr_account_user --id=1

Now you can download your photos’ data and, optionally, their original image/video ﬁles. See Management commands.

9 Django Ditto Documentation, Release 0.9.0

3.2 Models

The models available in ditto.flickr.models are: Account Representing a Flickr account that has API credentials and that we fetch Photos for. It has a one-to-one relationship with a User model. TaggedPhoto The through model relating Photos to tags. Photo A photo (or video) on Flickr. The media_type property of 'photo' or 'video' indicates which type this is. Photoset A photoset (album) containing Photos. User A single user on Flickr. There could be lots of Users but only one (or a few) will have associated Account objects – these are the Users we fetch Photos for.

3.3 Managers

3.3.1 Photos

There are several managers available for Photos. There are two main differentiators we want to restrict things by: • Public vs private: Whether a Photo is public or private (which includes those marked as for Friends and/or Family on Flickr.com). • Posted Photo vs Favorited Photo: In the future we may fetch Photos that have been favorited, rather than posted, by our Users that have Accounts. In this case we’d want to be able to differentiate between Photos that we posted, and Photos that we favorited. On public-facing web pages you should only use the mangaers starting Photo.public_ as these will ﬁlter out all Photos that are not public. Photo.objects.all() The default manager fetches all Photos in the system. Public, private, posted by you or (in the future) only favorited by you. Photo.public_objects.all() Gets all public Photos in the system, no matter who posted them to Flickr. Photo.photo_objects.all() Gets all Photos, public or private, that were posted by one of the Users associated with an API-credentialled Account. Photo.public_photo_objects.all() Gets public Photos that were posted by one of the Users associated with an API-credentialled Account. Of course, these can all be ﬁltered as usual. So, if you wanted to get all the public Photos posted by a particular User:

from ditto.flickr.models import User, Photo

user= User.objects.get(nsid='35034346050@N01') photos= Photo.public_photo_objects.filter(user=user)

NOTE: You can ﬁnd a user’s NSID (the ID used by Flickr) at http://idgettr.com

3.3.2 Users

User.objects.all() The default manager fetches all Users in the system. User.objects_with_accounts.all() This only gets Users that are associated with Accounts.

10 Chapter 3. Flickr Django Ditto Documentation, Release 0.9.0

3.4 Template tags

There are three assigment template tags and one simple template tag available for displaying Photos, Photosets and information about a Photo.

3.4.1 Annual Photo Counts

Get the number of Photos taken or uploaded per year for all or one User-with-Account. This fetches totals for all Users, by time uploaded:

{% load ditto_flickr %}

{% annual_photo_counts as counts %}

{% for row in counts %}

{{ row.year }}: {{ row.count }}

{% endfor %}

Both the year and count in each row are integers. To count by time taken instead:

{% annual_photo_counts count_by='taken_time' as counts %}

The alternative count_by value is equivalent to the default behaviour:

{% annual_photo_counts count_by='post_time' as counts %}

To restrict totals to a single User-with-Account, include their nsid:

{% annual_photo_counts nsid='35034346050@N01' count_by='taken_time' as counts %}

3.4.2 Day Photos

Gets public Photos posted or taken on a particular day by any of the Users-with-Accounts. In this example, my_date is a datetime.datetime.date type:

{% load ditto_flickr %}

{% day_photos my_date as photos %}

By default this will return photos posted on the date. To specify whether it uses post_time or taken_time use the time parameter:

{% day_photos my_date time='post_time' as photos %}

{% day_photos my_date time='taken_time' as photos %}

Or we can restrict this to Photos posted by a single User-with-an-Account:

{% day_photos my_date time='post_time' nsid='35034346050@N01' as photos %}

3.4. Template tags 11 Django Ditto Documentation, Release 0.9.0

3.4.3 Photosets

Gets Photosets. By default they are by any User and only 10 are returned.

{% load ditto_flickr %}

{% photosets as photoset_list %}

{% for photoset in photoset_list %}

{% endfor %}

You can restrict it to Photosets by a single User and/or change the number returned:

{% load ditto_flickr %}

{% photosets nsid='35034346050@N01' limit=300 as photoset_list %}

3.4.4 Photo license

When displaying information about a Photo you may want to display a user-friendly version of its license (indi- cating Creative Commons level, All rights reserved, etc). This tag makes that easy. Pass it the license property of a Photo object. Here, photo is a Photo object:

{% load ditto_flickr %}

This would create HTML something like this, depending on the license: Attribution-NonCommercial-ShareAlike License

3.4.5 Recent Photos

To display the most recent public Photos posted by any of the Users associated with Accounts. By default the most recent 10 are fetched:

{% load ditto_flickr %}

{% recent_photos as photos %}

{% for photo in photos %}

{% endfor %}

12 Chapter 3. Flickr Django Ditto Documentation, Release 0.9.0

The tag can also fetch a different number of Photos and/or only get Photos posted by a single User-with-an-Account. Here we only get the 5 most recent Photos posted by the User with an nsid of '35034346050@N01':

{% recent_photos nsid='35034346050@N01' limit=5 as photos %}

3.5 Management commands

3.5.1 Fetch Account User

This is only used when setting up a new Account object with API credentials. It fetches data about the Flickr User associated with this Account. Once the Account has been created in the Django admin, and its API credentials entered, run this (using the Django ID for the object instead of 1):

$ ./manage.py fetch_flickr_account_user --id=1

3.5.2 Fetch Photos

Fetches data about Photos (including videos). This will fetch data for ALL Photos for ALL Accounts:

$ ./manage.py fetch_flickr_photos --days=all

NOTE 1: This took about 75 minutes to fetch data for 3,000 photos on my MacBook. NOTE 2: Trying to run the same thing on a 512MB Digital Ocean machine resulted in the process being killed after fetching about 1,500 photos. See this bug. This will only fetch Photos uploaded in the past 3 days:

$ ./manage.py fetch_flickr_photos --days=3

Both options can be restricted to only fetch for a single Account by adding the NSID of the Account’s User, eg:

$ ./manage.py fetch_flickr_photos --account=35034346050@N01 --days=3

Whenever a Photo is fetched, data about its User will also be fetched, if it hasn’t been fetched on this occasion. Proﬁle photos of Users are downloaded and stored in your project’s MEDIA_ROOT directory. You can optionally set the DITTO_FLICKR_DIR_BASE setting to change the location. The default is:

DITTO_FLICKR_DIR_BASE='flickr'

If your MEDIA_ROOT was set to /var/www/example.com/media/ then the above setting would save the proﬁle image for the user with NSID 35034346050@N01 to something like this:

/var/www/example.com/media/flickr/46/05/35034346050N01/avatars/35034346050N01.jpg

3.5.3 Fetch originals

The fetch_flickr_photos management command only fetches data about the photos (title, locations, EXIF, tags, etc). To download the original photo and video ﬁles themselves, use the fetch_flickr_originals command, after fetching the photos’ data:

3.5. Management commands 13 Django Ditto Documentation, Release 0.9.0

$ ./manage.py fetch_flickr_originals

This took over 90 minutes for about 3,000 photos (and very few videos) for me. By default this command will fetch all the original files that haven’t yet been downloaded (so the first time, it will fetch all of them). To force it to download all the files again (if you’ve deleted them locally, but they’re still on Flickr) then:

$ ./manage.py fetch_flickr_originals --all

Both variants can be restricted to fetching ﬁles for a single account:

$ ./manage.py fetch_flickr_originals --account=35034346050@N01

Files will be saved within your project’s MEDIA_ROOT directory, as deﬁned in settings.py. There are two optional settings to customise the directories in which the ﬁles are saved. Their default values are as shown here:

DITTO_FLICKR_DIR_BASE='flickr' DITTO_FLICKR_DIR_PHOTOS_FORMAT='%Y/%m/ %d'

These values are used if you don’t specify your own settings. If your MEDIA_ROOT was set to /var/www/example.com/media/ then the above settings would save the Flickr photo 1234567_987654_o.jpg to something like this, depending on the Flickr user’s NSID and the date the photo was taken (not uploaded):

/var/www/example.com/media/flickr/35034346050N01/photos/2016/08/31/1234567_987654_o.

˓→jpg

Note that videos will have two “original” files downloaded: the video itself and a JPG image that Flickr created for it. Once you’ve downloaded the original image files, you can use these to generate all the different sizes of image required for your site, instead of linking direct to the image files on flickr.com. To do this, ensure imagekit is in your INSTALLED_APPS setting:

INSTALLED_APPS=( # ... 'imagekit', # ... )

And add this to your settings.py (its default value is False):

DITTO_FLICKR_USE_LOCAL_MEDIA= True

Any requests in your templates for the URLs of photo files of any size will now use resized versions of your downloaded original files, generated by Imagekit. The first time you load a page (especially if it lists many Flickr images) it will be slow, but the images are cached (in a CACHE directory in your media folder). For example, before changing this setting, the URL of small image (Photo.small_url) would be something like this: https://farm8.static.flickr.com/7442/27289611500_d0debff24e_m.jpg

After choosing to use local photos, it would be something like this:

/media/CACHE/images/flickr/35034346050N01/photos/2016/06/09/27289611500_d21f6f47a0_o/

˓→0ee894a3438233848e6e9d85e1985260.jpg

14 Chapter 3. Flickr Django Ditto Documentation, Release 0.9.0

If you change your mind you can switch back to using the images hosted on flickr.com by removing the DITTO_FLICKR_USE_LOCAL_MEDIA setting or changing it to False. Note that Ditto currently can’t do the same for videos, even if the original video file has been downloaded. No matter what the value of DITTO_FLICKR_USE_LOCAL_MEDIA the flickr.com URL for videos is always used.

3.5.4 Fetch Photosets

You can fetch data about your Photosets any time, but it doesn’t fetch detailed data about Photos within them. So, for best results, only run this after getting running fetch_flickr_photos. To fetch Photosets for all Accounts:

$ ./manage.py fetch_flickr_photosets

Or fetch for only one Account:

$ ./manage.py fetch_flickr_photosets --account=35034346050@N01

3.5. Management commands 15 Django Ditto Documentation, Release 0.9.0

16 Chapter 3. Flickr CHAPTER 4

Last.fm

You can fetch all your Scrobbles (listens) from one or more Last.fm accounts. You can then view: recent Scrobbles; the most popular Artists, Tracks and Albums (for different time periods); number of Scrobbles per year; all Scrobbles for a single day; etc.

4.1 Set-up

In the Django admin, create a new Account in the Last.fm app. Enter your Last.fm username, your full name, and a Last.fm API key from http://www.last.fm/api/account/create Now you can download your scrobbles. See Management commands.

4.2 Models

The models available in ditto.lastfm.models are: Account Representing a Last.fm account with an API key. Scrobble A single listen to a particular Track by an Artist at a certain time (post_time), optionally on an Album. Track A track by an Artist. Its scrobbles property is a set of Scrobble s; all the times it has been listened to. Tracks have no direct relationship to Album s. Artist A music artist. Its scrobbles property is a set of Scrobble s; all the times Track s by this artist have been listened to. Album An album by an Artist. Its scrobbles property is a set of Scrobble s; all the times Tracks on this album have been listened to. When the Last.fm API provides it, we store an mbid (MusicBrainz Identiﬁer) for each Track, Artist and Album.

17 Django Ditto Documentation, Release 0.9.0

The URL slug is used to differentiate one entity from another at Last.fm. While we store these as original_slug Ditto uses a lowercase version, slug, so that matching newly-fetched entities to existing ones is more reliable. (And because there’s no case-insensitive searching of unicode strings in SQLite.) Track s and Albums s aren’t directly related; their only connection is through Scrobble s. This is because the data doesn’t seem reliable enough to useful construct this relationship. Each Album has a tracks property that returns a QuerySet of Track s associated with it:

album= Album.objects.get(slug='bang+bang+rock+&+roll')

for track in album.tracks: print(track.name)

Similarly, to get the Album s on which a particular Track appears, use its albums property:

track= Track.objects.get(slug='emily+kane')

for album in track.albums: print(album.name)

See ditto.lastfm.models for more useful properties and methods.

4.3 Managers

The default manager for each of Track, Album and Artist has a with_scrobble_counts() method. This gives each returned object a scrobble_count property which is an aggregated count of the number of times that thing has been scrobbled. These results are also ﬁlterable by Account, minimum and maximum scrobble time, and (for Track and Album) by Artist. By default, the manager behaves as expected, eg:

# A QuerySet of all Track objects: tracks= Track.objects.all()

# A QuerySet of all Track objects by Art Brut: artist= Artist.objects.get(slug='art+brut') tracks= Track.objects.filter(artist=artist)

The default ordering is by name. To add number of scrobbles, and sort with the most-scrobbled tracks ﬁrst:

tracks= Track.objects.with_scrobble_counts().order_by('-scrobble_count')

artist= Artist.objects.get(slug='art+brut') tracks= Track.objects.with_scrobble_counts(artist=artist).order_by('-scrobble_count')

Here, each Track will have a scrobble_count property, an integer. Some more examples (d1 and d2 are both datetime.datetime s):

# All Artists scrobbled since d1: artists= Artist.objects.with_scrobble_counts(min_post_time=d1)

# All Albums scrobbled between d1 and d2,: albums= Album.objects.with_scrobble_counts(min_post_time=d1, (continues on next page)

18 Chapter 4. Last.fm Django Ditto Documentation, Release 0.9.0

(continued from previous page) max_post_time=d2)

# All Tracks scrobbled by this account since d1: account= Account.objects.get(username='gyford') tracks= Track.objects.with_scrobble_counts(min_post_time=d1, account=account)

# All Tracks by this artist scrobbled by this account, between d1 and d2: artist= Artist.objects.get(slug='art+brut') tracks= Track.objects.with_scrobble_counts(artist=artist, account=account, min_post_time=d1, max_post_time=d2)

4.4 Template tags

There are several template tags for getting common lists of things.

4.4.1 Annual Scrobble Counts

Get the number of scrobbles per year for all or one Account. This fetches totals for all Account s:

{% load ditto_lastfm %}

{% annual_scrobble_counts as counts %}

{% for row in counts %}

{{ row.year }}: {{ row.count }}

{% endfor %}

Both the year and count in each row are integers. To restrict totals to a single Account (assuming account is an Account object):

{% annual_scrobble_counts account=account as counts %}

4.4.2 Day Scrobbles

Get a QuerySet of all Scrobbles from a particular day for one or all Account s, earliest ﬁrst. In these examples, today can be either a datetime.datetime or a datetime.date.

{% load ditto_lastfm %}

{% day_scrobbles date=today as scrobbles %}

{% for scrobble in scrobbles %}

{{ scrobble.artist.name }} - {{ scrobble.track.name }} ({{ scrobble.post_time }}) (continues on next page)

4.4. Template tags 19 Django Ditto Documentation, Release 0.9.0

(continued from previous page)

{% endfor %}

To restrict scrobbles to a single Account (assuming account is an Account object):

{% day_scrobbles date=today account=account as scrobbles %}

4.4.3 Recent Scrobbles

Get a QuerySet of the most recent Scrobbles, by one or all Account s. The default quantity returned is 10.

{% load ditto_lastfm %}

{% recent_scrobbles as scrobbles %}

{# Then loop as in previous example. #}

To restrict scrobbles to a single Account (assuming account is an Account object), and increase the quantity returned to 30:

{% recent_scrobbles account=account limit=30 as scrobbles %}

4.4.4 Top Tracks

Get a QuerySet of the most-scrobbled Track s with the most-scrobbled ﬁrst. Can be restricted to: a single Account; a single day, week, month or year; tracks by a single Artist. By default 10 tracks are returned.

{% load ditto_lastfm %}

{% top_tracks as tracks %}

{% for track in tracks %}

{{ forloop.counter }}. {{ track.artist.name }} - {{ track.name }}: {{ track.scrobble_count }}

{% endfor %}

Examples of fetching for a single day, month or year, assuming my_date is either a datetime.datetime or a datetime.date:

{% top_tracks date=my_date period='day' as tracks %}

{% top_tracks date=my_date period='week' as tracks %}

{% top_tracks date=my_date period='month' as tracks %}

{% top_tracks date=my_date period='year' as tracks %}

For month and year, the calendar month/year around the date is used. e.g. if the supplied date was 2016-03-24 then period='month' would produce a chart for March 2016, and period='year' would produce a chart for all of 2016.

20 Chapter 4. Last.fm Django Ditto Documentation, Release 0.9.0

For week, it uses the Django setting FIRST_DAY_OF_WEEK, default being 0 (Sunday). Example of only fetching tracks by a single artist, assuming artist is an Artist object:

{% top_tracks artist=artist as tracks %}

Example of only fetching tracks scrobbled by a single Account:

{% top_tracks account=account as tracks %}

Example of fetching only 5 tracks by a single Artist, scrobbled by a single Account, during a single month:

{% top_tracks artist=artist account=account date=my_date period='month' limit=5 as

˓→tracks %}

Arguments can be in any order.

4.4.5 Top Albums

Get a QuerySet of the most-scrobbled Album s with the most-scrobbled ﬁrst. This works in exactly the same way as the top_tracks template tag, above, with identical arguments. e.g.:

{% load ditto_lastfm %}

{% top_albums artist=artist account=account date=my_date period='month' limit=5 as

˓→albums %}

{% for album in albums %}

{{ forloop.counter }}. {{ album.artist.name }} - {{ album.name }}: {{ album.scrobble_count }}

{% endfor %}

4.4.6 Top Artists

Get a QuerySet of the most-scrobbled Artist s with the most-scrobbled ﬁrst. This works in a similar way to the top_tracks and top_albums template tags, above. The only difference is that results cannot be ﬁltered by artist. e.g.:

{% load ditto_lastfm %}

{% top_artists account=account date=my_date period='month' limit=5 as artists %}

{% for artist in artists %}

{{ forloop.counter }}. {{ artist.name }}: {{ album.scrobble_count }}

{% endfor %}

4.4. Template tags 21 Django Ditto Documentation, Release 0.9.0

4.5 Management commands

There is only one Last.fm management command.

4.5.1 Fetch Scrobbles

Fetches Scrobbles for one or all Accounts. To fetch ALL Scrobbles for all Accounts (this could take a long time):

$ ./manage.py fetch_lastfm_scrobbles --days=all

To fetch Scrobbles for all Accounts from the past 3 days:

$ ./manage.py fetch_lastfm_scrobbles --days=3

Both options can be restricted to only fetch for a single Account by adding the Last.fm username. e.g.:

$ ./manage.py fetch_lastfm_scrobbles --account=gyford --days=3

It’s safe to re-fetch the same data. Duplicates will only occur if an Artist/Track/Album’s URL slug has changed. A change of case won’t cause duplicates, but anything more will. Subsequent fetches will update any other changed data, such as altered Artist names, new or different MBIDs, etc.

22 Chapter 4. Last.fm CHAPTER 5

Pinboard

You can fetch, store and display data about all your bookmarks for one or more Pinboard accounts.

5.1 Set-up

In the Django admin, add an Account in the Pinboard app with your API token from https://pinboard.in/settings/ password. Then use the Management commands to download your Bookmarks.

5.2 Models

The models available in ditto.pinboard.models are: Account A single Pinboard account. (Note: other services, like Twitter and Flickr, have separate User and Account models. Pinboard is currently simpler.) BookmarkTag A custom version of a Taggit Tag model, trying to match the way Pinboard creates slugs for tags. TaggedBookmark The through model linking Bookmarks and BookmarkTags. Bookmark A single URL added to Pinboard by a particular Account.

5.3 Managers

Bookmark models have several managers: Bookmark.objects.all() The default manager fetches all Bookmarks, posted by all Accounts, whether they’re public or private. Bookmark.public_objects.all() To display all Bookmarks on public-facing pages, public_objects should be used. It won’t return Bookmarks marked as private.

23 Django Ditto Documentation, Release 0.9.0

Bookmark.toread_objects.all() Returns all Bookmarks, public and private, that are marked as ‘To read’. Bookmark.public_toread_objects.all() Returns only public ‘To read’ Bookmarks. Private ‘To read’ Bookmarks will not be included. Of course, these can all be ﬁltered as usual. So, to display public ‘To read’ Bookmarks posted by a particular Account:

from ditto.pinboard.models import Account, Bookmark

account= Account.objects.get(username='philgyford') bookmarks= Bookmark.public_to_read_objects.filter(account=account)

5.4 Template tags

There are two template tags available for displaying Bookmarks in your templates.

5.4.1 Annual Bookmark Counts

Get the number of bookmarks per year for all or one Account. This fetches totals for all Accounts:

{% load ditto_pinboard %}

{% annual_bookmark_counts as counts %}

{% for row in counts %}

{{ row.year }}: {{ row.count }}

{% endfor %}

Both the year and count in each row are integers. To restrict totals to a single Account:

{% annual_bookmark_counts account='philgyford' as counts %}

5.4.2 Day Bookmarks

To display Bookmarks posted to Pinboard on a particular day, use day_bookmarks. By default it gets Bookmarks posted by all Accounts. In this example, my_date is a datetime.datetime.date type:

{% load ditto_pinboard %}

{% day_bookmarks my_date as bookmarks %}

{% for bookmark in bookmarks %}

{% endfor %}

Or we can restrict this to the Boomarks posted by a single account on that day:

{% day_bookmarks my_date account='philgyford' as bookmarks %}

24 Chapter 5. Pinboard Django Ditto Documentation, Release 0.9.0

5.4.3 Recent Bookmarks

To display the most recently-posted Bookmarks use recent_bookmarks. By default it gets the 10 most recent Bookmarks posted by all Accounts:

{% load ditto_pinboard %}

{% recent_bookmarks as bookmarks %}

{% for bookmark in bookmarks %}

{% endfor %}

The tag can also fetch a different number of Bookmarks and/or only get Bookmarks from a single Account. Here we only get the 5 most recent Bookmarks posted by the Account with a username of 'philgyford'.:

{% recent_bookmarks account='philgyford' limit=5 as bookmarks %}

5.4.4 Popular Bookmark Tags

To display the most common public Tags on public Bookmarks use popular_bookmark_tags. By default it gets the 10 Tags used most across all Accounts.

{% load ditto_pinboard %}

{% popular_bookmark_tags as tags %}

{% for tag in tags %}

{{ tag.name }} ({{ tag.num_times }})

{% endfor %}

The tag can also fetch a different number of Tags. Here we only get the 5 most popular Tags.:

{% popular_bookmark_tags limit=5 as tags %}

5.5 Management commands

Once you have set up an Account with your Pinboard API token (see above), you can fetch Bookmarks.

5.5.1 Fetch Bookmarks

Import all of your bookmarks:

$ ./manage.py fetch_pinboard_bookmarks --all

Periodically fetch the most recent bookmarks, eg 20 of them:

5.5. Management commands 25 Django Ditto Documentation, Release 0.9.0

$ ./manage.py fetch_pinboard_bookmarks --recent=20

Or fetch bookmarks posted on one date:

$ ./manage.py fetch_pinboard_bookmarks --date=2015-06-20

Or fetch a single bookmark by its URL (eg, if you’ve changed the description of a particular bookmark you’ve alread fetched):

$ ./manage.py fetch_pinboard_bookmarks --url=http://new-aesthetic.tumblr.com/

The above commands fetch bookmark(s) for all Accounts you’ve added. To restrict to a single account use --account with the Pinboard username, eg:

$ ./manage.py fetch_pinboard_bookmarks --recent=20 --account=philgyford

Be aware of the rate limits: https://pinboard.in/api/#limits

26 Chapter 5. Pinboard CHAPTER 6

Twitter

You can fetch, store and display data about your own Tweets and Liked Tweets for one or more Twitter accounts. By default the included models and templates will link to image and video ﬁles at Twitter, but you can also download the original ﬁles to store locally. These can then be used to serve different sizes of images locally. See the Fetch Files management command below.

6.1 Set-up

Go to https://apps.twitter.com/ and create a new Application for your Twitter account. You can leave the ‘Callback URL’ field empty. When viewing the Application Settings, click ‘manage keys and access tokens’. On the next screen click the ‘Create my access token’ button. Now, go to the Django admin, and add a new Account in the Twitter app. Copy the Consumer Key, Consumer Secret, Access Token and Access Token Secret from Twitter into the Django admin (ignore the other fields), and save the account. A new User object will be created, reflecting the Twitter user your API credentials are associated with. Once this is done you’ll need to import a downloaded archive of Tweets, and/or fetch Tweets from the Twitter API. See Management commands below.

6.2 Models

The models available in ditto.twitter.models are: Account Representing a Twitter account that has API credentials and that we fetch Tweets and/or Likes for. It has a one-to-one relationship with a User model. Media A photo, video or Animated GIF that was attached to one or more Tweets. Its media_type property differentiates between 'photo', 'video' and 'animated_gif'. (Yes, the plural model name is a bit confusing, sorry.) The ﬁles for photos and animated GIFs can be fetched from Twitter and used locally, although this isn’t the default. See the Fetch Files management command below.

27 Django Ditto Documentation, Release 0.9.0

Tweet A Tweet. It may have been posted by one of the Users with an Account that we fetch Twets for. Or it might have been posted by a different User and favorited/liked by one of the Users with an Account. User A Twitter user. Every time a Tweet is fetched a new User object is created/updated for the person who posted it. One or more of these Users will have an associated Account, as these are the user(s) we fetch data for. So, you will end up with many User objects in the system. But only one (or a few) will be associated with Account objects – these are the Users for which you fetch Tweets or favorited/liked Tweets.

6.3 Managers

6.3.1 Media

For Media objects there is only the standard manager available: Media.objects.all() Fetches all Media objects, attached to Tweets by anyone, whether public or private. There is no concept in Twitter of a ‘private’ photo or video. The same photo could be posted by different public and private Twitter users. Exercise caution when displaying Media objects to public webpages.

6.3.2 Tweet

Tweet models have several managers. There are two main differentiators we want to restrict things by: • Public vs private: Whether a Tweet was posted by a public or private Twitter user. • Posted Tweet vs Favorited Tweet: Sometimes we’ll want to only fetch Tweets posted by one of the Users with an API-credentialed Account. Sometimes we’ll want to only fetch Tweets those Accounts have favorited/liked. Sometimes we’ll want to fetch ALL Tweets, whether posted or favorited. On public-facing web pages you should only use the managers starting Tweet.public_ as these will ﬁlter out all Tweets posted by private Twitter users. Tweet.objects.all() The default manager fetches all Tweets in the system, posted by all Users, whether they’re public or private, and whether posted by an Account’s User or favorited by them. Tweet.public_objects.all() Only gets Tweets posted by any Users who are not private. This includes Tweets that have simply been favorited/liked by one of your Accounts. Tweet.tweet_objects.all() Only gets Tweets posted by a User with an Account, not favorited by them. And from both public and private Users. Tweet.public_tweet_objects.all() Only gets Tweets posted by a User with an Account, not favorited by them, but only for Users who aren’t private. For use on public pages. Tweet.favorite_objects.all() Only gets Tweets favorited by a User with an Account, whether those Tweets are posted by public or private Users. Tweet.public_favorite_objects.all() Only gets Tweets favorited by a User with an Account, and only Tweets posted by Users who aren’t private. Of course, these can all be ﬁltered as usual. So, if you wanted to get all the public Tweets posted by a particular User:

from ditto.twitter.models import User, Tweet

user= User.objects.get(screen_name='philgyford') tweets= Tweet.public_tweet_objects.filter(user=user)

And to get public Tweets that user has favorited:

28 Chapter 6. Twitter Django Ditto Documentation, Release 0.9.0

favorites= Tweet.public_favorite_objects.filter(user=user)

6.4 Template tags

There are four assigment template tags available for displaying Tweets in your templates.

6.4.1 Annual Favorite Counts

Get the number of favorites per year by all or one of the non-private Users-with-Accounts (only counting public Tweets):

{% load ditto_twitter %}

{% annual_favorite_counts as counts %}

{% for row in counts %}

{{ row.year }}: {{ row.count }}

{% endfor %}

Both the year and count in each row are integers. Or we can restrict this to favorites by a single User-with-an-Account:

{% annual_favorite_counts screen_name='philgyford' as counts %}

NOTE: The date used is the date the Tweets were posted on, not the date on which they were favorited.

6.4.2 Annual Tweet Counts

Get the number of Tweets per year by all or one of the non-private Users-with-Accounts:

{% load ditto_twitter %}

{% annual_tweet_counts as counts %}

{% for row in counts %}

{{ row.year }}: {{ row.count }}

{% endfor %}

Both the year and count in each row are integers. Or we can restrict this to Tweets posted by a single User-with-an-Account:

{% annual_tweet_counts screen_name='philgyford' as counts %}

6.4. Template tags 29 Django Ditto Documentation, Release 0.9.0

6.4.3 Day Favorites

Use this to fetch Tweets posted on a particular day by non-private Users, which have been favorited/liked by any of the Users-with-Accounts. Again, my_date is a datetime.datetime.date type:

{% load ditto_twitter %}

{% day_favorites my_date as favorites %}

NOTE: The date is the date the Tweets were posted on, not the date on which they were favorited. Again, we can restrict this to Tweets favorited by a single User-with-an-Account:

{% day_favorites my_date screen_name='philgyford' as favorites %}

6.4.4 Day Tweets

Gets Tweets posted on a particular day by any of the non-private Users-with-Accounts. In this example, my_date is a datetime.datetime.date type:

{% load ditto_twitter %}

{% day_tweets my_date as tweets %}

Or we can restrict this to Tweets posted by a single User-with-an-Account:

{% day_tweets my_date screen_name='philgyford' as tweets %}

6.4.5 Recent Favorites

This works like recent_tweets except it only fetches Tweets favorited/liked by our Users-with-Accounts, which were posted by public Twitter users:

{% load ditto_twitter %}

{% recent_favorites as favorites %}

Similarly, we can change the number of Tweets returned (10 by default), and only return Tweets favorited by a particular User:

{% recent_favorites screen_name='philgyford' limit=5 as favorites %}

6.4.6 Recent Tweets

To display the most recent Tweets posted by any of the non-private Users with an API-credentialled Account. By default the most recent 10 are fetched:

{% load ditto_twitter %}

{% recent_tweets as tweets %}

{% for tweet in tweets %} (continues on next page)

30 Chapter 6. Twitter Django Ditto Documentation, Release 0.9.0

(continued from previous page)

{{ tweet.user.name }} (@{{ tweet.user.screen_name }})
{{ tweet.text_html|safe }}

{% endfor %}

(There’s a lot more to displaying Tweets in full; see the included templates for examples.) The tag can also fetch a different number of Tweets and/or only get Tweets from a single User-with-an-Account. Here we only get the 5 most recent Tweets posted by the User with a screen_name of 'philgyford':

{% recent_tweets screen_name='philgyford' limit=5 as tweets %}

6.5 Management commands

6.5.1 Import Tweets

If you have more than 3,200 Tweets, you can only include older Tweets by downloading your archive and importing it. To do so, request your archive at https://twitter.com/settings/account . When you’ve downloaded it, do:

$ ./manage.py import_twitter_tweets --path=/Users/phil/Downloads/12552_

˓→dbeb4be9b8ff5f76d7d486c005cc21c9faa61f66 using the correct path to the directory you’ve downloaded and unzipped (in this case, the unzipped directory is 12552_dbeb4be9b8ff5f76d7d486c005cc21c9faa61f66). This will import all of the Tweets found in the archive.

6.5.2 Update Tweets

If you’ve imported your Tweets (above), you won’t yet have complete data about each one. To fully-populate those Tweets you should run this (replacing philgyford with your Twitter screen name):

$ ./manage.py update_twitter_tweets --account=philgyford

This will fetch data for up to 6,000 Tweets. If you have more than that in your archive, run it every 15 minutes to avoid rate limiting, until you’ve fetched all of the Tweets. This command will fetch data for the least-recently fetched. It’s worth running every so often in the future, to fetch the latest data (such as Retweet and Like counts).

6.5.3 Fetch Tweets

Periodically you’ll want to fetch the latest Tweets. This will fetch only those Tweets posted since you last fetched any:

$ ./manage.py fetch_twitter_tweets --account=philgyford --recent=new

You might also, or instead, want to fetch more than that. Here we fetch the most recent 200:

$ ./manage.py fetch_twitter_tweets --account=philgyford --recent=200

This would update data such as the Retweet and Like counts for all of the 200 fetched Tweets, even if they’re older than your last fetch. It’s ﬁne to fetch data about Tweets you’ve already fetched; their data will be updated.

6.5. Management commands 31 Django Ditto Documentation, Release 0.9.0

If you have more than one Twitter Account in Ditto, the above commands can be run across all of them by omitting the --account option. eg:

$ ./manage.py fetch_twitter_tweets --recent=new

6.5.4 Fetch Favorites

To fetch recent Tweets that all your Accounts have favorited/liked, run one of these:

$ ./manage.py fetch_twitter_favorites --recent=new $ ./manage.py fetch_twitter_favorites --recent=200

Or restrict to Tweets favorited/liked by a single Account:

$ ./manage.py fetch_twitter_favorites --account=philgyford --recent=new $ ./manage.py fetch_twitter_favorites --account=philgyford --recent=200

6.5.5 Update Users

When a Tweet of any kind is fetched, its User data is also stored, and the User’s proﬁle photo (avatar) is downloaded and stored in your project’s MEDIA_ROOT directory. You can optionally set the DITTO_TWITTER_DIR_BASE setting to change the location. The default is:

DITTO_TWITTER_DIR_BASE='twitter'

If your MEDIA_ROOT was set to /var/www/example.com/media/ then the above setting would save the proﬁle image for the user with a Twitter ID 12345678 to something like this:

/var/www/example.com/media/twitter/avatars/56/78/12345678/my_avatar.jpg

You may periodically want to update the stored data about all the Twitter users stored in Ditto. (quantity of Tweets, descriptions, etc). Do it like this:

$ ./manage.py update_twitter_users --account=philgyford

This requires an account as the data is fetched from that Twitter user’s point of view, when it comes to privacy etc.

6.5.6 Fetch Files (Media)

Fetching Tweets (whether your own or your favorites/likes) fetches all the data about them, but does not fetch any media files uploaded with them. There’s a separate command for fetching images and the MP4 video files created from animated GIFs. (There’s no way to fetch the videos, or original GIF files.) You must first download the Tweet data (above), and then you can fetch the files for all those Tweets:

$ ./manage.py fetch_twitter_files

This will fetch the files for all Tweets whose files haven’t already been fetched. So, the first time, it’s all the Tweets’ files, which can take a while. If you want to force a re-fetching of all files, whether they’ve already been downloaded or not:

$ ./manage.py fetch_twitter_files --all

32 Chapter 6. Twitter Django Ditto Documentation, Release 0.9.0

Each image/MP4 is associated with the relevant Tweet(s) and saved within your project’s MEDIA_ROOT directory, as deﬁned in settings.py. There’s one optional setting to customise the directory in which the ﬁles are saved. Its default value is as shown here:

DITTO_TWITTER_DIR_BASE='twitter'

Files are organised into separate directories according to the ﬁnal characters of their ﬁle names (so as not to have too many in one directory). eg, an image might be saved in:

/var/www/example.com/media/twitter/media/6T/ay/CRXEfBEWUAA6Tay.png

Every uploaded image, animated GIF and video should have a single image. Animated GIFs will also have an MP4 file. Once you’ve downloaded the original files, you can use these to generate all the different sizes of image required for your site, instead of linking direct to the image files on Twitter. To do this, ensure imagekit is in your INSTALLED_APPS setting:

INSTALLED_APPS=( # ... 'imagekit', # ... )

And add this to your settings.py (its default value is False):

DITTO_TWITTER_USE_LOCAL_MEDIA= True

Any requests in your templates for the URLs of photo files of any size will now use resized versions of your downloaded original files, generated by Imagekit. The first time you load a page (especially if it lists many images) it will be slow, but the images are cached (in a CACHE directory in your media folder). For example, before changing this setting, the URL of small image (Media.small_url) would be something like this:

https://pbs.twimg.com/media/CjuCDVLXIAALhYz.jpg:small

After choosing to use local photos, it would be something like this:

/media/CACHE/images/twitter/media/Lh/Yz/CjuCDVLXIAALhYz/

˓→5a726ea25d3bbd1b35b21b8b61b98c4c.jpg

If you change your mind you can switch back to using the images hosted on Twitter by removing the DITTO_TWITTER_USE_LOCAL_MEDIA setting or changing it to False. Animated GIFs are converted into MP4 videos when ﬁrst uploaded to Twitter. Ditto downloads and uses these in a similar way to images. ie, by default the video_url property of a Media object that’s an Animated GIF would be like:

https://pbs.twimg.com/tweet_video/CRXEfBEWUAA6Tay.mp4

If it’s been downloaded and DITTO_TWITTER_USE_LOCAL_MEDIA is True then calling video_url would return a URL like:

/media/twitter/media/6T/ay/CRXEfBEWUAA6Tay.mp4

However, there’s no way to download actual videos that were uploaded to Twitter, and so Ditto will always try to use videos hosted on Twitter, no matter what the value of DITTO_TWITTER_USE_LOCAL_MEDIA.

6.5. Management commands 33 Django Ditto Documentation, Release 0.9.0

6.5.7 Fetch Accounts

Deprecated: This only needs to be used whenever a new Account is added to the system. It fetches the User data for each Account that has API credentials, and associates the two objects.

$ ./manage.py fetch_twitter_accounts

I don’t think this is needed now.

34 Chapter 6. Twitter CHAPTER 7

Development

There’s a basic Django project in devproject/ to make it easier to work on the app. This might be enough to get things up and running (assumes pipenv is installed):

$ cd devproject $ pipenv install $ pipenv run ./manage.py migrate $ pipenv run ./manage.py runserver

7.1 Tests

Run tests with tox. Install it with:

$ pip install tox

You’ll need to have all versions of python available that are tested against (see tox.ini). This might mean deacti- vating a virtualenv if you’re using one with devproject/. Then run all tests in all environments like:

$ tox

To run tests in only one environment, specify it. In this case, Python 3.5 and Django 1.9:

$ tox -e py35-django19

To run a speciﬁc test, add its path after --, eg:

$ tox -e py35-django19 -- tests.flickr.test_views.HomeViewTests.test_home_templates

Running the tests in all environments will generate coverage output. There will also be an htmlcov/ directory containing an HTML report. You can also generate these reports without running all the other tests:

$ tox -e coverage

35 Django Ditto Documentation, Release 0.9.0

7.2 Other notes for development

7.2.1 Coverage

Using coverage.py to check test coverage:

$ coverage run --source='.' ./manage.py test $ coverage report

Instead of the in-terminal report, get an HTML version:

$ coverage html $ open -a "Google Chrome" htmlcov/index.html

7.2.2 Documentation

You’ll need sphinx installed. You could do this using pipenv and the Pipﬁles:

$ cd docs $ pipenv install

Build the documentation (assuming usage of pipenv):

$ cd docs $ pipenv run make html

7.2.3 Packaging

Set version number in ditto/__init__.py. Rebuild documentation (which includes the version number). Ensure CHANGES.rst is up-to-date for new version. Commit changes to git. Add a version tag:

$ python setup.py tag

Then:

$ python setup.py publish

36 Chapter 7. Development