3
@page architecture Architecture
5
@section programming-language Programming Language
7
The media scanner is be implemented in C++, using Boost C++ and glib where useful. This is intended to match the typical APIs used in Ubuntu TV.
9
@section monitoring Monitoring
11
We monitor the standard XDG_VIDEOS_DIR directory, its sub folders, and any plugged-in removable media. The exact set of folders to be monitored are configurable via dconf.
13
The media scanner schedules a quick re-scan upon start up and when detecting the mounting of removable media because it cannot predict changes that happened while not monitoring those folders.
15
We usually fully trust this combination of fast re-scanning and monitoring to reliably find any media files. However, we support additionally scheduled scans, with the timing being configurable. These scheduled scans are disabled by default, and by default we prevent scheduled scans when the system is running on low battery. Battery status is monitored via libupower-glib.
17
@subsection monitoring-implementation Monitoring Implementation
19
We use GFileMonitor to watch directory changes. This is the standard API for watching files in GLib. Its Linux backend uses inotify.
21
@note Because the number of watch handles is limited for inotify, we could hit resource and performance limits. Hitting the inotify watch handle limit would cause us to miss file additions, removals and changes. However this is unlikely if we (and other applications) only watch the standard directories. Sadly fanotify cannot yet be used for file system monitoring since it lacks far too many features, such as notifications on file renaming or removal. Despite its limitations inotify is therefore the best choice for file system monitoring on Linux today. The typical user should have far less than 65,536 movie folders, which is the configured limit for user watches on Ubuntu 12.04.
23
@note To reduce the impact of inotify's resource limits the media scanner checks the kernel settings regarding the maximum number of inotify watches per user. If too many watches must be created then only the most recently changed directories are watched, and periodic scanning of folders is enabled. The exact threshold for entering this fallback mode is configurable.
25
@subsection removable-media-detection Removable Media Detection
27
We monitor the presence of removable media with GVolumeMonitor.
29
@note We also considered using Grilo's optical-media plugin for volume monitoring, but it doesn't currently expose sufficient information to serve as the basis for the media scanner. For instance, it doesn't clearly distinguish between optical drives and USB sticks. Also it doesn't expose UUIDs or similar for reliable media identification.
31
When indexing a media file we also store the removable medium's UUID. Matches for detached removable media are automatically dropped from search results.
33
If a medium is detached we just keep the entry in the database, because there is no reliable way to always mark the database entries as being offline files. For instance, sudden power loss would not give us the chance to make any database changes.
35
Once a medium is inserted again, we first revalidate our existing records by checking for file existence and comparing basic file system meta data. If validation succeeds, the file will be included again in future search results. To distinguish verified files from other files on the removable medium we store a "last-verified" timestamp. As soon all such files have been validated, we then start scanning the media for new files.
37
@section image-files Image Files
39
Currently GstDiscoverer doesn't emit EXIF or similar picture tags. Therefore, as of this writing, we depend on a patch (see https://bugzilla.gnome.org/show_bug.cgi?id=676542#c5 ) to GStreamer.
41
@section cover-art-and-synopsis Cover Art and Synopsis
43
We have created a Grilo TMDb plugin, which is now part of the regular Grilo code base, which we use to discover further details about the media files.
45
However, other Grilo plugins may be provided. The media scanner will query them in order of their priorities.
47
@section data-store-api Data Store API
49
The media catalog is stored in a Lucene++ index. It is searchable, it is quick, and it uses disk space efficiently.
51
We don't require applications, or Grilo, to use Lucene++ directly. Instead we provide a minimal C++ library to:
52
- Locate media files matching specified criteria.
53
- Explicitly update selected media file properties in the media index.
54
- Explicitly request re-scanning of all, or just a subset, of the files.
55
- Receive notifications about additions, changes and removal of media files matching specified criterion.
57
This library provides the foundation for our Grilo combined meta-data and media plug-in, allowing 3rd-party applications to conveniently access the media catalog via Grilo's API.
59
Note that updates (writes) change the local index, but do not attempt to change the original files or change other original sources of metadata.
61
To reliably support concurrent access without implementing complex locking schemes, we apply the "single writer, multiple readers" pattern: For reading data, the backend directly accesses the Lucene++ index, but all writes are executed by the media scanner. The media scanner provides a D-Bus interface for changing document properties and watching gallery changes. This D-Bus interface is be intended only for use by the access library. It should not be used by applications.
65
Grilo allows applications to iterate over sets of media files using text matching accompanied by a set of filtering options to further narrow down the result. For more complex searches we define and implement a query language, to be used by the client. Grilo supports a basic set of properties for each file type by default, but our plug-in extends the the list of supported properties.
67
Grilo allows property values to be written to the index. We enable writing for properties such as comments, rating, lastPlayed, playCount. Enabling writing for a given property is just a matter of toggling a flag in the schema definition. Note that updates (writes) will change the local index, but will not attempt to change the original files or change other original sources of metadata.
69
@section Architecture Diagram
71
This diagram shows the various parts of the mediascanner and how applications may use it via Grilo.
73
@image html media-scanner.png "The Architecture of the Ubuntu TV Media Scanner" width=10cm
added
removed
docs/architecture.dox
