Response to the 2004 European Commission consultation on DRM

September 14th, 2004

This is a response to the 2004 European Commission consultation on DRM, as submitted to the Commission.

Dear Sir/Madam,

In response to the consultation announced on http://europa.eu.int/information_society/eeurope/2005/all_about/digital_rights_man/text_en.htm, I would like to make a number of observations and comments related to the Final Report of the High Level Group on Digital Rights Management.

First of all, I welcome the fact that the Commission is taking an interest in DRM and engaging in discussion of related issues, but would reflect that the Report appears to be based on a number of questionable assumptions about DRM, and fails to take into account the broad spectrum of views about it, including some of those expressed in earlier Workshops. In particular, the Report appears to focus on what the content industry believes to be the problems with DRM, rather than what consumers (who are after all the ultimate enabling force) perceive as the problems. In this response, I hope to clarify my concerns and offer a consumer perspective on these issues.

On the assumption that DRM is required to develop a digital marketplace:

The Report repeatedly discusses the supposed “enabling” functions of DRM. There is a limited observational truth in calling DRM an “enabling” technology, but only so far as noting that large content providers in particular appear to have been reluctant to engage in digital distribution without the perceived “protection” of DRM. However, this evidence alone is not sufficient to merit the implicit assumption that DRM is required in order to enable broader establishment of consumer services. The reality is that many large content providers have not even tried digital distribution without DRM. DRM is not called for, required or desired by consumers; rather, it is something largely being imposed on them by content providers. Thus to start from an assumption that DRM is required to enable growth in consumer services is incorrect. Statements that DRM enables a variety of trading models such as rental and sale are disingenuous; such models have thrived without DRM for a long time even in markets where unauthorised copying and distribution are easy and economically viable. Without substantial supporting evidence, it is just as reasonable to assume that it is reluctance on the part of content providers to engage with their customers in the digital marketplace which is hampering the growth of that marketplace, not the limitations of (or lack of) DRM. To the contrary, the mere presence of DRM can itself hamper the growth of those very markets, due to the inherent consumer-hostile nature of such technology, a point I shall return to later.

On the implicit assumption that the word “Rights” in Digital Rights Management refers only to the “rights” of licensors:

The Report fails to discuss or even acknowledge the long-term consequences of a heavily DRM-based content marketplace. In particular, it places heavy emphasis on the “rights” of content producers, but fails to acknowledge the effect of the balancing rights of consumers and society in general, either in the development of digital marketplaces based on DRM or in the desirability of encouraging DRM at a national or European level. What is necessary is to recognise and take into account the historical context of copyright, which is not an absolute property right but rather an artificial, limited-time monopoly granted by society which is limited in scope and grants certain rights to society as well as to the producers.

Modern DRM systems often rely, to a significant extent, on essentially ephemeral media such as computer storage for license/key management and so on. Indeed, some DRM systems work based on the assumption of the continued existence of and ability to communicate with the content provider. Whilst in the embryonic market, these issues have not risen to the surface, in time there will be an impact when businesses providing DRM-restricted content on such a basis disappear or otherwise cease providing services on the same basis that they did originally. Placing rights management outside the hands of consumers in this way seriously damages confidence when purchasing content.

For example, I am a willing and keen adopter of new online services. I have every desire to participate as a consumer in a marketplace for digital content. However, at the present time the presence of DRM makes that marketplace considerably less attractive than traditional “physical” sales. Consider a compact disc containing music for example: if I purchase a legitimate disc from a retailer, I know that my rights and freedom to use that disc are entirely under my control, constrained only by my obligations under law: once the disc is in my possession, there is no way that the content owner can physically restrict my ability to use it. Similarly, I can use it with any device, from any manufacturer of my choosing, which is capable of playing CDs. However, with digital content hampered by DRM, the content producer may arbitrarily restrict my rights in any number of ways: I may only be able to use specific software or devices (perhaps from a specific manufacturer) which may not even be available to me, I may be prevented from exercising normal rights that I have under copyright laws (for example, related to use of copyright material for the purposes of review or research) and, worst of all, it’s possible (dependent on the DRM system employed) that the content owner might in future be able to prevent my use of the content either by action or omission. (An example of action would be remotely updating a DRM license key to revoke my rights to use the content; an example of omission would be if the content provider went bankrupt and the DRM software was no longer able to verify the validity of my license and consequently prevented me accessing the content). Predictability and interoperability, as identified in the Report, are important factors to consider, but they do not alter the underlying nature of DRM.

What I hope is now apparent is that there is a fundamental contradiction lying at the heart of the Report: that DRM (at least as used and envisaged in the context of mass-produced consumer content) is fundamentally an anti-consumer technology, and yet the Report tries to examine how a marketplace can be developed which is acceptable to consumers, and provide an upbeat assessment of the future. The weak conclusions of the Report serve to illustrate a number of key fallacies that lie at the heart of DRM, which are:

  1. that it is possible to unambiguously define and enforce, technologically, a set of licensing rules for a given work. This is logically impossible within the constraints of what is “reasonable”, primarily because the delicate balance of copyright law requires subjective judgements on human behaviour such as “intent” to be made. A computer or other DRM device cannot do this; it cannot, for example, determine whether the distribution of part of a copyright work falls under a permitted use allowed by copyright law, or whether it is infringing, because the technological steps involved in doing either might be identical.
  2. that it is possible to employ DRM without restricting the freedom of the marketplace or of consumers. For DRM to work, it unequivocally requires consumers to give up certain freedoms, in particular their freedom of choice and control over the playback devices which they possess as well as the content media which they purchase. Even “open” DRM standards ultimately rely on “closed” technology to be effective. This is aggressively anti-consumer, and is a concept irreconcilably opposed to the genuinely enabling tide of technological innovation, openness and freedom which has come about in recent years with the proliferation of digital technology.
  3. that it is possible to solve societal problems (for example, a lack of respect for copyright) with technology. Again, DRM serves to aggravate the situation here rather than alleviating it. Alienating consumers with DRM can only lead to an adversarial relationship between producer and consumer. In a monopoly marketplace (as the marketplace for content mostly is, particularly in the area of films and music) where consumers cannot choose between suppliers, giving consumers a choice between “not purchasing” and “purchasing with unacceptable restrictions” serves only to alienate, annoy and potentially criminalise law-abiding consumers who only wish to purchase content and make reasonable use of it, whilst those who happily deal in infringing materials will continue to do so; the fact that they have to circumvent technological “protection” measures is unlikely to be much of a deterrent.

You will, no doubt, recognise that many of the issues are quite general in nature. Nevertheless, I feel that they are highly relevant in the context of the Report: until these fundamental issues are openly acknowledged and discussed, the lesser issues that the Report identifies such as interoperability, privacy and security, are mostly red herrings; addressing them will only mildly improve the palatability of DRM to consumers, whilst the core problems of the relationship between producers and consumers in a digital marketplace remain unaddressed.

The truth is that DRM is a hugely expensive, intrusive and problematic solution to a largely illusory problem. Contrary to the apocalyptic visions put forward by certain stakeholders, digital media technology, with or without DRM, will not destroy the content industry. Rather than giving legitimacy to DRM, the Commission’s time could be spent more constructively by researching and reporting on business models which harness the power of digital media without requiring such restrictive and adversarial technology, and undoing the damage done by the grossly reactive, imbalanced and ill-conceived European Copyright Directive.

Thank you for considering these views as part of your consultation.

Yours,

Tim Jackson

Exim – checking maildir quotas at SMTP RCPT time

August 22nd, 2004

(This doc written in August 2004 but updated Jan 2009 to correct an error in the description of how Jeremy Harris’s solution works)

A discussion in August 2004 on the exim-users list about checking quotas at SMTP RCPT time with Exim sparked an interesting discussion. To summarise:

  • By default, when using Exim’s inbuilt quota support, messages for users who are over-quota will be accepted at SMTP time, and a bounce message will be subsequently created and sent to the envelope-sender of the original mail.
  • Checking quotas at SMTP RCPT time and issuing a suitable rejection message would be a better way of dealing with the problem. Many people, myself included, do not like accepting then bouncing mail in today’s world of spam and viruses where envelope senders are, as often as not, fake. This often creates “collateral spam” to innocent third parties. (And, incidentally, wastes bandwidth too).
  • It is not easy for Exim’s inbuilt quota management to allow checking of quotas at SMTP RCPT time, due to both architectural considerations (quotas are defined in transports and checked in delivery processes, which have little to do with message reception) and, more significantly, security/permission ones (at message reception time, Exim is typically running as an unprivileged user yet the checking of quotas normally requires root privileges).

A number of solutions were proposed: (if I mis-explain or misquote anyone, it’s not intentional – please let me know so I can put the record straight)

  • Jeremy Harris: Do some clever, but rather complicated and unreadable stuff with ACLs and databases to check the maildirsize file directly from Exim, using the SIZE command (if it exists) from a remote SMTP client to check at RCPT time and the real (received) message size at DATA time. Relies on Jeremy’s particular configuration (though could be modified for others) and doesn’t scale well to multiple MXes and requires calculation of the maildirsize file after each RCPT command.
  • Peter Bowyer: Use an external socket daemon (for example, Alun Jones‘s Exim socket daemon) to do the checks (implementing your own logic to determine whether or not a given user is over-quota), and use some kind of ${readsocket…} call from the Exim ACLs. This is probably the most elegant and ‘perfect’ way of doing it, though not the simplest and it does require the quota to be calculated at each RCPT command.
  • Greg Woods: Periodically run a script which checks for over-quota users, and write an override redirect/alias file for any users that are overquota, containing things like user: :fail:User is overquota. Greg provided a sample script for use with Cyrus which uses the ‘quota‘ helper program to determine whether users are over-quota.

Now, I was considering this problem too, and was quite inspired by Greg’s solution which seemed to be efficient, elegant, robust and easily scalable (to scale to multiple MXes, you would merely have to synchronise the single file containing the list of overquota users). However, I wanted a version which would work directly with maildirs (e.g. in a typical Courier-IMAP ‘virtual user’ configuration). Plus, I didn’t really see the need to write an entire redirect line per user.

So, I came up with a solution which involves:

  • a small script which iterates through a directory of (assumed) maildirs, calculates the quota usage (using the Maildir++ ‘maildirsize’ file) and dumps a list of overquota users in a linear list to a text file.
  • an Exim router to check the above file. Used in conjunction with a ‘verify = recipient‘ directive in a RCPT ACL, this will prompt rejection of over-quota users at SMTP RCPT time.
  • using the Exim ‘quota_is_inclusive = false‘ directive on the maildir transport which delivers to local mailboxes

The only downside to this method is that it’s not ‘real-time’; there is an interval (according to the frequency at which you run the script to check quotas) during which users can be overquota but will not be determined as such. This means that there is a small window during which bounces might still be generated; the severity of the problem varies according to how often you can afford to run the checker script.

My solution is presented below in the hope that other people may find it useful. It’s very much a first attempt, so there may be problems I have overlooked. If so, please let me know so that I can fix them.

Step 1: The Exim router

I am assuming that you have some kind of clearly-delineated virtual mailsystem, where all mail to be delivered to IMAP mailboxes is ultimately addressed to a/some specific domain(s) for this purpose, listed in a domainlist called +maildir_domains (I use a private namespace for deliveries; all ultimate local maildir deliveries, for example, will be addressed to username@maildir and all ‘real’ mail addresses (e.g. info@example.com) are aliased to this). Therefore, before the router which routes your maildir users’ mail, insert a router similar to the following:

maildir_overquota:
  driver = redirect
  domains = +maildir_domains
  local_parts = lsearch;/etc/exim/maildir_quota_exceeded
  data = :fail:Mailbox quota exceeded
  allow_fail

This looks up the user to be delivered to in the linear file /etc/exim/maildir_quota_exceeded (which we will generate later – see next step; it could of course easily be converted to a DBM/cdb/etc. file for performance if necessary) and for any users listed in that file, it will redirect to the special address “:fail: Mailbox quota exceeded” which, in a typical configuration, will cause the error “Maildir quota exceeded” to be returned to any user trying to send mail to an over-quota user, either locally via a generated bounce message or at SMTP time. Note that it will also fail any messages from the over-quota user, which may or may not be desirable.

Step 2: The quota-checking script

The next, and most important, step is to generate the list of over-quota users. To do this, I wrote a script called maildir-check-quotas (follow the link to download). This (simple) script assumes you have all your maildir folders in a single directory (/home/vmail by default, though you can easily change that). It iterates through each folder and, if it finds a maildirsize file, works out the quota usage. If a user is over-quota, it writes that users’ name to the file /etc/exim/maildir_quota_exceeded (again, easily configurable). You should run this script periodically (e.g. from cron), for example every five minutes or so (perhaps more, if you can support the load).

By the way, the script is in PHP. I’m sure it can be converted to other languages pretty easily if you have a preference.

Note: This script assumes you have Exim’s transport option maildir_use_size_file set for maildir deliveries, though it will fallback gracefully (assuming no quota) for mailboxes that do not have a maildirsize file.

A sample file as output from this script (assuming users ‘fred’, ‘bob’ and ‘mary’ were over-quota when it was run) would be:

# List of maildir users who are over-quota
# Auto-generated by maildir-check-quotas v1.0
# Generated at Sun, 22 Aug 2004 17:40:00 +0100
bob
fred
mary

To get verbose information when running the script, pass it the -v option.

Step 3: Making sure that users can actually go over-quota!

Now, the above is all very well, but by default the maildir-check-quotas script checks to see if a user has actually exceeded their quota (or matched it exactly, but that’s unlikely). In a typical configuration, however, Exim treats self-imposed (i.e. non-filesystem) quotas in a similar way to system quotas, and tries to prevent the user ever exceeding the quota. This means that a mail which would send a user over-quota will be rejected. However, this means that no users will ever exceed their quota and therefore the quota checking script will never find any over-quota users! This rather defeats the object of the exercise. There are three obvious solutions:

  • Start rejecting mail at some percentage threshold (e.g. 99%) of actual allowed quota. Not ideal, as this is an inexact science (unless all your quotas are the same and you increase quotas to compensate) and it might mean that you end up rejecting mail which wouldn’t actually have sent a user over-quota. Additionally, there is the possibility of generating a limited number of bounces for an indefinite period (if a user is under the percentage threshold, but a new mail would take them over ‘absolute’ limit). However, if you want to use this method you can with my script – just set $MAILBOX_FULL_PERCENTAGE to something less than 100%.
  • A variation of above, but have a fixed ‘threshold’ (e.g. 1MiB) – you then set all quotas to be threshold bytes over the actual desired quota, and then make the script check for mailboxes exceeding (quota − threshold). This is slightly more complicated, but would give users a (maximum) ‘grace’ allocation of threshold over their allocated quota. This method also suffers from the problem of potentially generating bounces for an indefinite period. I don’t currently support this with my script.
  • Use the Exim transport option ‘quota_is_inclusive‘ and set it to false. (My preferred option). That way, the last mail which will cause a user to go over-quota will be accepted, and the maildir-check-quotas script will therefore ‘trip’ on the next run. The obvious downside to this is, of course, that your users can go arbitrarily over-quota depending on the size of the final mail that causes them to go over-quota (and subject to your normal message size limits). This method still suffers from the problem of possibly generating bounces, although in this case only during the period between a user exceeding their quota and the next run of the script.

Put all this together and you should have a system which checks quotas simply and effectively and allows SMTP time rejection. I think that a readsocket{} check and accompanying daemon is probably still the “best” way to attack this problem (though, perhaps, less efficient – especially in the face of abusive behaviour from remote hosts, though some gentle caching could probably alleviate things), and I may experiment with that at a later stage, but for now I thought this method might prove useful to some people.