1
/* Copyright (C) 2002-2003 Timo Sirainen */
4
Here's a description of how we handle Maildir synchronization and
7
We want to be as efficient as we can. The most efficient way to
8
check if changes have occured is to stat() the new/ and cur/
9
directories and uidlist file - if their mtimes haven't changed,
10
there's no changes and we don't need to do anything.
12
Problem 1: Multiple changes can happen within a single second -
13
nothing guarantees that once we synced it, someone else didn't just
14
then make a modification. Such modifications wouldn't get noticed
15
until a new modification occured later.
17
Problem 2: Syncing cur/ directory is much more costly than syncing
18
new/. Moving mails from new/ to cur/ will always change mtime of
19
cur/ causing us to sync it as well.
21
Problem 3: We may not be able to move mail from new/ to cur/
22
because we're out of quota, or simply because we're accessing a
29
Several checks below use MAILDIR_SYNC_SECS, which should be maximum
30
clock drift between all computers accessing the maildir (eg. via
31
NFS), rounded up to next second. Our default is 1 second, since
32
everyone should be using NTP.
34
Note that setting it to 0 works only if there's only one computer
35
accessing the maildir. It's practically impossible to make two
36
clocks _exactly_ synchronized.
38
It might be possible to only use file server's clock by looking at
39
the atime field, but I don't know how well that would actually work.
44
We have maildir_cur_dirty variable which is set to cur/ directory's
45
mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have
46
synchronized the directory.
48
When maildir_cur_dirty is non-zero, we don't synchronize the cur/
51
a) cur/'s mtime changes
52
b) opening a mail fails with ENOENT
53
c) time() > maildir_cur_dirty + MAILDIR_SYNC_SECS
55
This allows us to modify the maildir multiple times without having
56
to sync it at every change. The sync will eventually be done to
57
make sure we didn't miss any external changes.
59
The maildir_cur_dirty is set when:
61
- we change message flags
63
- we move mail from new/ to cur/
64
- we sync cur/ directory and it's mtime is
65
>= time() - MAILDIR_SYNC_SECS
67
It's unset when we do the final syncing, ie. when mtime is
68
older than time() - MAILDIR_SYNC_SECS.
73
If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize
74
it. maildir_cur_dirty-like feature might save us a few syncs, but
75
that might break a client which saves a mail in one connection and
76
tries to fetch it in another one. new/ directory is almost always
77
empty, so syncing it should be very fast anyway. Actually this can
78
still happen if we sync only new/ dir while another client is also
79
moving mails from it to cur/ - it takes us a while to see them.
80
That's pretty unlikely to happen however, and only way to fix it
81
would be to always synchronize cur/ after new/.
83
Normally we move all mails from new/ to cur/ whenever we sync it. If
84
it's not possible for some reason, we set maildir_have_new flag on
85
which instructs synchronization to check files in new/ directory as
86
well. maildir_keep_new flag is also set which instructs syncing to
87
not even try to move mails to cur/ anymore.
89
If client tries to change a flag for message in new/, we try to
90
rename() it into cur/. If it's successful, we clear the
91
maildir_keep_new flag so at next sync we'll try to move all of them
92
to cur/. When all of them have been moved, maildir_have_new flag is
93
cleared as well. Expunges will also clear maildir_keep_new flag.
95
If rename() still fails because of ENOSPC or EDQUOT, we still save
96
the flag changes in index with dirty-flag on. When moving the mail
97
to cur/ directory, or when we notice it's already moved there, we
98
apply the flag changes to the filename, rename it and remove the
99
dirty flag. If there's dirty flags, this should be tried every time
100
after expunge or when closing the mailbox.
105
This file contains UID <-> filename mappings. It's updated only when
106
new mail arrives, so it may contain filenames that have already been
107
deleted. Updating is done by getting uidlist.lock file, writing the
108
whole uidlist into it and rename()ing it over the old uidlist. This
109
means there's no need to lock the file for reading.
111
Whenever uidlist is rewritten, it's mtime must be larger than the old
112
one's. Use utime() before rename() if needed.
114
Only time you have to read this file is when assigning new UIDs for
115
messages, to see if they already have UIDs. If file's mtime hasn't
116
changed, you don't have to do even that.
121
Originally the middle identifier in Maildir filename was specified
122
only as <process id>_<delivery counter>. That however created a
123
problem with randomized PIDs which made it possible that the same
124
PID was reused within one second.
126
So if within one second a mail was delivered, MUA moved it to cur/
127
and another mail was delivered by a new process using same PID as
128
the first one, we likely ended up overwriting the first mail when
129
the second mail was moved over it.
131
Nowadays everyone should be giving a bit more specific identifier,
132
for example include microseconds in it which Dovecot does.
134
There's a simple way to prevent this from happening in some cases:
135
Don't move the mail from new/ to cur/ if it's mtime is >= time() -
136
MAILDIR_SYNC_SECS. The second delivery's link() call then fails
137
because the file is already in new/, and it will then use a
138
different filename. There's a few problems with this however:
140
- it requires extra stat() call which is unneeded extra I/O
141
- another MUA might still move the mail to cur/
142
- if first file's flags are modified by either Dovecot or another
143
MUA, it's moved to cur/ (you _could_ just do the dirty-flagging
146
Because this is useful only for very few people and it requires
147
extra I/O, I decided not to implement this. It should be however
148
quite easy to do since we need to be able to deal with files in new/
151
It's also possible to never accidentally overwrite a mail by using
152
link() + unlink() rather than rename(). This however isn't very
153
good idea as it introduces potential race conditions when multiple
154
clients are accessing the mailbox:
156
Trying to move the same mail from new/ to cur/ at the same time:
158
a) Client 1 uses slightly different filename than client 2,
159
for example one sets read-flag on but the other doesn't.
160
You have the same mail duplicated now.
162
b) Client 3 sees the mail between Client 1's and 2's link() calls
163
and changes it's flag. You have the same mail duplicated now.
165
And it gets worse when they're unlink()ing in cur/ directory:
167
c) Client 1 changes mails's flag and client 2 changes it back
168
between 1's link() and unlink(). The mail is now expunged.
170
d) If you try to deal with the duplicates by unlink()ing another
171
one of them, you might end up unlinking both of them.
173
So, what should we do then if we notice a duplicate? First of all,
174
it might not be a duplicate at all, readdir() might have just
175
returned it twice because it was just renamed. What we should do is
176
create a completely new base name for it and rename() it to that.
177
If the call fails with ENOENT, it only means that it wasn't a
187
#include "maildir-index.h"
188
#include "maildir-uidlist.h"
189
#include "mail-index-data.h"
190
#include "mail-index-util.h"
198
#include <sys/stat.h>
200
#define MAILDIR_SYNC_SECS 1
202
enum maildir_file_action {
203
MAILDIR_FILE_ACTION_EXPUNGE,
204
MAILDIR_FILE_ACTION_UPDATE_FLAGS,
205
MAILDIR_FILE_ACTION_UPDATE_CONTENT,
206
MAILDIR_FILE_ACTION_NEW,
207
MAILDIR_FILE_ACTION_NONE,
209
MAILDIR_FILE_FLAG_NEWDIR = 0x1000,
210
MAILDIR_FILE_FLAG_ALLOCED = 0x2000,
211
MAILDIR_FILE_FLAGS = 0x3000
214
struct maildir_hash_context {
215
struct mail_index *index;
216
struct mail_index_record *new_mail;
221
struct maildir_hash_rec {
222
struct mail_index_record *rec;
223
enum maildir_file_action action;
225
#define ACTION(hash) ((hash)->action & ~MAILDIR_FILE_FLAGS)
227
struct maildir_sync_context {
228
struct mail_index *index;
229
const char *new_dir, *cur_dir;
232
struct hash_table *files;
233
unsigned int new_count;
236
struct dirent *new_dent;
238
struct maildir_uidlist *uidlist;
239
unsigned int readonly_check:1;
240
unsigned int flag_updates:1;
241
unsigned int uidlist_rewrite:1;
242
unsigned int new_mails_new:1;
243
unsigned int new_mails_cur:1;
246
static int maildir_sync_cur_dir(struct maildir_sync_context *ctx);
248
/* a char* hash function from ASU -- from glib */
249
static unsigned int maildir_hash(const void *p)
251
const unsigned char *s = p;
252
unsigned int g, h = 0;
254
while (*s != ':' && *s != '\0') {
256
if ((g = h & 0xf0000000UL)) {
266
static int maildir_cmp(const void *p1, const void *p2)
268
const char *s1 = p1, *s2 = p2;
270
while (*s1 == *s2 && *s1 != ':' && *s1 != '\0') {
273
if ((*s1 == '\0' || *s1 == ':') &&
274
(*s2 == '\0' || *s2 == ':'))
279
static void maildir_update_filename_memory(struct mail_index *index,
284
if (index->new_filename_pool == NULL) {
285
index->new_filename_pool =
286
pool_alloconly_create("Maildir fname", 10240);
288
if (index->new_filenames == NULL) {
289
index->new_filenames =
290
hash_create(system_pool, index->new_filename_pool, 0,
291
maildir_hash, maildir_cmp);
294
new_fname = p_strdup(index->new_filename_pool, fname);
295
hash_insert(index->new_filenames, new_fname, new_fname);
298
static int maildir_update_filename(struct maildir_sync_context *ctx,
299
struct mail_index_record *rec,
300
const char *new_fname)
302
struct mail_index_update *update;
304
if (ctx->index->lock_type != MAIL_LOCK_EXCLUSIVE) {
305
maildir_update_filename_memory(ctx->index, new_fname);
309
update = ctx->index->update_begin(ctx->index, rec);
310
ctx->index->update_field(update, DATA_FIELD_LOCATION, new_fname, 0);
311
return ctx->index->update_end(update);
314
static int maildir_update_flags(struct maildir_sync_context *ctx,
315
struct mail_index_record *rec,
316
unsigned int seq, const char *new_fname)
318
enum mail_flags flags;
320
if (ctx->index->lock_type != MAIL_LOCK_EXCLUSIVE)
323
flags = maildir_filename_get_flags(new_fname, rec->msg_flags);
324
if (flags != rec->msg_flags) {
325
if (!ctx->index->update_flags(ctx->index, rec,
333
static int maildir_sync_open_uidlist(struct maildir_sync_context *ctx)
335
struct mail_index *index = ctx->index;
339
if (ctx->uidlist != NULL)
342
/* open it only if it's changed since we last synced it. */
343
path = t_strconcat(index->control_dir, "/" MAILDIR_UIDLIST_NAME, NULL);
344
if (stat(path, &st) < 0) {
345
if (errno == ENOENT) {
346
/* doesn't exist yet, create it */
347
switch (maildir_uidlist_try_lock(ctx->index)) {
351
ctx->uidlist_rewrite = TRUE;
357
return index_file_set_syscall_error(index, path, "stat()");
360
/* FIXME: last_uidlist_mtime should be in index headers */
361
if (st.st_mtime == index->last_uidlist_mtime)
364
ctx->uidlist = maildir_uidlist_open(index);
365
if (ctx->uidlist == NULL)
368
if (ctx->uidlist->uid_validity != index->header->uid_validity) {
369
/* uidvalidity changed */
370
if (!index->rebuilding && index->opened) {
371
index_set_corrupted(index,
372
"UIDVALIDITY changed in uidlist");
376
if (!index->rebuilding) {
377
index->set_flags |= MAIL_INDEX_FLAG_REBUILD;
381
index->header->uid_validity = ctx->uidlist->uid_validity;
382
i_assert(index->header->next_uid == 1);
385
if (index->header->next_uid > ctx->uidlist->next_uid) {
386
index_set_corrupted(index, "index.next_uid (%u) > "
387
"uidlist.next_uid (%u)",
388
index->header->next_uid,
389
ctx->uidlist->next_uid);
396
static int is_file_content_changed(struct mail_index *index,
397
struct mail_index_record *rec,
398
const char *dir, const char *fname)
400
#define DATA_HDR_SIZE (DATA_HDR_HEADER_SIZE | DATA_HDR_BODY_SIZE)
401
struct mail_index_data_record_header *data_hdr;
405
if ((rec->data_fields & DATA_HDR_INTERNAL_DATE) == 0 &&
406
(rec->data_fields & DATA_HDR_SIZE) != DATA_HDR_SIZE) {
407
/* nothing in cache, we can't know if it's changed */
412
path = t_strdup_printf("%s/%s", dir, fname);
414
if (stat(path, &st) < 0) {
416
index_file_set_syscall_error(index, path, "stat()");
422
data_hdr = mail_index_data_lookup_header(index->data, rec);
423
if (data_hdr == NULL)
426
if ((rec->data_fields & DATA_HDR_INTERNAL_DATE) != 0 &&
427
st.st_mtime != data_hdr->internal_date)
430
if ((rec->data_fields & DATA_HDR_SIZE) == DATA_HDR_SIZE &&
431
(uoff_t)st.st_size != data_hdr->body_size + data_hdr->header_size)
437
static void uidlist_hash_get_filenames(void *key, void *value, void *context)
439
buffer_t *buf = context;
440
struct maildir_hash_rec *hash_rec = value;
442
if (ACTION(hash_rec) == MAILDIR_FILE_ACTION_NEW)
443
buffer_append(buf, (const void *) &key, sizeof(const char *));
446
static int maildir_time_cmp(const void *p1, const void *p2)
448
const char *s1 = *((const char **) p1);
449
const char *s2 = *((const char **) p2);
450
time_t t1 = 0, t2 = 0;
452
/* we have to do numeric comparision, strcmp() will break when
453
there's different amount of digits (mostly the 999999999 ->
454
1000000000 change in Sep 9 2001) */
455
while (*s1 >= '0' && *s1 <= '9') {
456
t1 = t1*10 + (*s1 - '0');
459
while (*s2 >= '0' && *s2 <= '9') {
460
t2 = t2*10 + (*s2 - '0');
464
return t1 < t2 ? -1 : t1 > t2 ? 1 : 0;
467
static int maildir_full_sync_finish(struct maildir_sync_context *ctx)
469
struct mail_index *index = ctx->index;
470
struct maildir_uidlist *uidlist;
471
struct mail_index_record *rec;
472
struct maildir_hash_rec *hash_rec;
473
struct maildir_uidlist_rec uid_rec;
474
enum maildir_file_action action;
475
const char *fname, **new_files, *dir;
476
void *orig_key, *orig_value;
477
unsigned int seq, uid, last_uid, i, new_flag;
481
if (ctx->new_count > 0) {
482
/* new mails, either they're already in uidlist or we have
483
to add them there. If we want to add them, we'll need to
485
if (maildir_uidlist_try_lock(ctx->index) < 0)
488
if (!maildir_sync_open_uidlist(ctx))
493
rec = index->lookup(index, 1);
494
uidlist = ctx->uidlist;
497
memset(&uid_rec, 0, sizeof(uid_rec));
499
if (maildir_uidlist_next(uidlist, &uid_rec) < 0)
503
while (rec != NULL) {
504
seq++; uid = rec->uid;
506
/* skip over the expunged records in uidlist */
507
while (uid_rec.uid != 0 && uid_rec.uid < uid) {
508
if (maildir_uidlist_next(uidlist, &uid_rec) < 0)
512
fname = maildir_get_location(index, rec);
516
if (!hash_lookup_full(ctx->files, fname,
517
&orig_key, &orig_value)) {
521
hash_rec = orig_value;
524
if (uid_rec.uid == uid &&
525
maildir_cmp(fname, uid_rec.filename) != 0) {
526
index_set_corrupted(index,
527
"Filename mismatch for UID %u: %s vs %s",
528
uid, fname, uid_rec.filename);
532
if (uid_rec.uid > uid && hash_rec != NULL &&
533
(ACTION(hash_rec) == MAILDIR_FILE_ACTION_UPDATE_FLAGS ||
534
ACTION(hash_rec) == MAILDIR_FILE_ACTION_NONE)) {
535
/* it's UID has changed. shouldn't happen. */
536
i_warning("UID changed for %s/%s: %u -> %u",
537
index->mailbox_path, fname, uid, uid_rec.uid);
538
hash_rec->action = MAILDIR_FILE_ACTION_UPDATE_CONTENT |
539
(hash_rec->action & MAILDIR_FILE_FLAGS);
542
action = hash_rec != NULL ?
543
ACTION(hash_rec) : MAILDIR_FILE_ACTION_NONE;
545
case MAILDIR_FILE_ACTION_EXPUNGE:
546
if (!index->expunge(index, rec, seq, TRUE))
550
case MAILDIR_FILE_ACTION_UPDATE_FLAGS:
551
if (!maildir_update_filename(ctx, rec, orig_key))
553
if (!maildir_update_flags(ctx, rec, seq, orig_key))
556
case MAILDIR_FILE_ACTION_UPDATE_CONTENT:
557
if (!index->expunge(index, rec, seq, TRUE))
560
hash_rec->action = MAILDIR_FILE_ACTION_NEW |
561
(hash_rec->action & MAILDIR_FILE_FLAGS);
564
case MAILDIR_FILE_ACTION_NONE:
567
i_panic("BUG: %s/%s suddenly appeared as UID %u",
568
index->mailbox_path, (char *) orig_key, uid);
571
if (uid_rec.uid == uid) {
572
if (maildir_uidlist_next(uidlist, &uid_rec) < 0)
575
rec = index->next(index, rec);
578
if (seq != index->header->messages_count) {
579
index_set_corrupted(index, "Wrong messages_count in header "
581
index->header->messages_count);
585
/* if there's new mails which are already in uidlist, get them */
587
while (uid_rec.uid != 0) {
588
if (hash_lookup_full(ctx->files, uid_rec.filename,
589
&orig_key, &orig_value))
590
hash_rec = orig_value;
594
if (hash_rec != NULL &&
595
ACTION(hash_rec) == MAILDIR_FILE_ACTION_NONE) {
596
/* it's a duplicate, shouldn't happen */
597
i_error("%s: Found duplicate filename %s, rebuilding",
598
ctx->uidlist->fname, uid_rec.filename);
599
(void)unlink(ctx->uidlist->fname);
601
if (INDEX_IS_UIDLIST_LOCKED(index))
602
ctx->uidlist_rewrite = TRUE;
606
if (hash_rec != NULL) {
607
i_assert(ACTION(hash_rec) == MAILDIR_FILE_ACTION_NEW);
609
/* make sure we set the same UID for it. */
610
if (index->header->next_uid > uid_rec.uid) {
611
index_set_corrupted(index,
612
"index.next_uid (%u) > "
614
index->header->next_uid,
618
index->header->next_uid = uid_rec.uid;
620
new_flag = hash_rec->action & MAILDIR_FILE_FLAG_NEWDIR;
621
hash_rec->action = MAILDIR_FILE_ACTION_NONE | new_flag;
625
ctx->index->maildir_have_new = TRUE;
626
dir = new_flag != 0 ? ctx->new_dir : ctx->cur_dir;
628
if (!maildir_index_append_file(index, dir, orig_key,
633
if (maildir_uidlist_next(uidlist, &uid_rec) < 0)
637
if (ctx->uidlist != NULL) {
638
/* update our next_uid. it should have been checked for
642
i_assert(index->header->next_uid <= ctx->uidlist->next_uid);
643
index->header->next_uid = ctx->uidlist->next_uid;
645
/* uidlist is now synced, remember that. */
646
if (fstat(i_stream_get_fd(ctx->uidlist->input), &st) < 0) {
647
return index_file_set_syscall_error(index,
651
index->last_uidlist_mtime = st.st_mtime;
654
if (ctx->new_count == 0 || !INDEX_IS_UIDLIST_LOCKED(index)) {
655
/* all done (or can't do it since we don't have lock) */
659
ctx->uidlist_rewrite = TRUE;
661
/* then there's the completely new mails. sort them by the filename
662
so we should get them to same order as they were created. */
663
buf = buffer_create_static_hard(ctx->pool,
664
ctx->new_count * sizeof(const char *));
665
hash_foreach(ctx->files, uidlist_hash_get_filenames, buf);
666
i_assert(buffer_get_used_size(buf) ==
667
ctx->new_count * sizeof(const char *));
669
new_files = buffer_get_modifyable_data(buf, NULL);
670
qsort(new_files, ctx->new_count, sizeof(const char *),
673
if (!index->maildir_keep_new) {
677
/* this is actually slightly wrong, because we don't really
678
know if some of the new messages are in cur/ already.
679
we could know that by saving it into buffer, but that'd
680
require extra memory. luckily it doesn't really matter if
681
we say it's in new/, but it's actually in cur/. we have
682
to deal with such case anyway since another client might
683
have just moved it. */
686
ctx->index->maildir_have_new = TRUE;
689
for (i = 0; i < ctx->new_count; i++) {
690
if (!maildir_index_append_file(index, dir,
691
new_files[i], new_dir))
699
static int maildir_full_sync_init(struct maildir_sync_context *ctx,
702
struct mail_index *index = ctx->index;
703
struct mail_index_record *rec;
704
struct maildir_hash_rec *hash_rec;
709
/* FIXME: kludge. we want to have pointers to data file, so we must
710
make sure that it's base address doesn't change. this call makes
711
sure it's fully mmaped in memory even when we begin */
712
if (mail_index_data_get_mmaped(index->data, &size) == NULL)
715
if (index->header->messages_count >= INT_MAX/32) {
716
index_set_corrupted(index, "Header says %u messages",
717
index->header->messages_count);
721
/* read current messages in index into hash */
722
size = nearest_power(index->header->messages_count *
723
sizeof(struct maildir_hash_rec) + 1024);
724
ctx->pool = pool_alloconly_create("maildir sync", I_MAX(size, 16384));
725
ctx->files = hash_create(default_pool, ctx->pool,
726
index->header->messages_count * 2,
727
maildir_hash, maildir_cmp);
732
rec = index->lookup(index, 1);
733
while (rec != NULL) {
734
fname = maildir_get_location(index, rec);
738
if ((rec->index_flags & INDEX_MAIL_FLAG_MAILDIR_NEW) != 0)
742
(rec->index_flags & INDEX_MAIL_FLAG_MAILDIR_NEW) != 0) {
743
hash_rec = p_new(ctx->pool, struct maildir_hash_rec, 1);
745
hash_rec->action = MAILDIR_FILE_ACTION_EXPUNGE;
747
if (hash_lookup(ctx->files, fname) != NULL) {
748
index_set_corrupted(index,
749
"Duplicated message %s", fname);
753
/* WARNING: index must not be modified as long as
754
these hash keys exist. Modifying might change the
755
mmap base address. */
756
hash_insert(ctx->files, (void *) fname, hash_rec);
759
rec = index->next(index, rec);
762
index->maildir_have_new = have_new;
766
static int maildir_fix_duplicate(struct mail_index *index,
767
const char *old_fname, int new_dir)
769
const char *new_fname, *old_path, *new_path;
774
old_path = t_strconcat(index->mailbox_path, new_dir ? "/new/" : "/cur/",
777
new_fname = maildir_generate_tmp_filename(&ioloop_timeval);
778
new_path = t_strconcat(index->mailbox_path, "/new/", new_fname, NULL);
780
if (rename(old_path, new_path) == 0) {
781
i_warning("Fixed duplicate in %s: %s -> %s",
782
index->mailbox_path, old_fname, new_fname);
783
} else if (errno != ENOENT) {
784
index_set_error(index, "rename(%s, %s) failed: %m",
793
static void uidlist_hash_fix_allocs(void *key, void *value, void *context)
795
struct maildir_sync_context *ctx = context;
796
struct maildir_hash_rec *hash_rec = value;
798
switch (ACTION(hash_rec)) {
799
case MAILDIR_FILE_ACTION_NONE:
800
hash_remove(ctx->files, key);
802
case MAILDIR_FILE_ACTION_EXPUNGE:
803
if (hash_rec->action & MAILDIR_FILE_FLAG_ALLOCED) {
804
/* we're getting here because our recently
805
inserted node is traversed as well */
809
hash_rec->action |= MAILDIR_FILE_FLAG_ALLOCED;
810
hash_insert(ctx->files, p_strdup(ctx->pool, key), value);
817
static int maildir_full_sync_dir(struct maildir_sync_context *ctx,
818
const char *dir, int new_dir,
819
DIR *dirp, struct dirent *d)
821
struct maildir_hash_rec *hash_rec;
822
void *orig_key, *orig_value;
823
int check_content_changes, newflag;
825
newflag = new_dir ? MAILDIR_FILE_FLAG_NEWDIR : 0;
827
/* Do we want to check changes in file contents? This slows down
828
things as we need to do extra stat() for all files. */
829
check_content_changes = !ctx->readonly_check &&
830
getenv("MAILDIR_CHECK_CONTENT_CHANGES") != NULL;
833
if (d->d_name[0] == '.')
836
if (!hash_lookup_full(ctx->files, d->d_name,
837
&orig_key, &orig_value)) {
838
hash_rec = p_new(ctx->pool, struct maildir_hash_rec, 1);
840
hash_rec = orig_value;
841
if (ACTION(hash_rec) != MAILDIR_FILE_ACTION_EXPUNGE) {
842
if (!maildir_fix_duplicate(ctx->index,
849
if (hash_rec->rec == NULL) {
851
if (ctx->readonly_check)
855
ctx->new_mails_new = TRUE;
857
ctx->new_mails_cur = TRUE;
860
hash_rec->action = MAILDIR_FILE_ACTION_NEW | newflag;
861
hash_insert(ctx->files, p_strdup(ctx->pool, d->d_name),
866
if (!new_dir && (hash_rec->rec->index_flags &
867
INDEX_MAIL_FLAG_MAILDIR_NEW) != 0 &&
868
ctx->index->lock_type == MAIL_LOCK_EXCLUSIVE) {
869
/* mail was indexed in new/ but it has been
870
moved to cur/ later */
871
hash_rec->rec->index_flags &=
872
~INDEX_MAIL_FLAG_MAILDIR_NEW;
875
if (check_content_changes &&
876
is_file_content_changed(ctx->index, hash_rec->rec,
878
/* file content changed, treat it as new message */
880
MAILDIR_FILE_ACTION_UPDATE_CONTENT | newflag;
882
hash_insert(ctx->files, p_strdup(ctx->pool, d->d_name),
884
} else if (strcmp(orig_key, d->d_name) != 0) {
886
MAILDIR_FILE_ACTION_UPDATE_FLAGS | newflag;
888
hash_insert(ctx->files, p_strdup(ctx->pool, d->d_name),
890
ctx->flag_updates = TRUE;
892
hash_rec->action = MAILDIR_FILE_ACTION_NONE | newflag;
894
} while ((d = readdir(dirp)) != NULL);
896
/* records that are left to hash must not have any (filename) pointers
897
to index file. So remove none actions, and p_strdup() expunge
899
hash_foreach(ctx->files, uidlist_hash_fix_allocs, ctx);
904
static int maildir_new_scan_first_file(struct maildir_sync_context *ctx)
909
dirp = opendir(ctx->new_dir);
911
return index_file_set_syscall_error(ctx->index, ctx->new_dir,
915
/* find first file */
916
while ((d = readdir(dirp)) != NULL) {
917
if (d->d_name[0] != '.')
922
if (closedir(dirp) < 0) {
923
index_file_set_syscall_error(ctx->index, ctx->new_dir,
927
ctx->new_dirp = dirp;
934
static int maildir_full_sync_dirs(struct maildir_sync_context *ctx)
939
if (ctx->new_dirp == NULL &&
940
(ctx->index->maildir_have_new || ctx->index->maildir_keep_new)) {
941
if (!maildir_new_scan_first_file(ctx))
945
if (ctx->new_dent != NULL) {
946
if (!maildir_full_sync_dir(ctx, ctx->new_dir, TRUE,
947
ctx->new_dirp, ctx->new_dent))
949
ctx->new_dent = NULL;
952
dirp = opendir(ctx->cur_dir);
954
return index_file_set_syscall_error(ctx->index, ctx->cur_dir,
958
failed = !maildir_full_sync_dir(ctx, ctx->cur_dir, FALSE,
959
dirp, readdir(dirp));
961
if (closedir(dirp) < 0) {
962
return index_file_set_syscall_error(ctx->index, ctx->cur_dir,
969
static int maildir_sync_new_dir_full(struct maildir_sync_context *ctx)
971
if (!ctx->index->set_lock(ctx->index, MAIL_LOCK_EXCLUSIVE))
974
if (!maildir_full_sync_init(ctx, TRUE))
977
if (!maildir_full_sync_dir(ctx, ctx->new_dir, TRUE,
978
ctx->new_dirp, ctx->new_dent))
980
ctx->new_dent = NULL;
982
if (!maildir_full_sync_finish(ctx))
988
static int maildir_sync_new_dir(struct maildir_sync_context *ctx,
989
int move_to_cur, int append_index)
992
string_t *sourcepath, *destpath;
993
const char *final_dir;
996
if (ctx->index->maildir_have_new) {
997
/* some of the mails in new/ are already indexed.
998
we'll have to do a full sync. */
999
return maildir_sync_new_dir_full(ctx);
1002
if (!ctx->index->set_lock(ctx->index, MAIL_LOCK_EXCLUSIVE))
1005
switch (maildir_uidlist_try_lock(ctx->index)) {
1009
/* couldn't get a lock.
1010
no point in doing more. */
1014
/* make sure uidlist is up to date.
1015
if it's not, do a full sync. */
1016
if (!maildir_sync_open_uidlist(ctx))
1019
if (ctx->uidlist != NULL)
1020
return maildir_sync_cur_dir(ctx);
1022
ctx->uidlist_rewrite = TRUE;
1026
ctx->new_dent = NULL;
1028
sourcepath = t_str_new(PATH_MAX);
1029
destpath = t_str_new(PATH_MAX);
1031
final_dir = move_to_cur ? ctx->cur_dir : ctx->new_dir;
1034
if (d->d_name[0] == '.')
1037
str_truncate(sourcepath, 0);
1038
str_printfa(sourcepath, "%s/%s", ctx->new_dir, d->d_name);
1041
str_truncate(destpath, 0);
1042
str_printfa(destpath, "%s/%s", ctx->cur_dir, d->d_name);
1043
if (strchr(d->d_name, ':') == NULL)
1044
str_append(destpath, ":2,");
1046
if (rename(str_c(sourcepath), str_c(destpath)) < 0 &&
1048
if (ENOSPACE(errno))
1049
ctx->index->nodiskspace = TRUE;
1050
else if (errno == EACCES)
1051
ctx->index->mailbox_readonly = TRUE;
1053
index_set_error(ctx->index,
1054
"rename(%s, %s) failed: %m",
1060
ctx->index->maildir_keep_new = TRUE;
1061
if (!append_index) {
1066
/* continue by keeping them in new/ dir */
1067
final_dir = ctx->new_dir;
1068
move_to_cur = FALSE;
1074
ctx->index->maildir_have_new = TRUE;
1077
if (!maildir_index_append_file(ctx->index, final_dir,
1085
} while ((d = readdir(ctx->new_dirp)) != NULL);
1090
static int maildir_sync_cur_dir(struct maildir_sync_context *ctx)
1092
struct mail_index *index = ctx->index;
1094
if (ctx->new_dent != NULL && !index->maildir_keep_new) {
1095
/* there's also new mails. move them into cur/ first, if we
1096
can lock the uidlist */
1097
switch (maildir_uidlist_try_lock(index)) {
1101
if (!maildir_sync_new_dir(ctx, TRUE, FALSE))
1106
if (!index->set_lock(index, MAIL_LOCK_EXCLUSIVE))
1109
if (!maildir_full_sync_init(ctx, FALSE) ||
1110
!maildir_full_sync_dirs(ctx) ||
1111
!maildir_full_sync_finish(ctx))
1117
static int maildir_index_sync_context(struct maildir_sync_context *ctx,
1121
struct mail_index *index = ctx->index;
1123
time_t new_mtime, cur_mtime;
1125
if (!maildir_try_flush_dirty_flags(ctx->index, FALSE))
1128
if (index->fd != -1) {
1129
/* FIXME: file_sync_stamp should be in index file's headers.
1130
it should also contain maildir_cur_dirty. */
1131
if (fstat(index->fd, &st) < 0)
1132
return index_set_syscall_error(index, "fstat()");
1133
index->file_sync_stamp = st.st_mtime;
1136
if (stat(ctx->new_dir, &st) < 0) {
1137
index_file_set_syscall_error(index, ctx->new_dir, "stat()");
1140
new_mtime = st.st_mtime;
1142
if (stat(ctx->cur_dir, &st) < 0) {
1143
index_file_set_syscall_error(index, ctx->cur_dir, "stat()");
1146
cur_mtime = st.st_mtime;
1148
if (new_mtime != index->last_new_mtime ||
1149
new_mtime >= ioloop_time - MAILDIR_SYNC_SECS) {
1150
if (!maildir_new_scan_first_file(ctx))
1154
if (cur_mtime != index->file_sync_stamp ||
1155
(index->maildir_cur_dirty != 0 &&
1156
index->maildir_cur_dirty < ioloop_time - MAILDIR_SYNC_SECS)) {
1157
/* cur/ changed, or delayed cur/ check */
1158
if (changes != NULL)
1161
if (!maildir_sync_cur_dir(ctx))
1165
if (ctx->new_dent != NULL) {
1166
if (changes != NULL)
1169
if (!maildir_sync_new_dir(ctx, !index->maildir_keep_new, TRUE))
1172
/* this will set maildir_cur_dirty. it may actually be
1173
different from cur/'s mtime if we're unlucky, but that
1174
doesn't really matter and it's not worth the extra stat() */
1175
if (ctx->new_dent == NULL &&
1176
(ctx->new_count == 0 || !ctx->new_mails_new))
1177
cur_mtime = time(NULL);
1180
if (ctx->uidlist_rewrite) {
1181
i_assert(INDEX_IS_UIDLIST_LOCKED(index));
1183
if (!maildir_uidlist_rewrite(index, &index->last_uidlist_mtime))
1187
if (index->lock_type == MAIL_LOCK_EXCLUSIVE) {
1188
if (index->maildir_have_new)
1189
index->header->flags |= MAIL_INDEX_FLAG_MAILDIR_NEW;
1191
index->header->flags &= ~MAIL_INDEX_FLAG_MAILDIR_NEW;
1194
if (index->maildir_cur_dirty == 0 ||
1195
index->maildir_cur_dirty < ioloop_time - MAILDIR_SYNC_SECS) {
1196
if (cur_mtime >= ioloop_time - MAILDIR_SYNC_SECS)
1197
index->maildir_cur_dirty = cur_mtime;
1198
else if (ctx->new_count == 0 || !ctx->new_mails_cur)
1199
index->maildir_cur_dirty = 0;
1201
/* uidlist is locked, wait for a while before
1203
index->maildir_cur_dirty = ioloop_time;
1207
index->file_sync_stamp = cur_mtime;
1208
if (ctx->new_dent == NULL &&
1209
(ctx->new_count == 0 || !ctx->new_mails_new))
1210
index->last_new_mtime = new_mtime;
1215
static int maildir_full_sync_finish_readonly(struct maildir_sync_context *ctx)
1217
struct mail_index *index = ctx->index;
1218
struct mail_index_record *rec;
1219
struct maildir_hash_rec *hash_rec;
1220
void *orig_key, *orig_value;
1224
if (index->lock_type != MAIL_LOCK_EXCLUSIVE || !ctx->flag_updates)
1227
rec = index->lookup(index, 1); seq = 1;
1228
while (rec != NULL) {
1229
fname = maildir_get_location(index, rec);
1233
if (hash_lookup_full(ctx->files, fname, &orig_key, &orig_value))
1234
hash_rec = orig_value;
1238
if (hash_rec != NULL &&
1239
ACTION(hash_rec) == MAILDIR_FILE_ACTION_UPDATE_FLAGS) {
1240
if (!maildir_update_filename(ctx, rec, orig_key))
1242
if (!maildir_update_flags(ctx, rec, seq, orig_key))
1246
rec = index->next(index, rec); seq++;
1252
static int maildir_index_sync_context_readonly(struct maildir_sync_context *ctx)
1254
struct mail_index *index = ctx->index;
1256
i_assert(index->lock_type != MAIL_LOCK_UNLOCK);
1258
/* check only changes in file names. */
1260
/* if we can get exclusive lock, we can update the index
1261
directly. but don't rely on it. */
1262
(void)index->try_lock(index, MAIL_LOCK_EXCLUSIVE);
1264
if (!maildir_full_sync_init(ctx, FALSE) ||
1265
!maildir_full_sync_dirs(ctx) ||
1266
!maildir_full_sync_finish_readonly(ctx))
1272
static void maildir_index_sync_deinit(struct maildir_sync_context *ctx)
1274
if (ctx->uidlist != NULL)
1275
maildir_uidlist_close(ctx->uidlist);
1276
if (ctx->files != NULL)
1277
hash_destroy(ctx->files);
1278
if (ctx->pool != NULL)
1279
pool_unref(ctx->pool);
1281
if (ctx->new_dirp != NULL) {
1282
if (closedir(ctx->new_dirp) < 0) {
1283
index_file_set_syscall_error(ctx->index, ctx->new_dir,
1288
maildir_uidlist_unlock(ctx->index);
1291
static struct maildir_sync_context *
1292
maildir_sync_context_new(struct mail_index *index)
1294
struct maildir_sync_context *ctx;
1296
if (index->new_filenames != NULL) {
1297
hash_destroy(index->new_filenames);
1298
index->new_filenames = NULL;
1301
if (index->new_filename_pool != NULL)
1302
p_clear(index->new_filename_pool);
1304
ctx = t_new(struct maildir_sync_context, 1);
1306
ctx->new_dir = t_strconcat(index->mailbox_path, "/new", NULL);
1307
ctx->cur_dir = t_strconcat(index->mailbox_path, "/cur", NULL);
1311
int maildir_index_sync_readonly(struct mail_index *index,
1312
const char *fname, int *found)
1314
struct maildir_sync_context *ctx;
1315
struct maildir_hash_rec *hash_rec;
1319
/* fname may point into data file which syncing may break. */
1320
fname = t_strdup(fname);
1322
ctx = maildir_sync_context_new(index);
1323
ctx->readonly_check = TRUE;
1325
ret = maildir_index_sync_context_readonly(ctx);
1330
hash_rec = hash_lookup(ctx->files, fname);
1331
*found = hash_rec != NULL &&
1332
hash_rec->action != MAILDIR_FILE_ACTION_EXPUNGE;
1334
maildir_index_sync_deinit(ctx);
1339
int maildir_index_sync(struct mail_index *index, int minimal_sync,
1340
enum mail_lock_type data_lock_type __attr_unused__,
1343
struct maildir_sync_context *ctx;
1346
i_assert(index->lock_type != MAIL_LOCK_SHARED);
1348
if (changes != NULL)
1354
ctx = maildir_sync_context_new(index);
1355
ret = maildir_index_sync_context(ctx, changes);
1356
maildir_index_sync_deinit(ctx);
added
removed
src/lib-index/maildir/maildir-sync.c
