1
Oh Most Noble and Fragrant Emacs, please be in -*- outline -*- mode!
3
Newline Conversion and Keyword Substitution
4
===========================================
8
We've finally settled on a proposal articulated by Greg Hudson,
9
derived from Ben's original proposal plus much list discussion.
11
Here's Greg's mail with the proposal, then a few clarifying selections
12
from the followup discussion:
14
Alright, I'll make a proposal which is like yours but (in my
15
opinion) a little clearer. First, let's look at the different use
18
1. The most common case--text files which want native line endings.
19
These should be stored in the repository using LF line endings,
20
and in the working dir using native line endings.
22
2. Binary files. These files we don't want to touch at all.
24
3. Text files which, for one reason or another, want a specific
25
line ending format regardless of platform. These should be
26
stored in the repository and in the working directory using the
27
specified line ending. We probably don't have to worry so much
28
about data safety for these files since a particular, odd
29
behavior has been specified for them.
31
There are, of course, a hundred different ways we could arrange the
32
metadata. I propose an "svn:newline-style" property with the
33
possible values "none", "native", "LF", "CR", and "CRLF". The
36
none: Use case 2. don't do any newline translation
38
native: Use case 1. Store with LF in repository, and with native
39
line endings in the working copy.
41
LF, CR, CRLF: Use case 3. Store with specified format in the
42
repository and in the working copy.
44
On commit, we apply the following rules to transform the data
45
committed to the server:
47
If newline-style is none, do nothing.
49
If newline-stle is native, translate <native newline style> -> LF.
50
If we notice any CRs or LFs which aren't part of a native-style
51
newline, abort the commit.
53
If newline-style is LF, CR, or CRLF, translate <native newline
54
style> -> <requested newline style>. If we notice any CRs or LFs
55
which aren't part of a native-style newline and aren't part of a
56
requested-style newline, abort the commit. If the commit succeeds,
57
apply the <native newline style> -> <requested newline style>
58
translation to the working copy as well, so that it matches what we
59
would get from a checkout of the new rev.
61
On checkout, we translate LF -> <native newline style> if
62
newline-style is native; otherwise, we leave the file alone.
64
For now, let's say the default value of svn:newline-style is none.
65
In the future, we'll want to think about things like how to enable
66
newline-translation over the whole repository except for files
67
which don't appear to be text.
69
I think that's a complete proposal. Some possible variations:
71
Variation 1: If newline-style is native, on commit, translate
72
<first newline style seen> -> LF. If we see any CRs or LFs which
73
don't match the first newline style seen, abort the commit.
75
Variation 2: If newline-style is native, before commit, examine the
76
file to see if it uses only the native newline style. If it
77
doesn't, set the newline-style property to "none" and commit with
80
Variation 3: Combine variations 1 and 2; if newline-style is
81
native, then if before commit, examine the file to see if it uses a
82
single consistent newline style. If it does, translate <that
83
newline style> -> LF; if not, commit with newline-style set to
84
"none" and no translation.
86
Variation 4: If newline-style is native, then on commit, we edit a
87
property "svn:newline-conversion" to something like "CRLF LF" to
88
show what conversion we did. This enables mechanical reversal of
89
the translation if the file is later determined to be binary.
90
(Particularly useful with variations 1 or 3 where the transform
91
might not be obvious from the platform where the file was checked
94
We decided to hold off on doing any of the "variations". We'll just
95
do the basic proposal first, then see how well things work out.
97
Next, Greg responded to an observation by William Uther, in which
98
William pointed out that the behavior is not reversible when
99
svn:newline-style is LF or CRLF. Greg agrees, but explains why that's
102
On Fri, 2001-12-14 at 15:48, William Uther wrote:
103
> --On Friday, 14 December 2001 1:16 PM -0500 Greg Hudson
104
> <ghudson@MIT.EDU> wrote:
106
> > If newline-style is LF, CR, or CRLF, translate <native
107
> > newline style> -> <requested newline style>. If we notice any
108
> > CRs or LFs which aren't part of a native-style newline and
109
> > aren't part of a requested-style newline, abort the commit. If
110
> > the commit succeeds, apply the <native newline style> ->
111
> > <requested newline style> translation to the working copy as
112
> > well, so that it matches what we would get from a checkout of
115
> I don't think this preserves reversability. If a file contains
116
> BOTH <native-style newline> and <requested-style newline> then
117
> you neet to abort. If you translate just <native-style newline>
118
> then you can't undo the transformation - you don't know which
119
> newlines need to be untransformed.
121
This particular transform (for files marked CRLF, CR, or LF) is not
122
reversible. See where I said:
124
"We probably don't have to worry so much about data safety for
125
these files since a particular, odd behavior has been specified
128
However, let's add a possible variation to my proposal, for those
129
who are still uncomfortable with data-destroying transformations
130
applied to such flies:
132
Variation 5: If the file is marked CRLF, CR, or LF, we translate
133
<native-style newline> to <requested-style newline> during commit,
134
and abort the commit if we notice any kind of mixing of newline
135
styles. (Can also combine with variation 1.)
137
Colin Putney also followed up to William Uther's post, agreeing with
138
Greg and explaining why in even more detail:
140
> I don't think this preserves reversability. If a file contains
141
> BOTH <native-style newline> and <requested-style newline> then
142
> you neet to abort. If you translate just <native-style newline>
143
> then you can't undo the transformation - you don't know which
144
> newlines need to be untransformed.
146
> Stated simply: You should only translate when the newline style
147
> is entirely consistent. Anything else removes the inconsistency
148
> and hence loses information.
150
True, this scheme doesn't preserve reversibility. But in this case
151
that's OK, because the newline-style decrees what the newline style
152
must be. If there are native-style newlines mixed in with the
153
requested-style newlines, this is probably the result of corruption
154
by some native-newline-obsessive user tool. So the non-reversible
155
transform will actually undo the corruption.
157
For example, the file foo.dsp, which has newline-style of
158
CRLF. It's stored in the repository with CRLF newlines and on
159
checkout, no transformation is done. If Linus checks out the file
160
and edits it in an old version of emacs, any lines he adds will be
161
terminated with a bare LF. Since this is his native style of
162
newline, the transformation Greg described will undo this damage.
164
If the newline-style is set to a specific newline-style (ie. CR,
165
LF, or CRLF), then we know that (1) the file is text, not binary,
166
and (2), any other style of newline present is corruption.
168
A file should not be marked with a specific newline style unless
169
(1) user does so explicitly, or (2) it matches some heuristic when
170
it's added, *and* the file contents conform to that newline style.
172
So the only real possibility for corruption is if some user tool
173
creates a binary file that matches a heuristic for a specific
174
newline style. In our running example, William creates a vector
175
graphics file called foo.dsp and adds it. By chance, this file
176
happens to have CRLFs scattered though it, but no bare CRs, LFs,
177
'\0' characters or other harbingers of binary files. On the commit,
178
svn will notice the extension, set the newline-style to CRLF and
179
send it to the repository. William may get an error if he tries to
180
commit a change that introduces a bare CR or LF, but he won't
183
Linus can corrupt the file if he makes a change that introduces a
184
bare LF, which will get transformed into CRLF on
185
commit. Alternatively, Madeleine (was that her name?) Can introduce
186
a bare CR and commit, which will also corrupt the file.
188
That's a pretty long string of unlikely coincidences though, while
189
the opposite case, where this transformation *fixes* corruption, is
194
Finally, Greg followed up, confirming again what he meant by that
195
portion of the proposal, but also saying that he could go the other
196
way on the question too:
198
> +1 on Greg Hudson's latest proposal -- and I think we're now
199
> ready to Actually Do It. :-)
201
I hope so. For a while I was afraid we had hit our first failure
202
to achieve livable consensus. My apologies for not realizing the
203
reversability thing until two days and several thousand lines of
204
misguided debate had already gone by.
206
> My assumption is that "in the working copy" means both text-base
207
> and working file, for the sake of an efficient is-modified-p
208
> test, and since the repository file is just an automatic
209
> transform off the text-base anyway.
211
Actually, I was assuming that text-base would be a verbatim copy of
212
the repository contents. But that's kind of an implementation
213
detail; let's leave that up to Ben (assuming he's doing the
216
> Otherwise, then the is-modified-p check has to be tweaked in a
217
> way that will make modifiedness checks a lot slower in some
220
No... it just means that if the mod times force a contents check,
221
you have to translate the text-base contents as you compare them
222
against the normal contents. That's "a teeny tiny bit slower,"
225
> The second sentence of the above paragraph isn't about allowing
226
> mixed-style files. It's saying that if the entire file is native
227
> format, allow that (and transform when necessary), OR if the
228
> entire file is in the requested style, then allow that too. The
229
> latter situation could happen if someone used a LF-style tool
230
> under Windows, for example, so that when an LF-style file got
231
> saved, the whole thing would be LF-style now, not native style.
232
> No reason to disallow this.
236
See my last message, as well as Colin Putney's argument. In
237
summary, that's not actually what I meant, but I don't really care
240
We're implementing Greg's original meaning, since (as Colin Putney
241
pointed out) the chance of "bad" data corruption is very small, and
242
the chance that it would undo corruption is actually greater.
245
* Keyword substitution
247
Quick summary: there's one property, named "svn:keywords". It's value
248
is a whitespace-separated list of keywords to expand. Since keywords
249
have both long and short forms, both ways are allowed in the value of
250
the svn:keywords property. Here are all the keywords:
252
"LastChangedBy" "Author" ---> either one expands author
253
"LastChangedDate" "Date" ---> either one expands date
254
"LastChangedRevision" "Revision" "Rev" ---> any one expands rev
255
"HeadURL" "URL" ---> either one expands url
257
Here are some example values of the property:
259
"Rev LastChangedDate HeadURL"
261
"Author\nDate \n LastChangedRevision URL"
263
Unrecognized words are ignored; absence of the property is the same as
264
an empty value or a value with no valid keywords in it.
266
Keywords (long and short forms) are case-sensitive, as in CVS.
added
removed
notes/line-endings-and-keywords.txt
