Fossil

Check-in [30716e3c]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Fixed some Markdown formatting problems in www/emaildesign.md. Made a few minor grammar tweaks while in there.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256:30716e3c7dcdb4c194da43420e1e09b1b564ce28a1f871863e05d48d284868a8
User & Date: wyoung 2018-08-08 18:29:09
Context
2018-08-08
18:32
Typo fix check-in: 15b20343 user: wyoung tags: trunk
18:29
Fixed some Markdown formatting problems in www/emaildesign.md. Made a few minor grammar tweaks while in there. check-in: 30716e3c user: wyoung tags: trunk
18:17
Fix the backoffice processes on unix so that they close file descriptors 0, 1, and 2 and reopen them on /dev/null, so as not to interfere with parent processes in any way. Restore the default of backoffice-nodelay back to off. Remove the /test-backoffice-lease webpage, which did not work correctly. check-in: c09b2512 user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to www/emaildesign.md.

20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
..
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
..
99
100
101
102
103
104
105



106
107
108
109
110
111
112
...
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
are not created in new repositories by default.  The tables only
come into existance if email notification is configured and used.


  *  <b>SUBSCRIBER</b> &rarr;
     The subscriber table records the email address for people who
     want to receive email notifications.  Each subscriber has a
     "subscriberCode" which is a random 32-byte blob that uniquely
     identifies the subscriber.  There are also fields to indicate
     what kinds of notifications the subscriber wishes to receive,
     whether or not the email address of the subscriber has been
     verified, etc.

  *  <b>PENDING\_ALERT</b> &rarr;
     The PENDING\_ALERT table contains records that define events
................................................................................
the linkage between users and subscribers.  But it is also possible
to be a user without being a subscriber, or to be a subscriber without
being a user.

Sending Email Messages
----------------------

Fossil expects to interact with an external mail agent.
There are currently three different methods for sending outbound
email messages from Fossil to the external mail agent:

  1.  <b>"pipe"</b> &rarr; Invoke an external command that accepts
      the email message on standard input.  This is useful if the
      host computer has a command like /usr/sbin/sendmail that will
      accept well-formed email messages from standard input and forward
................................................................................
All emails are text/plain and use a transfer-encoding of base64.

There is a utility command-line program named 
["tools/decode-email.c"](/file/tools/decode-email.c) in
the Fossil source tree.  If you compile this program, you can use it
to convert the base64 transfer-encoding into human-readable output for
testing and debugging.




Receiving Email Messages
------------------------

Inbound email messages (for example bounces from failed notification
emails) should be relayed to the "fossil email inbound" command.  That
command is currently a no-op place-holder.  At some point, we will need
................................................................................

Email Address Verification
--------------------------

When anonymous passers-by on the internet sign up for email notifications,
their email address must first be verified.  An email message is sent to
the address supplied inviting the user to click on a link.  The link includes
the random 32-byte subscriberCode in hex.  If anyone visits the link, the
email address is verified.

There is no password.  Knowledge of the subscriberCode is sufficient to
control the subscription.  This is not a secure as a separate password,
but on the other hand it is easier for the average subscriber to deal
with in that they don't have to come up with yet another password.  Also,
even if the subscriberCode is stolen, the worst that can happens is that
the thief can change your subscription settings.  No PII (other than
the subscriber's email address) is available to an attacker with the
subscriberCode.  Nor can knowledge of the subscriberCode lead to a
email flood or other annoyance attack, as far as I can see.

If subscriberCodes are ever compromised, new ones can be generated
as follows:

>   UPDATE subscriber SET subscriberCode=randomblob(32);

Perhaps the system be enhanced to randomize the
subscriberCodes periodically - say just before each daily digest
is sent out?

User Control Of Their Subscription
----------------------------------

If a user has a separate account with a login and password for
the repository, then their subscription is linked to their account.
On the /login page is a link to a page to control their subscription.

For users without logins, they can request a link to a page for
controling their subscription on the /alerts or /unsubscribe page.
The link is sent via email, and includes the subscriberCode.

Internal Processing Flow
------------------------

Almost all of the email notification code is found in the "src/email.c"
source file.

When email notifications are enabled, a trigger is created in the schema
(the "email_trigger1" trigger) that adds a new entry to the
PENDING_ALERT table every time a row is added to the EVENT table.
During a "rebuild", the EVENT table is rebuilt from scratch, and we
do not want users to get notifications for every historical check-in,
so the trigger is disabled during "rebuild".

Email notifications are sent out by the email_send_alerts() function.
This function is can be called by having a cron job invoke the
"fossil email exec" command.  Or, if the email-autoexec setting is
enabled, then email_send_alerts() is invoked automatically after each
successful webpage is generated.  The latter approach is used on the
Fossil self-hosting repository.  The email_send_alerts() function is
a no-op (obviously) if there are no pending events to be sent.

Digests are handled by recording the time of the last digest in the
"email-last-digest" setting, and only sending a new digest if the
current time is one day or later after the last digest.

Individual emails are sent to each subscriber.  I ran tests and found
that I could send about 1200 emails/second, which is fast enough so that
I do not need to resort to trying to notify multiple subscribers with
a single email.  Because each subscriber gets a separate email, the
system can include information in the email that is unique to the
subscriber, such as a link to the page to edit their subscription.  That
link includes the subscriberCode.

Other Notes
-----------

The "fossil configuration pull subscriber" command pulls down the content
of the SUBSCRIBER table.  This is intended to as a backup-only.  It
is not desirable to have two or more systems sending emails to the
same people for the same repository, as that would mean users would
receive duplicate emails.  Hence the settings that control email 
notifications are not transmitted with the pull.  The "push", "export",
and "import" commands all work similarly







|







 







|







 







>
>
>







 







|


|



|


|





|


|










|
|




|



|
|
|

|

|

|
|

|



|



|




|




|
|


|
|
|
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
..
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
..
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
...
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
are not created in new repositories by default.  The tables only
come into existance if email notification is configured and used.


  *  <b>SUBSCRIBER</b> &rarr;
     The subscriber table records the email address for people who
     want to receive email notifications.  Each subscriber has a
     `subscriberCode` which is a random 32-byte blob that uniquely
     identifies the subscriber.  There are also fields to indicate
     what kinds of notifications the subscriber wishes to receive,
     whether or not the email address of the subscriber has been
     verified, etc.

  *  <b>PENDING\_ALERT</b> &rarr;
     The PENDING\_ALERT table contains records that define events
................................................................................
the linkage between users and subscribers.  But it is also possible
to be a user without being a subscriber, or to be a subscriber without
being a user.

Sending Email Messages
----------------------

Fossil expects to interact with an external [mail transfer agent][MTA].
There are currently three different methods for sending outbound
email messages from Fossil to the external mail agent:

  1.  <b>"pipe"</b> &rarr; Invoke an external command that accepts
      the email message on standard input.  This is useful if the
      host computer has a command like /usr/sbin/sendmail that will
      accept well-formed email messages from standard input and forward
................................................................................
All emails are text/plain and use a transfer-encoding of base64.

There is a utility command-line program named 
["tools/decode-email.c"](/file/tools/decode-email.c) in
the Fossil source tree.  If you compile this program, you can use it
to convert the base64 transfer-encoding into human-readable output for
testing and debugging.

[MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent


Receiving Email Messages
------------------------

Inbound email messages (for example bounces from failed notification
emails) should be relayed to the "fossil email inbound" command.  That
command is currently a no-op place-holder.  At some point, we will need
................................................................................

Email Address Verification
--------------------------

When anonymous passers-by on the internet sign up for email notifications,
their email address must first be verified.  An email message is sent to
the address supplied inviting the user to click on a link.  The link includes
the random 32-byte `subscriberCode` in hex.  If anyone visits the link, the
email address is verified.

There is no password.  Knowledge of the `subscriberCode` is sufficient to
control the subscription.  This is not a secure as a separate password,
but on the other hand it is easier for the average subscriber to deal
with in that they don't have to come up with yet another password.  Also,
even if the `subscriberCode` is stolen, the worst that can happens is that
the thief can change your subscription settings.  No PII (other than
the subscriber's email address) is available to an attacker with the
`subscriberCode`.  Nor can knowledge of the `subscriberCode` lead to a
email flood or other annoyance attack, as far as I can see.

If subscriberCodes are ever compromised, new ones can be generated
as follows:

        UPDATE subscriber SET subscriberCode=randomblob(32);

Perhaps the system be enhanced to randomize the
`subscriberCodes` periodically - say just before each daily digest
is sent out?

User Control Of Their Subscription
----------------------------------

If a user has a separate account with a login and password for
the repository, then their subscription is linked to their account.
On the /login page is a link to a page to control their subscription.

For users without logins, they can request a link to a page for
controling their subscription on the `/alerts` or `/unsubscribe` page.
The link is sent via email, and includes the `subscriberCode`.

Internal Processing Flow
------------------------

Almost all of the email notification code is found in the `src/email.c`
source file.

When email notifications are enabled, a trigger is created in the schema
(the `email_trigger1` trigger) that adds a new entry to the
`PENDING_ALERT` table every time a row is added to the `EVENT` table.
During a `fossil rebuild`, the `EVENT` table is rebuilt from scratch; since we
do not want users to get notifications for every historical check-in,
the trigger is disabled during `rebuild`.

Email notifications are sent out by the `email_send_alerts()` function.
This function is can be called by having a cron job invoke the
`fossil email exec` command.  Or, if the email-autoexec setting is
enabled, then `email_send_alerts()` is invoked automatically after each
successful webpage is generated.  The latter approach is used on the
Fossil self-hosting repository.  The `email_send_alerts()` function is
a no-op (obviously) if there are no pending events to be sent.

Digests are handled by recording the time of the last digest in the
`email-last-digest` setting, and only sending a new digest if the
current time is one day or later after the last digest.

Individual emails are sent to each subscriber.  I ran tests and found
that I could send about 1200 emails/second, which is fast enough that
I do not need to resort to trying to notify multiple subscribers with
a single email.  Because each subscriber gets a separate email, the
system can include information in the email that is unique to the
subscriber, such as a link to the page to edit their subscription.  That
link includes the `subscriberCode`., 

Other Notes
-----------

The `fossil configuration pull subscriber` command pulls down the content
of the `SUBSCRIBER` table.  This is intended to as a backup-only.  It
is not desirable to have two or more systems sending emails to the
same people for the same repository, as that would mean users would
receive duplicate emails.  Hence, the settings that control email 
notifications are not transmitted with the pull.  The `push`, `export`,
and `import` commands all work similarly.