Fossil

Check-in [85eaffb6]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Moved the chroot and loadmgmt sections of www/server.wiki into separate documents. This change also adds info on /proc to the chroot doc, which was missing in its prior server.wiki form. Also reduced a few other "details" sections of server.wiki to bullet points in the new "Further Details" list at the end of the document.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | server-docs
Files: files | file ages | folders
SHA3-256: 85eaffb6a59c0642ffad9c21d76ae21879b587d793b9b9422cdbb762b5b749f0
User & Date: wyoung 2019-08-16 12:53:13
Context
2019-08-16
14:25
An attempt to make the main server.wiki page simpler and yet self-contained, all at once. check-in: 5bb1e112 user: drh tags: server-docs
12:53
Moved the chroot and loadmgmt sections of www/server.wiki into separate documents. This change also adds info on /proc to the chroot doc, which was missing in its prior server.wiki form. Also reduced a few other "details" sections of server.wiki to bullet points in the new "Further Details" list at the end of the document. check-in: 85eaffb6 user: wyoung tags: server-docs
11:59
Changed "socket activation" to "socket listener" in all the new docs check-in: 3c9f811b user: wyoung tags: server-docs
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Added www/loadmgmt.md.



























































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# Managing Server Load

A Fossil server is very efficient and normally presents a very light
load on the server.  The Fossil [self-hosting server][sh] is a 1/24th
slice VM at [Linode.com][lin] hosting 65 other repositories in addition
to Fossil, including some very high-traffic sites such as
<http://www.sqlite.org> and <http://system.data.sqlite.org>. This small
VM has a typical load of 0.05 to 0.1. A single HTTP request to Fossil
normally takes less than 10 milliseconds of CPU time to complete, so
requests can be arriving at a continuous rate of 20 or more per second,
and the CPU can still be mostly idle.

However, there are some Fossil web pages that can consume large amounts
of CPU time, especially on repositories with a large number of files or
with long revision histories.  High CPU usage pages include
[`/zip`](/help/zip), [`/tarball`](/help/tarball),
[`/annotate`](/help/annotate), and others.  On very large repositories,
these commands can take 15 seconds or more of CPU time.  If these kinds
of requests arrive too quickly, the load average on the server can grow
dramatically, making the server unresponsive.

Fossil provides two capabilities to help avoid server overload problems
due to excessive requests to expensive pages:

1.  An optional cache is available that remembers the 10 most recently
    requested `/zip` or `/tarball` pages and returns the precomputed
    answer if the same page is requested again.

2.  Page requests can be configured to fail with a
    “[503 Server Overload][503]” HTTP error if an expensive request is
    received while the host load average is too high.

Both of these load-control mechanisms are turned off by default, but
they are recommended for high-traffic sites.

The webpage cache is activated using the [`fossil cache
init`](/help/cache) command-line on the server.  Add a `-R` option to
specify the specific repository for which to enable caching.  If running
this command as root, be sure to “`chown`” the cache database to give
the Fossil server write permission for the user ID of the web server;
this is a separate file in the same directory and with the same name as
the repository but with the “`.fossil`” suffix changed to “`.cache`”.

To activate the server load control feature visit the Admin → Access
setup page in the administrative web interface; in the “**Server Load
Average Limit**” box enter the load average threshold above which “503
Server Overload” replies will be issued for expensive requests.  On the
self-hosting Fossil server, that value is set to 1.5, but you could
easily set it higher on a multi-core server.

The maximum load average can also be set on the command line using
commands like this:

        fossil set max-loadavg 1.5
        fossil all set max-loadavg 1.5

The second form is especially useful for changing the maximum load
average simultaneously on a large number of repositories.

Note that this load-average limiting feature is only available on
operating systems that support the [`getloadavg()`][gla] API.  Most
modern Unix systems have this interface, but Windows does not, so the
feature will not work on Windows.

Because Linux implements `getloadavg()` by accessing the `/proc/loadavg`
virtual file, you will need to make sure `/proc` is available to the
Fossil server. The most common reason for it to not be available is that
you are running a Fossil instance [inside a `chroot(2)`
jail](./chroot.md) and you have not mounted the `/proc` virtual file
system inside that jail. On the [self-hosting Fossil repositories][sh],
this was accomplished by adding a line to the `/etc/fstab` file:

        chroot_jail_proc /home/www/proc proc ro 0 0

The `/home/www/proc` pathname should be adjusted so that the `/proc`
component is at the root of the chroot jail, of course.

To see if the load-average limiter is functional, visit the
[`/test_env`][hte] page of the server to view the current load average.
If the value for the load average is greater than zero, that means that
it is possible to activate the load-average limiter on that repository.
If the load average shows exactly "0.0", then that means that Fossil is
unable to find the load average. This can either be because it is in a
`chroot(2)` jail without `/proc` access, or because it is running on a
system that does not support `getloadavg()` and so the load-average
limiter will not function.


[503]: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4
[hte]: /help?cmd=/test_env
[gla]: https://linux.die.net/man/3/getloadavg
[lin]: http://www.linode.com
[sh]:  ./selfhost.wiki

Changes to www/server.wiki.

137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296


documents linked above: Apple’s <tt>launchd</tt>, Linux’s
<tt>systemd</tt>, Solaris’ SMF, etc. We would welcome [./contribute.wiki
| contributions] to cover these as well. We also welcome contributions
to fill gaps (❌) in the table above.
</blockquote>


<h2 id="ext">CGI Server Extensions</h2>

<blockquote>
    In addition to serving Fossil repositories via CGI, Fossil can
    itself [./serverext.wiki | launch other programs via CGI] to
    implement server extensions. Do not confuse these two concepts. This
    extension mechanism works regardless of the method above you choose
    to serve your Fossil repository.
</blockquote>


<h2 id="tls">Securing a repository with TLS</h2>

<blockquote>
  Fossil's built-in HTTP server (e.g. "fossil server") does not support
  TLS, but there are multiple ways to protect your Fossil server with
  TLS. All of this is covered in a separate document, <a
  href="./ssl.wiki">Using TLS-Encrypted Communications with Fossil</a>.
</blockquote>


<h2 id="chroot">The Fossil Chroot Jail</h2>

<blockquote>
If you run Fossil as root in any mode that serves data on the
network, and you're running it on Unix or a compatible OS, Fossil
will drop itself into a [https://en.wikipedia.org/wiki/Chroot |
chroot jail] shortly after starting up, after
it's done everything that requires root access. Most
commonly, you run Fossil as root to allow it to bind to TCP port 80
for HTTP service, since normal users are restricted to ports 1024
and up on OSes where this behavior occurs.

Fossil uses the owner of the Fossil repository file as its new user
ID when dropping root privileges.

When this happens, Fossil needs to have all of its dependencies
inside the chroot jail.  There are several things you typically need
in order to make things work properly:

<ul>
    <li>the repository file(s)

    <li><tt>/dev/null</tt> — create it with <tt>mknod(8)</tt> inside the
    jail directory ([https://fossil-scm.org/forum/forumpost/90caff30cb |
    Linux example])

    <li><tt>/dev/urandom</tt> — ditto

    <li>any shared libraries your <tt>fossil</tt> binary is linked to,
    such as <tt>/lib/libssl.so</tt>; consider
    <tt>[https://www.fossil-scm.org/fossil/doc/trunk/www/build.wiki |
    ./configure --static]</tt> to avoid the need for this
</ul> </blockquote>

<blockquote>
Fossil does all of this in order to protect the host OS. You can make it
bypass the jail part of this by passing <tt>--nojail</tt> to <tt>fossil server</tt>,
but you cannot make it skip the dropping of root privileges, on purpose.
</blockquote>


<h2 id="loadmgmt">Managing Server Load</h2>

<blockquote>
A Fossil server is very efficient and normally presents a very light
load on the server.
The Fossil [./selfhost.wiki | self-hosting server] is a 1/24th slice VM at
[http://www.linode.com | Linode.com] hosting 65 other repositories in
addition to Fossil (and including some very high-traffic sites such
as [http://www.sqlite.org] and [http://system.data.sqlite.org]) and
it has a typical load of 0.05 to 0.1.  A single HTTP request to Fossil
normally takes less than 10 milliseconds of CPU time to complete, so
requests can be arriving at a continuous rate of 20 or more per second,
and the CPU can still be mostly idle.

However, there are some Fossil web pages that can consume large
amounts of CPU time, especially on repositories with a large number
of files or with long revision histories.  High CPU usage pages include
[/help?cmd=/zip | /zip], [/help?cmd=/tarball | /tarball],
[/help?cmd=/annotate | /annotate] and others.  On very large repositories,
these commands can take 15 seconds or more of CPU time.
If these kinds of requests arrive too quickly, the load average on the
server can grow dramatically, making the server unresponsive.

Fossil provides two capabilities to help avoid server overload problems
due to excessive requests to expensive pages:

<ol>
  <li><p>An optional cache is available that remembers the 10 most recently
      requested /zip or /tarball pages and returns the precomputed answer
      if the same page is requested again.</p>
  <li><p>Page requests can be configured to fail with a
      [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4 | "503 Server Overload"]
      HTTP error if an expensive request is received while the host load
      average is too high.</p>
</ol>

Both of these load-control mechanisms are turned off by default, but they
are recommended for high-traffic sites.

The webpage cache is activated using the [/help?cmd=cache|fossil cache init]
command-line on the server.  Add a -R option to specify the specific repository
for which to enable caching.  If running this command as root, be sure to
"chown" the cache database (which is a separate file in the same directory
and with the same name as the repository but with the suffix changed to ".cache")
to give it write permission for the userid of the web server.

To activate the server load control feature
visit the Admin → Access setup page in the administrative web
interface; in the "<b>Server Load Average Limit</b>" box
enter the load average threshold above which "503 Server
Overload" replies will be issued for expensive requests.  On the
self-hosting Fossil server, that value is set to 1.5, but you could easily
set it higher on a multi-core server.

The maximum load average can also be set on the command line using
commands like this:
<blockquote><pre>
fossil set max-loadavg 1.5
fossil all set max-loadavg 1.5
</pre></blockquote>

The second form is especially useful for changing the maximum load average
simultaneously on a large number of repositories.

Note that this load-average limiting feature is only available on operating
systems that support the "getloadavg()" API.  Most modern Unix systems have
this interface, but Windows does not, so the feature will not work on Windows.
Note also that Linux implements "getloadavg()" by accessing the "/proc/loadavg"
file in the "proc" virtual file system.  If you are running a Fossil instance
inside a chroot() jail on Linux, you will need to make the "/proc" file
system available inside that jail in order for this feature to work.  On
the [./selfhost.wiki|self-hosting Fossil repositories], this was accomplished
by adding a line to the "/etc/fstab" file that looks like:

<blockquote><pre>
chroot_jail_proc /home/www/proc proc ro 0 0
</pre></blockquote>

The /home/www/proc pathname should be adjusted so that the "/proc" component is
in the root of the chroot jail, of course.

To see if the load-average limiter is functional, visit the [/test_env] page
of the server to view the current load average.  If the value for the load
average is greater than zero, that means that it is possible to activate
the load-average limiter on that repository.  If the load average shows
exactly "0.0", then that means that Fossil is unable to find the load average
(either because it is in a chroot() jail without /proc access, or because
it is running on a system that does not support "getloadavg()") and so the
load-average limiter will not function.

</blockquote>









|

|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
|
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
>
>
137
138
139
140
141
142
143
144
145
146



























































147


























































































148
149
documents linked above: Apple’s <tt>launchd</tt>, Linux’s
<tt>systemd</tt>, Solaris’ SMF, etc. We would welcome [./contribute.wiki
| contributions] to cover these as well. We also welcome contributions
to fill gaps (❌) in the table above.
</blockquote>


<h2 id="more">Further Details</h2>

  *  <a name="chroot"></a>[./chroot.md | The Server Chroot Jail]



























































  *  <a name="loadmgmt"></a>[./loadmgmt.md | Managing Server Load]


























































































  *  <a name="tls"></a>[./ssl.wiki | Securing a Repository with TLS]
  *  <a name="ext"></a>[./serverext.wiki | CGI Server Extensions]

Changes to www/server/any/cgi.md.

35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
    writable by the same account which executes the Fossil binary.
    (This might differ from the user the web server normally runs
    under.) The directory holding the repository file(s) needs to be
    writable so that SQLite can write its journal files.

*   Fossil must be able to create temporary files in a
    [directory that varies by host OS](../../env-opts.md#temp). When the
    CGI process is operating [within a chroot](../../server.wiki#chroot),
    ensure that this directory exists and is readable/writeable by the
    user who executes the Fossil binary.

Once the CGI script is set up correctly, and assuming your server is
also set correctly, you should be able to access your repository with a
URL like: <b>http://mydomain.org/cgi-bin/repo</b> This is assuming you
are running a web server like Apache that uses a “`cgi-bin`” directory







|







35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
    writable by the same account which executes the Fossil binary.
    (This might differ from the user the web server normally runs
    under.) The directory holding the repository file(s) needs to be
    writable so that SQLite can write its journal files.

*   Fossil must be able to create temporary files in a
    [directory that varies by host OS](../../env-opts.md#temp). When the
    CGI process is operating [within a chroot](../../chroot.md),
    ensure that this directory exists and is readable/writeable by the
    user who executes the Fossil binary.

Once the CGI script is set up correctly, and assuming your server is
also set correctly, you should be able to access your repository with a
URL like: <b>http://mydomain.org/cgi-bin/repo</b> This is assuming you
are running a web server like Apache that uses a “`cgi-bin`” directory

Changes to www/server/any/inetd.md.

22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
        fossil          12345/tcp  # fossil server

and use the symbolic name “`fossil`” instead of the numeric TCP port
number (“12345” in the above example) in `inetd.conf`.

Notice that we configured `inetd` to launch Fossil as root. See the
top-level section on “[The Fossil Chroot
Jail](../../server.wiki#chroot)” for the consequences of this and
alternatives to it.

You can instead configure `inetd` to bind to a higher-numbered TCP port,
allowing Fossil to be run as a normal user. In that case, Fossil will
not put itself into a chroot jail, because it assumes you have set up
file permissions and such on the server appropriate for that user.








|







22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
        fossil          12345/tcp  # fossil server

and use the symbolic name “`fossil`” instead of the numeric TCP port
number (“12345” in the above example) in `inetd.conf`.

Notice that we configured `inetd` to launch Fossil as root. See the
top-level section on “[The Fossil Chroot
Jail](../../chroot.md)” for the consequences of this and
alternatives to it.

You can instead configure `inetd` to bind to a higher-numbered TCP port,
allowing Fossil to be run as a normal user. In that case, Fossil will
not put itself into a chroot jail, because it assumes you have set up
file permissions and such on the server appropriate for that user.

Changes to www/ssl.wiki.

1
2
3
4
5
6
7
8
<title>TLS and Fossil</title>

<h2>Using TLS-Encrypted Communications with Fossil</h2>

If you are storing sensitive information in a repository accessible over
a network whose integrity you do not fully trust, you should use TLS to
encrypt all communications with it. This is most true for repositories
accessed over the Internet, especially if they will be accessed from
|







1
2
3
4
5
6
7
8
<title>Securing a Repository with TLS</title>

<h2>Using TLS-Encrypted Communications with Fossil</h2>

If you are storing sensitive information in a repository accessible over
a network whose integrity you do not fully trust, you should use TLS to
encrypt all communications with it. This is most true for repositories
accessed over the Internet, especially if they will be accessed from