Fossil

Check-in [893fca3a]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Preliminary documentation for the /ext mechanism.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 893fca3a15eff2a3a89dc39b81f8670ad901a7a43171c63d418762404213de77
User & Date: drh 2019-07-26 18:15:25
Context
2019-07-26
20:54
Do not allow /ext to return files whose names begin with "." or "-" even if those files occur in the top-level extroot directory. check-in: e91de28b user: drh tags: trunk
18:15
Preliminary documentation for the /ext mechanism. check-in: 893fca3a user: drh tags: trunk
13:35
Fix the fullcap() SQL function so that it correctly expands the capabilities of Admin and Setup users to include everything. check-in: 384fdb6b user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to www/server.wiki.

244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
notfound: http://url-to-go-to-if-repo-not-found/
</pre></blockquote>

Once deployed, a URL like: <b>http://mydomain.org/cgi-bin/repo/XYZ</b>
will serve up the repository "/home/fossil/repos/XYZ.fossil" (if it exists).

Additional options available to the CGI script are documented in the
source code.  As of 2017-07-02, the available options are described at
[/artifact/9a52a07b?ln=1777-1824|main.c lines 1777 through 1824].
</blockquote>

<h2 id="scgi">Fossil as SCGI</h2>
<blockquote>

The [/help/server|fossil server] command, described above as a way of
starting a stand-alone web server, can also be used for SCGI.  Simply add







|
|







244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
notfound: http://url-to-go-to-if-repo-not-found/
</pre></blockquote>

Once deployed, a URL like: <b>http://mydomain.org/cgi-bin/repo/XYZ</b>
will serve up the repository "/home/fossil/repos/XYZ.fossil" (if it exists).

Additional options available to the CGI script are documented in the
source code.  As of 2019-07-26, the available options are described at
[/artifact/183cf5e5ad2ae111?ln=1912-1959|main.c lines 1912 through 1959].
</blockquote>

<h2 id="scgi">Fossil as SCGI</h2>
<blockquote>

The [/help/server|fossil server] command, described above as a way of
starting a stand-alone web server, can also be used for SCGI.  Simply add

Added www/serverext.wiki.





































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
<title>CGI Server Extensions</title>

<h2>1.0 Introduction</h2>

If you choose to have a [./server.wiki|central server] for your Fossil-based
project (which is optional - Fossil can be used in a pure peer-to-peer
mode) then you can add CGI extensions to the Fossil website.  These
extensions work like any other CGI program, except that they also have access
to the Fossil login information, so that they can restrict access based on
the currently logged in user.  The CGI output can be a stand-alone webpage,
or it can be incorporated into the Fossil site using the standard Fossil
header and footer.

An example of where this is useful is the 
[https://sqlite.org/src/ext/checklist|checklist application] on
the [https://sqlite.org/|SQLite] project.  The checklist
helps the SQLite developers track which release tests have passed,
or failed, or are still to be done.  The checklist program began as a
stand-alone CGI which kept its own private user database and implemented
its on permissions and login system.  By converting checklist into
a Fossil extension, the same login that works for the
[https://sqlite.org/src|main SQLite source repository] also works
for the checklist.  And permission to change elements of the checklist
is based on permission to check-in to the main source repository.

<h2>2.0 How It Works</h2>

Extensions are disabled by default.
An administrator activates the CGI extension mechanism by specifying
an "Extension Root Directory" or "extroot" as part of the server setup.
If the Fossil server is itself run as CGI, then add a line to the CGI
script file that says:

<blockquote><pre>
extroot: <i>DIRECTORY</i>
</pre></blockquote>

Or, if the Fossil server is begin run as using the "fossil server" or
"fossil ui" or "fossil http" commands, then add an extra 
"--extroot <i>DIRECTORY</i>" option to that command.

The <i>DIRECTORY</i> is the DOCUMENT_ROOT for the CGI.
Files in the DOCUMENT_ROOT are accessed via URLs like this:

<blockquote>
https://example-project.org/ext/<i>FILENAME</i>
</blockquote>

In other words, access files in DOCUMENT_ROOT by appending the filename
relative to DOCUMENT_ROOT to the [/help?cmd=/ext|/ext]
page of the Fossil server.
Files that are readable but not executable are returned as static
content.  Files that are executable are run as CGI.

<h3>2.1 Example #1</h3>

The source code repository for SQLite is a Fossil server that is run
as CGI.  The URL for the source code repository is [https://sqlite.org/src].
The CGI script looks like this:

<blockquote><verbatim>
#!/usr/bin/fossil
repository: /fossil/sqlite.fossil
errorlog: /logs/errors.txt
extroot: /sqlite-src-ext
</verbatim></blockquote>

The "extroot: /sqlite-src-ext" line tells Fossil that it should look for
extension CGIs in the /sqlite-src-ext directory.  (All of this is happening
inside of a chroot jail, so putting the document root in a top-level
directory is a reasonable thing to do.)

When a URL like "https://sqlite.org/src/ext/checklist" is received by the
main webserver, it figures out that the /src part refers to the main
Fossil CGI script and so it runs that script.  Fossil gets the remainder
of the URL to work with: "/ext/checklist".  Fossil extracts the "/ext"
prefix and uses that to determine that this a CGI extension request.
Then it takes the leftover "/checklist" part and appends it to the
"extroot" to get the filename "/sqlite-src-ext/checklist".  Fossil finds
that file to be executable, so it runs it as CGI and returns the result.

The /sqlite-src-ext/checklist file is a
[https://wapp.tcl.tk|Wapp program].  The complete source code to the
this program can be seen at
[https://www.sqlite.org/checklistapp/file/checklist.tcl].

There is a cascade of CGIs happening here.  The webserver that receives
the initial HTTP request runs Fossil as a CGI based on the
"https://sqlite.org/src" portion of the URL.  The Fossil instance then
runs the checklist sub-CGI based on the "/ext/checklists" suffix.  The
output of the sub-CGI is read by Fossil and then relayed on to the
main webserver which in turn relays the result back to the original client.

<h3>2.2 Example #2</h3>

The [https://fossil-scm.org/home|Fossil self-hosting repository] is also
a CGI that looks like this:

<blockquote><verbatim>
#!/usr/bin/fossil
repository: /fossil/fossil.fossil
errorlog: /logs/errors.txt
extroot: /fossil-extroot
</verbatim></blockquote>

The extroot for this Fossil server is /fossil-extroot and in that directory
is an executable file named "fileup1" - another [https://wapp.tcl.tk|Wapp]
script.  (The extension mechanism does not have to use Wapp.  You can use
any kind of program you like.  But the creator of SQLite and Fossil is fond
of Tcl/Tk and so he tends to gravitate toward Tcl-based technologies like
Wapp.)  The fileup1 script is a demo program that lets the user upload
a file using a form, and then displays that file in the reply.  There is
a link on the page that causes the fileup1 script to return a copy of its
own source-code, so you can see how it works.

<h2>3.0 CGI Inputs</h2>

The /ext extension mechanism is an ordinary CGI interface.  Parameters
are passed to the CGI program using environment variables.  The following
standard CGI environment variables are supported:

  *  AUTH_TYPE
  *  AUTH_CONTENT
  *  CONTENT_LENGTH
  *  CONTENT_TYPE
  *  DOCUMENT_ROOT
  *  GATEWAY_INTERFACE
  *  HTTP_ACCEPT
  *  HTTP_ACCEPT_ENCODING
  *  HTTP_COOKIE
  *  HTTP_HOST
  *  HTTP_IF_MODIFIED_SINCE
  *  HTTP_IF_NONE_MATCH
  *  HTTP_REFERER
  *  HTTP_USER_AGENT
  *  PATH_INFO
  *  QUERY_STRING
  *  REMOTE_ADDR
  *  REMOTE_USER
  *  REQUEST_METHOD
  *  REQUEST_URI
  *  SCRIPT_DIRECTORY
  *  SCRIPT_FILENAME
  *  SCRIPT_NAME
  *  SERVER_NAME
  *  SERVER_PORT
  *  SERVER_PROTOCOL

In addition, Fossil adds the following environment variables:

  *  FOSSIL_CAPABILITIES
  *  FOSSIL_REPOSITORY
  *  FOSSIL_USER

The FOSSIL_USER string is the name of the logged-in user.  This variable
is missing or is an empty string if the user is not logged in.  The
FOSSIL_CAPABILITIES string is a list of 
[/setup_ulist_notes|Fossil capability letters] that
indicate what permissions the user has on the Fossil repository.
The FOSSIL_REPOSITORY environment variable gives the filename of the
Fossil repository that is running.

The [https://sqlite.org/ext/checklist|checklist application] uses the
FOSSIL_USER environment variable to determine the name of the user and
the FOSSIL_CAPABILITIES variable to determine if the user is allowed to
mark off changes to the checklist.  Only users with check-in permission
to the Fossil repository are allowed to mark off checklist items.  That
means that the FOSSIL_CAPABILITIES string must contain the letter "i".

If the HTTP request includes content (for example if this is a POST request)
then the CONTENT_LENGTH value will be positive and the data for the content
will be readable on standard input.

<h2>4.0 CGI Outputs</h2>

CGI programs construct a reply by writing to standard output.  The first
few lines of output are parameters intended for the webserver that invoked
the CGI.  These are followed by a blank line and then the content.

Typical parameter output looks like this:

<blockquote><verbatim>
Status: 200 Ok
Content-Type: text/html
</verbatim></blockquote>

CGI programs can return any content type they want - they are not restricted
to text replies.  It is OK for a CGI program to return (for example)
image/png.

The fields of the CGI response header can be any valid HTTP header fields.
Those that Fossil does not understand are simply relayed back to up the
line to the requester.

Fossil takes special action with some content types.  If the Content-Type
is "application/x-fossil-wiki" or "application/x-markdown" then Fossil
converts the content from Fossil-wiki or Markdown into HTML, adding its
own header and footer text according to the repository skin.  If the
content-type is "text/html", that is normally passed straight through
unchanged.  However, if the HTML content is of the form:

<blockquote><verbatim>
<div class='fossil-doc' data-title='DOCUMENT TITLE'>
... HTML content there ...
</div>
</verbatim></blockquote>

In other words, if the outer-most markup of the HTML is a &lt;div&gt;
element with a single class of "fossil-doc", 
then Fossil will adds its own header and footer to the HTML.  The
page title contained in the added header will be extracted from the
"data-title" attribute.

Except for the three cases noted above, Fossil makes no changes or
additions to the CGI-generated content, but simply passes the verbatim
content back up the stack towards the requester.

<h2>5.0 Trouble-Shooting Hints</h2>

Remember that the /ext will return any file in the extroot directory
hierarchy as static content if the file is readable but not executable.
When initially setting up the /ext mechanism, it is sometimes helpful
to verify that you are able to receive static content prior to working
on getting your CGIs working.  Also remember that CGIs must be
executable files.

Fossil likes to run inside a chroot jail, and will automatically put
itself inside a chroot jail if it can.  The sub-CGI program will also
run inside this same chroot jail.  Make sure all embedded pathnames
have been adjusted accordingly and that all resources needed by the
CGI program are available within the chroot jail.

If anything goes wrong while trying to process an /ext page, Fossil
returns a 404 Not Found error with no details.  However, if the requester
is logged in as a user that has Debug privilege (capability letter "D")
then additional diagnostic information may be included in the output.

If the /ext page has a "fossil-ext-debug=1" query parameter and if
the requester is logged in as a user with Debug privilege, then the
CGI output is returned verbatim, as text/plain and with the original
header intact.  This is useful for trying diagnosing problems with the
CGI script.