About this page
This page describes the design and development of BILE, a program that I wrote to give HTML pages a common look and feel. BILE was written in C and has been compiled for both Windows and Linux.
- Application model
- Known issues
- Further directions
- Document history
I wrote BILE originally to help produce a satirical Web site (hence the name) that never got off the ground. This was some years ago before blogging software and Web-based Content Management Systems were commonplace and before Web hosting services routinely offered CGI or other server-side interactivity. For this reason, BILE was created as an “offline” tool; the intention is that you run BILE over your Web pages and upload the results to your Web site. Therefore, BILE is suitable in the following situations:
- If your hosting service only permits you to host static content
- If your Web site is small and doesn’t need a full Content Management System
- If your HTML documents are intended to be accessed off line; a manual or slideshow, for example.
BILE can be used to style dynamic content such as PHP or ASP pages but, as with static content, BILE must be run on these files before they are uploaded to the server.
At the time BILE was conceived, the most commonly-available technology for giving Web pages on a site a consistent look and feel was Server Side Includes (SSI). However, SSI+ requires the Web page author to embed special comments into each and every page on which they want common page elements (such as sidebars and footers) to appear. Instead of doing this, BILE uses a template file into which the body of each page is “poured”; see the section on BILE’s application model for a more detailed description.
The first version of BILE was written in QBasic, the free version of QuickBasic that Microsoft distributed with DOS 6 and Windows 95. This version of BILE offered the user very little control over the output unless they were willing to change the BASIC code. It was also limited in the kinds of metadata it could extract from the input files because of limitations in QBasic itself.
Because of these limitations, I decided to rewrite BILE in a more powerful language. Specifically, I wanted to add the following improvements:
- A way of extracting metadata from files of different types
- Support for multiple directories and multiple templates. The original version of BILE operated only on a single directory and used only one template file.
- A proper template language with conditional statements, variables, etc. (Note however that the template language has no general looping construct or equivalent to the “goto” statement; it is not intended as a general-purpose programming language)
After some experiments, I decided on C as the implementation language.
The BILE source currently consists of about 6,000 lines of C. The source is fairly portable with little conditional compilation. I build the code using the GNU C Compiler on Linux and the MinGW32 port of same on Windows. The build is controlled using GNU make but I don't use the GNU autoconf tools. In addition, I use the following software for version control and change tracking:
- Git (originally, CVS) for version control
- CVSTrac, a bug-tracking and Wiki tool that integrates with CVS and Git
- Doxygen, to generate source code documentation
- Git Extensions, a shell extension for Windows that allows Git operations to be carried out from Windows Explorer
BILE was originally intended to be used to produce a news-type Website and its terminology reflects this. A BILE “project” is referred to as a publication. A publication is broken into sections which contain stories. Each section can have one or more indexes which are used to generate lists of links, tables of contents, etc. Stories can also have tags like those found on a number of Web sites. In the present implementation of BILE, publications and sections map to directories and stories map to files. There only difference between the publication and its sections is that indexes defined in the publication configuration file (see below) apply to all files in all sections and subsections, whereas indexes defined in section configuration files only index those files in the section directory itself.
Each BILE entity — publication, section, index and story —
has a number of variables associated with it. Some of these variables
are created by BILE itself on startup, some are read from file metadata and
configuration files and some are modified by BILE during processing.The
entities form a set of nested scopes. For example, if BILE is processing a
story file’s template and it encounters a reference to a variable
$var, it will first check if there is a local variable called
$var. If there is not, it will check the section’s
variables, the parent section’s variables and so on up to the
publication’s variables. Finally, it will check the computer’s
environment variables. If no variable can be found, it will create a new local
variable in the story called
$var and assign it a blank value. If
BILE code in an “inner” scope attempts to change a variable in an
“outer” scope, BILE will create a local variable with the same name
and containing the modified value. This is to prevent side effects as BILE
doesn’t guarantee the order in which it processes files. However, this
behaviour can be circumvented if necessary by using the
command described below.
BILE is invoked as follows:
bile [-f] [-v] -i input directory -o output directory -t template directory
||Optional. Force regeneration of all output pages even if the input hasn’t changed.|
||Optional. Verbose mode. Generate progress information while running.|
||Mandatory. Specifies the input directory.|
||Mandatory. Specifies the output directory.|
||Mandatory. Specifies the template directory.|
One of the benefits of using BILE is its ability to automatically generate
index pages which act as tables of contents for a site. An index
can be added to any template using the
block command but a separate file or set of files can be generated for an index
by setting the
variables for in the index’s definition in the publication or section
Normally, an index page will consist of only a single output file. This can
be overriden by changing the value of the
SET command as you are changing
the global state) and then using the
command. BILE checks the value of the
$index_file variable after
it has output the index page and if it has changed, re-runs the template
with the new file name.
Note: While inside an
INDEX block, BILE changes
the way it searches its scope for variables. When processing a file normally,
the search looks like this:
story → section → … publication → environment
INDEX block, the search looks like this:
story → index → section → … publication → environment
Configuration filesBILE configuration files are used to store information about the publication and each section in the publication. They also store the index definitions for the publication and its sections. They have a
.bilefile extension. The publication configuration file is called
publication.bileand is located in the publication’s top-level directory. Each subdirectory in this direction can have a section-specific configuration file called
The configuration files have the following format:
# A comment; ignored
$var_name1 = `A literal`
$var_name2 = `A valid ` . ucase(`bile`) . ` expression`
# Index definition
$sort_by = `-file_date`
Note the following:
- Blank lines and those beginning with a “#” character are ignored
- Variable names are prefixed with a “$” character, similar to PHP or Perl. Any valid BILE template language expression can appear on the right-hand side of an assignment.
- Indexes are defined by the
indexkeyword and the definition is terminated with the
Mode of operation
For BILE to run, it needs three arguments: the input directory from which files are to be read, the output directory where processed files are to be created, and the template directory where template files are stored. When BILE is run, it performs two main phases of processing: reading the data it needs to generate the output, and the generation phase itself. The first phase proceeds as follows:
- BILE initialises the publication structure, creating some global variables
- It reads the
publication.bilefile and creates the publication indexes.
- For each normal file BILE encounters, it extracts the metadata from it
into a story and adds it to the appropriate index. Some kinds of file
are never indexed (these include standard Web files like
favicon.icoas well as files with the extension
.incand any file whose name begins with a “.” character. Additionally, a file can prevent itself from being added to an index by setting the
- For each directory it finds, it creates a section and reads the
configuration from that directory’s
section.bilefile, creating any indexes for this section. It does this recursively so that all subdirectories of the input directory are read.
At the end of this operation, BILE has everything it needs to do its job. It then proceeds to the generation phase:
- For each input file, it checks the
$use_templatevariable if a template file is to be used.
- If a template is to be used and BILE has not used this template before, it reads the file from the templates directory and “compiles” into an internal form. BILE caches its templates in this form so they do not have to be read for each file.
- BILE then “executes” the template, passing the input file
to it. The result will be written to the output directory.
- BILE preserves the directory structure of the input directory
when it copies files to the output directory. That is, a file
in the input directory called, say,
input/news/latest.htmlwill be copied to
- If the file already exists in the output directory, BILE checks
if the input file and the template have changed since the
the date on the ouput file. If they have not, it will not
change the output file. This behaviour can be overriden by
specifying the “force” option (-f) on
the command line.
If the file doesn’t already exist in the output directory, BILE will set the variable
$is_new. If the file already exists but the input file is newer, BILE will set the variable
$is_modified. If the template file is newer, BILE will update the output file but it will not set
- BILE preserves the directory structure of the input directory when it copies files to the output directory. That is, a file in the input directory called, say,
- Any subdirectories of the template directory are copied to the output directory so that any images, stylesheets or other files required by the templates will be available.
- For every index defined in the publication and its sections, BILE checks if an index page is to be generated and generates the index pages if necessary.
Controlling the output
BILE allows you to specify variables in the configuration file or metadata to give finer control over what gets output.
||This is the variable that tells BILE to use a template on the input file.|
The location of the template file to use, if
BILE template files are simply text files with commands enclosed in
double square brackets,
[[like this]]. Template commands can be
simple commands or block commands that enclose other commands. Blocks
are closed by preceding the command name with a slash, for example,
[[if]] ... [[/if]]. Some commands are immediate; that is,
they are evaluated when the template is loaded, not when it is
executed. Immediate commands are prefixed with a “!”
The following commands are defined in BILE:
||A comment. Everything after the “#” is ignored. No output is generated|
||Writes the result of the expression to the output, escaping any HTML special characters.|
||Writes the result of the expression to the output without escaping any characters.|
Writes the body of the input file to the output. What constitutes the body
of a file varies depending on the input file type. For example, for an
HTML file, only the parts of the file between the
||Prints a “breadcrumb trail” for the input file using the result of the expression as a separator. The “breadcrumb trail” will consist of a link to the index page of the first index defined in the input file’s section, parent section, etc., all the way to the publication level.|
Leaves the current block.
Note: Unlike C and its descendents, a BILE
In this case, the
||Leaves the current block if the expression is true.|
||Conditional block command. Syntax:
Note that there is no
||Includes the filename given as the the result of the expression in the template including any BILE commands. Immediate command, so executed once when the template is loaded.|
||Block command. Evaluates the expression and looks for an index of that name. If an index is found, the block is evaluated for each file in the index. Used to generate tables of contents. If included in an index template, the expression may be omitted.|
Assigns the value of the expression to the local variable
||For HTML files, outputs any text that occurs before the opening <HTML> tag. Useful for PHP files which may have setup code before the <HTML> tag.|
||Generates a list of sections defined in the publication.|
Assigns the value of the expression to the global variable
||Block command. For each tag defined in the input file, evaluate the block.|
BILE has simple expression evaluator based on Jack Crenshaw’s series of articles entitled, “Let’s build a compiler”. The syntax is similar to that of PHP’s with some peculiarities described below.
Like PHP, BILE variables are prefixed with a “$” character. Prefixing a variable name with two “$” characters works as it does in PHP, allowing a simple form of indirection, for example:
[[let $a = `Test`]]
[[let $b = `a`]]
This will write “Test” to the output. This also works for functions, for example:
[[let $func = iif($do_uppercase, `ucase`, `lcase`)]]
will write “TEST” to the output if the variable
String literals may be delimited by single quote ('), double quote (") or backquote (`) characters. BILE does not interpolate variable names in double-quoted strings like PHP does.
BILE recognises the Boolean literals
The following additional values are regarded as Boolean False:
- The empty string
- Any string that is convertible to the number zero
- The literal string, “False”.
All other values are regarded as True in a Boolean context.
Like PHP, “.” is the preferred string concatenation operator. Using “+” may not work.
BILE’s arithmetic operators are (in decreasing order of precedence):
The logical operators have higher precedence than the arithmetic operators and are (in decreasing order of precedence):
BILE’s comparison operators are somewhat unorthodox. This is because BILE is often embedded in HTML code in which the standard comparison symbols like “<” and “>” have special meanings and must be escaped. Although the BILE parser could be modified to recognise the escaped form of the operators, I felt this would reduce the readibility of BILE code. Therefore, I decided to use two-letter names for the comparison operators which FORTRAN and DCL programmers might recognise, but will probably be unfamiliar to everyone else!
|Conventional operator||BILE operator||Description|
|=, ==||eq||equal to|
|<>, !=||ne||not equal to|
|<=||le||less than or equal to|
|>=||ge||greater than or equal to|
BILE has a number of built-in functions. These functions have been added on a more-or-less ad-hoc basis as I needed them so they are something of a “mixed bag”. The functions in BILE are stored in a table of function pointers so it is fairly straighforward to add new ones.
Removes the directory part of
Note: This function is equivalent to the Oracle function of the same name.
Returns True if a variable called
Returns the directory part of
Returns an SGML entity reference. For example,
Runs the external program
Reads the file
Returns True if the file
Note: This function is equivalent to the VB/VBA function of the same name.
Checks the publication for an index called
These functions are used to generate Previous/Next links on pages.
Returns the length of
||Returns the current time as the number of seconds since midnight of Jan 1 1970.|
Given two absolute paths
Formats a time value. Accepts the same format as the
Returns the substring of
Note: BILE counts the characters in strings starting from zero.
Returns an SGML element (tag) with the specified attributes. For
Note: In order to simplify the parser, there can be no space between a function name and the opening bracket.
BILE works by being able to extract metadata from its input files. There is a default file handler that works on all files and extracts information common to all files such as their name, size and modification date. There are two additional handlers for processing HTML and image files.
The general file handler will create the following variables in the file’s scope:
The HTML file handler parses the
<HEAD> element of the HTML
file and will create a variable called
$title equal to the contents
<TITLE> element. In addition, it will create variables
<META> element it finds in the
replacing any characters that are illegal in BILE variable names with
underscores. For example, if an HTML file contains the following
<META HTTP-EQUIV="Content-Type" CONTENT="text/html">
<META NAME="Keywords" CONTENT="home page">
BILE will create the following variables:
$title = `Home Page`
$content_type = `text/html`
$keywords = `home page`
The image file handler can parse a GIF, JPEG or PNG image and extract the
image’s type and dimensions. These will be stored in the variables
$content_type (as a MIME type),
$image_height. For GIF and JPEG images, it will check for
embedded comments in the image and store them in the variable
$comments. For PNG images, the handler will check for
tEXt chunks. These chunks contain image metadata in key/value
pairs and the handler will create a BILE variable for each key/value pair it
There are a number of bugs and other problems with BILE:
- Logging is currently broken, so specifying the -v (verbose) switch on the command line doesn’t work.
As it stands, BILE serves my needs pretty well. I use it to maintain this Web site. However, there are a number of features that could be added to make it more useful.
- Build a better expression evaluator, perhaps using flex. However, this would be something of an academic exercise. The rather flaky nature of the BILE language at present discourages writing large quantities of BILE code — which is probably no bad thing. If you find yourself writing lots of BILE code, it’s a good sign your requirements have surpassed BILE’s rather modest abilities and it’s probably time to look at a more powerful templating system.
- Support metadata extraction from more file types (e.g. OLE compound documents, audio files, EXIF image files). Also, the existing file handlers are very crude. They scan the input files using brute force rather than using external libraries like libpng or libjpeg which are probably more forgiving of errors.
- Have BILE generate a script that automatically uploads changed pages to a Web site using FTP or HTTP.
- BILE turns a blind eye to encoding issues. This needs to be addressed if BILE is to work robustly in multilingual environments.
- Allow plugins for additional file types and functions so BILE can be extended without having to recompile.
- BILE is written in straight C, albeit using a pseudo-OO style. This suggests that it might make more sense to reimplement in a proper OO language like C++, Java or C#.
|1.1||Ken Keenan||06 January 2016||Updated details|
|1.0||Ken Keenan||13 August 2007||Initial version|