SITE LOG

TODO

TOOLS

build_site.py


While making this website, I found myself in need of a static site builder. Writing html by hand is possible, but some content (like blogposts) quickly becomes a chore to edit in html form. There are many site builders available, but I found all of them unsatisfactory for one of the following reasons:

  • It forces you to use a particular folder structure, requiring you to either store blogposts and templates in specific directories, or to have an 'input' and 'output' folder.
  • It makes assumptions about the files you're dealing with and how to preprocess them
  • It's bloated and requires you to read a long wiki in order to do what you want
  • It's slow (looking at you, jekyll)

So I made my own: build_site.py ~600 lines. It is simple to use while also being easily configurable to do exactly what you want via python. Okay, technically every open source application is 'configurable' if you think of changing the source code as 'configuration', but this script is made with this in mind, and is based on the understanding the best static site builder is actually just python. In fact, you can write python code wrapped in {%% %%} and use echo() calls to build the html, similar to php files. The only real assumption it makes is that your html template filenames are prefixed with an underscore, though even that can be turned off.
There are two ways this script can be used:

  1. 'Aggregating' content via {% for %} loops, to e.g create an index page of all blogposts
  2. Creating many individual html files via %foreach property in the frontmatter, to e.g create individual post html files from markdown files

By default, it reads all html and php files starting with an underscore, applies the template, and outputs the built file without the underscore to the same directory. You can override this by setting the --input and --output options, or by setting the %output key in the frontmatter at the start of the template file to output to a custom path.

Details

Usage:

python build_site.py path/to/project

/path/to/project is taken as the 'root' path of the project. The script can also take an -i or --input argument (among others) to only process a subdirectory of root. The reason why there are two input-related arguments is to allow the use of absolute paths in templates, similarly to how they can be used on the website, e.g /data/file.md.
To process only a single template, the --input can be set to the filepath, or, if the template does not use absolute paths, you can just provide the filepath as the root argument.

Template syntax:

Key-value:

{{item.key}}

Substituted with the corresponding key from the frontmatter of the file. {{item.content}} stores the preprocessed contents of the file (usually, depending on the preprocessor). The default markdown preprocessor also adds a few other keys such as item.firstheader and item.filename, see the code for details.

Custom function:

{{function(input)}}

Custom function. Substituted with the output of the custom function defined in the Functions class. input can be the item name, e.g 'item' (passes the python dict), 'item.key' (value of the corresponding frontmatter property), or *%* (all data dict). Comma separated inputs will be split and passed as separate arguments. Any other input will be passed to the function as a string.

For loop:

{% for item in path %}..{% endfor %}

For loop. Contents are added for every item in path. By default, path can be either a folder or a csv file, though it's easy to extend the script with a custom iterator (see the extending section).
Example

If condition:

{% if cond %}..{% elif cond2 %}..{% else %}..{% endif %}

If condition. Contents are added if cond evaluates to true. cond is a python expression.
Example

Shorthand if:

{{cond ? trueVal : falseVal}}

Shorthand if. cond is a python expression. If true, trueVal is substituted, else falseVal.
Example

Import directive:

{% import path %}, {% import path as item %}

Pastes the file contents of path after preprocessing. When using with as, saves the file data into the item variable instead of pasting the contents, after which the keys will be accessible like usual with {{item.key}}.
Example

Export directive:

{% export item as template_path to output_path %}

Create an html file from an item and template. Can be used in place of the %foreach frontmatter property.
Example

Python code:

{%% python code %%}

Use python and echo() calls to build the html.
Example

Frontmatter special keys:

There are two special keys which can be defined in the frontmatter of any template file:

  • %output: path - Overrides the default output path. Set to %none to ignore the file.
  • %foreach: path - Special directive that will create a new html for every item at the path. The html files will be output to the path defined in %output. The keys can be accessed via page, e.g {{page.content}}. See the example.

Preprocessors / Iterators / Functions:

There are three classes defined at the start of build_site.py:

  • Preprocessors - Defines functions for preprocessing different file formats. The field extdict maps file extensions to the appropriate preprocessor function. By default, markdown files are converted to html, and html files are pasted as they are. To define a custom preprocessor, write a function taking a path and returning a dict and add the corresponding file extension to extdict.
  • Iterators - Defines iterators for {% for %} loops. extdict maps file extensions to the appropriate iterator. There is an iterator for folders and one for csv files by default. To define a custom iterator, write a function taking a path and returning a list of dicts, and add the corresponding file extension to extdict.
  • Functions - Defines custom functions for use inside {{}}. To define a custom function simply write a new function returning a string in the class. The default function py will evaluate the passed argument as a python expression and return the result. Of course, the functions themselves can be set up to do anything – for example, for loops, if conditions and the import and export directives can be easily implemented as functions.

For preprocessors and iterators you can also define functions that modify the input in a certain way. To do this, write a function taking a string (which will be the contents of the parentheses) and returning a dict or list of dicts, and add the key functionName(*) to extdict. For preprocessors, the function raw can be used to import files without preprocessing, e.g {% import raw(/path/file.md) %}. For iterators, the function reverse can be used to reverse any list, e.g {% for file in reverse(/path/folder) %}. The function range only takes two numbers as input and returns a list where the elements are dicts of the form {'index':num}, e.g {% for x in range(1,10) %}Number:{{x.index}}{% endfor %}

Examples:

Iterate over folder:

<div class="index-links">
    {% for item in /data/blog %}
        <a href="#{{item.id}}">{{item.title}}</a>
    {% endfor %}
</div>

Reverse iterate over csv file:

This will iterate over the rows of a csv file in reverse and only add the ones where the 'hidden' column is not 'true':

{% for row in reverse(/data/links.csv) %}
    {{row.hidden != 'true' ? <a href="#{{row.url}}">{{row.text}}</a> : }}
{% endfor %}

or, with python:

{%%
    for row in reversed(Iterators.get_data('/data/links.csv')):
        if row.hidden != 'true': 
            echo(f'<a href="{item.id}">{item.title}</a>')
%%}

Note that
Iterators.get_data('reverse(/data/links.csv)')
= Iterators.reverse('/data/links.csv')
= reversed(Iterators.get_data('/data/links.csv'))
= reversed(Iterators.csv(get_real_path('/data/links.csv')))

Blogposts index page:

Create a simple blog index page showing the first three lines of every post:

{% for item in /data/blog %}
    <input id="{{item.id}}-check" type="checkbox" {{'hidden' not in item.classes ? checked : }}>
    <div id="{{item.id}}" class="post {{item.classes}}">
        <h1>{{item.title}}</h1>
        <span>
            <a href="{{item.id}}.html">View post</a>
            <span>Posted: {{item.date}}</span>
            <span>{{wordCount(item.content)}} words</span>
        </span>
        <div class="contents">
            {{firstThreeLines(item.content)}}
        </div>
    </div>
{% endfor %}
<style>
    .post .contents { display: hidden; }
    {% for item in /data/blog %}
        #{{item.id}}-check:checked~#{{item.id}} .contents { display: block; }
    {% endfor %}
</style>

Example markdown file in /data/blog:

---
title: born to die
classes: hidden
date: 2053-09-02
---
# world is a fuck
blergh

Individual posts:

A template with a %foreach frontmatter property will make it output many html files, and can be used to create the individual blog post pages:

---
%foreach: /data/blog
%output: /html/blog/{{page.id}}.html
---
<!doctype html>
<head>
    ...
    <title>{{page.title}}</title>
</head>
...
<div id="{{page.id}}" class="post {{page.classes}}">
    <h1 class="title">{{page.title}}</h1>
    <div class="contents">
        {{page.content}}
    </div>
</div>

Individual posts (alternative):

Instead of using the %foreach frontmatter property in the post template, it's possible to 'export' the items in a for loop:

{% for item in /data/blog %}
    {% export item as /html/_post.html to /html/blog/{{item.id}}.html %}
{% endfor %}

This page:

<div class="container">
    <div id="sitelog" class="section">
        {% import /data/sitelog/log.md %}
    </div>
    <div id="todo" class="section">
        {% import /data/sitelog/todo.md %}
    </div>
    <div id="tools" class="section">
        {% import /data/sitelog/tools.md %}
    </div>
</div>

Import + css selector:

You can import a file as an item, then do other logic like so:

<div class="container">
    {% import /data/post.md as post %}
    {{css(#my-id > h1,post)}}
</div>

Index Pagination:

In _index.html, put

<div id="blog-index">
{%%
    paginate = 5
    offset = 0 if 'page' not in locals() else page.offset
    data = Iterators.get_data('/data/blog')[::-1]

    echo <div style="background: wheat;">
    echo <p>Showing post {offset+1}-{min(offset+paginate, len(data))} of {len(data)}</p>
    if offset > 0:
        echo <a href="index{offset//paginate-1 if offset > paginate else ""}.html">Previous</a>
    if offset + paginate < len(data):
        echo <a href="index{offset//paginate+1}.html">Next</a>
    echo </div><br>

    for i in range(offset, min(offset + paginate, len(data))):
        echo <a href="{data[i].id}.html"><h2 style="background: wheat;">{data[i].title}</h1></a>

    if offset + paginate < len(data):
        do_export(page_item={'offset': offset + paginate}, template='_index.html', out=f'index{offset//paginate + 1}.html')
%%}
</div>

Here we create paginated index files to show a maximum of 5 posts per page. In the first block, we define constants and get the list of (reversed) posts from the data folder. In the second block, we add links to the previous and next index pages, provided it's not already the first or last page. In the third block, we add links to the posts until we reach the pagination limit. Finally, we create the next page with the same _index.html template (recursively) by calling the do_export function, which is the function responsible for the {% export %} command.

Note that echo string gets converted to echo(f'string'), this is the only preprocessing that is done before the code is executed.

Longer examples:

Remarks

  • Using custom functions is recommended over many if statements and py() calls because they don't have to be eval'd.
  • If you're somehow still reading this and want to use it, let me know if you encounter any bugs

convertbox.py


50 years of software development have all culminated in the single greatest application ever written.
When run, convertbox.py shows a small window where one can drag and drop files, which will then be processed by a command defined in the input box at the bottom. For example, if you want a quick png to jpeg converter, you put magick convert -quality 80 {path} {pathnoext}.jpg in the input box and just DRAG AND DROP image files into the window.
I am a fucking genius.

Download

playlist-convert.py


Input:
    An m3u playlist file path.
Output:
    - If ffmpeg is available: mp3 files with embedded images (quickly), and a m3u playlist containing song details.
    - If ffmpeg is unavailable: An error message.

Download