The making of hackersanddesigners.nl

MainNavigation	No

Intro

This article shares the development of the H&D website and describes its new and expanded functionalities. It is addressed to readers already familiar with coding, as well as anyone interested in learning about using Mediawiki as an infrastructure, H&D's work process, and our approach to making and breaking things. The new website reflects the diversity of the H&D activities and proposes an online space for workshopping and a growing archive.

The H&D website runs on a Mediawiki installation. This is the same software on which the free online encyclopedia Wikipedia is built. Wiki websites function as both frontend and backend spaces. The frontend serves and present the content in a selective manner. The backend is used for storage, uploading, and editing. The wiki enables a variety of users to work and expand the wiki database while having simultaneous access to the same resources (e.g. reading through published articles).

Providing the wiki content through a frontend website allows users to easily and quickly access a large amount of information. This is in comparison to having to move through the wiki’s idiosyncratic backend navigation architecture, which is more of a search tool than a navigation bar. Still, wiki’s architecture is interesting and enables a rethinking of web design, dominant backend systems, and their imposed structures.

Hierarchy & structure

I spent quite some time getting a sense of how this wiki had been shaped over the previous three years of its existence when I began to work on the new version of H&D website. Wikis often promote a flat hierarchy, such as linking and backlinking between pages. H&D’s wiki makes use of that approach while ignoring all those red links that are automatically created when a page does not yet exist. Everything requires to be linked and needs to exist as a page. This process promotes a flat ontology for each piece of data.

H&D’s wiki was missing a straightforward way of creating new content. As part of the explorative approach taken from the beginning, you can either make an event page or an article page. Both methods, however, require you to copy-paste particular syntaxes from previously created pages, which is very prone to causing errors. Nonetheless, Mediawiki offers ‘forms’ and ‘templates’ as a way to define the content structure of a page. This outlines what kind of information is required to create that page.

An immediate benefit of this is de-duplicating variants of the same tag or token is that you are adding to fill out a field (was it ‘meetup’ or ‘Meetup’ or ‘meet-up’? Wonder no more). This is not just petty grammar. The wiki engine will otherwise turn that link to ‘Meetup’ red if it’s not spelled correctly! The page is missing! And then you make yet another page about ‘Meetup’ but spelled this time as ‘meet-ups.’

Considering encyclopedic minds and needs, this all sounds banal. But, what if you are just defining and adding content while you go with it, as in H&D’s wiki case?

Manually adding all the information necessary to create the article is cumbersome, especially when the content is being constantly modified. My solution was to introduce built-in fields, where the editor can select metadata such as dates, names of collaborators, and categories. That metadata gives articles a home and makes content creation more intuitive without being too pedantic. This is helpful, as H&D’s wiki often does not follow the wiki mindset on how to layout and organize information.

After three years of usage, it was the perfect time to review H&D’s wiki structure. There was enough content and trials to work with, making it possible to decide where to leave freedom and where improvements were needed.

For those who want to learn more about wiki editing and mark-up, a wiki tutorial may be found here.

Toolkit & front-end

The frontend runs as a Python app. I never wrote Python before, but the language turned out to be part of H&D’s practice, so I just went for it. I opted to avoid frameworks and forced myself to learn almost everything necessary to make a website from scratch. I started with server CGI and then embraced Python’s WSGI.

Concepts

One of the bigger challenges was how to create a proper navigation menu.

The wiki lies on a flat surface, linking articles between each other. While the wiki gives access to special search and filter pages where you can look at all the pages with particular categories, templates, properties, etc., there is no way to fetch that view through `json.`

As I expected, Semantic Mediawiki has an option to create collections of pages with common metadata elements. For example, it gave the option to find and group together all pages within a date range AND with x, y, z properties but NOT with a, b, c properties.

This feature is called Concepts. It was the smartest way I found to let editors define site sections directly from the wiki. These sections would then be fetched as navigation links on the frontend. This avoided the need to set up hard-coded navigation items. As such, it keeps a fluid control of the content in the frontend from and through the wiki.

Tools

A tool that I used extensively for building the frontend was BeautifulSoup. At its core, BeautifulSoup is a web-scraper plugin. In the case of building the wiki, I exploited it as a fine-grain filter tool to clean up undesired `html` tags fetched by each `json` call. It worked magnificently, providing a very detailed level of control. This fine control made it much easier to add atomic `css` classes to all `html` tags, thus forming a template. Because Mediawiki outputs so much wrapping `divs,` though, I felt obliged to get rid of them from the final template. I still wonder why this happens. It could be a problem of historical heritage due to maintaining a big and old framework that’s being modernised over time, or maybe Mediawiki developers just fell into the div wormhole and never got out.

Using Python instead of, say, Javascript, felt like a better choice for this project mostly because it helped me to dive into data structures. It was my first time making a restful API app. Although `json` exists because of Javascript, Python feels conceptually more aligned when learning programming and working with datasets. As Python suggests, use `try` and see what happens (`except`) instead of assuming that some particular data might be in there by using `if` this exist (`else`...). It is a different mindset that I appreciated and learned to adopt over the three months of the project.

Some thoughts on working with the wiki

Because of H&D’s activities and structure, the wiki has a lot of potential in the workshopping context. When needed, a hundred new users can sign up and make use of the space to document, draft, write, and edit content, all at the same time and without a fuss.

The frontend, conversely, becomes an easier place to access and check for upcoming events, as well as for browsing through the wiki content in a focused and well-presented manner. The H&D website becomes both a workshop AND a library.

Repository

hackersanddesigners / had-py / readme.md

hackersanddesigners.nl

Website for H&D, Amsterdam.

Side A

The wiki and the backend are running on MediaWiki, acting as open-door cms, workshop space, archive for projects and knowledge.

A wiki comes with no predetermined hierarchy, which lets you create your own logic for a navigation. On the the H&D wiki we work with Forms and Templates for a better editing experience; Semantic MediaWiki for organising all bits of data and metadata on each page and Concepts to create sections and the navigation for the frontend website.

Side B

The front-end is built in Python 3.6 with:

• Werkzeug to wrap the website inside an app and put it on the internet through the WSGI interface
• Requests to fetch all json from the MediaWiki backend
• Beautiful Soup to parse and clean up wiki’s extra markup, and to add atomic, functional css classes to html tags
• Jinja to build the templates (works hand in hand w/ Werkzeug)

To run the frontend website

Requirements

• terminal basics (introduction)
• Python (introduction)
• pip Python package manager (instructions)
• venv to setup a Python environment (check this for instructions)
• git!

Then

• open terminal
• clone the repository: git clone git@github.com:hackersanddesigners/had-py.git
• setup a new Python 3.6 environment inside the repo folder: python3.6 -m venv env. To activate it, type: source env/bin/activate
• type: python had.py
• you will be asked to install a bunch of dependencies using Python’s pip package manager. Do pip install name-of-package and keep running python had.py until everything is installed
• type: python had.py again, the Python app should start and print something like http://127.0.0.1:5000, copy that in your browser to see the H&D’s frontend website ☺︎

Some thoughts on the process

Improving and organising the MediaWiki demanded a good reading of the extensive (at times confusing) documentation, installing some plugins (in particular PageForms on top of Semantic MediaWiki) and manage the data through the MediaWiki interface. The most challenging part was to understand how Mediawiki actually treated data, and to decide how much complexity adding to the existing wiki. Overall, the aim was to be able to create new articles and to edit them in a pleasant manner but not limited the editing activities too much.

The wiki is fully able to parse different syntaxes whenever you need to embed an image, a video, an audio clip, an etherpad, etc, and it supports your writing and mark-up. One instance is the token autocompletion: for example some fields suggest you already used words (tokens) while you are typing.

Check Page Forms:Input types for an overview of what’s available.

Currently, the Event page receives data with this template

{{{for template|Event}}}

{| class="formtable"

! Name

| {{{field|Name}}}

|-

! Location

| {{{field|Location|input type=combobox}}}

|-

! Date

| {{{field|Date}}}

|-

! Time

| {{{field|Time}}}

|-

! PeopleOrganisations

| {{{field|PeopleOrganisations}}}

|-

! Type

| {{{field|Type|input type=combobox}}}

|-

! Web

| {{{field|Web|input type=checkbox}}}

|-

! Print

| {{{field|Print|input type=checkbox}}}

|}

{{{end template}}}

'''Text'''

{{{standard input|free text|rows=15|editor=wikieditor}}}

The combobox input type (see) looks up what has been already saved in that particular field from all the previous pages, and suggests possible completion.

± ± ±

Working with the frontend instead, requires to have / get some knowledge of:

• restful APIs (ref)
• how to handle page request, setup page routing, serve the app to the web, etc. — in this case with Python
• how to use a templating library

As we decided not to use a framework, the second point in particular gives you the chance to manually fiddle with stuff than if you mostly use a (php-based) cms, usually you can avoid or don’t have to touch at all.

Working with a restful API introduces you to the powerful world of json (intro), a very flexible data format able to convert anything into anything else—and back.

When working with Python, there are two very useful commands that let you inspect a json dataset:

• type(data), eg print(type(data)) — to check if the current object is a dict, or a lists, a string and so forth
• data.keys(), eg print(data.keys()) — to check the object’s keys (which is how json structures data)

Alternatively, you can access to a pretty-printed version of the json request you made to MediaWiki, by visiting that url, eg

https://en.wikipedia.org/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=json

and adding fm after format=json, like format=jsonfm (see docs) to have an indented, more readable version

https://en.wikipedia.org/w/api.php?action=query&titles=Main%20Page&prop=revisions&rvprop=content&format=jsonfm

This double approach lets you inspect better what’s going on in case you are unsure which data you are working with.

± ± ±

Once you get acquainted with MediaWiki’s design language (Forms, Templates, etc), json data structures and some basic Python, you can play around with the website.

± ± ±

custom mediawiki styles

• /mediawiki/resources/src/mediawiki.action/mediawiki.action.history.styles.css

this add more padding and take out list styles to the revision list

#pagehistory {
    margin-left: 0;
    padding-top: 0.25rem;
}
#pagehistory li {
    list-style: none;
    padding: 0.5rem;
    margin-bottom: 0.25rem;
}
#pagehistory li:hover {
    background-color: #f8f9fa;
    border: 1px dashed #a2a9b1;
}

• /mediawiki/resources/src/mediawiki.action/mediawiki.action.history.css

this set the ‘compare selected revisions’ to the bottom-right, always at reach and more prominent (before there were two buttons at the top and bottom of the revision list, very much buried amongst the rest of the text)

.historysubmit {
    position: fixed;
    bottom: 0.5rem;
    right: 1rem;
    padding: 0.5rem;
    font-size: 100%;
}

.historysubmit:hover, .historysubmit:active {
    background-color: yellow;
    cursor: pointer;
}

↗ view more tools ↗↗ view in GitHub ↗