How to document a summer academy: Difference between revisions

From Hackers & Designers
No edit summary
No edit summary
 
(41 intermediate revisions by 2 users not shown)
Line 1: Line 1:
''Choosing the right technical workflow for your hybrid publication''
'''About the process of making this publication by [[Vicky De Visser]], [[Anja Groten]] and [[James Bryan Graves]]'''


'''by [[Vicky De Visser]] and [[James Bryan Graves]]'''
During the course of the Summer Academy the participants, tutors and organizers actively documented in various ways. We quickly realized this parallel mode of collective production was developing a life of it's own and introduced an interesting form of exchange between all stakeholders. People were using tools like PiratePad for collective note-taking, a local wireless network created with a raspberry pi to share files, swapped usb sticks, and used git repositories to archive scripts and images. However the variety of documentation methods made the following processing of the documented material confusing.<br>
The proposition of this printed matter is to create a documentation of an on and offline documentation process. In addition the intention is to develop an open and central publishing infrastructure before approaching the next H&D Summer Academy in 2016. In the following you will find some considerations that drove the production of this specific hybrid publication.


===Choose an appropriate workflow for your hybrid publishing process===
Before starting with your publication, you will need to inventory your documentation material. Subsequently think about possible future occasions that will require documentation.


==Content creation==
In the case of the H&D Summer Academy documentation we filtered out 3 crucial moments to contemplate:
Before planning on how to gather the documentation we had to ask ourselves: what is to be documented?
 
There are three different moment where content can be created and has to be documented:  


'''Before the summer academy'''  
'''Before the summer academy'''  


Although the academy wasn’t started yet, there was already a lot of content that was generated by the organising staff. They did research, choose a subject, plan schedules and dates, get the word out by printed and online media and process motivations of applicants. It can be good to already document this process. By documenting the organisation flow you can create a base or structure for the upcoming summer school program and it is useful for sharing information about the future academy with applicants.  
Although the academy had not even started yet, a lot of content had been generated already. We had acquired research, chose a subject, planned schedules, got the word out by printed and digital media and processed motivation letters of applicants. It would have been helpful for future editions of the summer academy and possible new collaborators to document that process thoroughly.  


'''During the summer academy'''  
'''During the summer academy'''  


During the summer academy different formats of content were provided to the students. Analyse and estimate this before your academy starts. This way you can start looking for a suitable platform where everything can be documented. It wil save you a lot of time after the summer school is finished.
During the summer academy workshop materials were provided by the tutors, using different platforms and formats. While currently working with WikiMedia for assembling, writing and editing the documented content we realize this could have been the right platform to collect workshop descriptions beforehand, gather materials used during the workshops, put presentations there and add notes and scripts produced during workshops. As the summer academy progressed we more and more missed a central platform where participants could always go back to in order to catch up, acquire practical information or dump some interesting references. Pictures were taken both by the organization and participants. After the academy had been accomplished we emailed every participant to ask them to send their pictures. Having created this publishing infrastructure we are considering to offer an immediate access to the actual publishing process in the future, which would save time.  
Organising this summer academy was our first time and we had to learn the hard way. It took us quiet some time to set up a good platform after the summer academy was over.
 
For collaborative note taking we experimented with Pirate Pad and Ether Pad. Both are online text editors where different users can take and edit notes in the same document. We really liked these editors.
 
Pictures were taken both by the organisation as participants. After the academy we had to email every participant several times to ask them to send their pictures. Using a platform that can be used during the academy can make it easier to upload the photo’s or other content at once.  


'''After the summer academy'''  
'''After the summer academy'''  


After the academy is finished, you can ask participants and teachers to write a review. This can be helpful feedback if you want to organise an other one the year after.
After the summer academy program had been completed we did not wait long to ask the participants and tutors for feedback. To engage people in creating useful feedback you need to make the feedback process as low barrier as possible by for example asking very specific questions, creating forms that are easy to fill out or by interviewing people directly. A wiki could be a convenient platform to process that feedback. See also: [[Feedback from participants of the HDSA2015]]
 
==Choosing the right back-end to collect the documentation==
To compile all the content we wanted to bundle co-writing, co-editing and content management all on the same platform. We looked for a back-end where we could do some real time documenting on, that was easy accessible, had a good way of ordering information, had a short learning curve for participants and supported several kinds of mediafiles.
The lectures, workshops, screenings, excursions, party and exhibition provided a big variety of media such as videos, sound files, notes, images, and code as well as objects and long texts that wouldn’t be suitable for only an online publication.
 
We chose to use WikiMedia, which supports the several file formats and is easy editable by different users. It’s easy understandable structure and user guide is beneficial for changing collaborations and teams. This academy we didn’t have the opportunity to test WikiMedia as a back-end and content collector for students to upload their content as a WikiMedia page because we still had to built it. We used it after the academy to gather the information collected by writing emails and uploading code and pictures to a dropbox repository. During this filling of our WikiMedia back-end we encountered several hierarchy and workflow problems.
 
==Categorisation of the content==


We wanted to sort the documentation to be able to publish a well structured and logic documentation. We sorted the content by categorising the WikiMedia pages. Andre Castro recommended us to use a categorisation system sorted by state, media and topic.  
===Content managing===
In order to compile the documentation material we strived to bundle processes such as co-writing, co-editing and content management optimally all on the same platform. We looked for a back-end that provided real time editing process, that was accessible, provided a comprehensible way of ordering information, had a small learning curve for participants and supported a variety of media files. <br>
The workshops, lectures, screenings, excursions, and the final presentation were documented with media such as videos, sound files, notes, images, and code as well as objects, longer and shorter texts. <br>
Eventually we chose to use WikiMedia, which supports the several file formats and is easy editable by different users. Wikimedia is an easy infrastructure and is well documented. Preparing user guides along the way is beneficial for changing collaborations and teams. <br>
For the first edition of the H&D Summer Academy we didn’t have the opportunity to test WikiMedia as a back-end and content collector for students. We started using it only after the academy to gather the information collected by writing emails and uploading code and pictures to a github and dropbox repositories. During the process of filling the Wiki we did encounter some minor hierarchy and workflow challenges, that we had to get used to as an editorial team.


Because not all media such as videos, sound files and links aren’t suitable for print publication we added a media category so we could divide articles in a suitable print version and not suitable print version.  
===Content categorizing===
To be able to publish a well structured and logical documentation we started sorting content by categorizing the wiki pages we created. We applied a categorization system that was recommended to us, which sorts content by state, media and topic. <br>
The media category is necessary because not all media, for instance video, sound files or links are suitable for a printed publication. Furthermore very long texts are not suitable for the Hackers & Designers website. <br>
It is important to critically sort the content upfront considering the advantages and disadvantages of different media.  


It is important to consider that making a structure can inherit the danger of guiding the contributed content too much. Leave space for adding things that weren’t thought of before.
'''H&D Wiki categories structure:''' <br>
 
===H&D Wiki categories structure:===
'''State'''  
'''State'''  
* EditMe
* EditMe
Line 46: Line 38:
* Published
* Published
* Ready to be published
* Ready to be published
'''Media'''
'''Media'''
* Print
* Print
* Web
* Web
'''Topic / Tags'''
'''Topic / Tags'''
* eg Arduino
* eg Arduino
* eg HTML 2 Print
* eg HTML 2 Print


==Co-editing==
===Co-writing and editing===
After receiving the contributions, the organisation and participants must then select the most viable and implementable ones. The challenge of the selection process is that most submissions are not always useful, have an other hierarchy or are difficult to implement in the publication. Organisations have to deal with the submitted ideas in a very subtle way as throughout the process they don’t want to reject submissions and risk of alienating them which may eventually lead to disengagement.
After receiving all contributions, the participants and eventually the editorial team must select the most viable material. The challenge of the selection process lies in recognizing that some submissions might not be useful, have another logic or are difficult to implement in the publication for other reasons. The challenge consists to include as much of the content produced as possible without alienating it. <br>
It’s advised to give the participants a short introduction in how to collect and document the content. This way the organisers can prevent an extensive editing process afterwards.  
We advise to give the participants a short introduction beforehand in how to collect and document the content. This way the editors prevent an extensive editing process afterwards. <br>
 
WikiMedia lets users adjust articles and write comments on the changes that were made. Participants have accordingly the possibility to go back in time and compare older versions of the same article. That way content will not get lost after editing an article.
Wikimedia let’s users adjust articles and write comments on the changes that were made in an article. Participants have the possibility to go back in time and compare older versions of the same article. This way content will not get lost after over-editing an article.  
 
==Publishing==


After building and filling our back-end documentation platform we had to translate the content to a website and print publication format. For print we looked into two options: creating a publication from the website and creating a publication directly from the WikiMedia page. The process will be discussed here.  
===Editorial considerations===
After building and filling the wiki with content we revisited all articles created and considered an appropriate translation of the content for a website and a printed publication format. For print we looked into two options: 1. creating a publication from the website and 2. creating a publication directly from the wiki.  


'''Currently used set-up'''
'''Current set-up'''  
'''Hackers & Designers documentation platform.'''
{| class="wikitable" border="2"
 
{| class="wikitable" border="1"
|-
|-
|Back-End
|CMS
|WikiMedia Page
|[http://wiki.hackersanddesigners.nl wiki.hackersanddesigners.nl]


|-
|-
|Front-End
|Front-End
|Website of Hackers & Designers
|[http://hackersanddesigners.nl hackersanddesigners.nl]


|-
|-
|Print publication
|Print publication
|Generated from the wiki
|[http://wiki.hackersanddesigners.nl/mediawiki/index.php/Book_sprint_2015 wiki.hackersanddesigners.nl > Book sprint 2015]
|}
|}


'''Workflow'''
'''Editorial quest'''  
organisation > preparation > communication > collaborative note taking and input assembly > editing > publication > printing > distribution  
* collecting & sorting       
 
* prepping (i.e. retouching or re-taking foto's)         
'''Translation Wiki to website'''
* developing an editorial voice (in our case: The format of''How to'' manuals are the unifying component for every article)     
* generating content: co-writing and editing       
* publishing (online)       
* printing       
* distributing (publishing offline)       


'''Translation wiki to website'''
{| class="wikitable" border="1"
{| class="wikitable" border="1"
|-
|-
|Pandoc
|Javascript
|translates wiki-markup to the HTML markup language
|Requests content from wiki


|-
|-
|Python Scripts
|Mediawiki API
|Initiates several actions
|Returns wiki content


|-
|-
|Lay-out website
|HTML/CSS
|CSS
|Layout of the website
|}
|}


'''Translation Wiki to print'''
'''Translation wiki to print'''
 
{| class="wikitable" border="1"
{| class="wikitable" border="1"
|-
|-
|Pandoc
|Pandoc
|translates wiki markup to LaTeX mark-up
|Translates wiki markup to LaTeX mark-up


|-
|-
|Python Scripts
|Python Script
|Initiates several actions
|Initiates several actions


|-
|-
|LaTeX
|LaTeX
|Lay-outing of content from WikiMedia page and generating PDF
|Lay-out of wiki content and generating PDF
|}
|}


===Publishing online===
===Publishing online===
The wiki content is published on the Hackers & Designers website in real-time. Meaning every time the website is loaded or pages are requested the web browser queries the MediaWiki back-end using the MediaWiki provided API, along with installed wiki semantic extensions and their APIs, through a Javascript client-side web application (AngularJS).


The Hackers & Designers website is published always in real-time.  Meaning every time the website is loaded or pages are requested the web browser queries the MediaWiki back-end using the MediaWiki provided API, along with installed wiki semantic extensions and their APIs, through a Javascript client-side web application (AngularJS).
'''Example API Requests''' <br> 
 
''See also [http://wiki.hackersanddesigners.nl/mediawiki/index.php/How_to_document_a_summer_academy#Publishing wiki.hackersanddesigners.nl > How to document a summer academy]''   
 
'''Example API Requests'''


'''Request a page:'''
'''Request a page:'''
     curl 'http://wiki.hackersanddesigners.nl/mediawiki/api.php?action=parse&page=How_to_document_a_summer_academy&format=json&disableeditsection=true&prop=wikitext|images|links'
     curl 'http://wiki.hackersanddesigners.nl/
    mediawiki/api.php?action=parse&
    page=How_to_document_a_summer_academy&
    format=json&disableeditsection=true&
    prop=wikitext|images|links'


'''Request an image:'''
'''Request an image:'''
     curl 'http://wiki.hackersanddesigners.nl/mediawiki/api.php?action=query&titles=File:Chicken-and-Potato-Soup.png&prop=imageinfo&&iiprop=url&format=json'
     curl 'http://wiki.hackersanddesigners.nl/
 
    mediawiki/api.php?action=query&
Code is available on [https://github.com/hackersanddesigners/handd-web Github].
    titles=File:Chicken-and-Potato-Soup.png&
 
    prop=imageinfo&&iiprop=url&format=json'
===Publishing print===
'''1a. Web to print: HTML to Print with CSS and print preview'''


One way to publish a print publication from an online environment is printing a webpage by using CSS. A user can just click ‘Print Page’, a pop-up print preview will appear and the custom CSS will translate the webpage and media to a printable, in pages divided print publication.
'''The code is available on [https://github.com/hackersanddesigners/handd-web github.com/hackersanddesigners/handd-web].'''
The design is also made with code/text, which means that you can use text editors such as Etherpad to collaborate on content with several people at the same time. The negative side on using this set-up is that you if you decide to not use a single page website you’ll have to print each article separate and later bind it into a book. This way no table of contents will be created or there will be no option to apply different lay outing for indexes, special chapters, or glossaries.  


'''Differences Between CSS For The Web And CSS For Print
'''
===Publishing in print===
 
'''Web to print: HTML to print with CSS and print preview''' <br>
You also have to consider the differences between a webpage and a printed publication. What will happen with image galleries, sliders, menu’s and the like? If your webpage has a slideshow with ten images at the top, that’s not going to translate well to paper. The most basic level of interaction on the web is a link. This too becomes problematic. On your computer, you can simply click a link to see where it goes, on paper this functionality is lost so you need a good way to take all those inline links and show the reader where they lead.
One way to publish a print publication from an online environment is printing a webpage by using CSS. A user can just click ‘Print Page’, a pop-up print preview will appear and the custom CSS will translate the webpage and media to a printable print publication divided in pages.
The design is also made with code/text, which means that you can use text editors such as EtherPad to collaborate on content with several people at the same time. The negative side on using this set-up is that you if you decide to not use a single page website you will have to print each article separate and later bind it into a book. This way no table of contents will be created or there will be no option to apply different lay outing for indexes, special chapters, or glossaries.  


'''Differences between CSS for the web and CSS for print
''' <br>
You also have to consider the differences between a webpage and a printed publication. What will happen with image galleries, sliders, menu’s and the like? If your webpage has a slideshow with ten images at the top, that’s not going to translate well to paper. The most basic level of interaction on the web is a link. This too becomes problematic. On your computer, you can simply click a link to see where it goes, on paper this functionality is lost so you need a good way to take all those inline links and show the reader where they lead. <br>
The biggest difference, and conceptual shift, is that printed documents refer to a page model that is of a fixed size. Whereas on the web we are constantly reminded that we have no idea of the size of the viewport, in print the fixed size of each page has a bearing on everything that we do. Due to this fixed page size, we have to consider our document as a collection of pages, paged media, rather than the continuous media that is a web page.
The biggest difference, and conceptual shift, is that printed documents refer to a page model that is of a fixed size. Whereas on the web we are constantly reminded that we have no idea of the size of the viewport, in print the fixed size of each page has a bearing on everything that we do. Due to this fixed page size, we have to consider our document as a collection of pages, paged media, rather than the continuous media that is a web page.
Paged media introduces concepts that make no sense on the web. For example, you need to be able to generate page numbers, put chapter titles in margins, break content appropriately in order that figures don’t become disassociated from their captions. You might need to create cross-references and footnotes, indexes and tables of content from your document. You could import the document into a desktop publishing package and create all of this by hand, however, the work would then need redoing the next time you update the copy. This is where CSS comes in, whose specifications are designed for use in creating paged media.
Paged media introduces concepts that make no sense on the web. For example, you need to be able to generate page numbers, put chapter titles in margins, break content appropriately in order that figures don’t become disassociated from their captions. You might need to create cross-references and footnotes, indexes and tables of content from your document. You could import the document into a desktop publishing package and create all of this by hand, however, the work would then need redoing the next time you update the copy. This is where CSS comes in, whose specifications are designed for use in creating paged media.


'''Print Preview'''
'''Print preview''' <br>
 
If you want to print your website a print preview window appears. The back-end code in the browser needs to do heavy lifting to support all the requests coming from the front-end.
When you want to print your website a print preview window appears. The back-end code in the browser needs to do heavy lifting to support all the requests coming from the front-end.


The majority of the work in the back-end involves restructuring the HTML to support preview. Once all the platforms are brought in line, the printing pipeline needs to be broken up into two pieces: PDF generation and printing. The user has to select a printer before a print render is generated for a preview. When printing, the renderer generates printing metadata one page at a time, so it can generate them while the PrintJobWorker in the browser process sends them to the printer at the same time.
The majority of the work in the back-end involves restructuring the HTML to support preview. Once all the platforms are brought in line, the printing pipeline needs to be broken up into two pieces: PDF generation and printing. The user has to select a printer before a print render is generated for a preview. When printing, the renderer generates printing metadata one page at a time, so it can generate them while the PrintJobWorker in the browser process sends them to the printer at the same time.


'''From CSS and HTML to print with Prince.'''  <br>
After you prepared your HTML and CSS for the website, and the layout for the book is set with CSS, Prince or another API can be used for translating the HTML language to a PDF. <br>
You can download a free trial of Prince and install it with Terminal. The Terminal runs Prince and uses it to translate the website into a pdf. Prince is not an open source program.


'''From CSS and HTML to print with Prince.'''
'''Wiki to print''' <br>
 
'''Collection and the Offline Content Generator''' <br>
After you prepared your HTML and CSS for the website, and the layout for the book is set with CSS, Prince or another API can be used for translating the HTML language to a PDF.
After doing some research we found out that WikiMedia had already developed an open source plug-in, called Collections created by The Wiki Publisher Project. The extension makes it possible to select pages to create pdf’s or compile books. The user can create different chapters and arrange articles. Then the compiled pages are send to the OCG (Offline Content Generator). The OCG invokes a /Bundler to spider the articles and fetch all images, stylesheets, etc required to render them. One of several /Backends are then invoked to convert the bundle into a PDF, ZIM file, or other appropriate output format. This conversion to a pdf happens on wiki servers. We wanted to use this open source plug-in and adjust the code to fit our needs. (eg layout, page size, content table,…) <br>
 
'''Prince'''
 
You can download a free trial of the ‘Prince’ application and install it with ‘Terminal’. The Terminal runs ‘Prince’ and uses it to translate the website into a pdf. Prince is not an open source program but can be used as a trail. This means the front page of the pdf includes a logo of the prince software.
 
 
'''2. Wiki to print'''
 
'''Collection and the Offline Content Generator'''
After doing some research we found out that WikiMedia had already developed an open source plug-in, called Collections created by The Wiki Publisher Project.The extension makes it possible to select pages to create pdf’s or compile books. The user can create different chapters and arrange articles. Then the compiled pages are send to the OCG (Offline Content Generator). The OCG invokes a /Bundler to spider the articles and fetch all images, stylesheets, etc required to render them. One of several /Backends are then invoked to convert the bundle into a PDF, ZIM file, or other appropriate output format. This convertion to a pdf happens on wiki servers. We wanted to use this open source plug-in and adjust the code to fit our needs. (eg layout, page size, content table,…)  
 
For rendering on your own server (not using the wiki content generator) we installed the Offline Content Generator. This way we were able to change the layout of the output files. (pdf’s)  
For rendering on your own server (not using the wiki content generator) we installed the Offline Content Generator. This way we were able to change the layout of the output files. (pdf’s)  


'''Offline content generator the package:'''
'''Offline content generator the package:'''
* Service
* Service
* Bundler: This tool grabs all the dependencies for a given set of articles and creates a directory or zip file.  
* Bundler: This tool grabs all the dependencies for a given set of articles and creates a directory or zip file.  
* Latexer
* Latexer


'''Latex'''
'''Latex''' <br>
 
Latex is a high-quality typesetting system; it includes features designed for the production of technical and scientific documentation. LaTeX is available as free language.
LaTex is a high-quality typesetting system; it includes features designed for the production of technical and scientific documentation. LaTeX is available as free language.
* Several extensions are available for LaTex called: ‘packages’ or ‘styles’ (better image placement, implementation of mathematic formula,...)
* Several extensions are available for LaTex called: ‘packages’ or ‘styles’ (better image placement, implementation of mathematic formula,...)
* Create a text file in LaTex mark-up, which LaTex reads to produce the final document.
* Create a text file in LaTex mark-up, which LaTex reads to produce the final document.
Line 186: Line 168:
To be able to use the LaTeX mark-up, you have to install LaTeX distribution and an Editor. We used the TexMaker editor for Mac because this editor enables you to see quick previews of the Pdf file.
To be able to use the LaTeX mark-up, you have to install LaTeX distribution and an Editor. We used the TexMaker editor for Mac because this editor enables you to see quick previews of the Pdf file.


'''Bookshelf'''
'''Bookshelf''' <br>
This WikiMedia plug-in allows previous compiled books to be added to the bookshelf. The bookshelves can be used to print previously published editions of a publication by other users. This way we can compile one 'official' documentation book and let other people print it.
 
'''Pandoc'''  <br>
This program converts files from one mark-up language to another. In this case, we used it convert the WikiMedia mark-up to HTML for our website, and to LaTex mark-up for the print publication.


This WikiMedia plug-in allows previous compiled books to be added to the bookshelf. The bookshelves can be used to print previously published editions of a publication by other users. This way we can compile one 'official' documentation book and let other people print it.
'''Mark-up'''  <br>
A mark-up language is a notation used to annotate a document’s content to give information regarding the structure of the text or instructions for how it is to be displayed.


'''Pandoc'''
===Concluding: The Hackers & Designers approach===
Hackers & Designers through evaluating all the options available opted for using a Python script to collect MediaWiki content and then to use Pandoc together with a Latex template to generate the final PDF.


This program converts files from one mark-up language to another. In this case, we used it convert the WikiMedia mark-up to HTML for our website, and to LaTex mark-up for the print publication.
'''Step 1''' <br>
Download repository from: [https://github.com/hackersanddesigners/handd-book ''github.com/hackersanddesigners/handd-book'']


'''Mark-up'''


A mark-up language is a notation used to annotate a document’s content to give information regarding the structure of the text or instructions for how it is to be displayed.
'''Step 2''' <br>
Type into your terminal: ''python get_wiki.py'' <br>
Missing python dependencies can be installed with:


===Hackers & Designers Approach===
    pip install <module_name>


Hackers & Designers through evaluating all the options available opted for using an Python script to collect MediaWiki content and then to use Pandoc in collaboration with a Latex template to generate a final PDF artifact.
'''Step 3''' Type into your terminal the following Pandoc command: <br>
    pandoc --template handd-book.template handd-book.wiki -o handd-book.pdf --toc \
    -M fontsize=12pt -M author='Hackers \& Designers' -M title='About Bugs, Bots and Bytes' \
    --latex-engine xelatex -M mainfont='Helvetica' -M papersize='a5paper' -V links-as-notes


'''Python script:'''
Missing latex dependencies can be installed with:
    # -*- coding: utf-8 -*-
    import json
    import urllib2
    import os
    import re
   
    wikiUrl = 'http://wiki.hackersanddesigners.nl/mediawiki/'
   
    f = open('handd-book.wiki', 'w') 
    pages = [
      'How_to_organize_a_summer_academy',
      ...
      'Credits'
    ]
   
    def get_image(filename):
      if os.path.exists(filename):
        return
      wikiJson = json.load(urllib2.urlopen(wikiUrl + 'api.php?action=query&titles=File:' + filename + '&prop=imageinfo&&iiprop=url&format=json'))
      print wikiJson
      try:
        pages = wikiJson['query']['pages']
        print pages
        for key, value in pages.iteritems():
          url = value['imageinfo'][0]['url']
          print url
          img_res = urllib2.urlopen(url)
          img_file = open(filename, 'wb')
          img_file.write(img_res.read())
          img_file.close()
      except Exception, e:
        print e
   
    for page in pages:
      pageUrl = wikiUrl + 'api.php?action=parse&page=' + page + '&format=json&disableeditsection=true&prop=wikitext|images|links'
      print pageUrl
      wikiJson = json.load(urllib2.urlopen(pageUrl))
      wikistr = &#39;&#39;
      try:
        title = wikiJson['parse']['title'].encode('utf-8').strip()
        print title
        wikistr += '\n\n=' + title + '=\n\n'
      except Exception, e:
        print e
   
      try:
        # Get images - JBG
        imgs = wikiJson['parse']['images']
        for img in imgs:
          img = img.encode('utf-8').strip()
          print ' - ' + img
          get_image(img)
      except Exception, e:
        print e
   
      try:
        wikistr += wikiJson['parse']['wikitext']['*'].encode('utf-8').strip()
        wikistr = re.sub(r'\|\d*(x\d*)?px', '', wikistr) # Remove px info from images - JBG
        wikistr = re.sub(r'{{[A-Za-z0-9#:|/.?= \n&\-\\\”\{\}]*}}', '', wikistr) # Remove youtube links - JBG
   
        f.write(wikistr)
      except Exception, e:
        print e


'''Pandoc command:'''
    tlmgr install <pkgname>


    pandoc --template handd-book.template handd-book.wiki -o handd-book.pdf --toc -M fontsize=12pt -M author='Hackers \& Designers' -M title='About Bugs, Bots and Bytes' --latex-engine xelatex -M mainfont='Helvetica' -M papersize='a5paper' -V links-as-notes


Code is available on [https://github.com/hackersanddesigners/handd-book Github].
[[File:documentation-workflow.jpg|500px|Diagram: How this publication came into being]]


[[Category:DIY Manual]]
===Dos and Don'ts===     
[[Category:WriteMe]]
* Adding links to titles will cause errors   
[[Category:Print]]
* Image galleries will cause errors   
* Using & and other glyphs in titles might cause errors     
* Try to name your images in a meaningful way (no cryptical numbers)       
* Avoid spaces in file names     
* In your wiki mark-up: avoid single ''=…='' to mark-up titles. Start your hierarchy with ''==…==''

Latest revision as of 09:05, 17 December 2015

About the process of making this publication by Vicky De Visser, Anja Groten and James Bryan Graves

During the course of the Summer Academy the participants, tutors and organizers actively documented in various ways. We quickly realized this parallel mode of collective production was developing a life of it's own and introduced an interesting form of exchange between all stakeholders. People were using tools like PiratePad for collective note-taking, a local wireless network created with a raspberry pi to share files, swapped usb sticks, and used git repositories to archive scripts and images. However the variety of documentation methods made the following processing of the documented material confusing.
The proposition of this printed matter is to create a documentation of an on and offline documentation process. In addition the intention is to develop an open and central publishing infrastructure before approaching the next H&D Summer Academy in 2016. In the following you will find some considerations that drove the production of this specific hybrid publication.

Choose an appropriate workflow for your hybrid publishing process

Before starting with your publication, you will need to inventory your documentation material. Subsequently think about possible future occasions that will require documentation.

In the case of the H&D Summer Academy documentation we filtered out 3 crucial moments to contemplate:

Before the summer academy

Although the academy had not even started yet, a lot of content had been generated already. We had acquired research, chose a subject, planned schedules, got the word out by printed and digital media and processed motivation letters of applicants. It would have been helpful for future editions of the summer academy and possible new collaborators to document that process thoroughly.

During the summer academy

During the summer academy workshop materials were provided by the tutors, using different platforms and formats. While currently working with WikiMedia for assembling, writing and editing the documented content we realize this could have been the right platform to collect workshop descriptions beforehand, gather materials used during the workshops, put presentations there and add notes and scripts produced during workshops. As the summer academy progressed we more and more missed a central platform where participants could always go back to in order to catch up, acquire practical information or dump some interesting references. Pictures were taken both by the organization and participants. After the academy had been accomplished we emailed every participant to ask them to send their pictures. Having created this publishing infrastructure we are considering to offer an immediate access to the actual publishing process in the future, which would save time.

After the summer academy

After the summer academy program had been completed we did not wait long to ask the participants and tutors for feedback. To engage people in creating useful feedback you need to make the feedback process as low barrier as possible by for example asking very specific questions, creating forms that are easy to fill out or by interviewing people directly. A wiki could be a convenient platform to process that feedback. See also: Feedback from participants of the HDSA2015

Content managing

In order to compile the documentation material we strived to bundle processes such as co-writing, co-editing and content management optimally all on the same platform. We looked for a back-end that provided real time editing process, that was accessible, provided a comprehensible way of ordering information, had a small learning curve for participants and supported a variety of media files.
The workshops, lectures, screenings, excursions, and the final presentation were documented with media such as videos, sound files, notes, images, and code as well as objects, longer and shorter texts.
Eventually we chose to use WikiMedia, which supports the several file formats and is easy editable by different users. Wikimedia is an easy infrastructure and is well documented. Preparing user guides along the way is beneficial for changing collaborations and teams.
For the first edition of the H&D Summer Academy we didn’t have the opportunity to test WikiMedia as a back-end and content collector for students. We started using it only after the academy to gather the information collected by writing emails and uploading code and pictures to a github and dropbox repositories. During the process of filling the Wiki we did encounter some minor hierarchy and workflow challenges, that we had to get used to as an editorial team.

Content categorizing

To be able to publish a well structured and logical documentation we started sorting content by categorizing the wiki pages we created. We applied a categorization system that was recommended to us, which sorts content by state, media and topic.
The media category is necessary because not all media, for instance video, sound files or links are suitable for a printed publication. Furthermore very long texts are not suitable for the Hackers & Designers website.
It is important to critically sort the content upfront considering the advantages and disadvantages of different media.

H&D Wiki categories structure:
State

  • EditMe
  • WriteMe
  • Published
  • Ready to be published

Media

  • Print
  • Web

Topic / Tags

  • eg Arduino
  • eg HTML 2 Print

Co-writing and editing

After receiving all contributions, the participants and eventually the editorial team must select the most viable material. The challenge of the selection process lies in recognizing that some submissions might not be useful, have another logic or are difficult to implement in the publication for other reasons. The challenge consists to include as much of the content produced as possible without alienating it.
We advise to give the participants a short introduction beforehand in how to collect and document the content. This way the editors prevent an extensive editing process afterwards.
WikiMedia lets users adjust articles and write comments on the changes that were made. Participants have accordingly the possibility to go back in time and compare older versions of the same article. That way content will not get lost after editing an article.

Editorial considerations

After building and filling the wiki with content we revisited all articles created and considered an appropriate translation of the content for a website and a printed publication format. For print we looked into two options: 1. creating a publication from the website and 2. creating a publication directly from the wiki.

Current set-up

CMS wiki.hackersanddesigners.nl
Front-End hackersanddesigners.nl
Print publication wiki.hackersanddesigners.nl > Book sprint 2015

Editorial quest

  • collecting & sorting
  • prepping (i.e. retouching or re-taking foto's)
  • developing an editorial voice (in our case: The format ofHow to manuals are the unifying component for every article)
  • generating content: co-writing and editing
  • publishing (online)
  • printing
  • distributing (publishing offline)

Translation wiki to website

Javascript Requests content from wiki
Mediawiki API Returns wiki content
HTML/CSS Layout of the website

Translation wiki to print

Pandoc Translates wiki markup to LaTeX mark-up
Python Script Initiates several actions
LaTeX Lay-out of wiki content and generating PDF

Publishing online

The wiki content is published on the Hackers & Designers website in real-time. Meaning every time the website is loaded or pages are requested the web browser queries the MediaWiki back-end using the MediaWiki provided API, along with installed wiki semantic extensions and their APIs, through a Javascript client-side web application (AngularJS).

Example API Requests
See also wiki.hackersanddesigners.nl > How to document a summer academy

Request a page:

   curl 'http://wiki.hackersanddesigners.nl/
   mediawiki/api.php?action=parse&
   page=How_to_document_a_summer_academy&
   format=json&disableeditsection=true&
   prop=wikitext|images|links'

Request an image:

   curl 'http://wiki.hackersanddesigners.nl/
   mediawiki/api.php?action=query&
   titles=File:Chicken-and-Potato-Soup.png&
   prop=imageinfo&&iiprop=url&format=json'

The code is available on github.com/hackersanddesigners/handd-web.

Publishing in print

Web to print: HTML to print with CSS and print preview
One way to publish a print publication from an online environment is printing a webpage by using CSS. A user can just click ‘Print Page’, a pop-up print preview will appear and the custom CSS will translate the webpage and media to a printable print publication divided in pages. The design is also made with code/text, which means that you can use text editors such as EtherPad to collaborate on content with several people at the same time. The negative side on using this set-up is that you if you decide to not use a single page website you will have to print each article separate and later bind it into a book. This way no table of contents will be created or there will be no option to apply different lay outing for indexes, special chapters, or glossaries.

Differences between CSS for the web and CSS for print

You also have to consider the differences between a webpage and a printed publication. What will happen with image galleries, sliders, menu’s and the like? If your webpage has a slideshow with ten images at the top, that’s not going to translate well to paper. The most basic level of interaction on the web is a link. This too becomes problematic. On your computer, you can simply click a link to see where it goes, on paper this functionality is lost so you need a good way to take all those inline links and show the reader where they lead.
The biggest difference, and conceptual shift, is that printed documents refer to a page model that is of a fixed size. Whereas on the web we are constantly reminded that we have no idea of the size of the viewport, in print the fixed size of each page has a bearing on everything that we do. Due to this fixed page size, we have to consider our document as a collection of pages, paged media, rather than the continuous media that is a web page. Paged media introduces concepts that make no sense on the web. For example, you need to be able to generate page numbers, put chapter titles in margins, break content appropriately in order that figures don’t become disassociated from their captions. You might need to create cross-references and footnotes, indexes and tables of content from your document. You could import the document into a desktop publishing package and create all of this by hand, however, the work would then need redoing the next time you update the copy. This is where CSS comes in, whose specifications are designed for use in creating paged media.

Print preview
If you want to print your website a print preview window appears. The back-end code in the browser needs to do heavy lifting to support all the requests coming from the front-end.

The majority of the work in the back-end involves restructuring the HTML to support preview. Once all the platforms are brought in line, the printing pipeline needs to be broken up into two pieces: PDF generation and printing. The user has to select a printer before a print render is generated for a preview. When printing, the renderer generates printing metadata one page at a time, so it can generate them while the PrintJobWorker in the browser process sends them to the printer at the same time.

From CSS and HTML to print with Prince.
After you prepared your HTML and CSS for the website, and the layout for the book is set with CSS, Prince or another API can be used for translating the HTML language to a PDF.
You can download a free trial of Prince and install it with Terminal. The Terminal runs Prince and uses it to translate the website into a pdf. Prince is not an open source program.

Wiki to print
Collection and the Offline Content Generator
After doing some research we found out that WikiMedia had already developed an open source plug-in, called Collections created by The Wiki Publisher Project. The extension makes it possible to select pages to create pdf’s or compile books. The user can create different chapters and arrange articles. Then the compiled pages are send to the OCG (Offline Content Generator). The OCG invokes a /Bundler to spider the articles and fetch all images, stylesheets, etc required to render them. One of several /Backends are then invoked to convert the bundle into a PDF, ZIM file, or other appropriate output format. This conversion to a pdf happens on wiki servers. We wanted to use this open source plug-in and adjust the code to fit our needs. (eg layout, page size, content table,…)
For rendering on your own server (not using the wiki content generator) we installed the Offline Content Generator. This way we were able to change the layout of the output files. (pdf’s)

Offline content generator the package:

  • Service
  • Bundler: This tool grabs all the dependencies for a given set of articles and creates a directory or zip file.
  • Latexer

Latex
Latex is a high-quality typesetting system; it includes features designed for the production of technical and scientific documentation. LaTeX is available as free language.

  • Several extensions are available for LaTex called: ‘packages’ or ‘styles’ (better image placement, implementation of mathematic formula,...)
  • Create a text file in LaTex mark-up, which LaTex reads to produce the final document.
  • User needs to know the commands for the LaTex mark-up
  • LaTex automatically adjusts fonts, text size, line heights or text flow.
  • LaTex is not flexible for lay outing, but you’re able to write your own macro’s or download them from others at CTAN
  • pdfTex: engine or pdf compiler. This converts the LaTeX mark-up to a pdf.

To be able to use the LaTeX mark-up, you have to install LaTeX distribution and an Editor. We used the TexMaker editor for Mac because this editor enables you to see quick previews of the Pdf file.

Bookshelf
This WikiMedia plug-in allows previous compiled books to be added to the bookshelf. The bookshelves can be used to print previously published editions of a publication by other users. This way we can compile one 'official' documentation book and let other people print it.

Pandoc
This program converts files from one mark-up language to another. In this case, we used it convert the WikiMedia mark-up to HTML for our website, and to LaTex mark-up for the print publication.

Mark-up
A mark-up language is a notation used to annotate a document’s content to give information regarding the structure of the text or instructions for how it is to be displayed.

Concluding: The Hackers & Designers approach

Hackers & Designers through evaluating all the options available opted for using a Python script to collect MediaWiki content and then to use Pandoc together with a Latex template to generate the final PDF.


Step 1
Download repository from: github.com/hackersanddesigners/handd-book


Step 2
Type into your terminal: python get_wiki.py
Missing python dependencies can be installed with:

   pip install <module_name>

Step 3 Type into your terminal the following Pandoc command:

   pandoc --template handd-book.template handd-book.wiki -o handd-book.pdf --toc \ 
   -M fontsize=12pt -M author='Hackers \& Designers' -M title='About Bugs, Bots and Bytes' \
   --latex-engine xelatex -M mainfont='Helvetica' -M papersize='a5paper' -V links-as-notes

Missing latex dependencies can be installed with:

   tlmgr install <pkgname>


Diagram: How this publication came into being

Dos and Don'ts

  • Adding links to titles will cause errors
  • Image galleries will cause errors
  • Using & and other glyphs in titles might cause errors
  • Try to name your images in a meaningful way (no cryptical numbers)
  • Avoid spaces in file names
  • In your wiki mark-up: avoid single =…= to mark-up titles. Start your hierarchy with ==…==