Convert WriteFreely posts to Hugo blog - adding tags with a script

Mission:
I’m interested in migrating all my WriteFreely posts to Hugo but have hit a wall with dealing with tags.

  1. I exported a CSV file from WriteFreely. (CSV file data: id,slug,blog,url,created,title,body)
  2. Wrote very basic bash script to convert CSV file to .MD files.
#! /bin/bash
#Script to convert WriteFreely CSV to Hugo MD files

#WriteFreely CSV columns
#id,slug,blog,url,created,title, body (column1-8)

while IFS="," read -r column1 column2 column3 column4 column5 column6 column7
do
  echo "---" | tee "${column2}".md
  echo "title: \"${column6}\"" | tee -a "${column2}".md
  echo "date: $column5" | tee -a "${column2}".md
  echo "draft: false" | tee -a "${column2}".md
  echo "---" | tee -a "${column2}".md
  echo  " "  | tee -a "${column2}".md
  echo -e "$column7" | tee -a "${column2}".md

done < <(tail -n +2 data.csv)

This works to create .MD files for posts in Hugo. The only thing that is missing are the tags.

This is an example of a file that the script created.

---
title: "The Watermelon Boys by Ruqaya Izzidien"
date: 2019-02-19T12:17:29Z
draft: false
---
 
"Book Review

>The present is an arrogant time in which to live, always has been. Humans of the present look back at their people, land, and history, and whisper to themselves with glee,
[more review here]

My Rating: **★★★★★**
My 2019 Reading Challenge: Book #11

#TranslatedFiction #BookReview"

As you see there are tags at the end of each post #tag
They need to be inserted in the frontmatter for each post.

The top needs to be like this:

---
title: "The Watermelon Boys by Ruqaya Izzidien"
date: 2019-02-19T12:17:29Z
tags: ["TranslatedFiction", "BookReview"]
draft: false
---

Goal:
I’d like to automate adding the tags with another script, or modify my original script.

Any suggestions on the best/quickest/easiest way to do this? I have 300+ posts I need to do this for.

Thanks for reading!

2 Likes

Also if you do use WriteFreely, I learned something important while trying to do the conversion from CSV to .MD

Always start your posts in Writefreely with # TITLE. If you don’t, there will be no title in the exported CSV.

I had almost all my posts started with ## TITLE, and had to fix the CSV in LibreCalc.

Here is my strategy to put the tags in:

  • modify above script to add a dummy line in the frontmatter
    tag: [ XXXX88 ]

  • use LibreCalc formula to split body of CSV from the right on new line (\n) and just grab the last line containing tags (#tag #tag2 #tag3)

  • create CSV file with two columns (slug, tags)

  • create new script:
    – load CSV
    – put column2 (tags) into an array variable, format it to be: “tag”, “tag2”, “tag3”
    – use sed(?) to replace XXXX88 in the $slug.md with the array data

Now I just need to figure out the code for all this, or if there is an easier way, please let me know :smiley:

Ok final script. Hope it helps others who are moving from WriteFreely to Hugo.

  1. convertWritefreely: converts the Writefreely CSV to Hugo MD files. Make sure you clean up the CSV file in a spreadsheet program first.
#! /bin/bash
#Script to convert WriteFreely CSV to Hugo MD files

#WriteFreely CSV columns
#id,slug,blog,url,created,title, body (column1-8)

while IFS="," read -r column1 column2 column3 column4 column5 column6 column7
do
  echo "---" | tee "${column2}".md
  echo "title: \"${column6}\"" | tee -a ${column2}.md
  echo "date: $column5" | tee -a "${column2}".md
  echo "tags: [XXXXXX]" | tee -a "${column2}".md
  echo "categories: [\"blog import\"]" | tee -a "${column2}".md
  echo "draft: false" | tee -a "${column2}".md
  echo "---" | tee -a "${column2}".md
  echo  " "  | tee -a "${column2}".md
  echo -e "$column7" | tee -a "${column2}".md

done < <(tail -n +2 data_full_body.csv)
  1. convertTags: grabs ‘tags’ from end of WriteFreely CSV and replaces the placeholder in the *.MD files. Again make sure you clean up CSV file before running.
#! /bin/bash
#Script to convert WriteFreely CSV tags and past into the Hugo MD files

#tags CSV columns
#id,slug,tags,body (column1-4)
newTags="temp"

#Read in the data form the CSV file
while IFS="," read -r column1 column2 column3 column4
do
  echo "starting tag data: ${column3}"
  #Replace # from tags with sed
  newTags=$(echo "$column3" | sed -e "s/n#//" | sed -e "s/\(#\)\1*//g")
  echo -e "finish tag data: ${newTags}"

  read -a tagArray <<< $newTags
  echo "tag array start: ${tagArray[@]}"

#Loop through array and add quotes
quotes="\""
#for ((i=0; i<"$length+1"; i++))
#    do
#      tagArray[i]=${quotes}${tagArray[i]}${quotes}
#    done

tagsFixed=()
count=0
for i in "${tagArray[@]}"
do
    echo $i
    tagArray[$i]=${quotes}${i}${quotes}   
    echo "${tagArray[$i]}"
    tagsFixed[$count]=${tagArray[$i]}
    count=$(($count+1))
done

  echo "tag array end: ${tagsFixed[@]}"
  echo ""
  echo "here will write these tags into the Hugo.md files"

for (( i = 0 ; i < ${#tagsFixed[@]} ; i++ )) do  tagsFixed[$i]=${tagsFixed[$i]}","; done

tagsOutput=${tagsFixed[@]%}

  echo "${tagsOutput}"
  sed -i "s:XXXXXX:$tagsOutput:" $column2.md #column2 = slug
  echo ""
done < <(tail -n +2 data_full_tags.csv)

You will have to modify these to work for your data.
There isn’t any error checking.
Also script fails with non-English characters.
Use at your own risk.

If you modify this or improve it, please share with the community, or even make a Github repo :smiley:

EDIT:
there is some problem when Hugo tries to parse the files, there is

title: "“blah”

use sed to remove

find -type f -exec sed -i -e 's|title: ""|title: "|g' {} \;

also for some reason all the # signs in the posts are //

use sed to remove

find -type f -exec sed -i -e 's|//|#|g' {} \;

or improve the script :smiley:

2 Likes

I see that you have moved to hugo. Happy that you found a good software for your blog.
You may write an article “hugo for beginners” :smirk:, it’ll be great.
You can also add your tuto to the hugo website

1 Like

Well I can share how I got this running on Yunohost.

  1. Create your website with hugo. Test is with hugo server.
  2. Install Custom Web app
  3. Use Rsync to upload the /public/ directory of your hugo site into the /www/ folder of my_webapp. Should be something like “/var/www/APP/www”

If you make changes to your website on your computer, parse the pages with hugo again, then rsync the /public/ folder to Yunohost again.

EDIT:
Made a more formal “tutorial” and put on my blog

1 Like

Great tutorial.
I liked the idea of the cron job, but I was just wondering if it would be possible to have more than one backup (filename"date".tar) as it does not take a lot of space and having multiple backups may be helpful one day.

Another thing about hugo is organising the posts. Categories, tags, folder structure. I am still reading about it so I don’t screw it up.
There is also the comments integration and moderation, but I didn’t look for the best option. And maybe the analytics.

True!

#!/bin/bash
now=$(date +"%F")
filename=arkadicloud_$now

echo "* ARKADI-CLOUD BLOG *"
echo "** DELETE BACKUP **"
find /home/yunohost.backup/archives/ -type f -iname 'arkadicloud_*' -mtime +7 -delete
echo "*** MAKE BACKUP ***"
sudo tar -cf /home/yunohost.backup/archives/$filename.tar /var/www/my_webapp/www/
echo "**** SYNC PAGES TO ARKADI-ONE WEB SERVER ****"
sudo rsync -ah --delete /home/yunohost.app/nextcloud/data/elias/files/Documents/arkadicloud/public/ /var/www/my_webapp/www/
echo "** FINISHED! **"

I was also almost paralyzed by the overwhelming choice of ways to organize my content in the beginning. Just learn about one thing at a time or think of a specific goal you want to achieve. Then, copy your whole website to another directory and try things out! Hugo server will immediately tell you if something is wrong.

I disabled comments in my config.yaml. I don’t like Discuss and all that jazz.

Categories are built in just add something like this to front matter:
categories: ["cool posts", "not cool posts"]
Then if your theme has somewhere to display this, it will.

For folder structure I think you want to be looking at archetypes. I am not too clear on how that works either. I have a simple blog though, I don’t even use pictures! That may change now once I figure out how to use them in hugo.

1 Like