• Spark for ESP8266 Wifi Display

    Last year I created a simple ESP8266 based Wifi display using a Wemos D1 Mini and a 4x20 character LCD - Video 1 - Video 2. I used a php script for the backend because I wanted it show the time without having to add an RTC, and I didn’t want to format messages on the board. The php script will generate a preformatted message for the board, so it just has to split out the 4 separate lines. This was great, but I want to change the messages a little bit easier.

    ESP8266 WiFi Display

    Instead of using php this time, I went with Java which I’m much more familiar with, and will run the server on a Raspberry Pi. I have also set the wifi board to fallback to the old php script anytime the new Java server is unavailable. Since I’m using a Pi, it will be up and down a lot as I work on other projects, and don’t want the time on the display to be too far off. The message will change when it falls back, but I don’t mind.

    Change message page

    Spark

    I created a Java app using Spark, which is a Java web app micro framework based on statics and Lambdas. It starts up super fast, typically only a couple hundred millis, and is very lightweight. This works great on something like a Pi or shared environments where resources are limited.

    Spark Helloworld

    import static spark.Spark.*;
    
    public class HelloWorld {
        public static void main(String[] args) {
            get("/hello", (req, res) -> "Hello World");
        }
    }
    

    Thymeleaf

    Spark also works with template engines so in addition to services, you can generate html pages. I used Thymeleaf to create a basic message editing page that will display the current message.

    To send a Thymeleaf template response with Spark, create a Thymeleaf TemplateEngine and return the result of the engine.render method in your response:

    private static final TemplateEngine thymeleaf = new ThymeleafTemplateEngine();
    ...
    Spark.get("/yourPath", (req, res) -> {
      Map<String, Object> model = new HashMap<>();
      model.put("templateVariableName", value);
      model.put("variable2", message2);
      return thymeleaf.render(
          new ModelAndView(model, "templateFileName")
      );
    });
    

    build.gradle Dependencies

    compile group: 'com.sparkjava', name: 'spark-core', version: '2.6.+'
    compile group: 'com.sparkjava', name: 'spark-template-thymeleaf', version: '2.5.5'
    

    Basic Auth

    I didn’t want just anyone to be able to change the message, so I added an HTTP Basic Auth filter using QMetrics Spark Authentication library to require a username and password. This was the easiest route to get a minimal level of security, but please note that basic auth is not safe for plain text http connections and should only be used over https. Basic auth actually sends the username and password as a Base64 encoded header string, so without https anyone can decode them. In this case though, there is minimal risk.

    Spark.before(new BasicAuthenticationFilter(
    		"/*", new AuthenticationDetails(user, pass)));
    

    build.gradle Dependencies

    compile 'com.qmetric:spark-authentication:1.4'
    

    Fatjar

    I wanted this app to be easy to run. Spark uses Jetty as an embedded http server, so it starts fast and there are only library dependencies, with nothing extra to install. A standard build will generate one jar with your code in it, but all the other libraries still need to be packaged together somehow. In the past, these would have been bundled together in a folder inside of a zip file, but a better approach is to use a fat jar.

    Gradle helps make this easy. You need to create a new build task to tell Gradle to include all the dependencies in the built jar, and to set the main class in the Manifest to make it a runnable jar. Now there is just one self contained jar to distribute which could be run with java -jar your-fat-jar.jar

    Gradle fatJar task

    task fatJar(type: Jar) {
    	manifest {
            attributes 'Implementation-Title': 'ESP8266 Wifi Display',
              'Implementation-Version': version,
              'Main-Class': 'com.mikelduke.java.wifidisplay.WifiDisplay'
        }
        baseName = 'spark-wifi-display-all'
        from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
        with jar
    }
    

    Gradle Build Command

    ./gradlew fatJar

    Run it

    I created a shell script to start the spark-wifi-display on my pi which can be run on startup. This script starts the java app in the background so it doesn’t need a user session and sets the property the app uses to set what port the server runs on.

    #!/bin/bash
    
    nohup java -Dserver.port=8081 -jar git/ESP8266WifiDisplay/spark-wifi-display/build/libs/spark-wifi-display-all-1.0.0.jar &
    

    Run on Heroku

    The spark-wifi-display app can even be deployed and run on Heroku. It’s probably not a good choice for a free dyno though, because the wifi module will hit it repeatedly. Heroku will sleep (turn off) free dynos for 6 hours a day, so it won’t be always on and will eat up the free pool of hours doing something which is fairly pointless, but it is possible.

    To enable running on Heroku, I had to add a stage task to the gradle build and create a Procfile to hold the run command.

    Gradle Build Task

    task stage(dependsOn: ['build', 'clean'])
    build.mustRunAfter clean
    stage.dependsOn(fatJar)
    

    Procfile

    web: java -Dsecured=true -Dserver.port=$PORT -jar spark-wifi-display/build/libs/spark-wifi-display-all-1.0.0.jar
    

    The secured property is set to true since this will be available on the internet after it’s deployed. The username and password can be set in the Heroku settings page as environment variables.

    Use it

    The edit message page is on the root ‘/’ for example http://your-ip:8080/ and the message formatted for the display is on ‘/message’ or as an example http://192.168.1.1:8080/message


  • Converting to Jekyll from Wordpress

    Converting from Wordpress to Jekyll was pretty easy. My wordpress was self hosted, so I had access to the database info I needed to export my old site.

    Export

    I followed the Jekyll guide where you run the ruby jekyll-importer with your database info. You might need to update ruby and install the jekyll-import gem and some other dependencies, and may need sudo for some environments.

    apt-get install ruby-full
    gem install jekyll
    gem install jekyll-import
    gem install mysql2
    gem install bundler
    
    ruby -rubygems -e 'require "jekyll-import";
        JekyllImport::Importers::WordPress.run({
          "dbname"   => "",
          "user"     => "",
          "password" => "",
          "host"     => "localhost",
          "port"     => "3306",
          "socket"   => "",
          "table_prefix"   => "wp_",
          "site_prefix"    => "",
          "clean_entities" => true,
          "comments"       => true,
          "categories"     => true,
          "tags"           => true,
          "more_excerpt"   => true,
          "more_anchor"    => true,
          "extension"      => "html",
          "status"         => ["publish"]
        })'
    

    I couldn’t get this to run from the Dreamhost shell, but I had a Raspberry Pi I could run it on instead, since I didn’t want to have to run through the Windows Jekyll install process. Now I use a Docker container to build on my Windows machine instead. No install needed.

    After exporting, I was able to build the site using all the default settings and had a folder called _posts

    The wordpress posts all exported as html files containing the html from each post. Some of these might need additional jekyll plugins or include files to display properly, or some manual tweeking. In my case, a lot of the html links didn’t format correctly, and the youtube embeds didn’t display.

    cd export-folder
    jekyll new .
    bundle exec jekyll build
    

    This should create a _site folder in the current directory with the build site. Now start looking for new themes and customizing the site.

    Customize

    The first thing I did was to configure some of the settings in _config.yaml like title, email, and baseurl. I also deleted a lot of the stub pages that were exported from Wordpress. Those were just there to hold the links on my title bar, but jekyll made them into empty pages.

    After that, look for a theme or customize one. I didn’t see any premade ones I really liked so I just made some tweaks to the default minima theme. To modify the current theme, you copy the files you want to override from theme’s install to your project in a corresponding folder. As an example, Minmina has _includes/header.html, if you have your own _includes/header.html it will be used instead.

    Youtube Embeds

    One thing that was missing was the youtube video embeds I had. Using wordpress you only had to include a link to the video on it’s own line and it would automatically add the html required.

    Adding extra html for use in the markdown files is straight forward. I added an include file _includes/youtubeEmbed.html with the youtbe html inside:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/" frameborder="0" allowfullscreen></iframe>
    

    And can include it in a post like this:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/VcFI91r2zU4" frameborder="0" allowfullscreen></iframe>
    
    

    Use Docker for builds

    If you are on Linux/macos this might not be needed, since ruby installs are pretty easy.

    1. Install Docker for your os
    2. Share the windows folder path with your site to VirtualBox if needed
    3. Run the Docker build
      docker run -it --volume PATH_TO_PROJECT:/srv/jekyll jekyll/builder jekyll build
      
    4. If on linux/mac and not using a VM, you can cd to the jekyll folder and use $PWD instead
      docker run -it --volume $PWD:/srv/jekyll jekyll/builder jekyll build
      

    This will generate the _site, now you can either scp/ftp it to a webhost or run it locally from any http server.

    If you want to run the site from the Docker image using the jekyll server:

    docker run -it --volume /git/jekyll-temp:/srv/jekyll -p 4000:4000 --expose 4000 jekyll/builder sh -c "bundle install && bundle exec jekyll serve --host=0.0.0.0"
    

    If using VirtualBox, remember to enable port forwarding for port 4000 VirtualBox Port-Forward

    Github pages

    If you dont want to have to build anything, you can just push the git repo to git hub with a name like username.github.io and they will autodetect and build the site out for you at https://username.github.io

    It is nice to build locally first to preview the changes. Having a build setup also lets you deploy the site to any other webhost too.


  • Docker Jekyll Builder

    I created a new project on Githib docker-jekyll-builder-git for a Docker containerized Jekyll project builder that I can run on a Heroku free Dyno. I put it together for use as part of a publishing pipeline instead of relying on the Github builder. This way I can use whatever plugins I want and can scp the built _site to my existing webhost. This lets me use my existing content and avoids having to change the DNS for my domain name.

    Most of the other Jekyll builders I’ve found for Heroku rely on using either Rack or the Jekyll embedded server for serving content. Neither of these are really neccessary since it is just static content and any http server should do. You could instead use Apache or NGINX.

    Running a static blog off an app dyno seems like a waste of resources unless you really need to scale it out dynamically. The content could just as easily be hosted on any other webhost or combined with other static sites. You would also need to upgrade to at least a hobby dyno ($7/mo) to avoid dyno sleeping for 6hrs/day, and the costs will add up with several sites.

    I haven’t pushed the Docker image to Docker hub yet, but will once I’m happy with it and make things a little more flexible.

    The image requires several environment variables and these can all be configured through Heroku’s web ui. Basically it will check out a git repo, run a bundle install and jekyll build, the scp the result somewhere else. You need to set the URLs to use and and SSH key for the transfers.

    It can run as a bare docker image for use locally or on Heroku, instructions are in the readme or below. Next step is I want to make it run automatically after a push to github. I’ll make a php script on my webhost that can be hit from a webhook on Github that will then make the rest api call to Heroku and trigger a build.


    jekyll-builder-git

    Docker image to checkout a git repo, run a jekyll build, and the SCP the generated _site folder to somewhere else.

    This lets you use a short lived free Heroku one-off dyno to build a Jekyll app that is hosted somewhere else.

    Uses docker image jekyll/builder from https://github.com/jekyll/docker

    Environment Variables

    Set these variables in a run command or elsewhere:

    • GIT_HOST - Hostname for git repo
    • GIT_REPO - Path to git repo
    • SCP_HOST - Hostname for SCP destination
    • SCP_DEST - SCP Copy Destination
    • KEY - Private key for Git over SSH and SCP

    Heroku

    • Create a new app heroku create jekyll-builder
    • Push the Docker Image heroku container:push builder --app jekyll-builder
    • Create the Environment variables above in the Heroku admin console
    • Run the builder heroku run -a jekyll-builder --type builder bash build.sh

    Heroku Trigger Script

    The script run-heroku-jekyll-build.sh is an example Heroku API curl request to trigger the jekyll builder.

    • Set your app name in the APP_ID_OR_NAME variable
    • Get a Heroku API Token by running heroku auth:token
    • Set TOKEN= to the token
    • Run ./run-heroku-jekyll-build.sh to trigger a build.
    • Logs can be viewed from the Heroku Dev console log viewer. Make sure you have it up when running the build, the logs are not saved.

    Docker

    • Docker Build docker build -t jekyll-builder-git .
    • Sample Docker Run Command
      docker run \
      --rm \
      -it \
      -e "GIT_HOST=github.com" \
      -e "SCP_HOST=yoursite.com" \
      -e "GIT_REPO=https://something" \
      -e "SCP_DEST=you@yoursite.com:/path/to/dest/" \
      -e "KEY=-----BEGIN RSA PRIVATE KEY-----
      1234567890
      -----END RSA PRIVATE KEY-----"  \
      jekyll-builder-git
      

  • New site running with jekyll

    I decided to get rid of the old wordpress backed site and convert to a new one using Jekyll. The new page should be faster since it will be statically built instead of dynamic.

    Jekyll is a Ruby based static website builder. Instead of requiring a database to store content and retrieving it on every page load, Jekyll uses templates and markdown to let you build a site that is static so it requires no database and uses much less resources. Github pages use Jekll for projects hosted on their site so there are lots of resources available.


  • ESP8266 WiFiDisplay Pt2 - Panel Design and Final Build


subscribe via RSS