Blog publishing with AppVeyor

Monday, January 16, 2017 | Posted in AppVeyor Blog Build Deploy nBuildKit

One part of improving my career and increasing my professional reach is to post more to this blog about the work that I am doing and the code and skills that come from this work. Unfortunately at the moment writing is still more of a chore than it is a pleasure so that means that if I want to publish more I need to make it very easy for me to do so, otherwise the publishing process will just add yet another obstacle that I need to get past while producing content.

With me being a build engineer by profession the obvious solution was not to follow the sensible path of a managed system but to create some kind of build system to automatically build and publish my blog posts. Troy Hunt is right in that a managed system is far quicker and easier to use, thereby saving time that you can use for improvements elsewhere. In my case however I feel that there is something additional to be gotten from setting my blog up to be automatically build and deployed. To be specific additional learning experiences, more knowledge and a new blog post about how I achieved the goal. I hope that those things will make up for the extra work I have to do in maintaining my own blog.

So the goals for this project were

  • Build every time I push to the source repository on an independent environment so that I can be sure that I can always build my blog and nothing is dependent on my machine being available.
  • Publishing should involve a minimal amount of work done from my side. Ideally all I would have to do is commit my changes, merge them to the master branch of the source repository and then sit back and see the new post appear online.
  • No maintenance is necessary for what ever system I use to achive this goal. When I build new versions of one of my libraries, applications or infrastructure resources I want full control over the build, test and deployment environments. For building my blog that is far less important because it is a simple process which does not require many different steps.
  • Finally it would be nice to make it interesting enough to get a blog post out of it.

After a few hours of tinkering that is where we are now, this blog post was build and published using a combination of AppVeyor as the build server and nBuildKit for the build and deploy scripts.

In order to achieve this the first step that I took was to set up the build scripts so that I could build on my local machine. Having a build that executes in the same way on a developer machine and the build server has many benefits.

  • Familiarity: developers understand what the build server does because it works exactly as on their machine.
  • Ease of debugging: it is easier to discover the source of any errors if you can execute the scripts on a developers machine because then you have have full access to the workspace which may not be the case on a build machine.
  • Ease of testing: even build scripts need to be tested. If this is done on a developer machine then the testing will most likely be able to be executed much quicker because there is less need to commit every change, push it to the server and wait for the build server to execute the build.

Configuring nBuildKit means copying the sample files to the root of the workspace and updating them with values relevant to the project.

In this case I updated the settings.props file, which contains the general information about the build, test and deploy processes, with product information.

I also enabled the automatic merging for the GitFlow process. This allows nBuildKit to automatically complete feature, hotfix and release branches when running on a build server so that each build also verifies that the final merge will be successful and the changes will not break anything after the merge.

The next file to update was the build.settings.props file, which describes what needs to be done during the build process, was updated. The first part of that was to define which steps should be taken during the build process.

  • Clear the workspace. While this is not necessary on a build server, after all a fresh workspace is used for each build, it is necessary on a developer machine because old artefacts may still be present.
  • Ensure that the build server has configured the workspace with the correct source control information. Many build systems do not perform a full clone at the start of a build but rather get only the minimal source control information in addition to the source files. This reduces time and IO demands, especially for repositories with extensive history. Different build systems have a different approach to what steps exactly are taken. The lack of source control information, e.g. the state of different branches, or the branch that the current commit is part of, is not a problem if the only step is to compiling code however issues can occur if the state of the local workspace is used to determine version numbers or VCS information. In those cases it is better to ensure that the workspace is in the same state as it was on the developers machine.
  • Gather VCS information like the current commit ID and the branch name. For my blog this information is embedded in the footer of every page to indicate which commit was build and what version of the blog you are looking at.
  • Complete the active GitFlow branch, if there is one, by merging the active branch to the appropriate target branches and adding tags if necessary.
  • Get version information with GitVersion. This is done after the merge process so that the version number is the release version.
  • Build the pages with Wyam.
  • Zip up the generated pages

The final change to the build.settings.props file was to enable the generation of the archive files by uncommenting the ArchivePackageSpecs section.

The last file to be changed was the deploy.settings.props file which describes what needs to be done during the deploy process. As with the build.settings.props file the first step here was to select the steps that should be taken during the deploy process.

  • Once again ensure that the build server workspace is configured with the correct source control information.
  • Gather version information in the same way as was done during the build process.
  • Push the archive files to AppVeyor.
  • Push the files contained in the archive file to the repository that contains the generated pages. This was simply done by enabling the DeployStepsPushToGitBranch deploy step and providing information about the repository and branch that the files should be pushed to. The commit message for this process currently comes from a text file in the root of the repository. Ideally there would be some way to discover a decent commit message, e.g. the name of the most recent post, however many different changes are pushed to the source repository so selecting the correct commit message is somewhat difficult.

By updating these three files we have provided nBuildKit with enough information to execute both the build and deploy processes.

Once the build scripts were complete and tested the only thing that remained was to configure AppVeyor. This can either be done through the UI or a YAML file. In this case I elected to use the YAML file so that all information describing my build and deployment process would exist in the repository.

In order to create the configuration file I copied the reference file and then removed all the bits I didn't need. The only tricky bit was setting up the AppVeyor environment so that the deploy process could push the final results back to GitHub. For that I added a deploy key to the target repository. The private part of this key was then encrypted and placed in the YAML file. Some before_deploy script magic allows the decryption of the key so that git can push the changes to the repository via SSH.

Goals for 2017

Friday, January 13, 2017 | Posted in Goals Life

In 2016 set myself a bunch of goals for the year for different areas of my life. Of all the goals I only managed to achieve one successfully but am still pretty happy with how the year turned out. Overall improvements were made and new things were learned so what that concerns it was a successful year.

Given that setting goals did sort of work for me in 2016 going into 2017 I decided to do this again and set some goals in order to actively progress in different areas. This time I also decided to write the goals down somewhere public.

For 2017 my goals are

  • Improve my overall fitness while maintaining my weight at or below 80kg.

    In 2016 my goal was to improve my fitness and lose some weight. Due to a shoulder and elbow injury I actually gained weight in the first half of the year but over the course of the last 6 months I managed to lose about 5kgs thanks to a lot of exercise and an improvement in my eating habits. As a bonus I gained a lot of strength and general endurance. Overall these improvements have made me feel a lot better, have a more defined set of muscles and helped me in my outdoor spot pursuits as well.

    For 2017 the goal is to keep the weight the same, just below 80kgs, while improving my general strength and fitness.

  • Make my career independent from my job

    The last two years I have been working for Vista as a software / build engineer responsible for the development tooling and infrastructure. While this has been a bit of a change from my previous jobs as software developer it has been a great experience. I have learned about many new technologies, created some awesome tooling and have many more things still to learn and create. However with this change from general developer to more of an internal tooling / infrastructure person it has also become clear to me that my technical capabilities will improve more if I make my toughts, opinions and code available for public scrutiny by the larger development community. I already have a number of open source projects but for 2017 I also plan to write and potentially speak more about the things I'm working on. One part of that is re-re-activating my blog and writing new posts about some of the cool technologies I'm currently working with and on.

  • Simplify life and my surroundings.

    The first few years after I moved to New Zealand I didn't have very much in the way of material possessions, after all I had just moved to the other side of the world with only the things I could take on the airplane. Additionally I moved almost every year to a new house so getting rid of useless possessions meant I had to move less stuff from the old house to the new one. Over time though I have acquired more things both materially and in habits. Some of those things are highly prized however some things are a waste of time and space. The goal for 2017 is to minimize the amount of useless stuff I have, like pieces of furniture which are never used, and useless things I do so that there is more time and space to spend on things that are useful and improve my life.

  • Improve my confidence, in general in life but especially for climbing and paragliding

    In general in life having confidence in yourself and your skills is important. For my outdoor activities, climbing and paragliding, it is even more important. Without confidence in yourself and your skils it is not possible to perform well in either activity. Due to injuries and some intense experiences my confidence in either my climbing or flying skills isn't very high at the moment so 2017 is the year that I will do my utmost best to improve my confidence in my professional skills, my climbing and paragliding capabilities and life in general.

  • Sports / achievment goals

    The final goals for 2017, at least as far as this list goes, is to actively improve my skils in climbing and flying.

    For a while I have been trying to improve my climbing skills, strength and endurance enough to be able to climb a grade 24 sports climb. So for 2017 that will be my goal, to improve enoguh to finally be able to climb a 24. Additionally over the last few years I have been doing more trad climbing I am also going to try to improve my skills in that area by setting myself a goal of climbing a grade 20 on trad.

    In my other outdoor activity, paragliding, the general goals are related to the distance your can fly cross country (XC). As I have so far not managed to get away from launch at all my goal for 2017 is to do an XC flight of more than 5km.

Exit Docpad, enter Wyam

Wednesday, January 11, 2017 | Posted in Blog

After not having done anything with my blog for over a year I have returned and will once again try to make this blogging thing happen. Part of my goals for 2017 is to start advancing my career independent from my job. And one part of that goal is to blog about all the things I'm learning.

For about a year and a half Docpad has been very helpful in adding content to my blog, however as I was trying to automate my blog more and more I came to the conclusion that being on a Windows operating system doesn't make dealing with node and NPM any easier. So I figured it might make sense to switch to a static generator which uses technologies that play better on Windows. Enter Wyam, a static generator written in .NET, a technology stack I'm very familiar with.

Getting started

To get started with Wyam I downloaded the latest version and generated a skeleton directory structure by running the wyam.exe new --recipe Blog command. Once the skeleton has been created you can create the pages by running wyam.exe -r Blog -t CleanBlog, and to create and be able to view the pages you can execute wyam -r Blog -t CleanBlog -p. These commands assume that you store your blog source in the input directory and that the website pages should be placed in the output directory. This however is where the build engineer in me started making disapproving sounds so I changed the input and output folders by specifying the appropriate command line parameters to make it happy again

wyam.exe --input src --output build\site -r Blog -t CleanBlog -p

Good that is much more inline with the standard paths used by nBuildKit. Another change that needs to be made is where Wyam stores the NuGet packages. Normally Wyam stores those in a global location, however I strongly feel that builds should always only write to their local workspaces, so lets redirect the packages to a folder that is local to the workspaces

wyam.exe --input src --output build\site --use-local-packages --packages-path packages -r Blog -t CleanBlog -p

That command line is getting longer but we can make it shorter again by moving some of the standard items into the Wyam configuration file, so I created a config.wyam file in the root of the workspace and added the following lines

// Preprocessor directives
#recipe Blog
#theme CleanBlog

And with this the command line to generate the website and start a local webserver is

wyam.exe --input src --output build\site --use-local-packages --packages-path packages

Moving the existing content

Moving the existing content from the Docpad generated site to the Wyam generated site required a bit of work although the initial results were quite easy to achieve. I started off moving the post files to the posts directory and updating all the files to be named so that Wyam is able to extract the date and time from the file name. In order for that to work Wyam wants the file name to follow the naming convention <YEAR>-<MONTH>-<DAY>-<TITLE>.md.

The next step was to update the front matter in the post files. Wyam would like to see the following format:

Title: <POST_TITLE_GOES_HERE>
Tags:
  - <TAG_1>
  - <TAG_2>

With this done Wyam can generate a website with blog posts with the correct titles and dates. This is a good start. Next are the non-post pages, like the home page and the project pages. These can simply be dropped in the desired directory. Wyam will automatically generate HTML pages from any Markdown file and simply copy the other files (like the favicon etc.). Any non-blog post pages will show up in the menu bar so for all pages that should not show up in the menu bar you can add the ShowInNavBar: false entry in the page front matter like so:

Title: My page that shouldn't show up in the NavBar
ShowInNavBar: false
---
# Content

Once that was done I created overrides for

  • The page and post footers by creating the _Footer.cshml and _PostFooter.cshtml files. The page footer was straight forward but the post footer required some additional work to get the links to the previous and next post to function correctly. Fortunately Dave Glick was very helpful on the gitter page for Wyam and pointed out that there are collections that you can iterate over to find out what the previous and next posts are.
  • The sidebar which contains my little ‘about me’ blurp.
  • The navbar which by default orders pages alphabetically and does not contain a link to the home page. In my Docpad blog both those things are not true, so I did a bit of Razor coding and generated the navbar items myself

These changes got most of the pages looking pretty close to what they should be, but yet not quite the same as it was.

Tweaking

If you don't have an existing layout for your blog or you don't mind changing the layout then after the previous steps you should have a very nice looking blog that works just fine. However it turns out I'm stuborn and specific in the way I want things to look. In this case I wanted my blog to not look any different than the existing one does, at least for the time being. So that meant I had to do a fair amount of tweaking.

The first major tweak was to get the sidebar looking like it should. This turned out to be a little harder than I expected because it required overriding the _Layout.cshtml page. While this is a bigger step than the other overrides it also meant I was able to make the pages look exactly like I wanted and I was able to re-use my CSS files.

One of the more tricky parts was to generate a list of the last 10 posts. Originally I was using the list of posts, however because this is the layout page that is used for all pages it might well be that some pages are generated before the full post list is available (say hello to the tags pages). Once again Dave Glick came through and pointed me to the RawPosts collection which is generated right at the beginning of the cycle.

Once the issues with the sidebar were resolved the next step was to show the last five blog posts on the home page. By default Wyam will place excepts on the home page with a link to the actual post in the post title. However what I wanted was the full post, excluding the comment section. Once again the RawPosts collection came to the rescue which allowed me to iterate over the last five posts and get their pre-rendered content.

Some additional hacking also allowed me to reuse the _PostHeader.cshtml file for both the posts on the home page, for which the title links to the actual post page, and for the post page in which the title is just a header.

With these changes the blog looks (almost) exactly like it used to look except that it is now being generated by Wyam. All in all I have to say I'm pretty happy with the result and with how easy it was to move from Docpad to Wyam. And as a bonus I understand much more of how Wyam works and how my own blog is generated.

nBuildKit release - V0.5.1

Saturday, May 23, 2015 | Posted in nBuildKit

Version V0.5.1 of the nBuildKit build library has been released.

This release adds the following features

  • 91: MsBuild.Projects: Common.props should do a more extensive search for settings.props
  • 90: MsBuild: Allow pushing symbols via NuGet enhancement
  • 89: MsBuild: Allow assembly versioning to be more custom defined.
  • 87: MsBuild: Rename template files to have normal extension
  • 86: MsBuild: Rename the DirMsBuildShared, DirMsBuildExtensions and DirMsBuildTemplates properties enhancement
  • 85: Unable to build projects which have a mixture of target frameworks
  • 83: MsBuild: Allow PreCleanScripts and PreCompilationScripts to define their own properties
  • 82: MsBuild: Update the Sample custom version files to match the current version gathering capabilities
  • 80: MsBuild: Combine top-level build.msbuild, deploy.msbuild and shared.locatenbuildkit.msbuild
  • 78: MsBuild: Automatically include NuGet dependencies if nuspec is in same directory as project
  • 72: MsBuild: Define default directory paths only in settings.props file
  • 68: Improve inline documentation of settings.props file
  • 66: MsBuild: Provide initialization/installation scripts for an empty workspace
  • 55: MsBuild: Allow assembly copyright to be more custom defined
  • 22: MsBuild: Allow each file to have their own base directory when creating ZIP archives
  • 19: MsBuild.Projects: Provide scripts for Visual Basic projects

And fixes the following issues:

  • 92: MsBuild.Projects: GetVersion, VcsInfo and Generate files also run when they're disabled in the settings.props file
  • 88: MsBuild.Projects: Excessive NuGet restore of global packages
  • 84: Not compatible with ncrunch
  • 54: MsBuild: First space trimmed from release note content

Finally the following tasks were completed:

  • 7: MsBuild: Set up tests

nBuildKit release - V0.4.1

Sunday, November 30, 2014 | Posted in nBuildKit

Version V0.4.1 of the nBuildKit build library has been released.

This release introduces the ability to mix custom build steps with the default provided ones in any order that is desired. On top of that additional changes were made that now make it possible to build multiple Visual Studio solutions in a specific order.

Finally the user can provide one or more scripts that will be called if the build or the deploy process fails. These scripts can be used to clean-up resources or provide notification on build failure. Do note that any failure in these scripts will lead to the immediate termination of the process.