Havenga.dev

Strange Bundler sub-dependency source selection

2022-01-03T16:38:00+02:00

Bundler implicit sub-dependencies prefer to pull from their highest declared non-default source, and it’s an absolute pain…

Context

If you’ve done any heavy enterprise Ruby work with shared features across multiple systems, logic that multiple systems need to be aware of simultaneously, or alternatively built some kind of internal cli tool in ruby, then you’ve probably built a gem in the past. There’s a further chance that you might not have been ready to opensource that particular gem for some reason, and thus opted to host it on some form of alternative gem registry. Gem Fury, Artifactory, or a simple gem server probably.

Around the beginning of 2021 it was realized that Bundler was vulnerable to something referred to as dependency confusion. While I won’t go too far into depth about about this attack, the TL:DR version is that when declaring a Gemfile, in the past Bundler would prioritize gems found on the default source rubygems.org. If you then added an additional source for some kind of internal gem, someone could release a opensource gem using the same name as your internal gem, which Bundler would then install instead of pulling your gem from your specified source.

While there were protections in place on rubygems.org to prevent malicious gems from being uploaded, this was still a hazardous outcome from the perspective of being, at worst as possible RCE, at best, still a rather massive nuisance.

So Bundler was patched to make it prefer the explicitly declared gem source for a particular gem over the default gem source. Great! In almost all cases, this is probably exactly the behaviour you want. But, it comes with a caveat…

The Problem

Bundler is a magnificent piece of software for helping ensure versions are locked at their intended versions, matched according to their dependency version specifications, and pulled from their effective sources, all without needing to be needlessly verbose about every single little thing that your app needs to run. If you’ve ever taken a peek at the Gemfile.lock file in a fresh rails app you might have been surprised to find close to or even over 100 dependencies, despite the default Gemfile declaring 15 or so. The reason for this is when a gem is created, a gemspec file is written up containing gem dependencies and versions for those dependencies that the gem should be able to work with. This is the information which Bundler combines with the declared version constraints within your project’s Gemfile in order to grab any dependencies for the gems you declare to make sure they work as intended, without you needing to explicitly declare the sub-dependencies of your dependencies yourself in your Gemfile. Brilliant right?

Now, here’s the question. If I declare a gem in my Gemfile, and configure it to come from a non-standard source, where do you expect that gem’s dependencies to come from?

source "https://rubygems.org"

gem "rspec"
gem "work-logic", source: "https://internal-gem-repo.awesome"

Intuitively, you might have guessed that the logical source of a gem’s dependencies would the same place as the gem itself right? So if I have a gem called, for example, work-logic, declared to come from https://internal-gem-repo.awesome, we would expect any dependencies of work-logic to similarly be sourced from https://internal-gem-repo.awesome. Makes perfect sense.

rspec from rubygems, work-logic from internal gem repo.

Now, what if one of the dependencies of my internal work-logic gem is a public gem? Perhaps the work-logic gem is actually a Rails engine, and thereby actually depends on rails.

Rails as a dependency of work-logic

Well, with the above behaviour, because work-logic is sourced from https://internal-gem-repo.awesome, and rails is a dependency of work-logic, bundler is going to try source rails from https://internal-gem-repo.awesome. And if you couldn’t be bothered to mirror rails from your gem server, then it’s going to fail.

“So just declare the rails source as rubygems.com explictly then…” you’re probably thinking.

source "https://rubygems.org"

gem "rspec"
gem "rails"
gem "work-logic", source: "https://internal-gem-repo.awesome"

Rails as a dependency of work-logic, explicitly sourced from rubygems.org

And that does indeed work, rails is then explicitly sourced from rubygems.org, work-logic is explicitly sourced from https://internal-gem-repo.awesome, and all sounds swell until we hit the next reason bundler fails. Rails has dependencies. Lots of them. So, in a situation where work-logic depends on rails but is sourced from https://internal-gem-repo.awesome, and rails, explicitly requested from rubygems.org, has all of it’s own implicit dependencies like active-record; where would you expect rails’s dependencies to be sourced from? Intuitively, one would probably say “well, those are rails dependencies, so those should come from rubygems.org.” Except that’s not what Bundler does. Bundler sees the sub dependencies of rails in this situation as part of the initial work-logic dependency tree. So even though we’ve explicitly requested rails be sourced from rubygems.org, bundler attempts to source rails’s dependencies from https://internal-gem-repo.awesome. Since there are mirrored versions of those gems on the internal gem repo, this technically works, but is not the desired outcome of our configuration.

Rails as an explicitly declared dependency of work-logic but sourced from ruby gems, but it’s implicit gems still source from our internal gem repo.

As it stands, there’s only 2 ways to fix this, some dependency jiggery pokery and potentially some manual modification of your Gemfile.lock to try force these sub-dependencies to source from rubygems.org, or explicitly declare every single gem that rails depends on in your Gemfile and it’s source as rubygems.org, eschewing much of the simplicity the initial Gemfile would have offered.

source "https://rubygems.org"

gem "rspec"
gem "rails"
gem "active-record"
gem "action-text"
gem "active-support"
... etc
gem "work-logic", source: "https://internal-gem-repo.awesome"

Ew…

The (Rough) Solution

Simply, in the process of generating the dependency graph and figuring out the versions and sources of gems, the closest explicitly declared parent of a gem should be used to determine it’s source. So as per the above context, the moment we declare rails explicitly, as being sourced from rubygems.org, any implicit dependencies of rails should then be sourced from rubygems.org as well, unless a more explicit directive on one of those lower dependencies is given.

I spent… way too much time reading the bundler source code to figure out why this was happening, but I think I’ve tracked down the problem to the SourceMap implementation.

You can delve into the depths of Bundler if you’re interested, but the issue seems to boil down to the Bundler:SourceMap#all_requirements method’s solution for resolving the source of indirect dependencies after the first pass of direct dependency resolution.

def all_requirements
  requirements = direct_requirements.dup

  unmet_deps = sources.non_default_explicit_sources.map do |source|
    (source.spec_names - pinned_spec_names).each do |indirect_dependency_name|
      previous_source = requirements[indirect_dependency_name]
      if previous_source.nil?
        requirements[indirect_dependency_name] = source
      else
        no_ambiguous_sources = Bundler.feature_flag.bundler_3_mode?

        msg = ["The gem '#{indirect_dependency_name}' was found in multiple relevant sources."]
        msg.concat [previous_source, source].map {|s| "  * #{s}" }.sort
        msg << "You #{no_ambiguous_sources ? :must : :should} add this gem to the source block for the source you wish it to be installed from."
        msg = msg.join("\n")

        raise SecurityError, msg if no_ambiguous_sources
        Bundler.ui.warn "Warning: #{msg}"
      end
    end

    source.unmet_deps
  end

  sources.default_source.add_dependency_names(unmet_deps.flatten - requirements.keys)

  requirements
end

def direct_requirements
  @direct_requirements ||= begin
    requirements = {}
    default = sources.default_source
    dependencies.each do |dep|
      dep_source = dep.source || default
      dep_source.add_dependency_names(dep.name)
      requirements[dep.name] = dep_source
    end
    requirements
  end
end

To contextualize the snippet, the SourceMap defines the fully realized set of dependencies and where they need to be pulled from for Bundler to ultimately resolve and pull your gems. direct_requirements looks at the list of dependencies you explicitly defined in your Gemfile, and if the Gem has an explicitly defined source by virtue of the source kwarg or a source block wrapping it’s definition in your Gemfile, then the source is defined. This dsl is evaluated to create a bundler Definition, which prior to resolution, generates a SourceMap using the above snippet to determine which source your respective gem will come from.

The problem however, is in these few lines:

def all_requirements
  requirements = direct_requirements.dup

  unmet_deps = sources.non_default_explicit_sources.map do |source|
    (source.spec_names - pinned_spec_names).each do |indirect_dependency_name|
      previous_source = requirements[indirect_dependency_name]
      if previous_source.nil?
        requirements[indirect_dependency_name] = source
      else
  ...

Bundler is grabbing unmet dependencies by cycling through your explicitly declared, non-default sources. Which is to say any gem source you’ve declared at all, that’s not RubyGems. It then goes through the spec list from that source, excluding any gems you explicitly declared in your Gemfile, and if that gem was already declared with a source elsewhere (such as your Gemfile or Gemfile.lock), it will just use that source. Otherwise it will just select the source it’s currently looking at as the source for your gem. This isn’t an illogical choice. If your gem is coming from this alternative repository, and your gem’s dependencies are present on this mirror, it’s probably reasonable you want your gem’s dependencies to come from there too.

The problem is, per our example above, we’ve explicitly declared rails with the expectation that it’s dependencies should be sourced from rubygems.org, but because some version of rails’s dependencies exist on https://internal-gem-repo.awesome, it’s prefering our explicitly declared gem source anyways.

So step one to fixing this bizarre behaviour, is to make Bundler explicitly prefer to fetch a gems dependencies from it’s closest explicitly declared parent:

def all_requirements
  requirements = direct_requirements.dup
  direct_dependency_parents = @dependencies.map{|dep| dep.to_spec.dependencies.map{|sub_dep| {sub_dep.name => dep.name} } }.compact.flatten.reduce(&:merge)

  unmet_deps = sources.non_default_explicit_sources.map do |source|
    (source.spec_names - pinned_spec_names).each do |indirect_dependency_name|
      requirements[indirect_dependency_name] = requirements[direct_dependency_parents[indirect_dependency_name]]

Without modifying bundler in crazy ways, the most straightforward way to do this is to take the list of explicit dependencies we’re trying to configure a SourceMap for and map through them to create a hash of the sub_dependencies as keys, and their values being the gem that depends on them, so that we can establish the parent of that gem. The requirements variable is a hash list of our explicitly declared gems and what their source is, already made by the SourceMap class. So we leverage this with our new method of determing the parent gem for our current gem, and use that to determine the source.

This largely resolves our issue, as now any gems that rails depends on are sourced from rubygem.org despite rails being a dependency of our work-logic gem. However, now that we’re here, we may go a step further. Our work-logic gem has some other dependencies that we don’t actually want to be sourced from https://internal-gem-repo.awesome as these are other open source gems we don’t have any good reason to mirror either. So I added an additional option to the gem command of the Gemfile dsl, preffered_spec_source. This required 3 particular changes.

First we add our preferred_spec_source as a variable to the Dependency object Bundler uses to track dependencies.

  class Dependency < Gem::Dependency
    attr_reader :autorequire
    attr_reader :groups, :platforms, :gemfile, :git, :github, :branch, :ref, :preferred_spec_source    

    ...

    def initialize(name, version, options = {}, &blk)
      type = options["type"] || :runtime
      super(name, version, type)

      @autorequire    = nil
      @groups         = Array(options["group"] || :default).map(&:to_sym)
      @source         = options["source"]
      @preferred_spec_source = options["preferred_spec_source"]
      @git            = options["git"]
      @github         = options["github"]
      @branch         = options["branch"]
      @ref            = options["ref"]
      @platforms      = Array(options["platforms"])
      @env            = options["env"]
      @should_include = options.fetch("should_include", true)
      @gemfile        = options["gemfile"]

      @autorequire = Array(options["require"] || []) if options.key?("require")
    end

Then we add preferred_spec_source as a VALID_KEY for the gem command in the DSL.

module Bundler
  class Dsl
    include RubyDsl

    def self.evaluate(gemfile, lockfile, unlock)
      builder = new
      builder.eval_gemfile(gemfile)
      builder.to_definition(lockfile, unlock)
    end

    VALID_PLATFORMS = Bundler::Dependency::PLATFORM_MAP.keys.freeze

    VALID_KEYS = %w[group groups git path glob name branch ref tag require submodules
                    platform platforms type source install_if gemfile preferred_spec_source].freeze

And then we need to modify the SourceMap implementation one last time so that, in the event the gem we’re looking at has an explicit parent, we need to check if the explicit parent declared a preferred_spec_source, and use that instead of anything we might have determined elsewise.

    def all_requirements
      requirements = direct_requirements.dup
      direct_dependency_parents = @dependencies.map{|dep| dep.to_spec.dependencies.map{|sub_dep| {sub_dep.name => dep.name} } }.compact.flatten.reduce(&:merge)

      unmet_deps = sources.non_default_explicit_sources.map do |source|
        (source.spec_names - pinned_spec_names).each do |indirect_dependency_name|
          previous_source = requirements[indirect_dependency_name]
          if previous_source.nil?
            explicit_parent = @dependencies.find{ |dependency| dependency.name == direct_dependency_parents[indirect_dependency_name] } 
            requirements[indirect_dependency_name] =  if explicit_parent && explicit_parent.preferred_spec_source
              sources.rubygems_sources.find{|source| source.name == "rubygems repository #{explicit_parent.preferred_spec_source}"}
            else
              requirements[direct_dependency_parents[indirect_dependency_name]]
            end
          else

This particular implementation is probably flaky as it does no validation that your preferred_spec_source isn’t garbage. (If it is, it seems to just ignore it.) With some refactoring and testing though,this would likely handle this particular situation robustly. I’d need to understand the bundler ecosystem far more than I do after my 2 day deep dive to do that though.

Conclusion

Figuring out how this functionality worked without ever having looked at how Bundler works internally has been quite the adventure. I’d like to take a moment to thank binding.pry for being the best developer tool I could ever recommend, as figuring this out ultimately required me to pull the bundler source code and just start jumping into various locations during the execution to determine the origin of my undesired source mappings. I’ve definitely gained a massive appreciation for the sheer complexity of what it is Bundler does so that we can, under 99% of all cases, just pull a bunch of correctly versioned and platformed gems from the right place, and move on with our lives without trudging through dependency hell.

These fixes are likely very naive in some regards, and will probably need to be refactored and tested to truly be useable in the wild so modify bundler at your own risk. I might consider opening a PR for bundler with a request for some guidance on any potential issues these changes could cause and how to prevent them.

Deterministic software development practices and the horror of losing your Docker images

2021-01-14T18:00:00+02:00

Sometimes when developing software, it can be hard to expect or account for some of the horrors that might arise as a result of a small and seemingly insignificant decision. Small, ostensibly innocuous decisions can have monumental ramifications down the line, as I’m certain any software developer with a few production systems under their belt can attest with a harrowed grimace. Over the past two days I had the joy of experiencing exactly such a consequence.

Deterministic Software Development Practices

While I continue the joys of pandemic-era South African enforced teetotalism despite my frayed nerves, it’s worth providing some context to my sudden desire to take a 6 month sabatical as a barista ~~or an alcoholic~~.

Good software development practice emphasises determinism and reproducability when it comes to the code you write, the system you develop and the applications you deploy. In an ideal world, all code does the same thing every time, every system deploys identically every time, and applications only do what you expect them to every time.

Much to our misery, such an ideal world is the stuff of fairy tales, and it’s only through the rigourous implementation of protocols and procedures that we can enforce some degree of reasonable expectation upon reality. Such efforts are the spawning pool of concepts such as Unit Testing, Test Driven Development (TTD), codefied deployments, Continuous Integration (CI) and Continuous Delivery (CD), Docker, and Infrastructure As Code, among others.

Effectively tested code provides a valuable safety net of surety that running that code will produce the desired results when executed as intended; and if you’re testing effectively, that it will fail in a reliable and expected fashion when executed in an unexpected way.

Codefied deployments with tools like Ansible, Chef or Salt are an effective means to remove the human element of error from potentially lengthy and complicated processes; making an invaluable assurance that your deployment processes will continue to work in a consistent manner even after John the DevOps whiz leaves for his new job in Silicon Valley (or quits software development entirely in favour of a more relaxing life as a carpenter).

This testing and sets of strictly defined deployments can be further established as a deterministic process through the introduction of effective CI and CD, which can ensure that all code written is held to an appropriate testing standard, validating that no past tests have been broken (and in better implementations, that newly added code is effectively tested as well), and that deployments follow an expected deployment procedure based on the merging and versioning of code to various version control branches.

This would normally take the form of the codefied deployment being executed with the newest versioned set of code pushed to a specific git branch, probably called “develop”, to a “Staging” server for manual validation and testing; while a versioned merge to a “~~master~~main” branch would trigger the deployment of that code to “Production”, with all manner of potential checks and balances to ensure a safe and succesful deployment… if you’re not too scared of the word “DevOps”.

The result of all these efforts is that you can ensure that whatever code you have deployed is, at the minimum, exactly as or more functional than it was in any prior deployment, and that in the event of disaster, you can effectively recover by following the same preceduralized steps, and deploy a functioning version of your application should your manager/boss/product owner/client demand it be done. (Though we do drift ever so slightly closer to that mystical land of fairytales here, as disaster recovery is rarely quite so straightforward.)

The Horror of losing your Docker images

This all brings us to my past 2 days, having enjoyed the safety net of stable CI/CD which would reliably run the unit tests suite for my applications; and in the event of succesful testing and merging, build the associated application’s docker image to be stored in our internal registry for later deployment. (No automated deployments just yet, but the dream lives on for the moment.)

For the uninitiated, Docker is a nifty containerization protocol that took the developer world by storm as an effective way to package up both your application and the environment in which it needs to be executed in, thereby ensuring yet another layer of determinism and an amusing and succinct response to the problem so well known by the phrase “it works on my machine.” Which is to say: “Well, then we’ll package your machine.”

In terms of deployment, having your historical docker containers tagged and stored is an effective measure in terms of disaster recovery, as it provides a relatively easy mechanism by which to roll back your exact deployment in the event of disaster (or an unfortunately dynsfunctional deployment.) It can also be highly beneficial for simplying the development of an application between developers who might have differing local environments as well.

Today’s horror boiled down to one seemingly innocuous decision. Due to space constraints, it was decided that our internal container registry would not be backed up, as these images could easily be rebuilt through their dockerfile defintions in any case, right? Right. Absolutely. In fact, I had made various efforts to script up these build processes and attach them to our CD processes. So when disaster struck and our registry was lost, it seemed like no real calamity at first.

But then there was a bug we needed to patch in one of our systems… A story in of itself, once we eventually tracked down the relevant issue, we applied the relevant patch to our code, wrote the related tests to ensure it’s validity, and pushed it up for the lovely automated scripts to validate and build the docker image for future deployment. All seemed like smooth sailing as we confidently awaited the opportunity to resolve this strange bug in our system with but a singular deployment; none the wiser to our impending stress level hike.

We deployed the newly built and packaged docker image to our staging server as the vastly important final sanity check prior to deploying it to master; but were surprised to find that weirdly, the application suddenly refused to connect to the database. Hmmm. Peculiar. So the debugging questions began? Did you change something? Did I change something? Is something out of date? The error seemed to imply TLS certificate validation is failing; maybe lets rerun the cert generation script? Okay. Did that, certs are all emplaced and it’s still broken.

What, oh what, could it possibly be? Our despondency only grew as we checked increasingly more minute details. Cert timestamps checked out. Were we mounting the certs to the correct containers. Seems so. Did they verify? Yes again. Did they match? Unfortunately, since we knew we needed to find another thing to check, yes. Each increasingly unlikely question was answered unequivocably with the word “yes”; leaving us to break for lunch with severe frustration and feeling increasingly nervous as we were asked once again for the status on the patch we’d been working on prior to this issue.

Let it be said, the wonders of taking a break for lunch can do wonders for even the desperate, as once we came back, my colleague had begun thinking about the thing that we had changed and thought nothing of. All the docker images we’d rebuilt, and more specifically, the base docker images we had rebuilt and published to our registry on top of which our code would be layered and deployed. I found the idea completely unlikely, until he used the docker image history to examine the sub layers of our deployed image, vs those of the last known good image we had been previously running on staging. What became evident was that the underlying operating system layers of the newer image were also newer.

This to be perfectly clear… is not very deterministic. The implication of this is that the base Ruby image we had previously built our internal base image atop of had been functionally modified, which meant that our new base images were not identical to the base images we’d had prior to the loss of our image registry’s contents. Rolling back to a Ruby image that had not been modified further proved this, as our mystical cert verfication error disappeared in a puff of smoke. This of course, left us feeling relieved, exhausted and, understandably, very concerned that we may be unable to trust our docker images to build and behave in a reliable, expected and deterministic fashion. Or at least for how long it takes to rebuild all our images and manually ensure that they still run in the expected fashion.

It is frustrating to realize that this core versioned Ruby image dependency would be changed right out from under us like this. While our internal image storage was shielding us from the results of this external influence, it’s worth understanding that any of your code or systems which rely on external dependency’s to build or deploy are at the mercy of decisions which may be made outside of your control, and may be actively deleterious to your own software development efforts. We need only refer back to the events of the left-pad debacle to recall how one small, meaningless dependency broke the internet.

So the moral of the story is quite simple. Your build artifacts are important, and your deployment mechanism’s reliance on external entities is an active risk to the succesful continuation and operation of your application, and potentially the business itself. Backup your docker containers, and make sure that whatever images or dependencies you might be using as a base from which to build your applications are, to the best of your ability, cached internally to shield them from external influence, be it mistaken or actively malicious by bad actors.

Building a Jekyll Blog as lazily as possible

2020-07-16T17:03:00+02:00

Many software developers have a personal blog that acts as something of a portfolio of the things they’ve worked on, researched or merely learned. What I am going to talk about is Jekyll and Github-Pages, two lovely pieces of technology which allowed me to put this little blog of my own together with the minimum effort possible.

Now before I get into detail; yes, “lazily” and “minimum effort possible” are perhaps not terms you want to be reading if you’re trying to appraise the value and behavioural habits of a Software Developer. However in this case I use the terms not to imply that I’ve avoided putting care and attention into the creation of the blog, but rather to indicate a preference towards more efficient and effective solutions when pragmatically appropriate.

In this case pragmatism speaks to the fact that while I am fully capable of fancy, frameworked front-end development, I don’t derive any particular enjoyment from it like I do with other developmental tasks. As such, the pragmatic path to facilitate me writing posts about the things I have a passion for, is to get this blog online as efficiently possible.

I suppose this is something of a moot point, since I do find some joy in finding efficient and effective approaches to problems, so I somehow found my itch scratched despite the task at hand.

Finally, the Jekyll stuff

In any case, you’re probably not here to listen to me talk about myself, so lets talk about Jekyll!

Jekyll is a Ruby Gem written with the express goal of minimizing the difficulty involved in writing and maintaining a blog by automating various aspects of common blog design and management, as well as providing a fairly simple and straightforward environment with which to apply blog themes kindly offered by the open-source community. (The link to the theme powering my blog in particular is present in the footer of this page.)

As far as it’s goal goes, it beats the mark quite substantially. Like all Ruby Gem’s, installation and creation of your very own Jekyll project is a simple as running the following in your terminal, provided you have a reasonably modern version of Ruby available within your development environment.

gem install jekyll && jekyll new <your_blog_name>

Likely your output from the jekyll new command will look something like this:

➜  jekyll new test
Running bundle install in /home/gregory/Projects/test... 
  Bundler: Fetching gem metadata from https://rubygems.org/..........
  Bundler: Fetching gem metadata from https://rubygems.org/.
  Bundler: Resolving dependencies...
  Bundler: Using public_suffix 4.0.5
  Bundler: Using addressable 2.7.0
  Bundler: Using bundler 2.0.1
  Bundler: Using colorator 1.1.0
  Bundler: Using concurrent-ruby 1.1.7
  Bundler: Using eventmachine 1.2.7
  Bundler: Using http_parser.rb 0.6.0
  Bundler: Using em-websocket 0.5.1
  Bundler: Using ffi 1.13.1
  Bundler: Using forwardable-extended 2.6.0
  Bundler: Using i18n 0.9.5
  Bundler: Using rb-fsevent 0.10.4
  Bundler: Using rb-inotify 0.10.1
  Bundler: Using sass-listen 4.0.0
  Bundler: Using sass 3.7.4
  Bundler: Using jekyll-sass-converter 1.5.2
  Bundler: Using listen 3.2.1
  Bundler: Using jekyll-watch 2.2.1
  Bundler: Using kramdown 1.17.0
  Bundler: Using liquid 4.0.3
  Bundler: Using mercenary 0.3.6
  Bundler: Using pathutil 0.16.2
  Bundler: Using rouge 3.22.0
  Bundler: Using safe_yaml 1.0.5
  Bundler: Using jekyll 3.8.7
  Bundler: Using jekyll-feed 0.15.0
  Bundler: Using jekyll-seo-tag 2.6.1
  Bundler: Using minima 2.5.1
  Bundler: Bundle complete! 6 Gemfile dependencies, 28 gems now installed.
  Bundler: Use `bundle info [gemname]` to see where a bundled gem is installed.
New jekyll site installed in /home/gregory/Projects/test. 
➜  cd test
➜  ls
404.html  about.md  _config.yml  Gemfile  Gemfile.lock  index.md  _posts
➜  ls _posts 
2020-08-16-welcome-to-jekyll.markdown

Congrats, you have a Jekyll blog! Well done! Lets knock off, head to the pub and share a pint in celebration! All you need to do is start it:

➜ jekyll s
Configuration file: /home/gregory/Projects/test/_config.yml
            Source: /home/gregory/Projects/test
       Destination: /home/gregory/Projects/test/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
       Jekyll Feed: Generating feed for posts
                    done in 0.505 seconds.
 Auto-regeneration: enabled for '/home/gregory/Projects/test'
    Server address: http://127.0.0.1:4000/
  Server running... press ctrl-c to stop.

Your wonderful new Jekyll blog.

Okay, okay, odds are good you’re fully aware I’m being full of nonsense and that, unfortunately, nothing is that simple. (As much as us lazy developers wish otherwise…)

Now that we’ve had our reality check, lets get on with actually turning our initial generated jekyll template into an actual blog. Naturally, nothing beats the actual documentation which is likely to be updated in the long term unlike this blog post, so you can find that here.

The lazyiest way to make the blog your own

First and foremost, as clean as the default aesthetic is, you’re undoubtedly in the mood to show off your creative flare and make sure your blog effectively mirrors your sunshiny disposition and meticulous attention to design and detail… or, like me, you just don’t want to burn your readers eyeballs out while being minimally offensive from a design perspective.

The following is the default configuration for Jekyll generated by the jekyll new command. The is essentially the central configuration underpinning the entire Jekyll blog. The blog’s title, url and it’s wonderful, skillful and not at all lazy author are all detailed in here, in addition to various other interesting configurations, including those specific to the theme you apply to your blog.

# Welcome to Jekyll!
#
# This config file is meant for settings that affect your whole blog, values
# which you are expected to set up once and rarely edit after that. If you find
# yourself editing this file very often, consider using Jekyll's data files
# feature for the data you need to update frequently.
#
# For technical reasons, this file is *NOT* reloaded automatically when you use
# 'bundle exec jekyll serve'. If you change this file, please restart the server process.

# Site settings
# These are used to personalize your new site. If you look in the HTML files,
# you will see them accessed via Havenga.dev, your-email@example.com, and so on.
# You can create any custom variable you would like, and they will be accessible
# in the templates via .
title: Your awesome title
email: your-email@example.com
description: >- # this means to ignore newlines until "baseurl:"
  Write an awesome description for your new site here. You can edit this
  line in _config.yml. It will appear in your document head meta (for
  Google search results) and in your feed.xml site description.
baseurl: "" # the subpath of your site, e.g. /blog
url: "" # the base hostname & protocol for your site, e.g. http://example.com
twitter_username: jekyllrb
github_username:  jekyll

# Build settings
markdown: kramdown
theme: minima
plugins:
  - jekyll-feed

# Exclude from processing.
# The following items will not be processed, by default. Create a custom list
# to override the default setting.
# exclude:
#   - Gemfile
#   - Gemfile.lock
#   - node_modules
#   - vendor/bundle/
#   - vendor/cache/
#   - vendor/gems/
#   - vendor/ruby/

You’ll notice the config contains a line written “theme: minima”, which applies the “minima” theme to your blog, no extra steps required. Changing the theme is simply a matter of making sure the theme files are available and then modifying this particular configuration. There’s a variety of wonderful open source themes to view and choose from, as well as details about how to make sure your gem based themes are available to your blog, all mentioned in the Jekyll theme documentation here

No Minimum effort prettiness

Of course, if you’re feeling the itch of laziness, then what you’ll be particularly interested in is this one subtle bit of configuration in the _config.yml for this very blog (at the time of writing at any case):

remote_theme: "mmistakes/minimal-mistakes@4.20.1"

The remote_theme configuration lets you, for specific themes that have been apropriately configured on Github to be supported, let you set them as your blogs chosen theme using this option, and then do literally nothing else in your pursuit of ~~laziness~~ efficiency. Job’s done and your blog suddenly looks magically different give or take a few configuration tweaks specific to the theme you chose.

Make sure to at least glance over the documentation for your theme since, for example, the minimal-mistakes theme in particular requires the default layout of your pages to be changed in order for them to be able to be pre-rendered by Jekyll.

From there, just a little config and you’re all set. Most of the themes including some magical logic that can use information provided in your _config.yml to populate all manner of fancy presentational details for you without you needing to write any HTML or CSS, such as the author sidebar with my handsome mug.

Deployment time… Oh no.

Yep, now we actually have to put this puppy on the internet! Easy right? Maybe you could spool up a Heroku app, push your blog over to there and present it just like that? Or you could purchase a server or compute instance from some provider like AWS, scp your code over to it and turn the ignition to immediate developer blogging success!

Orrrrrrrrrrr, if you’ve put as much effort into building this blog as I have, these still sound like too much work. Luckily for us, Github Pages is a thing.

Github Pages is an awesome little bit of functionality provided by Github to serve some perhaps not-very-basic informational webpages directly from the Github repository. In the case of this blog, this repository is found at https://github.com/OddballGreg/oddballgreg.github.io, which translates to the blog being hosted at https://oddballgreg.github.io by Github Pages.

You might notice that’s not the URL you’re at right now. I’d picked up havenga.dev at a good price from Namecheap a while back when I first thought about writing a dev blog, and after a day of learning to understand what on earth ANAME and CNAME records are, as well as a variety of other DNS related tidbits, set that domain to permanently redirect to the Github Pages url for this blog.

You’ll need to enable the Github Pages functionality (and inform Github of your desire to use your own URL if you care to) on the primary settings page of your repository. Make sure to enable HTTPS as well since, well, there’s really no reason your web assets shouldn’t be using HTTPS in this day and age. Your account will need to be verified and a few extra details filled in before GitHub will let you finish this configuration.

The Github repository settings for enabling Github Pages

Next up; the following is the “Gemfile” for this blog. One of these is automatically generated by the jekyll new command, though it’s not unique to blogs, or even web applications. Gemfiles are essentially Ruby’s way of performing dependency management. By listing the gems you want to use in your application and setting their specific versions as desired, you can effectively manage the dependencies and their versions that you expect to use when running the ruby application. Running the ruby command bundle in a modern ruby version (or installing it via gem install in an older one first) will then trigger “bundler” to resolve the dependency requirements of each gem listed in the Gemfile to make sure that none of the dependencies have mutually exclusive requirements; before writing the final set of resolved and agreed dependency versions to the “Gemfile.lock” file. Provided you commit this file to your source control, it provides you with a relatively strong assurance that anyone running the Ruby project will be doing so with same dependencies.

source "https://rubygems.org"

# Hello! This is where you manage which Jekyll version is used to run.
# When you want to use a different version, change it below, save the
# file and run `bundle install`. Run Jekyll with `bundle exec`, like so:
#
#     bundle exec jekyll serve
#
# This will help ensure the proper Jekyll version is running.
# Happy Jekylling!
# gem "jekyll", "~> 3.8.6"

# If you want to use GitHub Pages, remove the "gem "jekyll"" above and
# uncomment the line below. To upgrade, run `bundle update github-pages`.
gem "github-pages", group: :jekyll_plugins
gem 'faraday', '0.17.3'
gem "jekyll-github-metadata"
gem 'jekyll-include-cache'

# If you have any plugins, put them here!
group :jekyll_plugins do
  gem "jekyll-feed", "~> 0.6"
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
# and associated library.
install_if -> { RUBY_PLATFORM =~ %r!mingw|mswin|java! } do
  gem "tzinfo", "~> 1.2"
  gem "tzinfo-data"
end

# Performance-booster for watching directories on Windows
gem "wdm", "~> 0.1.0", :install_if => Gem.win_platform?

Longwinded explanation aside, the only thing your really need to care about is that for Github Pages to work with your Jekyll blog, you need to add the github-pages gem as a dependecy for the application, and then run bundle to resolve a version for it.

gem "github-pages", group: :jekyll_plugins

Provided all has gone well, you should be all set to push your code to it’s Github repository now and set off to demonstrate your programming prowess to the world.

All Done!

Erh, not really. You realize you still have to actually write some content for the blog now right? Maybe a dry attempt at humorously describing how lazy Jekyll and Github Pages let you be while setting up your blog?

In any case, I recommend taking it easy, putting on some calming music or YouTube video game playthroughs you don’t actually care to pay too much attention to, and just have fun with it. Having written for a blog every day for 2 years in my past, I can assure you, it’s all too easy to stress yourself out by applying too much pressure when it’s unneeded… kind’ve like trying to make a professional blog right off the bat when almost no-one is likely to be reading it in the beginning anyways.