Devlog I: Foundations

Back in March I began working on a Go web application called Househunt. It’s goal is to be a non-trivial reference code base.

While progress has been slower than I wanted, most of the foundational components are in place. They might see little tweaks and bugfixes, but I don’t expect major changes at this moment.

In this devlog I’ll list the major components and notes I made along the way.

But first, a screenshot of how the project looks now:

The parts

Config handling

Configuration is provided via env variables. There are a few required options and a lot of default values that can be overwritten.

Provided values are validated on a best effort basis.

The config handling is mainly implemented using a map of functions. Code can be found here.

Data storage: SQLite

Data is stored in a SQLite database. Househunt uses the mattn/go-sqlite3 package as the SQL driver, which does require it to use CGO.

I hadn’t really used SQLite before, so it tooks some config tweaking to make it suitable for web app workloads:

• This Github comment was very valuable. It describes how you need to run the driver using two different DB pools.
• My code for this can be found here.
• Shootout to @Staticfrost_ for great suggestions around the drivers and config tweaks.

SQLite surprised me in some ways:

• By default SQLite allows “flexible typing”, any type can go into any column.
• You explicitly enable foreign keys.
• Can store DB in memory, great for speedy tests.

Migrations

Some migration systems support “up” and “down” migrations. The idea is that “down” migrations allow you to roll back the database to an earlier version.

However, not all database operations can actually be undone, data that is deleted can’t be restored. Making the entire model a bit awkward IMO.

I prefer modelling my migrations as an “append only” list of changes that were executed against the database, if I need to roll back something I will add a new migration that undoes the previous one.

The code can be found here.

A small binary is available that only runs the migrations, giving you the option to run migrations in different ways:

• During startup of the server binary.
• As this seperate binary.

The repository also contains the full schema created by all migrations. This enables us to just browse Github to inspect the DB schema without having to combine the migrations in our heads.

If this generated schema is not up to date the build pipeline fails.

HTML Views

HTML Views are created by combining different templates. Househunt supports loading these templates in two ways:

• Parse, cache and execute the templates embedded in the binary.
• Parse and execute the templates from disk. This is done in an ad-hoc way, enabling frontend development without having to rebuild the binary.

The code can be found here.

HTTP Handlers

Earlier I wrote an article on generic HTTP handlers. The handlers in Househunt are an evolution on these ideas.

I’ll do a more in-depth write-up on them in the future, but the code can be found here.

Sessions

Househunt uses encrypted cookies to store session state. Courtesy of the classic gorilla/sessions package.

CSRF Protection

Same here, CSRF protection is implemented using the gorilla/csrf package.

Transactional emails

At the time of writing Househunt sends two kinds transactional emails:

• Account activation links.
• Password reset links.

The email subjects and bodies are constructed from templates in a similar way to the HTML views. The emails are then send via the Postmark API.

Code can be found here.

Password authentication

Househunt uses password authentication, users will need to register an account, activate it and can then login.

This is where a significant amount of time was spent, to make sure things are correct. The OWASP cheatsheets were a great help.

Notes on the auth package:

• Argon2 for password hashing. The Go package interface is a bit more technical as compared to bcrypt.
• Time sensitive operations are done async using go routines to prevent timing attacks.
• I’m using blind indexes for storing emails. This might be a bit overkill, but it’s mainly to protect myself from messing up with cloud based backups.
• Implemented common interfaces to prevent leaking data in logs.

If you find any issues, please let me know!

User stories

Next to unit tests, there is a load of “user story tests”.

These tests attempt to imitate the user navigating the app and submitting forms. The code can be found here.

Why is progress so slow?

So, this represents about 2 months of work. This definitely wasn’t full-time, but if I had to estimate there is a solid 2-3 weeks worth of effort in here.

What made progress slow:

• I was a bit too much of a perfectionist in the beginning.
• There were two real-life events that had me away from work for a couple of days, took a few days to recover from them and return to get to “full work mode” again.
• I’m trying to stick to the standard library as much as possible.

What is next?

Now with all these basic components in place, we can start building some actual business logic.

I think the most logical thing will be a way for agents to create listings and go from there.

So be sure to follow along as I begin building that next week :)