As I have spent most of March this year hassling with the tooling provided with Golang, I have not made half the progress on HALO I was supposed to. Moving into April with no end in sight, I have to stop for a moment and redo my assessment for HALO’s choice of language in backend server programming.
For those unfamiliar with Golang, it has this cornucopia called the “Go tool”, invoked on the command line using the go command. go handles getting and building programs, dependencies, test suites, and so forth. In exchange for this convenience, go expects developers to follow ‘best practises’ wherever relevant, which is dictated by Golang community consensus. I think the benefits of this should be somewhat obvious, but if they aren’t, the Golang community seems to be more than happy to explain everything they’ve decided on.
This month of development has been dominated by my efforts to take manual control of HALO’s build process using POSIX shell scripts for automation. The reasons for doing this include:
The first reason may be confounding to some. I will explain it in greater detail another day, but for now I hope it suffices to say, dependencies need to be handled with extraordinary care. Cutting-edge tooling like that provided with Golang doesn’t work without a consistent internet connection. It also introduces incalculable unpredictability with the total lack of contract enforcement (for example, with semver) and un-knowable circumstances a repository may find itself in in the future (for example, deleted or moved). For well-defined APIs and software honestly billing itself as reliable, this risk is unacceptable. Without active maintenance the software will inevitably become buggy if it doesn’t outright break because of volatile dependencies pulling the rug out from under its feet. The other problem with Golang’s dependency management is the control problem made infamous by npm the more dependencies one has, the less accountable the project becomes. While Golang is not as terrible in this context as JavaScript, it still isn’t geared to help developers grapple with the maintainers of the code they depend on.
The second problem is that Golang ‘best practise’ mandates putting all Golang source files in the root of the Git repository. It also mandates that tests be paired to source files 1:1 with a _test filename suffix before the .go extension. I found myself reminded of Java programming practises in trying to grapple with these arcane rules, and the reminiscience set in once I realised the third and final issue at hand: the tooling works to enforce these things every step of the way.
Originally, I worded the third reason a bit generously. My problem is more severe than Golang’s lack of support for doing things differently – had that been the extent of my issue I would have sailed right on, writing the scripts I need to automate the build process and get on with coding the application. No, the go tool actively obstructs me and undermines every effort I make to deal with the natural implications of a manual build. It turned a detour that needed a couple of days to complete into a month-long torture session with this program as my inquisitor, and I’d like to enumerate a few of the insanities this tool has brought to bat.
First, $GOPATH. go will deliberately halt without doing what you told it to if this variable contains two consecutive dots, or if it begins with a dot. Ergo, it thoroughly rejects relative paths, so don’t even bother unless you’ve got a fancy absolute path resolver. will also resign in protest if the $GOPATH directory doesn’t follow the prescribed hierarchy, which makes it impossible to reconfigure the variable to sanely point somewhere nearby for dependencies not in your original $GOPATH.
Within the last couple of versions, Golang recently received support for a new modules feature that sidesteps that circus. While this makes it practical to have projects with source code in a subdirectory of the repo (why wouldn’t you?), you’re still out-of-luck if you hoped to sidestep Golang’s dependency management involving an internet connection, Git clones, and everything else that comes with it. You cannot put shared code in a separate module folder, build it once as a static library, and pass that to the linker for each project that depends on it. You still need to follow the orthodoxy for tests.
If this tool were a human being, it would belong in a psych ward. There are a world’s worth of reasons substantiating best practises and its tooling support, but no reason with any technical substance justifies obstructing developers from disregarding those reasons and doing something different when necessary. Rules are made to be broken, and it seems the only recognised path for that process is through the Golang community itself, which is a hidden cost most people cannot bear. I am here to make HALO, not fix the problems of the programming language I happen to be writing it in.
While I held out as much optimism as I could, I have to admit I haven’t seen much better out of the community itself. Visiting Discord servers and IRC channels for Golang has shown me a kind of conceited hubris from the people active there that, in my opinion, more than explains the schizophrenia exibited by the tool, and this was to some extent corroborated by posts on help forums such as SO and GitHub. Lots of yapping about best practises, colouring inside the lines, the attitude of people acting like they know better no matter what the context or who they’re talking to. They’re of no help for dealing with programming problems when it requires thinking outside their predefined box of ‘best practises’. One Discord community of over 2,000 members even shared with me this jaw-dropping exchange: https://archive.vn/deMEm
This has cost a lot of time, and I have to make a call here on whether this is still worth pursuing through Golang instead of any other programming language. By far the most compelling benefits of Go lie with its language design, so it really couldn’t be more unfortunate to have that retarded by ideological convictions from its community.
As nice as it is to write Go, I can write in many other programming languages too. It’s time to make a feature assessment and choose another language to take Go’s place. Before I examine the contenders, I want to get out of the way languages not up for contest, and the (simpler) reasons for why they are ruled out.
Like Golang and D, Scala helps itself to performance through static typing, and like D its typing discipline is also strong for that. The JVM is also key in providing more performance boosts, and the language is incredibly expressive in both functional and object-oriented contexts. It also supports several concurrency strategies, backed up by its functional programming patterns, making it ideal for HALO’s bastion backends, especially with tools like Apache Spark.
Some downsides Scala presents relate to its functional expressivity, which is to a great extent unavoidable and those unable to cope should git gud. Arguably the largest drawback is its power curve: Scala absolutely requires competency and skill to effectively use, and those lacking that will struggle to use the language to its full effectiveness.
Python is a very popular and understandable programming language with very strong, diverse support for various problem domains. While HALO will probably forego the high-level Django framework, Python is still a compelling choice thanks to its library support, and its dependency management is as opt-in as anything.
Fine-grained optimisation is something that Python doesn’t support as cleanly. While this is not of immediate concern for HALO, it’s still important for the assessment as problems like this will crop up down the road. Compared to other languages, Cython and similar pathways for native performance are as clunky and ad-hoc as those of Node.js. This is important also because of Python’s typing discipline, as unlike Scala, D, and Go it doesn’t accomodate performance as well.
Python’s support for concurrency is present, but the expressivity of the language does not accomodate it as well as Haskell, Scala or other functional languages. Libraries are very prone to cause paradigm clashes, rendering them useless for their purpose and driving the developer to reinvent the wheel to solve their problems.
D’s syntax will be comfortably familiar for those versed in C♯, and veteran C++ developers will be pleasantly surprised by its extensive support for various paradigms, quality of life features like contracts and compile-time code execution, and of course its familiar syntax. Its main downside is its relative complexity and unpopularity, and dependencies aren’t always in stock, so C bindings may end up in order for special things like database drivers. D is as good as gold in regards to its community and ecosystem, so there should be no problems like those of Golang or Rust when it comes to dependencies and code accountability.
Although Scala well outdoes D in expressivity because of the latter’s derivation from C, it can still be very empowering for projects needing functional accomodations in the interest of concurrency. While Python presents problems with its open-endedness leading to paradigm clashes through third-party libraries, D suffers less from this as there isn’t much to count on outside the standard library, Phobos.
The strong suit of Haskell lies with its overtly functional design. It has Warp for an HTTP server, and there are implementations of OpenPGP and AES which can be integrated to start with. Its drawbacks mainly revolve around its unpopularity amongst the general public, and its build processes are more clunky than that of D. Thankfully, Haskell does not seem to suffer from performance problems, unlike most of its closest competitors Scala and Golang, and it does support C bindings to a similar extent as D. The main problem lies in what argument Haskell has in what it provides that cannot be found with Scala.
Like Python, JavaScript is a quite popular language, but it was born to develop the Worldwide Web. Using Node.js, server-side code can be written with rapid iteration, but care must be taken to work without npm, opting for manual dependency management. The async nature of JS can be difficult to apprehend sometimes, and concurrency support is also minimal.
It should be understood—especially in the open-source context of HALO—that we don’t have a paramount concern for popularity or the community of programming languages. Personally, I am well-equipped to deal with the inevitable people problems that will come up when dealing with JavaScript, as I am already avoiding ecosystem dependency for technical reasons besides. Cutting out people who obstruct the project is not a problem, even though JS is among the worst in presenting this issue for people.
At the end of the day, JavaScript was designed for the browser. Although Node has done wonders to make this work well on the server side, the language still has limitations simply because of how it was designed. Something more targeted to the class of problems HALO faces will generally outdo JS in the races, even gimped.
Before the rise of Node, Ruby on Rails filled much the same role JavaScript does on the server side, and with a lot of the same drawbacks, too. It also has an async event loop, minimal concurrency support, and a landfill packaging ecosystem, so it doesn’t have much besides its legacy and the language itself separating it from JS. Most of my criticisms of JavaScript apply the same to Ruby.
For HALO, the choice is Scala. I have given this a few days to simmer, bearing in mind everything explained above, and I have not found any other languages to be more appropriate.
It’s worth pointing out the largest criticisms of Scala—relating to thresholds of skill and competency—are not only non-issues for HALO as an open source project, but actually blessings in disguise: those who will be able to make contributions will be more skilled on the whole, and will be contributing better code in doing so. This benefit is also positively compounded by Scala’s functional, concise nature doing so much to minimise bugs in any case.
Until next time,
Άλέξανδερ Νιχολί