Writing A Job Runner (In Elixir) (Again) (10 years later)

5/23/2025 at 2:12:04 PM

FYI if anyone is looking for a production ready job runner in Elixir, I strongly recommend Oban. They have a paid pro version but the open source version is still absolutely fantastic.

by sbrother

5/23/2025 at 2:21:30 PM

Oban is great! This is definitely just met to be a learning exercise. But it could go quite a ways on its own in prod.

by rhgraysonii

5/23/2025 at 2:17:17 PM

Just to add to the parent the Open Source version and the Pro Version just differs in that the Pro Version has a few plugins that the normal version has (for example workflows where you can have multiple workers working together) but it is not something you need for most use-cases.

by victorbjorklund

5/23/2025 at 3:57:29 PM

It's more than just a few - even more basic things like rate limiting or concurrency controls are gated behind Pro. It works extremely well, but I've been reluctant to use it in open source projects because there's quite a bit in there I'd need to rebuild.

by pikdum

5/23/2025 at 5:20:19 PM

im curious about your use case -- it seems weird (to me) to use it in an open source project unless its some kind of turnkey full app -- is there a way to just release it and encourage people to bring their own oban keys? that way it looks good for the elixir ecosystem that it has found a way to support hybrid open source libraries and expands the obam ecosystem

by throwawaymaths

5/23/2025 at 5:48:09 PM

My open source projects are generally more applications than libraries, yeah.

by pikdum

5/23/2025 at 6:01:43 PM

Do you worry about enshittification down the road?

by freedomben

5/23/2025 at 10:41:12 AM

I wrote this up after having written a tutorial on doing this in GenStage around a decade ago, and thought it was interesting to have the two of them side by side to consider. I linked the original in the document linked to here.

Overall, I'm wildly impressed at how this Elixir code held up, and it was a joy to revisit this.

by rhgraysonii

5/23/2025 at 6:14:53 PM

Great write up, thanks for sharing! Nothing against Oban, but it is nice seeing someone in the Elixir community not just say, "run Oban" and drop the mic

by freedomben

5/23/2025 at 3:57:21 PM

Years ago I built something similar, a Pub/Sub notification system: https://github.com/huqedato/qnotix. It is still running (with small modifications), in production, at my ex-customer.

Such a pity the industry (customers) reacts with skepticism every time I propose solutions based in Elixir/Erlang. I always hear: "Elixir, what? We want Java/.Net/Python/php"

by huqedato

5/23/2025 at 6:11:12 PM

I've run into this a few times before too. Sometimes discussing maintenance/ongoing costs and how minimal they are with an Elixir/Phoenix app can be compelling, though not always. I prefer to be over-transparent with clients so I'll usually add that if they plan to switch contractors often then it's probably not (tho sometimes still is) a good idea to go with Elixir. When I compare the amount of ongoing effort to keep a PHP, Node, or Python app running securely in prod with the ongoing effort with a Phoenix app, it's usually a better deal for them.

The one Danger Zone area though is when they want something between WordPress and a "real" app. A lot of the clients I've talked to already have WordPress and are really trying to stretch the framework to places it isn't really made to go. Those customers often think "I should be able to get a full app built for <=$10,000" and often shop around until they find someone who will accept it, and then they usually end up getting a bad/buggy product that is slow as dirt. At that point it's often too late as well because nobody wants to pay to start over. For those clients though I wouldn't recommend Elixir because they would almost rather spend more in maintenance costs than in initial development because the numbers feel smaller when spread out, even if they add up to more in the long run and with a much worse product.

That is not to say of course that any Elixir app is going to be better - there are bad Elixir devs just like any stack, though I think Elixir's relative lack of popularity actually leads to an overall more skilled dev pool.

by freedomben

5/23/2025 at 4:12:49 PM

It means they don't trust your judgement, aka you're just there to execute their plans. You're the floor guy, not the architect, owner, or developer of a house.

by jbverschoor

5/23/2025 at 4:20:16 PM

Not really. I am usually the one they pay to solve their problems. It's not a matter of trusting my judgement, rather they are trapped in an institutional/corporate mindset (old patterns are most suitable, 'best practice', 'reliable, proven tech' and such)

by huqedato

5/23/2025 at 5:03:29 PM

> I am usually the one they pay to solve their problems.

In the context that doesn't really say a lot, they pay the floor guy and the architect to solve problems, just different problems.

>they are trapped in an institutional/corporate mindset (old patterns are most suitable, 'best practice', 'reliable, proven tech' and such)

While true, it is also correct. From their perspective Elixir is an exotic material, the next "one they pay to solve their problems" is not going to know how to work with it, scrap it, and start over. Therefore the Elixir solution isn't a good solution for them.

As a business man once told me, "I never saw a feature I liked so much that I was wiling to pay for it twice."

by stonemetal12

5/23/2025 at 4:43:10 PM

The problem is finding other people to work in less mainstream languages. Probably not an issue right now, but during better times...

by icedchai

5/23/2025 at 1:48:35 PM

As someone who has been learning Elixir on and off for over a year, this looks really exciting on first skim through. Excited to give this a deep read this weekend!

by dangerousnight

5/23/2025 at 2:22:22 PM

If you have any feedback or anything is unclear feel free to open an issue. I am thinking I am going to take this and expand the concepts to start as a beginners primer going through the primary concepts in brief akin to Elixir School, and then expand it into building this and a web service that is using it and offering some real time features.

by rhgraysonii

5/23/2025 at 2:41:27 PM

I’d love to see one for Elixir built around the Postgres pgmq and pg_crom extensions similar to what pgflow is doing: https://www.pgflow.dev/

by cpursley

5/23/2025 at 5:59:02 PM

pgflow can be used with any language/runtime, I just started with TypeScript and Supabase, as that's what I'm using.

The worker is stateless and "dumb" by design (currently it runs on serverless functions) - it just calls SQL functions: "poll_for_tasks" to get some tasks from the queue and then either "complete_task" or "fail_task" after executing user code - that's it, nothing more, so it should be relatively easy to adopt it to other runtimes.

I have written a small architecture primer on pgflow if anyone is interested in its simple but flexible design https://www.pgflow.dev/concepts/how-pgflow-works/

by jumski

5/23/2025 at 2:49:19 PM

That would be super cool. Pretty much everything Supabase is up to is awesome.

by rhgraysonii

5/23/2025 at 3:00:29 PM

Yeah, their realtime thing is powered by Elixir (listens to Postgres WAL).

by cpursley

5/23/2025 at 2:58:45 PM

The absolute facility with which you can do distributed applications in Elixir is the big thing that sold me in the first place. Nice to see this tutorial!

by jamal-kumar

5/23/2025 at 3:04:01 PM

Incredibly well written. You clearly put a lot of thought and effort into this. Thank you.

Also FOR UPDATE SKIP LOCKED is interesting.

by hmmokidk

5/23/2025 at 3:30:17 PM

You also could use an advisory exact lock, which would serialize the order of input and force everyone to wait in line, but the advantage here is that we are allowed shared concurrent reads while maintaining our lock. Thank you for the feedback. It was fun to revisit after 10 years. The initial piece came after a Columbus, OH Ruby meetup that Jose attended. It was quite fun to see him put it together as a whole and I ran with things a bit further than his first ideas with this.

by rhgraysonii

5/23/2025 at 7:04:44 PM

SKIP LOCKED is amazing for this kind of thing. I used it to build a transaction outbox a few years ago.

One thing worth looking into if you do this in production is adding a way to add partitions such that each partition is single threaded. It’s the only way to guarantee ordering if your jobs are doing anything non-deterministic.

by sarchertech

5/23/2025 at 7:30:24 PM

We have a system where each pod spins up around 30 scheduled job instances of one job each processing a "partition", then transaction outbox is queried with hash of identifier equating it to partition.

We increased partition counts on sale days and it works well for us.

Couple of gotchas we had were.

1) Using hashtext from postgres is sketchy.

2) Increasing partiton count is an orchestra which requires stopping the partition.

by boruto