How Blacksmith took CI off our plate

PyVista’s documentation includes thousands of examples that run real workflows and render 3D scenes. The rendered outputs (generated figures) of these examples are checked against an established baseline for every addition to the code base (as a PR check).

The hundreds of gallery examples and thousands of API docstring examples now provide a critical test corpus that, over time, has ensured PyVista’s API stability and given the researchers who rely on PyVista in their workflows confidence that figures and results generated across versions of PyVista are reproducible. This stability and reproducibility were at the heart of why we started PyVista: the same guarantees weren’t in place in the underlying graphics layer (more on this in a later blog post).

As the corpus of API examples and workflow examples grew to cover all the ways PyVista’s community uses the tools to solve multidisciplinary 3D questions, the open source infrastructure we relied on for pull requests was stretched thin. On GitHub’s standard hosted runners, a full docs build ran well over an hour. Back in 2023 we were already talking about waiting 90 minutes for CI checks. Add a queue and a round of review, and merging a contributor’s pull request could take a day or two, even for simple typo fixes while on the free tier and dealing with concurency limits.

Since PyVista had been an entirely volunteer effort with no corporate sponsorship or endorsements, we didn’t have the resources to pay for better infrastructure.

Disclosure: I’m a PyVista co-creator and now a co-founder of CoDimensional. CoDimensional has chosen to provide commercial stewardship for PyVista moving forward, which includes financial contributions like paying for CI infrastructure.

Self-hosted CI runners worked for a while

In 2025, Alex Kaszynski, fellow PyVista co-creator and core developer, stood up self-hosted runners on his own hardware. It helped, and an uncached docs build dropped from 1h 18m to 34m (#7848).

It also put the cost and the upkeep solely on Alex. Those machines needed patching, runners needed to stay registered, and when a server went offline, every open pull request waited on it or fell back to GitHub’s free-tier infrastructure that couldn’t keep up. There were also security concerns and many other complications (which I’ll let Alex write about his own experiences).

CI is a solved problem. What are we doing?

This past spring, one of those servers went down again, and I went looking for a better answer. At this point in my career, I had seen countless incredibly sophisticated CI pipelines orchestrated on any number of platforms (CircleCI, Jenkins, Azure, etc.) or even full scientific workflows run as part of CI on platforms like Neptune. All to say, this is a solved problem, and we don’t need to be maintaining a server in Alex’s basement… we have better things to put our brainpower on.

I’d seen Blacksmith come up on ThePrimeagen’s channel, so I thought I’d toy around with it.

The migration (#8706) came down to changing the runner label in our workflows:

runs-on: blacksmith-8vcpu-ubuntu-2204

In our test, the same docs build took 62 minutes on a standard GitHub-hosted runner. Today it finishes in about seven minutes on Blacksmith in our merge queue. The improvements include optimization work from the PyVista community, but we couldn’t have come anywhere close to this on GitHub’s standard runners, and none of it depends on anyone keeping hardware alive.

To be fair to GitHub, they also offer larger runners with more cores and memory. At list price, Blacksmith’s 8-vCPU runner still costs less per minute than GitHub’s 8-core runner ($0.016 vs. $0.022), and it runs on bare-metal gaming CPUs with high single-core performance, which Blacksmith says makes jobs about twice as fast. A faster runner at a lower rate made the choice easy.

Bringing it to CoDim

That experience is why CoDimensional’s CI runs on Blacksmith. We ship a lot of code across more than 50 internal repositories, and we take testing, correctness assurances, performance guarantees, and regression prevention incredibly seriously. So every change and upgrade across our stack runs full test suites (many of which include the same kinds of rendering baselines as PyVista’s).

We started out on paid GitHub runners and quickly found that CI turnaround decides how fast we can move and serve our customers.

What has mattered most to us:

• Drop-in setup. Moving a workflow over is a one-line change, and moving it back is the same line. We have some other hosted hardware we rely on, and being able to quickly change a variable in our GitHub organization to coordinate across those 50 repositories is a massive productivity boost.
• Fast feedback. Jobs start quickly and finish quickly, so we can stay focused, finish the task, and move on to the next thing.
• Visibility. Blacksmith reports test failures directly on the pull request and shows where the time goes in each job, which made it much easier for us to trim our own pipelines.
• A team that answers. When we’ve had questions, the Blacksmith team has been quick to help, whether that meant setting up a Slack channel or answering email.

If your CI is holding you back, we highly recommend swapping the runner label on one workflow and seeing what Blacksmith can do.