#sre

howdy, #hachyderm!

over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.

since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.

to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.

then, after we've flipped the switch and while DNS is propagating -- -- we can watch in real-time as different servers begin flipping over to the new provider.

we named the tool hachyboop and it's available publicly --> https://github.com/hachyderm/hachyboop

please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code.

attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.

#sre #devops #mastodon #infrastructure

Ugh slept like shit. The drama and stress from work is giving me insomnia. The boss who quit didn't give leave ANYTHING for my new boss to understand what I do with DevEx and incidents.

I had to bust my ass yesterday to basically hold the ground that I had worked extremely hard over the past four months to make it so that I could manage incidents. New boss wanted to yank me out of it and put me back on "SRE" infrastructure.

CTO says incidents are staying where they are, with Engineering. So I told him I want to transfer to Engineering.

And then I get the question "do want to be an SRE?"

Ridiculous. Makes me want to scream.

#SRE

Saw this very nice article on slack about making standalone #python scripts that use #uv to automatically create their own virtual environments and install their dependencies there.

https://thisdavej.com/share-python-scripts-like-a-pro-uv-and-pep-723-for-easy-deployment/

#sre @sre #devops @devops

I am curating the Supply Chain Security track at #Rootconf2025!

Got stories, tools, or lessons from the trenches? Come speak — or just show up and learn.

hasgeek.com/rootconf/2025/

#DevSecOps #SBOM #CI_CD #InfoSec #DevOps #OpenSource #CloudSecurity #SupplyChainSecurity #SRE

There'd be a lot less whining from devs about 'needing' root in prod if K & R had named that account janitor

@sre @devops #sre #devops

Okay, I've been asked this question to which I was not prepared: where do you go if you want to learn Kubernetes from the ground up? Consider a person who doesn't even know how to run "docker" in their own laptop, but they want to rise to a point where they can actually manage a Kubernetes cluster. Thoughts?
#HomeLab #Kubernetes #SysAdminLife #k8s #SRE

Anyone looking for a developer with crazy fullstack experience (back to the 1990s!), about two decades professional experience with both #Python and #Django, very solid with JS/frontend, some #SRE background, an absurd love for #Lisp, and who's an enthusiastic loony for new tech?

I'm your girl!

I'm looking to jump ship to somewhere much more fulfilling, and I'd love to check you out as long as you're diversity-friendly.

#fedihire #fedihired #getfedihired

System Administration

Week 8, The Simple Mail Transfer Protocol

In this video, we begin our discussion of E-Mail by looking at the components of the larger mail system (the Mail User Agent, Mail Transfer Agent, Mail Delivery Agent, Access Agent); we observe the packets involved in a simple #SMTP exchange and track an email from one system to the other, both through the logs and on the wire, before we then learn to speak SMTP via telnet(1).

https://youtu.be/Ai8rjqelwsI

#SysAdmin #DevOps #SRE

When I thought that, after several years, I knew a lot about Kubernetes...

I just found a new thing that blew my mind: transferring ownership of managed fields.

https://kubernetes.io/docs/reference/using-api/server-side-apply/#transferring-ownership

Fuck. My. Life.

#Kubernetes #DevOps #SRE #PlatformEngineer

My younger daughter wants to interview for a Lead Staff SRE position at a new company (but for an old boss). She'd like some ideas of what she might expect.

This is work she's currently doing at a large company, but due to the way her career has unfolded, she's never actually done a coding interview before.

There's 2 rounds, qualifying, and final.
Qualifying is leetcode stuff, data structures etc.
Final one is 4 parts, each one hour:

behavioral
Python/code review
system design
Linux

She's worried about coding for someone for the first time. And what algorithms/data structures they'll want (she doesn't have a formal CS background). She also isn't sure (nor am I) what the Linux segment would be likely to cover.

I know she has the skills and can talk through the problems well. She's survived lots of layoffs because she's the one who makes everything work and builds strong teams with focused goals. But now she's the last person left in her group and training folks in India, so the clock is ticking.

I can DM a link to the job description if that helps. Would rather not publish details publicly.

Thanks.

#SRE

Yes, #DevSecOps can work, if done right. #Cybersecurity is everyone's responsibility, dev to test to prod. Sharing is caring!

I know from leading some amazing #DevOps & #SRE teams. With the right people, attitudes, goals, training, incentives, metrics, tools, rewards, DevSecOps does wonders!

OH on slack:

every day a fresh yak to shave and while the ones from yesterday still have mullets

@sre @devops #sre #devops

OH:

If the computer can lie to me, it's moral to lie to it

#devops #sre @sre @devops #ai

howdy, folks - it's been a bit since our last #hachyderm infra check in.

stuff in motion:

- ditching #terraform cloud & tf for #opentofu and #atlantis. we are just about to import our dev environment and put it through its paces.
- bringing #postgresql under ansible management. the team has been doing awesome work, and we've started to spin up dev nodes using the new playbooks. soon: production!
- moving #DNS zones away from AWS route 53. we chose bunny DNS as our provider and have been doing basic tests in dev. we'll likely prep our records for production this week with a plan for a cutover in one of the coming weekends.

and if you filled out our volunteer form and haven't heard from me in a bit - you're still on the list. we'll onboard a new batch of folks in the next couple of weeks.

@hachyderm

#devops #sre #mastoadmin

Life of #SRE as a Salesperson

https://tech.trivago.com/post/2024-12-20-life-of-sre-as-a-salesperson

My employer is an EFF-approved independent ISP, and we're hiring a Senior Systems Engineer https://grnh.se/a6b53dbd1us
Key technologies involved: Linux (RedHat), MySQL/MariaDB, VMware,Proxmox, Ansible, Perl, Python and Shell.

Salary starts at $145k. Can be fully remote (within US), but preference if able to work in our northern california office one day a week.

See link for details (and better accuracy).
#syseng #sysadmin #sre #devops #hiring

Bueno, venga, va, nunca he hecho #presentación a pesar de que me he mudado más veces de las que puedo contar.

Soy #DevOps, #SRE o #PlatformEngineer, depende de cómo vaya el día. Mi trabajo no me define, pero me flipan los ordenadores desde lechón, así que un poco si.

Me gusta la broma y la chanza, pero si te ofendo con algún chiste, avísame para cambiarlo.

No me interesaba la política hasta que LOS PUTOS FASCISTAS me obligaron.

Para lo demás, la bio, o pregunta.

System Administration

Week 7, HTTP and CDNs

After discussing the DNS, we now move on to #HTTP and HTTPS. While we don't have videos for these sections, hopefully the lecture slides can help you get an idea of what we're covering there. We review the basic HTTP protocol, peek at #QUIC and H3, and talk about load balancing and content delivery networks:

https://stevens.netmeister.org/615/07.pdf

#SysAdmin #DevOps #SRE

OH on slack, re why some things never come out of backlog

We can't say "your crap is not important enough". We say "look there are more important/urgent tasks right now. We'll keep yours on the backlog and come back to them"

@sre @devops #sre #devops

oh on slack:

code isn't an investment, it's a liability. It rots and requires maintenance for new security vulns, to work with the latest software, etc.

#devops #sre @devops @sre