Index (RSS, JSON) | About | Projects
Southern California Linux Expo 20x

Southern California Linux Expo 20x

My Experience at the Southern California Linux Expo (SCaLE 20x)

This originally started as a presentation I made for work after visiting the SCaLE 20x in March of 2023. I santized this of work-specific information so that the notes could be published here and made useful. Then I threw in some background of my time in Pasadena (or well... todo anyway).

Intro picture

What is SCaLE 20x?

What is SCaLE? It's an event in Pasadena, CA going on for 20 years, although it's not really a “Linux” expo. 20x was a four-day event beginning Thursday (03/09) and ending Sunday (03/12).

Multiple “sub-conferences” took place:

lobby

What is at SCaLE?

image redacted until I can replace my friends' faces with Guy Fawkes masks

Weekday Talks

conference building

Thursday

Friday

Embedded Systems Workshop

embedded

Three instructors and one's of them kid led embedded talks. We were supposed to get Raspberry Pis, but we got BeagleBoards (BeagleBone) due to supply chain issues. This altered our schedule.

1st lesson: – info dump on theory

2nd lesson: – History and theory of u-boot (v. GRUB2, syslinux, etc.)

3rd and 4th lessons: – low-level board input and output

I also had the privilege of meeting a guy who has been going to SCALE for 17 years and what he has seen. The guy won prize two years in a row.

Kubernetes and Homelab B.O.F

homelab

Professional evaluation of homelabs:

The instructor's network:

I asked about his running his own dev solution. He said he's done enough of that for CI/CD stuff. I also asked for his network visualization solution. He keeps it simple but that was still a TODO… No recommendation.

Weekend Talks

intro

Saturday

Sunday

What Nix Can Do

NixOS

Nix is lately a buzzword in the Linux world… which means and does a lot of things… or it’s a meme. Nix is being rapidly adopted as a solution for trending "immutable systems" or "immutable distros". Nix is a secondary package manager and scripting language (which the build system is written in).

NixOS is a Unix distribution that uses Nix as its system package manager. It is touted as having the highest security of any Unix distribution.

Why is this relevant?

Devcontainers

devcontainers

It is a standard pushed by Microsoft, with support in IDEs and editors; VSCode, Jetbrains tools, vim, etc. Personally, some of our team have used them at our work.

Essentially it is a JSON file with three different fields:

They keeps the development environment more tracked and reproducible, and allows quick spin up and contributions.

(left off at slide 9)

EBPF: Superpowers for Security

eBPF

"What JavaScript is to the Browser, eBPF is to the kernel."

eBPF: extended Berkeley Packet Filter

This does more than its original namesake:

Bpf() system call loads compiled (bytecode), sends it to the ePBF verifier (in kernel), and then to the eBPF JITc. There are mulitple hooks at runtime:

Some examples of eBPF project-relevance and business are:

Linus was actually impressed. He probably has fewer merge requests to approve.

Your Infrastructure Needs to D.I.E

D.I.E. supersedes traditional C.I.A. where:

D.I.E. is distributed with “cattle” like containers, whenever possible, orchestrated with k8s, Helm, and other tools.

It is immutable in that no internal state of distributed nodes changes. Everything is RO. State is stored only in specially selected “pet” nodes (typically VMs) that have rock-hard security

It is ephemeral in that any node can be destroyed at any time and the deployment can continue to function and self-heal.

Bamboozle layer

If an attacker tampers with a node, the container is destroyed without explanation. One example combined an eBPF program to watch for any forbidden system calls and trigger death. The buzzword for this is “Security Chaos Engineering”

Highlights and Takeaways

A Jenkins plugin, “Lava”, for PXE booting with u-boot may be immediately useful for infrastructure projects.

A BeagleBone could be setup in the lab as embedded training platform if desired and permitted.

Devcontainers or Nix scripting/packaging (or GitPod, etc.) may be used per repo. It decreases spin-up time, captures development dependencies, and increases reproducibility. Bitbucket (and probably GitLab) may have a plug-in for this.

For intermittent bugs/failures during signal processing, eBPF programs may monitor and act on kernel operations. This is especially relevant when containerized workloads are depending on a shared kernel.

While migrating and upgrading various machines, VMs, and containers to a more regular k8s deployment, D.I.E architecture may enhance lab deployment and security.

Vault is a good choice for storing secrets (see: Confidential Story of Well-kept Secrets). There were additional, "exotic" resources mentioned during the Upscale Talks:

Exhibit Hall

exhibit hall

This was a warehouse-sized room of many, many booths ran from mid-Friday to mid-Sunday.

There were various organizational presence from:

Current and popular tech being pushed IRL:

The new technology here was: eBPF (Isovalent), Nix, NixOS

I got neat stuff here as well. :)

Game Night

Game Night

On Saturday, SCaLE hosted a game night. It was a good break from all the technical discussion. There were actually casino games, escape rooms, air hockey, billiards, cornhole, and arcade cabinets. There was also a retro computer showcase (interactive too!). There was also free food. :)

This was a new event to 20x to encourages nerds to mix and try new things, and it was also a family friendly event!

I had the opportunity to network with some folks and talk high-level about the company. I was surprised that people recognized each other despite its being an annual event.

Lessons Learned

Planning

Some industry/technical terms:

General lessons for travel and conference newbies:

Appendix

Thursday Talks

Friday Talks

Saturday Talks

Sunday Talks

Other Weekday Talks of Interest

Thursday:

Other Weekend Talks of Interest

Saturday: