Skip to content

lab-6-conclusion.md

Latest commit

 

History

History
775 lines (576 loc) · 18.9 KB

lab-6-conclusion.md

File metadata and controls

775 lines (576 loc) · 18.9 KB

Lab 6: Putting It All Together

Overview

Congratulations on making it this far! You've built a complete "Workstation as Code" system. In this final lab, we'll bring everything together, demonstrate the complete workflow, and explore the power of what you've created.

Time Required

20-30 minutes

Learning Objectives

  • Understand how all three layers work together
  • Practice the complete workflow
  • Experience disaster recovery
  • Learn best practices
  • Understand how to maintain the system
  • Plan next steps for your own environment

What You've Built

Let's review the complete system you now have:

Three Git Repositories

  1. bluefin-custom: Your operating system definition

    • Bluefin base image + Wireshark
    • Custom system files (MOTD, bashrc)
    • Automated builds via GitHub Actions
    • Container image for updates

  2. gopass-store: Your encrypted secrets (all encrypted with AGE)

    • SSH keypair
    • GitHub Personal Access Token
    • Other secrets you added
    • Backed up to GitHub (safely encrypted)

  3. dotfile-store: Your user environment

    • ZSH configuration
    • SSH configuration
    • Git configuration
    • Templates that pull secrets from Gopass
    • Deploys with one command

The Three Layers

┌─────────────────────────────────────────┐
│  Layer 1: OS (Bluefin + BlueBuild)      │
│  - System packages                      │
│  - System configuration                 │
│  - Atomic updates                       │
│  - Built: GitHub Actions                │
│  - Updated: rpm-ostree                  │
└─────────────────────────────────────────┘
                ↓
┌─────────────────────────────────────────┐
│  Layer 2: Secrets (Gopass + AGE)        │
│  - Encrypted with AGE                   │
│  - Stored in Git                        │
│  - Accessed via CLI                     │
│  - Provides: SSH keys, tokens, etc.     │
└─────────────────────────────────────────┘
                ↓
┌─────────────────────────────────────────┐
│  Layer 3: Dotfiles (Chezmoi)            │
│  - User configuration                   │
│  - Templated for flexibility            │
│  - Pulls secrets from Gopass            │
│  - Deployed to home directory           │
└─────────────────────────────────────────┘

Exercise 1: The Complete Workflow

Let's walk through making a change at each layer.

1.1 Change the OS Layer

Add a new package to your OS:

# On your host machine (or in a new terminal)
$ cd ~/bluefin-custom  # Or wherever you cloned it

# Edit recipe.yml
# Add a new package to the install list

Edit recipes/recipe.yml:

modules:
  - type: dnf
    ...
    install:
      ...
      - htop  # Add this new package
# Commit and push
$ git add recipes/recipe.yml
$ git commit -m "Add htop package"
$ git push

# Go to GitHub Actions and watch the build
# Build time: ~10-20 minutes

1.2 Change the Secrets Layer

Add a new secret (all commands below are performed inside your VM):

# Add a new API key
$ gopass insert services/example-api-key
# Enter a fake API key for demonstration

# Commit to Git (gopass does this automatically, but verify)
$ gopass sync

# Verify you can retrieve it
$ gopass show services/example-api-key

1.3 Change the Dotfiles Layer

Add a new alias to your ZSH config:

# Edit through chezmoi
$ chezmoi edit ~/.zshrc

# Add a new alias (append to file):
# alias weather='curl wttr.in'

# See what changed
$ chezmoi diff ~/.zshrc

# Apply the change
$ chezmoi apply ~/.zshrc

# Commit and push
$ chezmoi git add dot_zshrc
$ chezmoi git -- commit -m "Add weather alias"
$ git push

# Test it immediately
$ source ~/.zshrc
$ weather  # Should fetch weather (if internet is available)

1.4 Update Your VM

Now let's apply all the changes:

# Update OS to get htop
$ rpm-ostree upgrade
# Wait for download
$ systemctl reboot

# After reboot, verify htop is installed
$ which htop
$ htop  # Should launch

# Update dotfiles (pull latest from Git)
$ chezmoi update 
# This pulls from Git and applies

# Verify weather alias
$ weather

All three layers updated from Git!

Exercise 2: Disaster Recovery

This is where the magic happens. Let's simulate losing your machine and recovering.

2.1 The Scenario

Disaster: Your laptop is stolen/broken/corrupted.

Recovery plan:

  1. Get a new machine (or new VM in our case)
  2. Install Bluefin (original ISO)
  3. Rebase to your customer image
  4. Restore AGE key (from secure backup)
  5. Clone Gopass store
  6. Clone and apply Chezmoi dotfiles
  7. Back to productivity in < 1 hour

2.2 Simulate New Machine

Create a second VM or a new toolbox:

# Create a new toolbox to simulate a new machine
$ toolbox create recovery
$ toolbox enter recovery

# You're now on a "new machine" with nothing configured

2.3 Bootstrap the Recovery

Install base tools:

# Install required tools
$ sudo dnf install -y git age gopass chezmoi

# Verify
$ git --version
$ age --version
$ gopass --version
$ chezmoi --version

2.4 Restore AGE Key

In a real disaster, you'd retrieve your AGE key from your secure backup.

$ gopass age identities add
Enter the age identity starting in AGE-: <PASTE YOUR AGE SECRET KEY HERE>
Enter your PIN: <PASTE YOUR PASSPHRASE HERE>
Retype your PIN: <PASTE YOUR PASSPHRASE HERE>
⚠ New age identities are not automatically added to your recipient list, consider adding it using 'gopass recipients add age1m9nkvjx2hns6rha2pr62cmm323yacd8kmyued3uq9q5n9fy82guq38lym4'
⚠ If you do not add this recipient to the recipient list, make sure to re-encrypt using 'gopass fsck --decrypt' to properly support this identity

2.5 Restore Gopass Store

$ GIT_SSH_COMMAND="ssh -i ~/gopass-deploy-key -o IdentitiesOnly=yes" \
gopass --yes setup --remote git@github.com:YOUR_USERNAME/gopass-store.git \
 --name "James Freeman" --email "sysadmin@example.com" --crypto age

   __     _    _ _      _ _   ___   ___
 /'_ '\ /'_'\ ( '_'\  /'_' )/',__)/',__)
( (_) |( (_) )| (_) )( (_| |\__, \\__, \
'\__  |'\___/'| ,__/''\__,_)(____/(____/
( )_) |       | |
 \___/'       (_)

🌟 Welcome to gopass!
🌟 Initializing a new password store ...
🔐 Using crypto backend: age
💾 Using storage backend: gitfs
Enter your PIN:
Joining existing team ...
🌟 Configuring your password store ...
Enter your PIN:
✅ Configuration written
[gopass]Configuring git remote ...
[gopass]Cloning from the git remote ...
The authenticity of host 'github.com (20.26.156.215)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
[gopass]✅ Done. Joined Team "gopass"
[gopass]⚠ You still need to request access to decrypt secrets!

If you can see your secrets, Gopass is restored!

2.6 Restore Dotfiles with Chezmoi

This is the best part - one command:

# Initialize chezmoi from your repository and apply in one command
$ chezmoi init --apply git@github.com:YOUR_USERNAME/dotfile-store.git

That's it! Chezmoi:

  1. Cloned your dotfiles repository
  2. Evaluated all templates
  3. Retrieved secrets from Gopass
  4. Deployed everything to your home directory

2.7 Verify Recovery

Check that everything was restored:

# Check ZSH config
$ cat ~/.zshrc
$ source ~/.zshrc
# Should see your welcome message

# Check SSH keys
$ ls -la ~/.ssh/
$ cat ~/.ssh/workshop_id_ed25519.pub

# Check Git config
$ git config --global user.name
$ git config --global user.email

# Check SSH config
$ cat ~/.ssh/config

Everything is back! And it took < 10 minutes.

2.8 Recovery Time Comparison

Traditional approach:

  • Reinstall OS: 1-2 hours
  • Install packages: 1-2 hours (finding all the ones you had)
  • Configure settings: 2-4 hours (trying to remember what you changed)
  • Restore dotfiles: 1-2 hours (if you even have backups)
  • Total: 6-10 hours, and you'll still forget things

Workstation as Code:

  • Install OS from custom ISO: 30 minutes
  • Restore AGE key: 2 minutes
  • Clone Gopass store: 1 minute
  • Apply Chezmoi dotfiles: 2 minutes
  • Total: ~35 minutes, and everything is identical

Exercise 3: Multi-Machine Scenario

The real power comes when managing multiple machines.

3.1 Machine-Specific Configuration

Edit your Chezmoi dotfiles to handle multiple machines:

$ chezmoi edit ~/.zshrc

Add machine-specific configuration:

# At the end of the file, add:

# Machine-specific configuration
{{- if eq .chezmoi.hostname "workstation-as-code-lab" }}
# Work machine specific
export WORK_MODE=true
alias connect-vpn='echo "Connecting to work VPN..."'
{{- else if eq .chezmoi.hostname "personal-laptop" }}
# Personal machine specific
export PERSONAL_MODE=true
alias games='echo "Launching Steam..."'
{{- end }}

Apply and commit:

$ chezmoi apply ~/.zshrc
$ chezmoi git add dot_zshrc
$ chezmoi git -- commit -m "Add machine-specific configuration"
$ chezmoi git push

3.2 Shared Base, Custom Extensions

This allows you to:

  • Share base configuration across all machines
  • Have machine-specific customizations
  • Maintain one repository for all machines

3.3 Example: Work vs. Personal

dotfiles repo (shared)
    ├── dot_zshrc.tmpl (base + conditionals)
    ├── dot_gitconfig.tmpl (base + conditionals)
    └── private_dot_ssh/
            ├── config (base)
            └── work-specific keys (conditionally deployed)

Best Practices

OS Layer (Bluebuild)

Do:

  • Keep recipe.yml simple and well-commented
  • Test builds before deploying to main machine
  • Use semantic commit messages
  • Pin versions for critical packages
  • Document why packages are included

Don't:

  • Add packages you don't need (bloat)
  • Make changes without testing
  • Forget to update documentation
  • Ignore build failures

Secrets Layer (Gopass)

Do:

  • Back up your AGE key to multiple secure locations
  • Use strong, unique passwords
  • Regularly sync to Git
  • Audit your secrets periodically
  • Use descriptive secret names

Don't:

  • Share your AGE key
  • Commit AGE key to Git
  • Use public repositories for Gopass store
  • Store plaintext secrets anywhere
  • Forget to test recovery

Dotfiles Layer (Chezmoi)

Do:

  • Use templates for dynamic content
  • Pull secrets from Gopass, never hardcode
  • Test on a new machine/toolbox regularly
  • Comment complex template logic
  • Version control everything

Don't:

  • Hardcode secrets in templates
  • Make dotfiles OS-specific without conditionals
  • Forget to commit changes
  • Overcomplicate with too many templates
  • Skip testing chezmoi apply

Maintenance Workflow

Daily

# Update OS (happens automatically in background)
# rpm-ostree downloads updates automatically

# Add new secrets as needed
$ gopass insert path/to/secret

# Make dotfile changes
$ chezmoi edit ~/.someconfig
$ chezmoi apply

Weekly

# Sync everything to Git
$ chezmoi git push
$ goass sync
$ cd ~/bluefin-custom && git push

# Apply OS updates (reboot)
$ rpm-ostree status  # Check for staged updates
$ systemctl reboot  # Apply if needed

# Update dotfiles from Git
$ chezmoi update

Monthly

# Review and clean up
$ gopass ls  # Remove old/unused secrets
$ chezmoi managed  # Review managed files

# Test recovery
# Try restoring in a new toolbox/VM

# Update base image version
# Edit recipe.yml: image-version: 41 (when new Fedora releases)

Quarterly

# Audit secrets
$ gopass ls  # What secrets do you have?
# Remove old ones, update passwords

# Review packages
# Edit recipe.yml - remove unused packages

# Review dotfiles
# Clean up old configurations

# Test full disaster recovery
# Create a new VM and recover from scratch

Going to Production

For Your Daily Driver

Ready to use this on your main machine?

Before you commit:

  1. Test thoroughly in VM
  2. Backup current machine (traditional backup)
  3. Document current setup
  4. Have recovery plan ready

Migration path:

  1. Start with dotfiles only (lowest risk)

    • Initialize Chezmoi on current machine
    • Add existing dotfiles
    • Test apply workflow

  2. Add secrets gradually

    • Set up Gopass
    • Migrate passwords one at a time
    • Keep old password manager until confident

  3. Consider custom OS last

    • Most disruptive change
    • Test thoroughly in VM first
    • Plan maintenance window
    • Have rollback plan

Team Usage

Want to share with your team?

OS Images:

  • Fork base image repository
  • Team members customize their own forks
  • Share common packages in base repository

Secrets:

  • Individual Gopass stores (private)
  • Shared team stores (for shared credentials)
  • Use Gopass team features for selective sharing

Dotfiles:

  • Fork common dotfiles repository
  • Personal customizations in individual repos
  • Share useful templates back to team repo

Troubleshooting the Stack

When Something Breaks

OS won't boot:

# At GRUB menu, select previous deployment
# System will rollback automatically

# Or from running system:
$ rpm-ostree rollback
$ systemctl reboot

Gopass secrets inaccessible:

# Check AGE key exists
$ ls ~/.config/gopass/age/identities

# Check Gopass config
$ gopass config

# Re-initialize if needed
$ gopass init --crypto age

Dotfiles not applying:

# Check for errors
$ chezmoi apply -v  # Verbose mode

# See what would change
$ chezmoi diff

# Check template syntax
$ chezmoi execute-template "{{ .name }}"

Debug Mode

Enable debug output:

# Chezmoi
$ chezmoi apply --verbose

# Gopass
$ gopass --debug show secret

# RPM-ostree
$ rpm-ostree status --verbose

What You've Learned

Technical Skills

  • Container-based OS images with BlueBuild
  • Modern encryption with AGE
  • Password management with Gopass
  • Dotfile templating with Chezmoi
  • Git-based configuration management
  • Infrastructure as Code principles

Concepts

  • Declarative system configuration
  • Atomic updates and rollbacks
  • Secret management and encryption
  • Template-based configuration
  • Disaster recovery planning
  • Multi-machine management

Mindset Shift

  • From pets to cattle
  • From imperative to declarative
  • From fear to confidence
  • From manual to automated
  • From undocumented to version-controlled

Beyond the Workshop

Ideas to Explore

Expand your OS image:

  • Add development tools
  • Include fonts and themes
  • Configure system services
  • Add automatic update schedules

Enhance secret management:

  • Add more secrets (GPG keys, SSL certs)
  • Set up team sharing
  • Integrate with other tools
  • Automate secret rotation

Advanced dotfiles:

  • Machine-specific configurations
  • Conditional package installation
  • Complex templating logic
  • Integration with more tools

Automation:

  • Automatic builds on schedule
  • Automated testing of images
  • CI/CD for dotfiles
  • Automated recovery scripts

Community and Resources

Join the Community

Get Help

  • GitHub Discussions for each project
  • Discord servers (Bluefin, Universal Blue)
  • Reddit: r/Fedora, r/linuxadmin
  • Your questions help improve documentation!

Share Your Setup

  • Blog about your experience
  • Share your dotfiles (public repo)
  • Contribute improvements back
  • Help others get started

Final Challenge

Want to test your knowledge? Try this:

The Ultimate Test

  1. Document your current setup

    • What packages do you use daily?
    • What configurations are critical?
    • What secrets do you need?

  2. Build your real environment

    • Create your actual custom OS image
    • Set up your actual secrets
    • Migrate your actual dotfiles

  3. Test disaster recovery

    • Restore in a fresh VM
    • Time how long it takes
    • Document any issues

  4. Refine and iterate

    • Fix what didn't work
    • Add what you missed
    • Simplify what's complex

  5. Go to production

    • Deploy to your actual machine
    • Use it for a week
    • Refine based on experience

Conclusion

You now have the skills to:

  • Define your entire workstation as code
  • Recover from disasters in minutes
  • Deploy to new machines effortlessly
  • Version control every configuration
  • Maintain consistency across machines
  • Never manually configure a system again

The Power You Hold

Three Git repositories. That's it.

Those three repositories contain:

  • Your entire operating system definition
  • Every secret you need, encrypted safely
  • All your personal configurations

Clone those three repositories on any machine, and in less than an hour, you have your complete environment. Identical. Reproducible. Tested.

The Confidence You Gain

Breaking things is no longer scary. Trying new configurations is safe. New machines are no longer painful. Your computing environment is no longer a house of cards.

You have the same confidence in your workstation that you have in your production infrastructure.

Thank You

Thank you for attending this workshop at cfgmgmtcamp 2026! We hope you leave empowered to take control of your computing environment.

Your laptop is no longer a pet. It's cattle. And that's a good thing.

Verification Checklist

  • Understand how all three layers work together
  • Can modify OS image and deploy updates
  • Can add/retrieve secrets with Gopass
  • Can modify and apply dotfiles with Chezmoi
  • Successfully completed disaster recovery exercise
  • Understand best practices for maintenance
  • Know where to get help and learn more
  • Have ideas for extending your own setup

What's Next?

  1. Keep your workshop environment as a reference
  2. Plan your production migration on your own time
  3. Join the communities for support
  4. Share your experience and help others
  5. Iterate and improve your setup over time

Previous: Lab 5: Managing Dotfiles with Chezmoi Next: Appendix: Going Further

Workshop Feedback

We'd love to hear from you!

  • What worked well?
  • What was confusing?
  • What would you like to see added?
  • Will you use this in production?

Find us:

Thank you and happy automating!
⚡