|
| 1 | +puppetlabs/puppetdb - PuppetDB Management |
| 2 | +----------------------------------------- |
| 3 | + |
| 4 | +Purpose: Install and manage the PuppetDB server and database, and |
| 5 | + configure the Puppet master to use PuppetDB |
| 6 | +Module: puppetlabs/puppetdb (http://forge.puppetlabs.com/cprice404/puppetdb) |
| 7 | +Puppet Version: 2.7+ |
| 8 | +Platforms: RHEL6, Debian6, Ubuntu 10.04 |
| 9 | + |
| 10 | +One of the new projects that we at Puppet Labs are excited about right now is |
| 11 | +PuppetDB, our new “data warehouse” for managing storage and retrieval of all |
| 12 | +platform-generated data. (If you haven’t checked it out yet, have a look at |
| 13 | +[Nick Lewis’ blog |
| 14 | +post](http://puppetlabs.com/blog/introducing-puppetdb-put-your-data-to-work/) or |
| 15 | +the [PuppetDB documentation](http://docs.puppetlabs.com/puppetdb/).) Currently, |
| 16 | +it offers a huge performance improvement for exported and collected resources, |
| 17 | +as well as several other great features. We’re even more excited about some of |
| 18 | +the not-quite-released functionality that is in the pipeline, so stay tuned for |
| 19 | +more information! |
| 20 | + |
| 21 | +Installing and configuring PuppetDB isn’t *too* difficult, but we knew that it |
| 22 | +could and should be even easier than it was. That’s where the new |
| 23 | +`puppetlabs/puppetdb` module comes in. Whether you just want to throw PuppetDB |
| 24 | +onto a test system as quickly as possible so that you can check it out, or you |
| 25 | +want finer-grained access to managing the individual settings and configuration, |
| 26 | +this module aims to let you dive in at exactly the level of involvement that you |
| 27 | +desire. |
| 28 | + |
| 29 | +Here are some of the capabilities of the new 1.0 release of the `puppetdb` |
| 30 | +module; almost all of these are optional, so you are free to pick and choose |
| 31 | +which ones suit your needs: |
| 32 | + |
| 33 | +* Installs and manages the core PuppetDB server |
| 34 | +* Installs and manages the underlying database server (PostgreSQL or a simple |
| 35 | +embedded database) |
| 36 | +* Configures your Puppet master to use PuppetDB |
| 37 | +* Optional support for opening the PuppetDB port in your firewall on |
| 38 | +RedHat-based distros |
| 39 | +* Validates your database connection before applying PuppetDB configuration |
| 40 | +changes, to help make sure that PuppetDB doesn’t end up in a broken state |
| 41 | +* Validates your PuppetDB connection before applying configuration changes to |
| 42 | +the Puppet master, to help make sure that your master doesn’t end up in a broken |
| 43 | +state |
| 44 | + |
| 45 | +Installing the module |
| 46 | +--------------------- |
| 47 | + |
| 48 | +Installing the PuppetDB module is a breeze using the Puppet module tool |
| 49 | +(available in Puppet 2.7.14+ and Puppet Enterprise 2.5+): |
| 50 | + |
| 51 | + $ puppet module install puppetlabs/puppetdb |
| 52 | + puppet module install puppetlabs/puppetdb |
| 53 | + Preparing to install into /etc/puppet/modules ... |
| 54 | + Downloading from http://forge.puppetlabs.com ... |
| 55 | + Installing -- do not interrupt ... |
| 56 | + /etc/puppet/modules |
| 57 | + └─┬ puppetlabs-puppetdb (v0.1.1) |
| 58 | + ├── cprice404-inifile (v0.0.2) |
| 59 | + ├─┬ inkling-postgresql (v0.3.0) |
| 60 | + │ └── puppetlabs-stdlib (v3.0.1) |
| 61 | + └── puppetlabs-firewall (v0.0.4) |
| 62 | + $ |
| 63 | + |
| 64 | +Resource Overview |
| 65 | +----------------- |
| 66 | + |
| 67 | +Let’s take a quick peek at the main classes and types defined by the module. |
| 68 | +(We’ll take a more in-depth look, with examples, in the following section.) |
| 69 | + |
| 70 | +##### `puppetdb` class |
| 71 | + |
| 72 | +This is a sort of ‘all-in-one’ class for the PuppetDB server. It’ll get you up |
| 73 | +and running with everything you need (including database setup and management) |
| 74 | +on the server side. The only other thing you’ll need to do is to configure your |
| 75 | +Puppet master to use PuppetDB... which leads us to: |
| 76 | + |
| 77 | +##### `puppetdb::master::config` class |
| 78 | + |
| 79 | +This class should be used on your Puppet master node. It’ll verify that it can |
| 80 | +successfully communicate with your PuppetDB server, and then configure your |
| 81 | +master to use PuppetDB. |
| 82 | + |
| 83 | +***NOTE***: Using this class involves allowing the module to manipulate your |
| 84 | +puppet configuration files; in particular: `puppet.conf` and `routes.yaml`. The |
| 85 | +`puppet.conf` changes are supplemental and should not affect any of your existing |
| 86 | +settings, but the `routes.yaml` file will be overwritten entirely. If you have an |
| 87 | +existing `routes.yaml` file, you will want to take care to use the `manage_routes` |
| 88 | +parameter of this class to prevent the module from managing that file, and |
| 89 | +you’ll need to manage it yourself. |
| 90 | + |
| 91 | +##### `puppetdb::server` class |
| 92 | + |
| 93 | +This is for managing the PuppetDB server independently of the underlying |
| 94 | +database that it depends on; so it’ll manage the PuppetDB package, service, |
| 95 | +config files, etc., but will allow you to manage the database (e.g. postgresql) |
| 96 | +however you see fit. |
| 97 | + |
| 98 | +###### `puppetdb::database::postgresql` class |
| 99 | + |
| 100 | +This is a class for managing a postgresql server for use by PuppetDB. It can |
| 101 | +manage the postgresql packages and service, as well as creating and managing the |
| 102 | +puppetdb database and database user accounts. |
| 103 | + |
| 104 | +##### Low-level classes |
| 105 | + |
| 106 | +There are several lower-level classes in the module (e.g., `puppetdb::master::*` |
| 107 | +and `puppetdb::server::*` which you can use to manage individual configuration |
| 108 | +files or other parts of the system. In the interest of brevity, we’ll skip over |
| 109 | +those for now... but if you need more fine-grained control over your setup, feel |
| 110 | +free to dive into the module and have a look!) |
| 111 | + |
| 112 | +Example Usage |
| 113 | +------------- |
| 114 | + |
| 115 | +Enough with the gory details, let’s talk about how to actually use the thing! |
| 116 | + |
| 117 | +When you are first getting started with PuppetDB, there are a few decision |
| 118 | +you’ll have to make: |
| 119 | + |
| 120 | +* Which database back-end should I use? (The current choices are PostgreSQL or |
| 121 | +our embedded database; we’ll discuss this more a bit later on.) |
| 122 | +* Should I run the database on the same node that I run PuppetDB on? |
| 123 | +* Should I run PuppetDB on the same node that I run my master on? |
| 124 | + |
| 125 | +The answers to those questions will be largely dependent on your Puppet |
| 126 | +environment. How many nodes are you managing? What kind of hardware are you |
| 127 | +running on? Is your current load approaching the limits of your hardware? |
| 128 | + |
| 129 | +### The Simple Case |
| 130 | + |
| 131 | +Since I won’t be able to answer all of those questions for you, we’ll start off |
| 132 | +with the absolute simplest case: using our default database (PostgreSQL), and |
| 133 | +running everything (PostgreSQL, PuppetDB, Puppet master) all on the same node. |
| 134 | +This setup will be great for testing / experimental environment, and may be |
| 135 | +sufficient for many real-world deployments depending on the number of nodes |
| 136 | +you’re managing. So, what would our manifest look like in this case? |
| 137 | + |
| 138 | + node puppetmaster { |
| 139 | + # Configure puppetdb and its underlying database |
| 140 | + include puppetdb |
| 141 | + # Configure the puppet master to use puppetdb |
| 142 | + include puppetdb::master::config |
| 143 | + } |
| 144 | + |
| 145 | + |
| 146 | +That’s it! Obviously, you can provide some parameters for these classes if |
| 147 | +you’d like more control, but that is literally all that it will take to get you |
| 148 | +up and running with the default configuration. Here are the steps that this |
| 149 | +manifest will trigger: |
| 150 | + |
| 151 | +* Install PostgreSQL on the node if it’s not already there |
| 152 | +* Create the PuppetDB postgres database instance and user account |
| 153 | +* Validate the postgres connection and, if successful, install and configure |
| 154 | +PuppetDB |
| 155 | +* Validate the PuppetDB connection and, if successful, modify the Puppet master |
| 156 | +config files to use PuppetDB |
| 157 | +* Restart the Puppet master so that it will pick up the config changes |
| 158 | + |
| 159 | +If your logging level is set to INFO or finer, you should start seeing |
| 160 | +PuppetDB-related log messages appear in both your Puppet master log and your |
| 161 | +PuppetDB log as subsequent agent runs occur. |
| 162 | + |
| 163 | +Note: If you’d prefer to use PuppetDB’s embedded database rather than |
| 164 | +PostgreSQL, have a look at the database parameter on the puppetdb class. The |
| 165 | +embedded db can be useful for testing and very small production environments, |
| 166 | +but is not recommended for production environments as it consumes a great deal |
| 167 | +of memory as your number of nodes increases. |
| 168 | + |
| 169 | +### A Distributed Setup |
| 170 | + |
| 171 | +In many cases, you’ll prefer not to install PuppetDB on the same node as the |
| 172 | +Puppet master. Your environment will be easier to scale if you are able to |
| 173 | +dedicate hardware to the individual system components. You may even choose to |
| 174 | +run the PuppetDB server on a different node from the PostgreSQL database that it |
| 175 | +uses to store its data. So let’s have a look at what a manifest for that |
| 176 | +scenario might look like: |
| 177 | + |
| 178 | + # This is an example of a very basic 3-node setup for PuppetDB. |
| 179 | + |
| 180 | + # This node is our Puppet master. |
| 181 | + node puppet { |
| 182 | + # Here we configure the puppet master to use PuppetDB, |
| 183 | + # and tell it that the hostname is ‘puppetdb’ |
| 184 | + class { 'puppetdb::master::config': |
| 185 | + puppetdb_server => 'puppetdb', |
| 186 | + } |
| 187 | + } |
| 188 | + |
| 189 | + # This node is our postgres server |
| 190 | + node puppetdb-postgres { |
| 191 | + # Here we install and configure postgres and the puppetdb |
| 192 | + # database instance, and tell postgres that it should |
| 193 | + # listen for connections to the hostname ‘puppetdb-postgres’ |
| 194 | + class { 'puppetdb::database::postgresql': |
| 195 | + listen_addresses => 'puppetdb-postgres', |
| 196 | + } |
| 197 | + } |
| 198 | + |
| 199 | + # This node is our main puppetdb server |
| 200 | + node puppetdb { |
| 201 | + # Here we install and configure PuppetDB, and tell it where to |
| 202 | + # find the postgres database. |
| 203 | + class { 'puppetdb::server': |
| 204 | + database_host => 'puppetdb-postgres', |
| 205 | + } |
| 206 | + } |
| 207 | + |
| 208 | +That’s it! This should be all it takes to get a 3-node, distributed |
| 209 | +installation of PuppetDB up and running. Note that if you prefer, you could |
| 210 | +easily move two of these classes to a single node and end up with a 2-node setup |
| 211 | +instead. |
| 212 | + |
| 213 | +### Cross-node Dependencies |
| 214 | + |
| 215 | +If you’re playing along at home, you may have spotted some cross-node |
| 216 | +dependencies here and you’ve probably recognized that the order that these nodes |
| 217 | +check in with the puppet master will have serious implications for getting |
| 218 | +everything up and running. It would be very bad to configure the master to use |
| 219 | +the PuppetDB server before that server was up and running. Likewise, it |
| 220 | +wouldn’t be great to try to start up the PuppetDB server pointing to a Postgres |
| 221 | +server that isn’t actually running Postgres yet. |
| 222 | + |
| 223 | +The module handles this problem for you by taking a sort of “eventual |
| 224 | +consistency” approach. There’s nothing that the module can do to control the |
| 225 | +order in which your nodes check in, but the module *can* check to verify that |
| 226 | +the services it depends on are up and running before it makes configuration |
| 227 | +changes--so that’s what it does. |
| 228 | + |
| 229 | +When your Puppet master node checks in, it will validate the connectivity to the |
| 230 | +PuppetDB server before it applies its changes to the Puppet master config files. |
| 231 | + If it can’t connect to PuppetDB, then the puppet run will fail and the previous |
| 232 | +config files will be left intact. This prevents your master from getting into a |
| 233 | +broken state where all incoming Puppet runs fail because the master is |
| 234 | +configured to use a PuppetDB server that doesn’t exist yet. The same strategy |
| 235 | +is used to handle the dependency between the PuppetDB server and the postgres |
| 236 | +server. |
| 237 | + |
| 238 | +What does this all mean to you, as a user? Well, it basically means that the |
| 239 | +first time you add this stuff to your manifests, you may see a few failed Puppet |
| 240 | +runs on the affected nodes. This should be limited to 1 failed run on the |
| 241 | +PuppetDB node, and up to 2 failed runs on the Puppet master node. After that, |
| 242 | +all of the dependencies should be satisfied and your puppet runs should start to |
| 243 | +succeed again. |
| 244 | + |
| 245 | +If you prefer, you can manually trigger puppet runs on the nodes in the correct |
| 246 | +order (Postgres, PuppetDB, Puppet master) and you should avoid any failed runs. |
| 247 | + |
| 248 | +Configuring the module |
| 249 | +---------------------- |
| 250 | + |
| 251 | +The module supports a large number of configuration options. If you’d like more |
| 252 | +control over things like: |
| 253 | + |
| 254 | +* whether or not to open the PuppetDB port on the firewall |
| 255 | +* what address the PuppetDB server should listen on |
| 256 | +* what version of PuppetDB to use |
| 257 | +* what address the PostgreSQL server should listen on |
| 258 | +* PostgreSQL database name, username, password, etc. |
| 259 | +* custom paths to various configuration files |
| 260 | + |
| 261 | +and more, please take a peek at the individual classes. They expose a large |
| 262 | +number of parameters and should hopefully be documented fairly well. (We won’t |
| 263 | +cover them here since this post has already gotten a bit long-winded, if I do |
| 264 | +say so myself, but perhaps we’ll do a follow-up blog post in the future that |
| 265 | +goes into greater detail.) |
| 266 | + |
| 267 | +Conclusion |
| 268 | +---------- |
| 269 | + |
| 270 | +That’s about it for now. We hope that this module makes it So Darn Easy to get |
| 271 | +up and running with PuppetDB that you simply can’t come up with any more excuses |
| 272 | +not to go ahead and do it right now! We think you’ll be happy you did--not only |
| 273 | +because of its current power and features, but also because of all of the great |
| 274 | +things we have in store for it in the near future. |
| 275 | + |
| 276 | +If you have any questions, suggestions, or feedback, please send them to Ryan |
| 277 | +or Chris! If there’s a setting that you’d like to be able to manage that we |
| 278 | +haven’t exposed yet, let us know, or better yet, file a pull request to the |
| 279 | +module project: https://github.com/puppetlabs/puppetlabs-puppetdb |
0 commit comments