config.md

Configuration

Next, you'll need to create a configuration file with your details. The extract and load scripts in our pipeline will utilise the details here.

Setup

1. Create a configuration file under ~/Reddit-API-Pipeline/airflow/extraction/ called configuration.conf:

   touch ~/Reddit-API-Pipeline/airflow/extraction/configuration.conf

2. Copy in the following:

   [aws_config]
   bucket_name = XXXXX
   redshift_username = awsuser
   redshift_password = XXXXX
   redshift_hostname =  XXXXX
   redshift_role = RedShiftLoadRole
   redshift_port = 5439
   redshift_database = dev
   account_id = 
   aws_region = us-east-1
   
   [reddit_config]
   secret = XXXXX
   developer = XXXXX
   name = XXXXX
   client_id = XXXXX
   

3. Change XXXXX values

   • If you need a reminder of your aws_config details, change folder back into the terraform folder and run the command. It will output the values you need to store under aws_config. Just be sure to remove any " from the strings.

     terraform output

   • For reddit_config these are the details you took note of after setting up your Reddit App. Note the developer is your Reddit name.

---

Previous Step | Next Step

or

Back to main README