Cloud Formation Modification

DataSunrise provides a dedicated script for HA infrastructure deployment within Amazon Web Services based on Amazon Cloud Formation Service. The following template creates all the necessary resources to support the DataSunrise HA environment (the EC2 servers under the AutoScaling group, Configurational Audit and Dictionary RDS databases, the Load Balancer, etc.) and can be customized according to the customer’s needs.

In this article, we are going to show how the Cloud Formation template can be modified to support the customized configuration. 2 scenarios will be presented.

Scenario 1.

We release the minor DataSunrise and AMIs update weekly to keep our solution up-to-date and the most optimized. For example, update ODBC drivers, fix some database vulnerabilities, update the necessary libraries, etc. Therefore our customers might be required to update the AMIs that they are currently using, to the latest one.

This can be done by manual template edition. Detailed instruction is provided below:

1. Open the template file and proceed to the RegionMap section:
   

   Picture 1. The template file

2. Choose the region where your environment is deployed.
3. Proceed to our official GitHub repository

   We update the script files there every week once the new DataSunrise release is out.

4. Proceed as well to the RegionMap section and the region where your resources are deployed:
   

   Picture 2. The DataSunrise repository

5. You can update the AMI ID only for the region that you are using. Therefore copy the AMI ID for the required region.
6. Proceed to your current template file and replace the current AMI ID for the required region with the value copied from our GitHub repository.
7. Go to the AWS CF page and choose the option to Update your CF stack.
8. Select Replace current template and upload the edited version of your template (step 6). Click Next.

Now you can check if the update process was finished successfully. For example, If it was required to update ODBC drivers you can perform a test connection to your target database to check if the updated drivers were used:

Configuration -> Databases -> Your_target_database -> Test connection.

Picture 3. Test connection

If the new version of the ODBC driver is used, the AMI update is finished successfully.

Otherwise, you can recheck your steps or contact us for further investigation.

Scenario 2.

Sometimes our customers need to install some additional tools, for example, ODBC drivers, change the server’s configuration, or apply the latest infrastructural fixes. This can be done by modifying the UserData part of the CF template file.

Let us check how this can be done. We will configure a DB2 ODBC driver that is not included by default in our AWS images.

In the CF template in the UserData section, several scripts are being imported that contain all the necessary configurational settings:

Picture 4. CF template with scripts

Each one sets up its’ own set of environmental components:

1. vm-creads.sh – this script is required to retrieve and attach an IAM role to the current EC2 machine.
2. cf-params.sh – this script is used to transfer the values of the CF stack resources parameters into the script so they can be easily used during the script workflow.
3. ds-manip.sh – this script is required for some administrative operations such as granting permissions to the Datasunrise user to any files, log in/out to the Datasunrise web console, etc.
4. ds-setup.sh – this script contains all the necessary functions that are required to configure DataSunrise successfully. For example, setting up admin password, dictionary database configuration, proxy configuration, and so on.
5. aws-ds-setup.sh – this script sets up AWS-based resources such as CloudWatch metrics and log upload.
6. pre-setup.sh – this script is necessary to prepare the environment for DataSunrise installation: download necessary packages, builds, set up a license for the DataSunrise application, etc.

In today’s case, we are going to install the DB2 ODBC driver that will be used during the target database instance initialization. Therefore we will apply the changes to the pre-setup.sh file.

In our scenario, the necessary commands are taken from the DataSunrise official documentation User Guide

1. Let us paste the necessary commands to the pre-setup.sh file.
   • But first, we need to upload the DB2 ODBC driver to the S3 bucket. So, please, download the drivers from the official website. You will need to log in to your IBM account or create a new account. Once the driver is downloaded, upload it to the S3 bucket, the one that you have specified as BackupS3BucketName during the Cloud Formation script deployment.
   • Now, let us paste the necessary commands as it is shown below:
     

     Picture 5. Commands in the pre-setup.sh file

     Here are the commands:

     logBeginAct "Install DB2 ODBC driver..."
     aws s3 cp "s3://$BackupS3BucketName/ibm_data_server_driver_for_odbc_cli_linuxx64_v11.1.tar.gz" ibm_data_server_driver_for_odbc_cli_linuxx64_v11.1.tar.gz --only-show-errors
     tar -xfv ibm_data_server_driver_for_odbc_cli_linuxx64_v11.1.tar.gz
     echo "[DB2]" | sudo tee -a /etc/odbcinst.ini
     echo "Driver   	= /home/user/clidriver/lib/libdb2o.so" | sudo tee -a /etc/odbcinst.ini
     echo "Description  = DB2 Driver" | sudo tee -a /etc/odbcinst.ini
     echo "FileUsage   	= 1" | sudo tee -a /etc/odbcinst.ini
     echo "DontDLClose  = 1" | sudo tee -a /etc/odbcinst.ini
     RETVAL=$?
     logEndAct "Install DB2 ODBC driver result - $RETVAL"

   • In order to monitor the progress on the ODBC driver installation, you can use our logging mechanism:
     • use the logBeginAct function to start logging and separate one part of the installation process from another:

       the line in the template: logBeginAct “Install DB2 ODBC driver…”

       the output in the cloud-init-output.txt file: [DS ]: Install DB2 ODBC driver…

       * cloud-init-output.txt file is a file that stores all the log messages from DataSunrise Cloud Formation script deployment. It can be found in /var/log/messages/. You can check it whenever you need to troubleshoot your cloud formation stack deployment.

     • once you have opened the logBeginAct part, inside that part you can use the log function to log any information or statuses to the cloud-init-output.txt file.

       We will not use it in our pasted part but you can take a look at another example:

       

       Picture 6. Example of the log function

     • To finish the logging part you can enter:

       logEndAct “Install DB2 ODBC driver result – $RETVAL”

       

       Picture 7. Example of $RETVAL

       *RETVAL variable is used to check the exitcode of the previous command execution.

       RETVAL=$?

2. Save the file. Go to the AWS CF page and choose the option to Update your CF stack.
3. Select Replace current template and upload the edited version of your template (step 2). Click Next.
4. Restart the application and perform the test connection to make sure that the DB2 driver is being used now:

Picture 8. DataSunrise Web Console

The customization of the template was finished successfully. The cloud formation template can be customized as well for further use cases. Most of the modifications are usually done to the UserData scripts section. Feel free to contact us for assistance if you need any help with Cloud Formation customization.