soletechie

Amazon connect offers: - built in reports i.e., historical and real-time reports.  We can customize these reports, schedule them and can integrate with any BI tool of our requirement to query and view the connect data.  Sample solution provided by AWS: 1. Make sure Connect is exporting the CTR data using Kinesis Data Stream 2. Use Kinesis Firehose to deliver the CTR that are in KDS to S3. (CTR's can be delivered as batch of records, so one s3 object might have multiple CTR's). AWS Lambda is used to add a new line character to each record, which makes object easier to parse.  3. s3 Event Notifications are used to send an event to modify the CTR record and saves it in S3. 4. Athena queries the modified CTR's using SQL. Use partitions to restrict the amount of data scanned by each query, improving performance and reducing cost. Lambda function is used to maintain the partitions.  5. Quicksight is used to visualize the modified CTRs.  Solution variations: Convert re...

 - Created a gitlab group: Group name : SoleTechie Group URL : http://gitlab.com/soletechie1 Visibility level : private          Group ID : 52826632    - Creating a Gitlab project: Project name : cicd-demo Project URL : https://gitlab.com/soletechie/ Project slug : cicd-demo Project description : Setting up git lab runners and trying to implement the CI-CD flow Project deployment target (optional) : Infrastructure provider (Terraform) Visibility level : Private      

 We will learn how to setup AWS account, how to access AWS resources using AWS CLI, how to leverage VS Code to view AWS resources.  AWS documentation links for getting started guides : https://aws.amazon.com/getting-started/?e=gs2020&p=console/#Get_to_Know_the_AWS_Cloud https://aws.amazon.com/getting-started/guides/setup-cdk/ https://aws.amazon.com/getting-started/?e=gs2020&p=console/#Launch_Your_First_Application Setting up AWS account : 1. Create Amazon Free Tier account :  https://portal.aws.amazon.com/billing/signup?refid=ps_a131l0000085ejvqam&trkcampaign=acq_paid_search_brand&redirect_url=https%3A%2F%2Faws.amazon.com%2Fregistration-confirmation#/start - Provide your details (email, username, billing information, and make sure you select basicsupport-free option). - Upon successful signup, we will be seeing a confirmation like this: 2. Signin as root user : provide your login information (email, password) and we will be able to see our aws dashboard....

Here are the list of VS Code extensions that anyone working on cloud technologies can use to speed up their development.  To download any extension, refer to the extension tab on your VS code window: As we will manage all our cloud resources using Terraform, we will start with Terraform Autocomplete Extension.  1. Terraform Extensions Terraform: to manage terraform resources directly from VS Code.  Terraform Autocomplete : useful when we are creating terraform resources. 2. Docker : To build, manage and deploy docker containers from VS Code. 3. Python : extension that provides python interpreter 4. Prettier-Code formatter : 5. Markdown Preview :  6. Git :   Git History : Git Graph : Now we can select the below extensions, and click on install.  AWS VSCode Extensions: 1. AWS Toolkit : To interact with AWS resources directly from VS Code. Helpful in taking a look of AWS resources without having to login into console, provides us with a very cool UI to g...

 If we are using terraform, terraform state file is the heart of all the infrastructure that we spin up using terraform templates.  There are several ways to deploy the infrastructure using terraform: 1. Using CLI (setup terraform and then run terraform commands) 2. Automated Build (terraform scripts integrated as part of your jenkins pipeline) No matter of the way we chose, we must make sure that we are using the same terraform state file, so that we are having a sync and proper checklists of the resources that we used.  I would like to share the terraform commands that we do on a daily basis: terraform init = the basic/starting command which initializes the terraform (make sure the proper provider is provided. In my case, I use AWS).  terraform workspace select <workspace name > (creates a new workspace, useful in scenarios where we have different terraform modules - database, servers, logs, storage) terraform state list = shows the list of terraform resour...

What are Modules?  - self contained pieces of IAC that abstract the infrastructure deployments  - use clear organization and DRY (Dont Repeat Yourself)  - helps in writing composable, shareable and reusable infrastructures   Scope the requirements into appropriate modules : - When building a module, consider 3 areas:   1. Encapsulation - Group infrastructure that is always deployed together   2. Privileges - Restrict modules to privilege boundaries   3. Voltatility - Separate long lived infrastructure from short-lived (Ex: Database-static vs Application Servers-dynamic)  Create the module MVP : * Always aim to deliver a module that works for 80% of usecases  * Never code for edge cases. A module should be a reusable block of code.  * Avoid conditional expressions in MVP  * Module should only expose the most commonly modified arguments as variables.   Scoping Example - A team wants to provision thei...

  While running a databricks job, especially running a job with large datasets and longer running queries that creates a lot of temp space - we might be facing below issue if we have a minimal configuration set to the cluster.  The simple way to fix this would be changing the spark driver config in the databricks cluster tab spark.driver.maxResultSize = 100G (change the GB based on your cluster size)