OpenSearch

     2: Snapshot Stored location: We choose snapshot frequently 1 hour in AWS in which location snapshot are getting stored ?

To get list of automated snapshots run the below API:

If your domain encrypts data at rest—>GET _snapshot/cs-automated-enc/_all
If your domain doesn’t encrypt data at rest—>GET _snapshot/cs-automated/_all

Snapshot Restore from Kibana console

Syntax:

POST /_snapshot/location/snapshot_name/_restore?wait_for_completion=false 

POST /_snapshot/instaclustr-production/es-a4f4eb4a-271c-4300-84f4-0794bb498a50-snapshot-5ff78b07-1d94-4df0-866b-d821f5a3a812/_restore?wait_for_completion=false 

{ 

  "indices": "index_name", 

  "ignore_unavailable": true, 

  "include_global_state": false, 

  "include_aliases": true 

} 

Data Engineering

 What is PySpark?

Python API for Apache spark

It's openSource

Distributed computing framework

set of libraries for real-time and large-scale data processing

Apache Spark is written in Scala

Key Features:

Rapid processing

Effectiveness with RDD -> Resilient Distributed DataSet

  Objects are immutable and it runs in distributed

  Fault tolerance, as a result of node failure it re-process

PySpark is needed

pyspark can be done in python or scala

Python is superior in readability, comprehensive, and maintainable, compare to scala which gives complex code

It also provides machine-learning libraries

Ease of learning python with simple syntax compare to scala

PySpark vs Scala 

Courtesy link: https://www.youtube.com/watch?v=6F2doPE0-vc

Docker

 Install MySQL 5.7 in Docker

bash> docker pull mysql/mysql-server:5.7
bash> docker images
bash> docker run --name=mysql57 -d mysql/mysql-server:5.7
bash> docker logs mysql57
bash> docker exec -it mysql57 mysql -uroot -p
 mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'newpassword';
 mysql> update mysql.user set host = '%' where user='root';
Quit from mysql client
docker restart mysql57

Connect from MySQL WorkBench

jacoco plugin usage with maven

From the place where you have the pom.xml run the below line to get the  site > jacoco under target for index.html to see the report on the package. 

mvn -B org.jacoco:jacoco-maven-plugin:prepare-agent clean verify org.jacoco:jacoco-maven-plugin:report

ElasticSearch Commands and Support

Curl to GET index data includes header and replicas  with sort by replica.

curl -X GET "http://IP:9200/_cat/indices?h=index,rep&s=rep"

Check the cluster health

curl -X GET "http://IP:9200/_cluster/health?pretty=json"

Turn of cluster settings for wildCard index execution

curl -XPUT "http://IP:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{"persistent":{"action.destructive_requires_name":"false"}}'

curl -XDELETE "http://IP:9200/1.9.2__idx_test__y*"

curl -XPUT "http://IP:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{"persistent":{"action.destructive_requires_name":"true"}}'

Update index replication

curl -X PUT "http://IP:9200/index_name/_settings" -H 'Content-Type: application/json' -d '{"index":{"refresh_interval":"30s","number_of_replicas":1}}'

RED ElasticSearch Cluster (Debug with explain to see the current allocation).

curl -XGET IP:9200/_cluster/allocation/explain?pretty

Find Red Indices: 
health can be red, green, yellow

curl -XGET 'http://IP:9200/_cat/indices?health=red'

GET Indices and its store size

curl -XGET "http://IP:9200/_cat/indices?format=json&s=store.size:desc" > cat_indices_feb9.json

Steps to Simple Docker File creation with centos 7 images includes jdk, node, maven and push to AWS ECS

Steps to Simple Docker File to create centos 7 install node, maven

$ vi Dockerfile

FROM centos:7

RUN yum -y update ; yum -y install java-1.8.0-openjdk-devel wget which

RUN yum install -y gcc gcc-c++ \

                   libtool libtool-ltdl \

                   make cmake \

                   git \

                   pkgconfig \

                   sudo \

                   automake autoconf \

                   yum-utils rpm-build && \

    yum clean all

RUN curl -sL https://rpm.nodesource.com/setup_10.x | bash - 

RUN yum -y install nodejs

RUN npm install -g @angular/cli

RUN wget https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz

RUN tar xvf apache-maven-3.6.3-bin.tar.gz -C /usr/lib/

RUN echo 'M2_HOME="/usr/lib/apache-maven-3.6.3"' >> /etc/profile

RUN echo 'export M2_HOME' >> /etc/profile

RUN echo 'M2="$M2_HOME/bin"' >> /etc/profile

RUN echo 'MAVEN_OPTS="-Xms512m -Xmx1024m"' >> /etc/profile

RUN echo 'export M2 MAVEN_OPTS' >> /etc/profile

RUN echo 'PATH=$M2:$PATH' >> /etc/profile

RUN echo 'export PATH' >> /etc/profile

RUN echo "source /etc/profile" >> /root/.bashrc

ENV FLAVOR=rpmbuild OS=centos DIST=el7

CMD ["/bin/bash"]

Build docker image and push to AWS. Make sure docker and AWS CLI installed

1. Create repository in AWS ECS. 

2. In AWS select repository (view push commands) to see the commands 

Retrieve an authentication token and authenticate your Docker client to your registry.

aws ecr get-login-password --region us-west-2 --profile <profile> | docker login --username AWS --password-stdin gubendran.dkr.ecr.us-west-2.amazonaws.com

Note: If you receive an error using the AWS CLI, make sure that you have the latest version of the AWS CLI and Docker installed.

Build your Docker image using the following command. For information on building a Docker file from scratch see the instructions here . You can skip this step if your image is already built:

docker build -t test_build:2.5.0 .

After the build completes, tag your image so you can push the image to this repository:

docker tag test_build:2.5.0 gubendran.dkr.ecr.us-west-2.amazonaws.com/test_build:2.5.0

Run the following command to push this image to your newly created AWS repository:

docker push gubendran.dkr.ecr.us-west-2.amazonaws.com/test_build:2.5.0

Describe AWS ECR image from the repository created

aws ecr describe-images --repository-name test_build

Install NPM HighCharts, Elastic Dump

Install NPM HighCharts

Verify node and npm. If missing then install 

node -v

npm -v

Install

curl -sL https://rpm.nodesource.com/setup_10.x | bash - 

yum -y install nodejs

node -v

npm -v

yum install bzip2

yum install fontconfig freetype freetype-devel fontconfig-devel libstdc++

npm install -g highcharts-export-server --ignore-scripts 

cd /usr/lib/node_modules/highcharts-export-server/node_modules/phantomjs-prebuilt/ 

node install.js

cd /usr/lib/node_modules/highcharts-export-server 

node build.js

To Start highcharts-export-server

highcharts-export-server -enableServer 1 &

ElasticDump

npm install -g elasticdump