Saturday, December 7, 2019

GCP Study Notes 2: start running jupyter in 5 mins from GCP, Bigqery example

The most easy way to run jupyter notebook in GCP
There are several ways to run jupyter notebook in GCP, you can starting from creating dataproc cluster, or VM, then install the anaconda package there. It involves quite a few configue setting and might not that easy for the traditional data scientist with more statitics background. Here is the most easy way I found you can run the jupyter session in GCP via a fewest clicks.

Go to left menu of GCP, find "AI platform", then click "notebook", you can easily create a notebook with a few clicks. After you create the notebook, click "open jupyterlab", then you are in the traditional jupyter environment, you can create .txt file and rename to .py file, running like python spyder.

Another powerful part is the tutorial there, once you are in the jupyter environment, you can see a "tutorial" folder, which has a lot of tutorial examples you can try there.

Error: "cannot import name automl_v1beta1", "cannot import name automlwrapper","ImportError: No module named 'automlwrapper'"

When you are in the jupyter emvironment, you were trying to import automlwrapper or automl_v1beta1, you might find the above error message. Here is some quick way to fix:
#run the following in the terminal
pip install --user --upgrade  google-cloud-automl
#then re-start the kernel in jupyter, run import again:
from import storage
from import automl_v1beta1 as automl
from automlwrapper import AutoMLWrapper

Here are some example code of creating dataproc cluster and pre-install python.
set -euo pipefail

[[ ! -d "${DIR}" ]] && DIR="${PWD}"
readonly DIR

echo "Creating dataproc cluster..."

function usage {
  cat << EOF
usage: $0 [-h] [-c=cluster-name]
  -h                 display help
  -c=cluster-name    specify unique dataproc cluster name to launch
  exit 1

for i in "$@"
  case $i in
      shift # past argument=value

[[ -c "${DATAPROC_CLUSTER_NAME}" ]] && usage
echo "Using following cluster name: ${DATAPROC_CLUSTER_NAME}"


# NOTE: change the following parameters to suit your needs
# master-boot-disk-size
# worker-boot-disk-size
# num-workers
gcloud beta dataproc clusters create ${DATAPROC_CLUSTER_NAME} \
    --image-version 1.4 \
    --zone=us-west1-a \
    --bucket=edg-dsa-users \
    --enable-component-gateway \
    --metadata 'MINICONDA_VARIANT=3' \
    --metadata 'MINICONDA_VERSION=latest' \
    --metadata 'CONDA_PACKAGES=jupyterlab' \
    --metadata 'PIP_PACKAGES=pandas patsy datetime pyhive pandas_gbq argparse sklearn gcsfs pandasql keras tensorflow seaborn pandas_profiling google-cloud-storage paramiko google-cloud-automl' \
    --metadata 'PYTHON_VERSION=3.7' \
    --subnet "projects/****-network-sbx-****/regions/us-east1/subnetworks/****-data-svcs-1-us-w1-sbx-subnet" \
    --scopes '' \
    --tags allow-ssh,egress-nat-gce \
    --project $PROJECT_ID \
    --service-account $SERVICE_ACCOUNT \
    --num-workers=2 \
    --num-preemptible-workers=0 \
    --master-machine-type=n1-highmem-96 \
    --worker-machine-type=n1-standard-1 \
    --master-boot-disk-size=1000GB \
    --worker-boot-disk-size=15GB \
    --properties "yarn:yarn.scheduler.maximum-allocation-mb=241664,yarn:yarn.nodemanager.resource.memory-mb=241664" \
    --initialization-actions \

What's the difference between Kubernetes and Docker?

There are two kinds of usage: committed and sustained discount usage.

vCPU = unit of measure of CPU.
RAM is measure in unit of Gigabite.

There are 3 different ways to do dev on GCP: command line(gcloud), REST API, Google Cloud Consule.

HDD: harddisk device.
SSD: Solid State Device. SSD is more expensive than HDD, physically more closely to the VM.

2 kinds of VM: standard and preemptible: preempitble virtual machine(PVM) are more cheap(up to 80% cheaper than regular instances), technically the same as standard, but charged on the excess usage, is always terminated after 24 hours and when the GCP usage spikes, also terminates, whichever comes first. This is the reason they might be stopped at any time.
gcloud compute instances create my-vm --zone us-central1-b --preemptible

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Kubernetes was built by Google based on their experience running containers in production using an internal cluster management system called Borg.

In GCP: Compute engine ~ Virtual Machine = AWS (EC2)

Here is some example using papermill to run a collection of jupyter notebooks automatically, output to GCS, and then shut down to save the cost.
The following command starts execution of a Jupyter notebook stored in a Cloud Storage bucket:

# Compute Engine Instance parameters
export IMAGE_FAMILY="tf-latest-cu100" 
export ZONE="us-central1-b"
export INSTANCE_NAME="notebook-executor"
export INSTANCE_TYPE="n1-standard-8"
# Notebook parameters
export INPUT_NOTEBOOK_PATH="gs://my-bucket/input.ipynb"
export OUTPUT_NOTEBOOK_PATH="gs://my-bucket/output.ipynb"
export PARAMETERS_FILE="params.yaml" # Optional
export PARAMETERS="-p batch_size 128 -p epochs 40"  # Optional
gcloud compute instances create $INSTANCE_NAME \
        --zone=$ZONE \
        --image-family=$IMAGE_FAMILY \
        --image-project=deeplearning-platform-release \
        --maintenance-policy=TERMINATE \
        --accelerator='type=nvidia-tesla-t4,count=2' \
        --machine-type=$INSTANCE_TYPE \
        --boot-disk-size=100GB \
        --scopes= \
gcloud --quiet compute instances delete $INSTANCE_NAME --zone $ZONE

The above commands do the following:

Create a Compute Engine instance using TensorFlow Deep Learning VM and 2 NVIDIA Tesla T4 GPUs
Install the latest NVIDIA GPU drivers
Execute the notebook using Papermill

Upload notebook result (with all the cells pre-computed) to Cloud Storage bucket in this case: “gs://my-bucket/”

Terminate the Compute Engine instance
After clicking activate google cloud Shell, in the terminal, you can view your roles by running:

gcloud iam roles describe roles/viewer
gcloud iam roles describe roles/editor
gcloud iam roles describe roles/owner

Google Cloud Dataproc is a fast, easy-to-use, fully-managed cloud service for running Apache Spark and Apache Hadoop clusters in a simple, cost-efficient way.
Cloud Dataproc easily integrates with other Google Cloud Platform (GCP) services, giving you a powerful and complete platform for data processing, analytics and machine learning

Apache Spark is an analytics engine for large scale data processing. Logistic regression is available as a module in Apache Spark's machine learning library, MLlib.
Spark MLlib, also called Spark ML, includes implementations for most standard machine learning algorithms such as k-means clustering, random forests,
alternating least squares, k-means clustering, decision trees, support vector machines, etc. Spark can run on a Hadoop cluster, like Google Cloud Dataproc, in order to process very large datasets in parallel.

Relational databases are also called Relational Database Management Systems (RDBMS) or SQL databases. Historically,
the most popular of these have been Microsoft SQL Server, Oracle Database, MySQL, and IBM DB2.
Relational databases are designed to run on a single server in order to maintain the integrity of the table mappings and avoid the problems of distributed computing.

Non-relational databases are also called NoSQL databases. NoSQL has become an industry standard term,
but the name is beginning to lose popularity since it doesn’t fully cover the complexity and range of non-relational data stores that are available.
Some of the most known NoSQL or non-relational DBs that Serra discussed are MongoDB, DocumentDB, Cassandra, Coachbase, HBase, Redis, and Neo4j.

Hadoop is a file system with components made up of Hadoop Distributed File System (HDFS), Yarn, and MapReduce.

Cloud Spanner is the first scalable, enterprise-grade, globally-distributed, and strongly consistent database service built for the cloud specifically to combine the benefits of relational database structure with non-relational horizontal scale.

Difference between horizontally scaling and vertically scaling:

In simple layman language, Horizontal scaling is more complex than the vertical scaling(adding more cpu,memory at the existing machine).

Horizontal scaling means that you scale by adding more machines into your pool of resources
whereas Vertical scaling means that you scale by adding more power (CPU, RAM) to an existing machine.

Horizontal Scaling, also referred to as "scale-out" is basically the addition of more machines or setting up a cluster or a distributed environment for your software system.
This usually requires a load-balancer program which is a middle-ware component in the standard 3 tier client-server architectural model.

Vertical Scaling - also referred to as "scale-up" approach is an attempt to increase the capacity of a single machine :
By adding more processing power By adding more storage More memory etc.

Horizontal scaling comes with overhead in the form of cluster setup, management, and maintenance costs and complexities.
The design gets increasingly complex and programming model changes.

Relational Databases:

Pros: Relational databases work with structured data.
They support ACID transactional consistency and support “joins.”
They come with built-in data integrity and a large eco-system.
Relationships in this system have constraints.
There is limitless indexing. Strong SQL.

Relational Databases do not scale out horizontally very well (concurrency and data size), only vertically, (unless you use sharding).
Data is normalized, meaning lots of joins, which affects speed.
They have problems working with semi-structured data.

They scale out horizontally and work with unstructured and semi-structured data. Some support ACID transactional consistency.
Schema-free or Schema-on-read options.
High availability.
While many NoSQL databases are open source and so “free”, there are often considerable training, setup, and developments costs. There are now also numerous commercial products available.

Weaker or eventual consistency (BASE) instead of ACID.
Limited support for joins.
Data is denormalized, requiring mass updates (i.e. product name change).
Does not have built-in data integrity (must do in code).
Limited indexing.

What is SSH?
Secure Shell, sometimes referred to as Secure Socket Shell, is a protocol which allows you to connect securely to a remote computer or a server by using a text-based interface.

When a secure SSH connection is established, a shell session will be started, and you will be able to manipulate the server by typing commands within the client on your local computer.

ImportError: No module named

1. pip install --upgrade google-cloud-storage
2. restart the kerkel.

if you run the following: pip install google-cloud, it might not work.

check GCP creditial:
import os
print('Credendtials from environ: {}'.format(os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')))

What's the product matching between AWS and GCP?


Service Category Service AWS Google Cloud Platform
Compute IaaS Amazon Elastic Compute Cloud(EC2) Compute Engine
PaaS AWS Elastic Beanstalk App Engine
Containers Amazon Elastic Container Service Google Kubernetes Engine
Serverless Functions AWS Lambda Cloud Functions
Managed Batch Computing AWS Batch N/A
Network Virtual Networks Amazon Virtual Private Cloud Virtual Private Cloud
Load Balancer Elastic Load Balancer Cloud Load Balancing
Dedicated Interconnect Direct Connect Cloud Interconnect
Domains and DNS Amazon Route 53 Google Domains, Cloud DNS
CDN Amazon CloudFront Cloud CDN
Storage Object Storage Amazon Simple Storage Service(S3) Cloud Storage
Block Storage Amazon Elastic Block Store Persistent Disk
Reduced-availability Storage Amazon S3 One Zone-Infrequent
Cloud Storage Nearline
Archival Storage Amazon Glacier Cloud Storage Coldline
File Storage Amazon Elastic File System Cloud Filestore (beta)
Database RDBMS Amazon Aurora Cloud SQL,
Cloud Spanner
NoSQL: Key-value Amazon DynamoDB Cloud Firestore,
Cloud Bigtable
NoSQL: Indexed Amazon SimpleDB Cloud Firestore
Big Data & Analytics Batch Data Processing Amazon Elastic MapReduce, AWS Batch Cloud Dataproc,
Cloud Dataflow
Stream Data Processing Amazon Kinesis Cloud Dataflow
Stream Data Ingest Amazon Kinesis Cloud Pub/Sub
Analytics Amazon Redshift, Amazon Athena BigQuery
Workflow Orchestration Amazon Data Pipeline, AWS Glue Cloud Composer
Application Services Messaging Amazon Simple Notification Service Cloud Pub/Sub
Management Services Monitoring Amazon CloudWatch Stackdriver Monitoring
Logging Amazon CloudWatch Logs Stackdriver Logging
Deployment AWS CloudFormation Cloud Deployment Manager
Machine Learning Speech Amazon Transcribe Cloud Speech-to-Text
Vision Amazon Rekognition Cloud Vision
Natural Language Processing Amazon Comprehend Cloud Natural Language
Translation Amazon Translate Cloud Translation
Conversational Interface Amazon Lex Dialogflow
Enterprise Edition
Video Intelligence Amazon Rekognition Video Cloud Video Intelligence
Auto-generated Models N/A Cloud AutoML (beta)
Fully Managed ML Amazon SageMaker Cloud Machine Learning Engine

Example of using google bigquery on GCP in 15 min
#running in the bigquery API SQL: 
create table `maximal-inkwell-#####.sample1.test`  
CLUSTER BY cluster_col 
SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 10

#here is the syntax: 
  column_name column_schema[, ...]
[PARTITION BY partition_expression]
[CLUSTER BY clustering_column_list]
[AS query_statement]

##Here are the python code: 
from import bigquery

# Construct a BigQuery client object.
client = bigquery.Client()

# TODO(developer): Set table_id to the ID of the destination table.
# table_id = "your-project.your_dataset.your_table_name"

job_config = bigquery.QueryJobConfig(destination=table_id)

sql = """
    SELECT corpus
    FROM `bigquery-public-data.samples.shakespeare`
    GROUP BY corpus;

# Start the query, passing in the extra configuration.
query_job = client.query(sql, job_config=job_config)  # Make an API request.
query_job.result()  # Wait for the job to complete.

print("Query results loaded to the table {}".format(table_id))

#you can also define a function to create table in bigquery via sql
import time

from import bigquery
from import Table
from import Dataset

def create_table_as_select(dataset_name, table_name, sqlQuery, project=None):
    job_config = bigquery.QueryJobConfig()

    # Set configuration.query.destinationTable
    dataset_ref = bigquery_client.dataset(dataset_name)
    table_ref = dataset_ref.table(table_name)

    job_config.destination = table_ref

    # Set configuration.query.createDisposition
    job_config.create_disposition = 'CREATE_IF_NEEDED'

    # Set configuration.query.writeDisposition
    job_config.write_disposition = 'WRITE_APPEND'

    # Start the query
    job = bigquery_client.query(sqlQuery, job_config=job_config)

    # Wait for the query to finish

    returnMsg = 'Created table {} .'.format(table_name)

    return returnMsg

except Exception as e:
    errorStr = 'ERROR (create_table_as_select): ' + str(e)

We can also run bigquery sql from the cloud shell:
bq query "select string_field_10 as request, count(*) as requestcount 
from logdata.accesslog group by request order by requestcount desc"

#===========================================================# Create a dataset using bigquery API:
bigquery_client  = bigquery.Client() #Create a BigQuery service object
dataset_id = 'my_dataset' 
dataset_ref = bigquery_client.dataset(dataset_id) # Create a DatasetReference using a chosen dataset ID.
dataset = bigquery.Dataset(dataset_ref)  # Construct a full Dataset object to send to the API.
dataset.location = 'US' # Specify the geographic location where the new dataset will reside. 
Remember this should be same location as that of source data set from where we are getting data to run a query

# Send the dataset to the API for creation. Raises google.api_core.exceptions.AlreadyExists if the Dataset already exists within the project.
dataset = bigquery_client.create_dataset(dataset)  # API request
print('Dataset {} created.'.format(dataset.dataset_id))


No comments:

Post a Comment

Data Science Study Notes: recommendation engine notes 1: Deep matrix factorization using Apache MXNet

Deep matrix factorization using Apache MXNet ( notes from Oreilly , github notebook ) Recommendation engines are widely used models th...