Setup Kubeflow Cluster in a Shared VPC on Google Cloud Platform

Prepare the Environment

gcloud container subnets list-usable \                 
--project <Service project ID> \
--network-project <Host Project ID>
# make a bin folder to contain the kfctl
mkdir ~/bin
# download kfctl from the releases page
wget -P /tmp https://github.com/GoogleCloudPlatform/training-data-analyst/raw/master/courses/data-engineering/kubeflow-resources/kfctl_v1.0-0-g94c35cf_linux.tar.gz
# unzip to the bin folder
tar -xvf /tmp/kfctl_v1.0–0-g94c35cf_linux.tar.gz -C ~/bin
# add the kfctl binary to the PATH
export PATH=$PATH:~/bin
# Set your GCP project ID and the zone where you want to create 
# the Kubeflow deployment:
export PROJECT=<your GCP project ID>
export ZONE=<your GCP zone>
gcloud config set project ${PROJECT}
gcloud config set compute/zone ${ZONE}
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml"
export CLIENT_ID=<CLIENT_ID from OAuth page>
export CLIENT_SECRET=<CLIENT_SECRET from OAuth page>
  • The CLIENT_ID and CLIENT_SECRET can be obtained from the Cloud Console by selecting APIs & Services -> Credentials
export KF_NAME=<your choice of name for the Kubeflow deployment>
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}
  • For example, your kubeflow deployment name can be ‘my-kubeflow’ or ‘kf-test’.
  • Set base directory where you want to store one or more Kubeflow deployments. For example, ${HOME}/kf_deployments.

Deploy Kubeflow with Customization

export CONFIG_FILE="kfdef.yaml"
mkdir -p ${KF_DIR}
cd ${KF_DIR}
curl -L -o ${CONFIG_FILE} https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml
```
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.project' ${PROJECT}
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.zone' ${ZONE}
yq w -i ${CONFIG_FILE} 'metadata.name' ${KF_NAME}
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_FILE}
cluster:
name: {{ CLUSTER_NAME }}
network: projects/<host project ID>/global/networks/<network name>
subnetwork: projects/<host project ID>/regions/<region>/subnetworks/<subnet name>
initialClusterVersion: "{{ properties['cluster-version'] }}"
{ if properties['securityConfig']['privatecluster'] }
ipAllocationPolicy:
createSubnetwork: false
useIpAliases: true
clusterSecondaryRangeName: <name of secondary ip range for pods>
servicesSecondaryRangeName: <name of secondary ip range for services>
privatecluster: true
gkeApiVersion: v1beta1
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}
  • {KF_NAME}-storage
  • {KF_NAME}kubeflow-qwiklab
  • In the Workloads section, a number of Kubeflow components
  • In the Services section, a number of Kubeflow services
gcloud container clusters get-credentials ${KF_NAME} --zone ${ZONE} --project ${PROJECT}
kubectl -n kubeflow get all

Delete Kubeflow Deployment

kfctl delete -f ${CONFIG_FILE} — delete_storage
kfctl delete -f ${CONFIG_FILE}

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Lei Feng

Lei Feng

Big Data, Google Cloud Platform, Machine Learning, Operations Research