github
/
k3s
mirror of https://github.com/k3s-io/k3s
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
k3s/examples/spark/images/Makefile

62 lines
2.9 KiB
Makefile
Raw Normal View History

Spark: Bump images to 1.5.1, add Makefile
2015-10-24 13:48:41 +00:00
all: push
Zeppelin: Add Zeppelin image to Spark example This adds a very basic Zeppelin image that works with the existing Spark example. As can be seen from the documentation, it has a couple of warts: * It requires kubectl port-forward (which is unstable across long periods of time, at least for me, on this app, bug incoming). See * I needed to roll my own container (none of the existing containers exactly matched needs, or even built anymore against modern Zeppelin master, and the rest of the example is Spark 1.5). The image itself is *huge*. One of the further refinements we need to look at is how to possibly strip the Maven build for this container down to just the interpreters we care about, because the deps here are frankly ridiculous. This might be a case where, if possible, we might want to open an upstream request to build things dynamically, then use something like probably the cut the image down considerably. (This might already be possible, need to poke at whether you can late-bind interpreters later.)
2015-11-05 22:56:15 +00:00
push: push-spark push-zeppelin
Zeppelin: Compress the image by doing the build on a separate image Pull the tar through to the image we need it on. Addresses #17231
2015-11-15 00:44:27 +00:00
.PHONY: push push-spark push-zeppelin spark zeppelin zeppelin-build
Spark: Bump images to 1.5.1, add Makefile
2015-10-24 13:48:41 +00:00
# To bump the Spark version, bump the version in base/Dockerfile, bump
Zeppelin: Add Zeppelin image to Spark example This adds a very basic Zeppelin image that works with the existing Spark example. As can be seen from the documentation, it has a couple of warts: * It requires kubectl port-forward (which is unstable across long periods of time, at least for me, on this app, bug incoming). See * I needed to roll my own container (none of the existing containers exactly matched needs, or even built anymore against modern Zeppelin master, and the rest of the example is Spark 1.5). The image itself is *huge*. One of the further refinements we need to look at is how to possibly strip the Maven build for this container down to just the interpreters we care about, because the deps here are frankly ridiculous. This might be a case where, if possible, we might want to open an upstream request to build things dynamically, then use something like probably the cut the image down considerably. (This might already be possible, need to poke at whether you can late-bind interpreters later.)
2015-11-05 22:56:15 +00:00
# the version in zeppelin/Dockerfile, bump this tag and reset to
# v1. You should also double check the native Hadoop libs at that
# point (we grab the 2.6.1 libs, which are appropriate for
# 1.5.1-with-2.6). Note that you'll need to re-test Zeppelin (and it
# may not have caught up to newest Spark).
Spark: Don't daemonize, don't pretend to tail logs we don't use Along the way: Fixup port mismatch in worker container
2015-10-30 18:37:29 +00:00
TAG = 1.5.1_v2
Spark: Bump images to 1.5.1, add Makefile
2015-10-24 13:48:41 +00:00
Zeppelin: Add Zeppelin image to Spark example This adds a very basic Zeppelin image that works with the existing Spark example. As can be seen from the documentation, it has a couple of warts: * It requires kubectl port-forward (which is unstable across long periods of time, at least for me, on this app, bug incoming). See * I needed to roll my own container (none of the existing containers exactly matched needs, or even built anymore against modern Zeppelin master, and the rest of the example is Spark 1.5). The image itself is *huge*. One of the further refinements we need to look at is how to possibly strip the Maven build for this container down to just the interpreters we care about, because the deps here are frankly ridiculous. This might be a case where, if possible, we might want to open an upstream request to build things dynamically, then use something like probably the cut the image down considerably. (This might already be possible, need to poke at whether you can late-bind interpreters later.)
2015-11-05 22:56:15 +00:00
# To bump the Zeppelin version, bump the version in
# zeppelin/Dockerfile and bump this tag and reset to v1.
Use upstream zeppelin 0.5.5 release
2015-11-20 16:44:14 +00:00
ZEPPELIN_TAG = v0.5.5_v3
Zeppelin: Add Zeppelin image to Spark example This adds a very basic Zeppelin image that works with the existing Spark example. As can be seen from the documentation, it has a couple of warts: * It requires kubectl port-forward (which is unstable across long periods of time, at least for me, on this app, bug incoming). See * I needed to roll my own container (none of the existing containers exactly matched needs, or even built anymore against modern Zeppelin master, and the rest of the example is Spark 1.5). The image itself is *huge*. One of the further refinements we need to look at is how to possibly strip the Maven build for this container down to just the interpreters we care about, because the deps here are frankly ridiculous. This might be a case where, if possible, we might want to open an upstream request to build things dynamically, then use something like probably the cut the image down considerably. (This might already be possible, need to poke at whether you can late-bind interpreters later.)
2015-11-05 22:56:15 +00:00
spark:
Spark: Bump images to 1.5.1, add Makefile
2015-10-24 13:48:41 +00:00
docker build -t gcr.io/google_containers/spark-base base
docker tag gcr.io/google_containers/spark-base gcr.io/google_containers/spark-base:$(TAG)
docker build -t gcr.io/google_containers/spark-worker worker
docker tag gcr.io/google_containers/spark-worker gcr.io/google_containers/spark-worker:$(TAG)
docker build -t gcr.io/google_containers/spark-master master
docker tag gcr.io/google_containers/spark-master gcr.io/google_containers/spark-master:$(TAG)
docker build -t gcr.io/google_containers/spark-driver driver
docker tag gcr.io/google_containers/spark-driver gcr.io/google_containers/spark-driver:$(TAG)
Use upstream zeppelin 0.5.5 release
2015-11-20 16:44:14 +00:00
# This target is useful when needing to use an unreleased version of Zeppelin
Zeppelin: Compress the image by doing the build on a separate image Pull the tar through to the image we need it on. Addresses #17231
2015-11-15 00:44:27 +00:00
zeppelin-build:
docker build -t gcr.io/google_containers/zeppelin-build zeppelin-build
docker tag -f gcr.io/google_containers/zeppelin-build gcr.io/google_containers/zeppelin-build:$(ZEPPELIN_TAG)
Use upstream zeppelin 0.5.5 release
2015-11-20 16:44:14 +00:00
zeppelin:
Zeppelin: Add Zeppelin image to Spark example This adds a very basic Zeppelin image that works with the existing Spark example. As can be seen from the documentation, it has a couple of warts: * It requires kubectl port-forward (which is unstable across long periods of time, at least for me, on this app, bug incoming). See * I needed to roll my own container (none of the existing containers exactly matched needs, or even built anymore against modern Zeppelin master, and the rest of the example is Spark 1.5). The image itself is *huge*. One of the further refinements we need to look at is how to possibly strip the Maven build for this container down to just the interpreters we care about, because the deps here are frankly ridiculous. This might be a case where, if possible, we might want to open an upstream request to build things dynamically, then use something like probably the cut the image down considerably. (This might already be possible, need to poke at whether you can late-bind interpreters later.)
2015-11-05 22:56:15 +00:00
docker build -t gcr.io/google_containers/zeppelin zeppelin
docker tag -f gcr.io/google_containers/zeppelin gcr.io/google_containers/zeppelin:$(ZEPPELIN_TAG)
push-spark: spark
Spark: Bump images to 1.5.1, add Makefile
2015-10-24 13:48:41 +00:00
gcloud docker push gcr.io/google_containers/spark-base
gcloud docker push gcr.io/google_containers/spark-base:$(TAG)
gcloud docker push gcr.io/google_containers/spark-worker
gcloud docker push gcr.io/google_containers/spark-worker:$(TAG)
gcloud docker push gcr.io/google_containers/spark-master
gcloud docker push gcr.io/google_containers/spark-master:$(TAG)
gcloud docker push gcr.io/google_containers/spark-driver
gcloud docker push gcr.io/google_containers/spark-driver:$(TAG)
Zeppelin: Add Zeppelin image to Spark example This adds a very basic Zeppelin image that works with the existing Spark example. As can be seen from the documentation, it has a couple of warts: * It requires kubectl port-forward (which is unstable across long periods of time, at least for me, on this app, bug incoming). See * I needed to roll my own container (none of the existing containers exactly matched needs, or even built anymore against modern Zeppelin master, and the rest of the example is Spark 1.5). The image itself is *huge*. One of the further refinements we need to look at is how to possibly strip the Maven build for this container down to just the interpreters we care about, because the deps here are frankly ridiculous. This might be a case where, if possible, we might want to open an upstream request to build things dynamically, then use something like probably the cut the image down considerably. (This might already be possible, need to poke at whether you can late-bind interpreters later.)
2015-11-05 22:56:15 +00:00
push-zeppelin: zeppelin
gcloud docker push gcr.io/google_containers/zeppelin
gcloud docker push gcr.io/google_containers/zeppelin:$(ZEPPELIN_TAG)
Spark: Bump images to 1.5.1, add Makefile
2015-10-24 13:48:41 +00:00
clean:
Add cleaning to clean target
2015-11-20 14:45:51 +00:00
docker rmi gcr.io/google_containers/spark-worker:$(TAG) || :
docker rmi gcr.io/google_containers/spark-worker || :
docker rmi gcr.io/google_containers/spark-master:$(TAG) || :
docker rmi gcr.io/google_containers/spark-master || :
docker rmi gcr.io/google_containers/spark-driver:$(TAG) || :
docker rmi gcr.io/google_containers/spark-driver || :
docker rmi gcr.io/google_containers/spark-base:$(TAG) || :
docker rmi gcr.io/google_containers/spark-base || :
docker rmi gcr.io/google_containers/zeppelin:$(ZEPPELIN_TAG) || :
docker rmi gcr.io/google_containers/zeppelin || :