github
/
k3s
mirror of https://github.com/k3s-io/k3s
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
k3s/examples/elasticsearch/README.md

322 lines
14 KiB
Markdown
Raw Normal View History

Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
# Elasticsearch for Kubernetes
This directory contains the source for a Docker image that creates an instance
of [Elasticsearch](https://www.elastic.co/products/elasticsearch) 1.5.2 which can
be used to automatically form clusters when used
Added inline links to "services" "pods" "namespaces" "secrets" and "replication controllers" (using relative linking ../../folder/filename.md)
2015-06-02 17:24:27 +00:00
with [replication controllers](../../docs/replication-controller.md). This will not work with the library Elasticsearch image
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
because multicast discovery will not find the other pod IPs needed to form a cluster. This
Added inline links to "services" "pods" "namespaces" "secrets" and "replication controllers" (using relative linking ../../folder/filename.md)
2015-06-02 17:24:27 +00:00
image detects other Elasticsearch [pods](../../docs/pods.md) running in a specified [namespace](../../docs/namespaces.md) with a given
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
label selector. The detected instances are used to form a list of peer hosts which
are used as part of the unicast discovery mechansim for Elasticsearch. The detection
of the peer nodes is done by a program which communicates with the Kubernetes API
server to get a list of matching Elasticsearch pods. To enable authenticated
Added inline links to "services" "pods" "namespaces" "secrets" and "replication controllers" (using relative linking ../../folder/filename.md)
2015-06-02 17:24:27 +00:00
communication this image needs a [secret](../../docs/secrets.md) to be mounted at `/etc/apiserver-secret`
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
with the basic authentication username and password.
Here is an example replication controller specification that creates 4 instances of Elasticsearch which is in the file
[music-rc.yaml](music-rc.yaml).
```
update examples/elasticsearch to v1
2015-06-09 21:57:25 +00:00
apiVersion: v1
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
kind: ReplicationController
metadata:
labels:
name: music-db
namespace: mytunes
name: music-db
spec:
replicas: 4
selector:
name: music-db
template:
metadata:
labels:
name: music-db
spec:
containers:
- name: es
image: kubernetes/elasticsearch:1.0
env:
- name: "CLUSTER_NAME"
value: "mytunes-db"
- name: "SELECTOR"
value: "name=music-db"
- name: "NAMESPACE"
value: "mytunes"
ports:
- name: es
containerPort: 9200
- name: es-transport
containerPort: 9300
volumeMounts:
- name: apiserver-secret
mountPath: /etc/apiserver-secret
readOnly: true
volumes:
- name: apiserver-secret
secret:
secretName: apiserver-secret
```
The `CLUSTER_NAME` variable gives a name to the cluster and allows multiple separate clusters to
exist in the same namespace.
The `SELECTOR` variable should be set to a label query that identifies the Elasticsearch
nodes that should participate in this cluster. For our example we specify `name=music-db` to
match all pods that have the label `name` set to the value `music-db`.
The `NAMESPACE` variable identifies the namespace
to be used to search for Elasticsearch pods and this should be the same as the namespace specified
for the replication controller (in this case `mytunes`).
Before creating pods with the replication controller a secret containing the bearer authentication token
should be set up. A template is provided in the file [apiserver-secret.yaml](apiserver-secret.yaml):
```
update examples/elasticsearch to v1
2015-06-09 21:57:25 +00:00
apiVersion: v1
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
kind: Secret
metadata:
name: apiserver-secret
namespace: NAMESPACE
data:
token: "TOKEN"
```
Replace `NAMESPACE` with the actual namespace to be used and `TOKEN` with the basic64 encoded
versions of the bearer token reported by `kubectl config view` e.g.
```
$ kubectl config view
...
- name: kubernetes-logging_kubernetes-basic-auth
...
token: yGlDcMvSZPX4PyP0Q5bHgAYgi1iyEHv2
...
$ echo yGlDcMvSZPX4PyP0Q5bHgAYgi1iyEHv2 | base64
eUdsRGNNdlNaUFg0UHlQMFE1YkhnQVlnaTFpeUVIdjIK=
```
resulting in the file:
```
update examples/elasticsearch to v1
2015-06-09 21:57:25 +00:00
apiVersion: v1
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
kind: Secret
metadata:
name: apiserver-secret
namespace: mytunes
data:
token: "eUdsRGNNdlNaUFg0UHlQMFE1YkhnQVlnaTFpeUVIdjIK="
```
which can be used to create the secret in your namespace:
```
kubectl create -f apiserver-secret.yaml --namespace=mytunes
secrets/apiserver-secret
```
Now you are ready to create the replication controller which will then create the pods:
```
$ kubectl create -f music-rc.yaml --namespace=mytunes
replicationcontrollers/music-db
```
update examples/elasticsearch to v1
2015-06-09 21:57:25 +00:00
It's also useful to have a [service](../../docs/services.md) with an load balancer for accessing the Elasticsearch
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
cluster which can be found in the file [music-service.yaml](music-service.yaml).
```
update examples/elasticsearch to v1
2015-06-09 21:57:25 +00:00
apiVersion: v1
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
kind: Service
metadata:
name: music-server
namespace: mytunes
labels:
name: music-db
spec:
selector:
name: music-db
ports:
- name: db
port: 9200
targetPort: es
update examples/elasticsearch to v1
2015-06-09 21:57:25 +00:00
type: LoadBalancer
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
```
Let's create the service with an external load balancer:
```
$ kubectl create -f music-service.yaml --namespace=mytunes
services/music-server
```
Let's see what we've got:
```
$ kubectl get pods,rc,services,secrets --namespace=mytunes
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE
rename resize to scale
2015-05-21 21:10:25 +00:00
music-db-0fwsu 10.244.2.48 kubernetes-minion-m49b/104.197.35.221 name=music-db Running 6 minutes
es kubernetes/elasticsearch:1.0 Running 29 seconds
music-db-5pc2e 10.244.0.24 kubernetes-minion-3c8c/146.148.41.184 name=music-db Running 6 minutes
es kubernetes/elasticsearch:1.0 Running 6 minutes
music-db-bjqmv 10.244.3.31 kubernetes-minion-zey5/104.154.59.10 name=music-db Running 6 minutes
es kubernetes/elasticsearch:1.0 Running 19 seconds
music-db-swtrs 10.244.1.37 kubernetes-minion-f9dw/130.211.159.230 name=music-db Running 6 minutes
es kubernetes/elasticsearch:1.0 Running 6 minutes
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
music-db es kubernetes/elasticsearch:1.0 name=music-db 4
NAME LABELS SELECTOR IP(S) PORT(S)
music-server name=music-db name=music-db 10.0.138.61 9200/TCP
rename resize to scale
2015-05-21 21:10:25 +00:00
104.197.12.157
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
NAME TYPE DATA
apiserver-secret Opaque 2
```
This shows 4 instances of Elasticsearch running. After making sure that port 9200 is accessible for this cluster (e.g. using a firewall rule for GCE) we can make queries via the service which will be fielded by the matching Elasticsearch pods.
```
$ curl 104.197.12.157:9200
{
"status" : 200,
"name" : "Warpath",
"cluster_name" : "mytunes-db",
"version" : {
"number" : "1.5.2",
"build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
"build_timestamp" : "2015-04-27T09:21:06Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
$ curl 104.197.12.157:9200
{
"status" : 200,
"name" : "Callisto",
"cluster_name" : "mytunes-db",
"version" : {
"number" : "1.5.2",
"build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
"build_timestamp" : "2015-04-27T09:21:06Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
```
We can query the nodes to confirm that an Elasticsearch cluster has been formed.
```
$ curl 104.197.12.157:9200/_nodes?pretty=true
{
"cluster_name" : "mytunes-db",
"nodes" : {
"u-KrvywFQmyaH5BulSclsA" : {
"name" : "Jonas Harrow",
...
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
},
...
"name" : "Warpath",
...
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
},
...
"name" : "Callisto",
...
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
},
...
"name" : "Vapor",
...
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : [ "10.244.2.48", "10.244.0.24", "10.244.3.31", "10.244.1.37" ]
...
```
Let's ramp up the number of Elasticsearch nodes from 4 to 10:
```
rename resize to scale
2015-05-21 21:10:25 +00:00
$ kubectl scale --replicas=10 replicationcontrollers music-db --namespace=mytunes
scaled
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
$ kubectl get pods --namespace=mytunes
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE
rename resize to scale
2015-05-21 21:10:25 +00:00
music-db-0fwsu 10.244.2.48 kubernetes-minion-m49b/104.197.35.221 name=music-db Running 33 minutes
es kubernetes/elasticsearch:1.0 Running 26 minutes
music-db-2erje 10.244.2.50 kubernetes-minion-m49b/104.197.35.221 name=music-db Running 48 seconds
es kubernetes/elasticsearch:1.0 Running 46 seconds
music-db-5pc2e 10.244.0.24 kubernetes-minion-3c8c/146.148.41.184 name=music-db Running 33 minutes
es kubernetes/elasticsearch:1.0 Running 32 minutes
music-db-8rkvp 10.244.3.33 kubernetes-minion-zey5/104.154.59.10 name=music-db Running 48 seconds
es kubernetes/elasticsearch:1.0 Running 46 seconds
music-db-bjqmv 10.244.3.31 kubernetes-minion-zey5/104.154.59.10 name=music-db Running 33 minutes
es kubernetes/elasticsearch:1.0 Running 26 minutes
music-db-efc46 10.244.2.49 kubernetes-minion-m49b/104.197.35.221 name=music-db Running 48 seconds
es kubernetes/elasticsearch:1.0 Running 46 seconds
music-db-fhqyg 10.244.0.25 kubernetes-minion-3c8c/146.148.41.184 name=music-db Running 48 seconds
es kubernetes/elasticsearch:1.0 Running 47 seconds
music-db-guxe4 10.244.3.32 kubernetes-minion-zey5/104.154.59.10 name=music-db Running 48 seconds
es kubernetes/elasticsearch:1.0 Running 46 seconds
music-db-pbiq1 10.244.1.38 kubernetes-minion-f9dw/130.211.159.230 name=music-db Running 48 seconds
es kubernetes/elasticsearch:1.0 Running 47 seconds
music-db-swtrs 10.244.1.37 kubernetes-minion-f9dw/130.211.159.230 name=music-db Running 33 minutes
es kubernetes/elasticsearch:1.0 Running 32 minutes
Elasticsearch Discovery with Kubernetes
2015-05-07 00:29:24 +00:00
```
Let's check to make sure that these 10 nodes are part of the same Elasticsearch cluster:
```
$ curl 104.197.12.157:9200/_nodes?pretty=true | grep name
"cluster_name" : "mytunes-db",
"name" : "Killraven",
"name" : "Killraven",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Tefral the Surveyor",
"name" : "Tefral the Surveyor",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Jonas Harrow",
"name" : "Jonas Harrow",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Warpath",
"name" : "Warpath",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Brute I",
"name" : "Brute I",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Callisto",
"name" : "Callisto",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Vapor",
"name" : "Vapor",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Timeslip",
"name" : "Timeslip",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Magik",
"name" : "Magik",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
"name" : "Brother Voodoo",
"name" : "Brother Voodoo",
"name" : "mytunes-db"
"vm_name" : "OpenJDK 64-Bit Server VM",
"name" : "eth0",
Add ga-beacon analytics to gendocs scripts hack/run-gendocs.sh puts ga-beacon analytics link into all md files, hack/verify-gendocs.sh verifies presence of link.
2015-05-14 22:12:45 +00:00
```
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/examples/elasticsearch/README.md?pixel)]()