Configuration change and rolling restart with MySQL Cluster 7.0

This article will describe how we start this cluster after a shut down, changing configuration and do a rolling restart.

The configuration

The config.ini stored on the first management node:

[NDBD DEFAULT]
Datadir=/data2/users/geert/cluster/master
NoOfReplicas=2
DataMemory=80M
IndexMemory=10M

[NDB_MGMD DEFAULT]
Datadir=/data2/users/geert/cluster/master

[NDB_MGMD]
Id=1
Hostname = machine-1

[NDB_MGMD]
Id=2
Hostname = machine-2

[NDBD]
Id=3
Hostname = machine-3

[NDBD]
Id=4
Hostname = machine-4

[API]
[API]
[API]
[API]

Starting MySQL Cluster 7.0 with 2 management nodes

We assume that the cluster from previous post was shut down and we need to restart it. Here are the instructions to do so.
We start both management node process the same way, without the options --initial or --reload!

# machine-1
 shell> ndb_mgmd --configdir=/path/to/configcache/dir
 ..NDB Cluster Management Server. mysql-5.1.35 ndb-7.0.7
 ..Loaded config from '/path/to/configcache/dir/ndb_1_config.bin.1'

# machine-2
shell> ndb_mgmd --configdir=/path/to/configcache/dir
..NDB Cluster Management Server. mysql-5.1.35 ndb-7.0.7
..Loaded config from '/path/to/configcache/dir/ndb_2_config.bin.1'

Data nodes are started on machine-3 and machine-4:

# machine-3
shell> ndbd -c machine-1
..Configuration fetched from 'machine-1', generation: 1

# machine-4
shell> ndbd -c machine-2
..Configuration fetched from 'machine-2', generation: 1

That should bring your cluster back up, ready for some experiments!

Rolling restart after configuration change

Lets assume we want more memory to store data and index information. We change the following in the configuration file config.ini, which you find on the first management node, machine-1:

[NDBD DEFAULT]
DataMemory=160M
IndexMemory=20M

Save your new config.ini and kill the ndb_mgmd process on machine-1, followed by starting it again with the--reload option:

 # machine-1
 shell> killall ndb_mgmd
 shell> ndb_mgmd -f config.ini --reload --configdir=/path/to/configcache/dir
 ..NDB Cluster Management Server. mysql-5.1.35 ndb-7.0.7
 ..Loaded config from '/path/to/configcache/dir/ndb_1_config.bin.1'

The above output might be a bit confusing: we started with a changed config.ini but it said it loaded config from the previous cached version. This is normal. It needs to first read the old to know the changes from the new. The real magic is shown in the cluster log ndb_1_cluster.log on machine-1 (simplified for this blog post):

.. Detected change of config.ini on disk, will try to set it 
when all ndb_mgmd(s) started. This is the actual diff:
[ndbd(DB)]
NodeId=3
-IndexMemory=10485760
+IndexMemory=20971520

[ndbd(DB)]
NodeId=4
-IndexMemory=10485760
+IndexMemory=20971520
..
Node 2 connected
Starting configuration change, generation: 1
Configuration 2 commited
Config change completed! New generation: 2

Notice that currently in MySQL Cluster 7.0.7 there is bug that when changing 2 parameters, only 1 will show up when the difference is shown in the logs.
On the second management node you’ll find in ndb_2_cluster.log something like this:

..Node 2: Node 1 Connected
..
..Configuration 2 commited

Both management nodes have agreed on the same configuration, and both have it binary cached in files named likendb_2_config.bin.*.
We continue now with restarting the data nodes, while connected to either management node do the following:

 # machine-1 or machine-2
 shell> ndb_mgm
 ndb_mgm> ALL REPORT MEMORY USAGE
 Node 3: Data usage is 0%(4 32K pages of total 2560)
 Node 3: Index usage is 0%(8 8K pages of total 1312)
 Node 4: Data usage is 0%(4 32K pages of total 2560)
 Node 4: Index usage is 0%(8 8K pages of total 1312)
 ndb_mgm> 3 RESTART
 Node 3: Node shutdown initiated
 Node 3: Node shutdown completed, restarting, no start.
 Node 3 is being restarted
 Node 3: Started (version 7.0.7)
 ndb_mgm> 4 RESTART
 Node 4: Node shutdown initiated
 Node 4: Node shutdown completed, restarting, no start.
 Node 4 is being restarted
 Node 4: Data usage decreased to 0%(0 32K pages of total 5120)
 Node 4: Started (version 7.0.7)
 ndb_mgm> ALL REPORT MEMORY USAGE
 Node 3: Data usage is 0%(6 32K pages of total 5120)
 Node 3: Index usage is 0%(8 8K pages of total 2592)
 Node 4: Data usage is 0%(6 32K pages of total 5120)
 Node 4: Index usage is 0%(8 8K pages of total 2592)

The ALL REPORT MEMORY USAGE output shows that the configuration took effect and a rolling restart was succesful.

Laat een reactie achter

Het e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *