This article will describe how we start this cluster after a shut down, changing configuration and do a rolling restart.
The configuration
The config.ini stored on the first management node:
[NDBD DEFAULT] Datadir=/data2/users/geert/cluster/master NoOfReplicas=2 DataMemory=80M IndexMemory=10M [NDB_MGMD DEFAULT] Datadir=/data2/users/geert/cluster/master [NDB_MGMD] Id=1 Hostname = machine-1 [NDB_MGMD] Id=2 Hostname = machine-2 [NDBD] Id=3 Hostname = machine-3 [NDBD] Id=4 Hostname = machine-4 [API] [API] [API] [API]
Starting MySQL Cluster 7.0 with 2 management nodes
We assume that the cluster from previous post was shut down and we need to restart it. Here are the instructions to do so.
We start both management node process the same way, without the options --initial or --reload!
# machine-1 shell> ndb_mgmd --configdir=/path/to/configcache/dir ..NDB Cluster Management Server. mysql-5.1.35 ndb-7.0.7 ..Loaded config from '/path/to/configcache/dir/ndb_1_config.bin.1' # machine-2 shell> ndb_mgmd --configdir=/path/to/configcache/dir ..NDB Cluster Management Server. mysql-5.1.35 ndb-7.0.7 ..Loaded config from '/path/to/configcache/dir/ndb_2_config.bin.1'
Data nodes are started on machine-3 and machine-4:
# machine-3 shell> ndbd -c machine-1 ..Configuration fetched from 'machine-1', generation: 1 # machine-4 shell> ndbd -c machine-2 ..Configuration fetched from 'machine-2', generation: 1
That should bring your cluster back up, ready for some experiments!
Rolling restart after configuration change
Lets assume we want more memory to store data and index information. We change the following in the configuration file config.ini, which you find on the first management node, machine-1:
[NDBD DEFAULT] DataMemory=160M IndexMemory=20M
Save your new config.ini and kill the ndb_mgmd process on machine-1, followed by starting it again with the--reload option:
# machine-1 shell> killall ndb_mgmd shell> ndb_mgmd -f config.ini --reload --configdir=/path/to/configcache/dir ..NDB Cluster Management Server. mysql-5.1.35 ndb-7.0.7 ..Loaded config from '/path/to/configcache/dir/ndb_1_config.bin.1'
The above output might be a bit confusing: we started with a changed config.ini but it said it loaded config from the previous cached version. This is normal. It needs to first read the old to know the changes from the new. The real magic is shown in the cluster log ndb_1_cluster.log on machine-1 (simplified for this blog post):
.. Detected change of config.ini on disk, will try to set it when all ndb_mgmd(s) started. This is the actual diff: [ndbd(DB)] NodeId=3 -IndexMemory=10485760 +IndexMemory=20971520 [ndbd(DB)] NodeId=4 -IndexMemory=10485760 +IndexMemory=20971520 .. Node 2 connected Starting configuration change, generation: 1 Configuration 2 commited Config change completed! New generation: 2
Notice that currently in MySQL Cluster 7.0.7 there is bug that when changing 2 parameters, only 1 will show up when the difference is shown in the logs.
On the second management node you’ll find in ndb_2_cluster.log something like this:
..Node 2: Node 1 Connected .. ..Configuration 2 commited
Both management nodes have agreed on the same configuration, and both have it binary cached in files named likendb_2_config.bin.*.
We continue now with restarting the data nodes, while connected to either management node do the following:
# machine-1 or machine-2 shell> ndb_mgm ndb_mgm> ALL REPORT MEMORY USAGE Node 3: Data usage is 0%(4 32K pages of total 2560) Node 3: Index usage is 0%(8 8K pages of total 1312) Node 4: Data usage is 0%(4 32K pages of total 2560) Node 4: Index usage is 0%(8 8K pages of total 1312) ndb_mgm> 3 RESTART Node 3: Node shutdown initiated Node 3: Node shutdown completed, restarting, no start. Node 3 is being restarted Node 3: Started (version 7.0.7) ndb_mgm> 4 RESTART Node 4: Node shutdown initiated Node 4: Node shutdown completed, restarting, no start. Node 4 is being restarted Node 4: Data usage decreased to 0%(0 32K pages of total 5120) Node 4: Started (version 7.0.7) ndb_mgm> ALL REPORT MEMORY USAGE Node 3: Data usage is 0%(6 32K pages of total 5120) Node 3: Index usage is 0%(8 8K pages of total 2592) Node 4: Data usage is 0%(6 32K pages of total 5120) Node 4: Index usage is 0%(8 8K pages of total 2592)
The ALL REPORT MEMORY USAGE output shows that the configuration took effect and a rolling restart was succesful.