Table of Contents
This chapter describes Tibero Active Cluster's concepts, components, and processes, and explains how to use it.
Tibero Active Cluster (hereafter TAC) is the main feature of Tibero for providing high scalability and availability. All instances executed in a TAC environment execute transactions using a shared database. Access to the shared database is mutually controlled for data consistency and conformity.
Processing time can be reduced because a larger job can be divided into smaller jobs, and then the jobs can be performed by several nodes.
Multiple systems share data files based on shared disks. Nodes act as if they use a single shared cache by sending and receiving the data blocks necessary to organize TAC through a high speed private network that connects the nodes.
Even if a node stops while operating, other nodes will continue their services. This transition happens quickly and transparently.
The structure of TAC is shown below:
TAC consists of the following modules:
-
Global Wait-Lock Adapter (GWA)
-
Sets and manages the CWS Lock Status Block (hereafter LKSB), the handle to access CWS, and its parameters.
-
Changes the lock mode and timeout used in Wlock depending on CWS, and registers the Complete Asynchronous Trap (hereafter CAST) and Blocking Asynchronous Trap (hereafter BAST) used in CWS.
-
-
-
Controls access to data blocks in a cluster. DLM is embedded.
-
CR Block Server, Current Block Server, Global Dirty Image, and Global Write services are included.
-
The Cache layer can access CCC through GCA (Global Cache Adapter). The related background threads are: CATH, CLGC, and CRCF.
-
-
-
Provides an interface that allows the Cache layer to use the CCC service.
-
Sets and manages CCC LKSB, the handle to access CCC, and its parameters. It also changes the block lock mode used in the Cache layer for CCC.
-
Saves data blocks and Redo logs for the lock-down event of CCC and offers an interface for DBWR to request a Global write and for CCC to request a block write from DBWR.
-
CCC sends and receives CR blocks, Global dirty blocks, and current blocks through GCA.
-
-
Message Transmission Control (MTC)
-
Solves the problem of message loss between nodes and out-of-order messages.
-
Manages the retransmission queue and out-of-order message queue.
-
Guarantees the reliability of communication between nodes in modules such as CWS and CCC by providing General Message Control (GMC).
Inter-Instance Call (IIC), Distributed Deadlock Detection (hereafter DDD), and Automatic Workload Management currently use GMC for communication between nodes.
-
-
Inter-Node Communication (INC)
-
Provides network connections between nodes.
-
Transparently provides network topology and protocols to users of INC and manages protocols such as TCP and UDP.
-
-
-
Manages weights that show the workload and information received from TBCM such as the node ID, IP address, port number, and incarnation number.
-
Provides a function to look up, add, or remove node membership. The related background thread is NMGR.
-
One process (ACSD, Active Cluster Service Daemon) is created in TACUAC. This process consists of ACCT, NMGR, DIAG, WRCF, CRCF, WLGC, CLGC, WATH, CATH, and CMPT threads. Each thread is included in the following groups:
-
Active Cluster Control Thread
-
ACCT is the thread responsible for message communication between clusters. It receives lock operation and reconfiguration requests of CWS and CCC from a remote node and sends them to CMPT. Or, it receives a message from a session of its own node and sends it to a remote node. It is a main thread of the ACSD process, which is responsible for monitoring other threads.
-
-
Diagnostic Thread
-
Cluster Message Processor (Message handler)
-
CMPT is a thread that processes a request sent from another node. The number of CMPT threads created in the common pool equals the value of the ACF_CMPT_CNT initialization parameter.
CMPT performs the following tasks after receiving a message from ACCT.
-
Receives a CR block request, creates a CR block for a given snapshot, and then transfers this to the requester.
-
Receives a current block request, reads a current block that exists in the local block cache, and then transfers it to the requester.
-
Receives a global write request and makes BLKW write this to disk if the given data block is dirty.
-
Receives a MTC IIC request and handles it.
-
Receives MLD (Master Lookup Directory) lookup and deletion requests and handles them.
-
-
-
Asynchronous Thread
-
Handles asynchronous jobs in place of the working threads that are responsible for sessions in CWS and CCC.
The thread WATH and CATH have the following characteristics:
-
When they receive a BAST request or release a lock, they notify the master node of a lock downgrade after receiving a disk write notification from BLKW or a log flush reply from LOGW.
This can only be performed by the CATH thread.
-
BLKW handles global write requests that were received from the master node and then notifies CATH, and sends write notifications to the master node.
This can only be performed by the CATH thread.
-
Sends requests to remove MC locks to the master node before reclaiming shadow resource blocks. After sending the request, it receives a response and handles it.
-
-
-
Garbage Collector
-
Manages lock resources and checks timeout periodically.
The threads WLGC and CLGC have the following characteristics:
-
Periodically checks if lock waiter has timed out before performing Distributed Deadlock Detection (DDD). If there is a deadlock, it forces DDD to start.
-
Periodically checks messages in the MTC retransmission queue and resends messages that have timed out.
-
Periodically performs synchronization of TSN. This can only be performed by the WLGC process.
-
If the available shared memory for lock resources is low, secures the necessary resources by reclaiming resource blocks.
-
-
-
Reconfigurator
The configuration for TAC includes settings for a single instance by default. Apart from the basic settings, the following initialization parameters should also be specified to use TAC:
<$TB_SID.tip>
MEMORY_TARGET=6144M TOTAL_SHM_SIZE=4096M DB_CACHE_SIZE=2048M CLUSTER_DATABASE=Y THREAD=0 UNDO_TABLESPACE=UNDO0 LOCAL_CLUSTER_ADDR=192.168.1.1 LOCAL_CLUSTER_PORT=12345 CM_PORT=30000
| Initialization Parameter | Description |
|---|---|
MEMORY_TARGET | Size of the total memory. This value includes shared memory, memory used in sort and hash operations, and other memory used in the DBMS. Shared memory is allocated when the DBMS starts up. Remaining memory is allocated and deallocated as needed. |
TOTAL_SHM_SIZE | Total size of shared memory. |
DB_CACHE_SIZE | TAC needs shared memory for other purposes besides buffer cache. Therefore, the total size of buffer cache must be less than the shared memory of a single instance. Typical size is about half of the total shared memory. |
CLUSTER_DATABASE | Option to use TAC. Must be set to Y. |
THREAD | Number of Redo threads. A unique number is allocated to each instance. |
UNDO_TABLESPACE | Name of the Undo tablespace. A unique number is allocated to each instance. |
LOCAL_CLUSTER_ADDR | IP address used to communicate between TAC instances. |
LOCAL_CLUSTER_PORT | Internal port number used to communicate between TAC instances. This port must be open between nodes. |
CM_PORT | Port number for communication with CM. Must be the same as the value set in CM_UI_PORT, the CM initialization parameter. |
Note the following about TAC:
-
Redo thread numbers and the Undo tablespace name must be unique among servers that use the same database. The name should be the same as the name created in “10.5. Creating a Database for TAC”.
-
CONTROL_FILES and DB_CREATE_FILE_DEST must be the same physical file or specified as the same device in the configuration files of instances in all servers.
TAC is a clustered database based on a shared disk. The database only needs to be created once by a single server because instances of several database servers use the same physical database files.
Instances of all servers read and write the same control and data files. However, in TAC, each instance has a separate file for Redo logs and Undo information to minimize data access on the shared disk. Redo logs and Undo information are saved by an instance of each server in a separate file, but they must exist on the shared disk because other instances also read them for recovery.
The process to create a database for TAC is as follows:
-
Set a configuration file for Tibero and then start TBCM. For information about configuring and starting up CM, refer to “Chapter 9. Tibero Cluster Manager”.
$ tbcm -bTBCM starts, but it cannot directly control Tibero.
-
Start Tibero in NOMOUNT mode.
$ tbboot -t NOMOUNT -c -
Connect with the SYS user and create a database with the CREATE DATABASE statement.
[tibero@tester ~]$ tbsql sys/tibero SQL> CREATE DATABASE "ac" ... (1) ... USER sys IDENTIFIED BY tibero MAXINSTANCES 8 ... (2) ... MAXDATAFILES 256 CHARACTER SET MSWIN949 LOGFILE GROUP 0 'log001' SIZE 50M, GROUP 1 'log011' SIZE 50M, GROUP 2 'log021' SIZE 50M MAXLOGFILES 100 MAXLOGMEMBERS 8 NOARCHIVELOG DATAFILE 'system001' SIZE 512M AUTOEXTEND ON NEXT 8M MAXSIZE 3G DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE 'temp001' SIZE 512M AUTOEXTEND ON NEXT 8M MAXSIZE 3G EXTENT MANAGEMENT LOCAL AUTOALLOCATE UNDO TABLESPACE UNDO0 DATAFILE 'undo001' SIZE 512M AUTOEXTEND ON NEXT 8M MAXSIZE 3G EXTENT MANAGEMENT LOCAL AUTOALLOCATE Database created.
(1) Specify the database name.
(2) The maximum number of instances for a server is specified as 8.
This is a standard CREATE DATABASE statement. However, note that the MAXINSTANCES parameter indicates the maximum number of instances that share the database files. This parameter value affects the headers of control and data files. Because only the number of instances of Tibero specified in this parameter can be created, allocate a sufficient amount of instances when the database is created.
As described above, an instance of each server needs to have a separate Redo and Undo space. The Undo tablespace and Redo log file that are created when the CREATE DATABASE statement is executed are used for the first instance. Separate Redo log groups and Undo tablespaces need to be created for the other instances to access the database.
The created Redo log group automatically becomes the zeroth Redo thread. Therefore, the initialization parameters
THREADandUNDO_TABLESPACEhave to be specified in the server instance's environment configuration file $TB_SID.tip when the CREATE DATABASE statement is executed, as follows:THREAD=0 UNDO_TABLESPACE=UNDO0
-
After the CREATE DATABASE statement has been executed, Tibero is automatically terminated. The following example connects with the SYS user, creates an Undo tablespace and a Redo log group, and executes the DDL statement after Tibero restarts:
[tibero@tester ~]$ tbboot [tibero@tester ~]$ tbsql sys/tibero SQL> CREATE UNDO TABLESPACE UNDO1 DATAFILE 'undo011' SIZE 512M AUTOEXTEND ON NEXT 8M MAXSIZE 3G EXTENT MANAGEMENT LOCAL AUTOALLOCATE Tablespace 'UNDO1' created.
SQL> ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 3 'log031' size 50M; Database altered. SQL> ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 4 'log041' size 50M; Database altered. SQL> ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 5 'log051' size 50M; Database altered. SQL> ALTER DATABASE ENABLE PUBLIC THREAD 1; ... (1) ... Database altered.(1) The DDL statement that activates the Redo thread is executed.
This DDL statement is the same as the standard DDL statement to add a Redo log group. However, note that it designates a THREAD number. This example adds the Undo tablespace UNDO1 used by the second instance and the Redo log group for the Redo thread 1 and activates them.
Redo threads are designated with numbers starting from 1. The Redo log group created when the CREATE DATABASE statement is executed becomes the zeroth thread. The zeroth thread is automatically activated when the CREATE DATABASE statement is executed.
Note
Note that the numbering of Redo log groups must be unique across the entire database, not just within a Redo thread. Therefore, 0, 1, and 2 cannot be used. At least two Redo log groups must exist to activate the corresponding Redo thread. To add another instance, create an Undo tablespace and a Redo thread and activate the thread as shown above.
-
If DB_CREATE_FILE_DEST is not properly specified in a TAC raw device environment or a shared file system, Tibero may not start normally.
If Tibero does not start normally, first add the tablespace (SYSSUB) where TPR-related information is saved as follows:
[tibero@tester ~]$ tbsql sys/tibero SQL> CREATE TABLESPACE SYSSUB DATAFILE '<SYSSUB Path>/syssub001.dtf' ...; Tablespace 'SYSSUB' created.
Note
If this step is omitted and the next step is executed, refer to "Chapter 7. TAC Installation and Uninstallation" and "Appendix A. Troubleshooting" in Tibero Installation Guide.
-
Execute the script file system.sh in the directory $TB_HOME/scripts. In the Windows environment, system.vbs should be executed.
[tibero@tester scripts]$ system.sh $TB_HOME/bin/tbsvr Creating the role DBA... Creating system users & roles... Creating virtual tables(1)... Creating virtual tables(2)... Granting public access to _VT_DUAL... Creating the system generated sequences... Creating system packages: Running /home/tibero/tibero6/scripts/pkg_standard.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_output.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_lob.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_utility.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_obfuscation.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_transaction.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_random.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_lock.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_system.sql... Running /home/tibero/tibero6/scripts/pkg_dbms_job.sql... Running /home/tibero/tibero6/scripts/pkg_utl_raw.sql... Running /home/tibero/tibero6/scripts/pkg_utl_file.sql... Running /home/tibero/tibero6/scripts/pkg_tb_utility.sql... Creating public synonyms for system packages... ............................................
This section describes preparations for TAC execution and explains how to create a database and start and monitor TAC.
TAC can be executed if there is shared disk space that can be used by all instances and a network that enables communication between instances.
The hardware requirements and OS settings for executing TAC are as follows:
-
Shared Disk Space
-
At least seven shared files are necessary to execute and operate TAC:
-
A control file
-
Two Redo log files
-
An undo log file
-
A system tablespace file
-
A temporary tablespace file
-
A TBCM file
-
-
At least three shared files are necessary whenever an instance is added.
-
Two Redo log files
-
An Undo log file
-
-
-
Shared Disk Permissions
If a shared file system is used, permissions for the directory used by TAC must be specified. If a RAW file system is used, TAC must be specified to read and write the permissions of each RAW file.
-
Internal Network Settings
It is recommended that the internal network be separate from the external network to ensure that TAC has better database performance. Check the internal network interfaces, IP addresses, and speed.
Note
TAC does not support using a crossover cable to set up an internal network .
When TAC starts, a database should be created in a node that will be used in the cluster. TBCM is required when Tibero starts with TAC. Therefore, start TBCM first, and then start Tibero in NOMOUNT mode. For information about configuring CM, refer to “Chapter 9. Tibero Cluster Manager”.
The following example initializes TBCM, and then starts Tibero in NOMOUNT mode:
tac1@tester ~]$ tbboot -t NOMOUNT listener port = xxxx change core dump dir to /home/tibero6/bin/prof Tibero 6 TmaxData Corporation Copyright (c) 2008-. All rights reserved. Tibero instance started up (NOMOUNT mode). tac1@tester ~]$
A database is created by accessing the first TAC
instance that starts in NOMOUNT mode, and then Redo and Undo logs for
other instances are created. They must have the same names as specified
in the initialization parameters THREAD and
UNDO_TABLESPACE in the configuration file $TB_SID.tip.
TAC can start after a database is created and a configuration file for other instances is created.
The following example executes all instances and starts TAC in the order of TBCM and Tibero:
tac2@tester ~]$ tbcm -b CM Guard daemon started up. import resources from 'resource file path in CM TIP file'... Tibero 6 TmaxData Corporation Copyright (c) 2008-. All rights reserved. Tibero cluster manager started up. Local node name is (cm0:xxxx). tac2@tester ~]$ tbboot listener port = xxxx change core dump dir to /home/tibero6/bin/prof Tibero 6 TmaxData Corporation Copyright (c) 2008-. All rights reserved. Tibero instance started up (NORMAL mode). tac2@tester ~]$
The status of nodes and instances can be monitored using the script file tbcm_stat.sh in the directory $TB_HOME/scripts. The current status and IP information of nodes that participate in TAC can be periodically monitored.
Tibero provides a global view. All monitoring views used in a single instance can be viewed by TAC instances.
The following example monitors sessions connected to all clusters. The following SQL statement is executed using the tbSQL utility.