Merge branch 'release/InitRepo'

This commit is contained in:
Samuele Locatelli
2026-02-12 18:43:14 +01:00
14 changed files with 452 additions and 82 deletions
+14
View File
@@ -0,0 +1,14 @@
SELECT CONCAT(
'ALTER TABLE ', TABLE_SCHEMA, '.', TABLE_NAME, ' MODIFY ', COLUMN_NAME, ' ', COLUMN_TYPE, ' NOT NULL DEFAULT \'\';'
) AS query_di_correzione
FROM INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_SCHEMA = 'eqn_prod'
AND DATA_TYPE IN (
'blob',
'tinyblob',
'mediumblob',
'longblob'
)
AND IS_NULLABLE = 'NO'
AND COLUMN_DEFAULT IS NULL;
+19
View File
@@ -0,0 +1,19 @@
-- Parametri di configurazione (Imposta il tuo database qui)
SET @TARGET_DB = 'eqn_prod';
SET @TARGET_CHARSET = 'utf8mb4';
SET @TARGET_COLLATION = 'utf8mb4_unicode_ci';
SET @DRY_RUN = 0; -- Cambia a 0 per ottenere i comandi pronti all'esecuzione
SELECT
CASE
WHEN @DRY_RUN = 1 THEN CONCAT('-- [DRY-RUN] Tabella: ', TABLE_NAME, ' (Attuale: ', TABLE_COLLATION, ')')
ELSE CONCAT('ALTER TABLE ', TABLE_SCHEMA, '.', TABLE_NAME,
' CONVERT TO CHARACTER SET ', @TARGET_CHARSET,
' COLLATE ', @TARGET_COLLATION, ';')
END AS sql_output
FROM
INFORMATION_SCHEMA.TABLES
WHERE
TABLE_SCHEMA = @TARGET_DB
AND TABLE_TYPE = 'BASE TABLE'
AND (TABLE_COLLATION != @TARGET_COLLATION OR TABLE_COLLATION IS NULL);
+12
View File
@@ -0,0 +1,12 @@
SELECT
CONCAT('ALTER TABLE ', TABLE_SCHEMA, '.', TABLE_NAME,
' MODIFY ', COLUMN_NAME, ' ', COLUMN_TYPE,
' NOT NULL DEFAULT 0;') AS query_di_correzione
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_SCHEMA = 'eqn_prod'
AND DATA_TYPE IN ('int', 'tinyint', 'smallint', 'mediumint', 'bigint', 'float', 'double', 'decimal')
AND IS_NULLABLE = 'NO'
AND COLUMN_DEFAULT IS NULL
AND EXTRA NOT LIKE '%auto_increment%';
+11
View File
@@ -0,0 +1,11 @@
SELECT
CONCAT('ALTER TABLE ', TABLE_SCHEMA, '.', TABLE_NAME,
' MODIFY ', COLUMN_NAME, ' ', COLUMN_TYPE,
' NOT NULL DEFAULT \'\';') AS query_di_correzione
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_SCHEMA = 'eqn_prod' -- Sostituisci con il nome del tuo database
AND DATA_TYPE IN ('text', 'mediumtext', 'longtext', 'tinytext', 'varchar')
AND IS_NULLABLE = 'NO'
AND COLUMN_DEFAULT IS NULL;
+80
View File
@@ -0,0 +1,80 @@
# EQN MariaDb
Resoconto attività conversione, script, dettaglio sommario impiego
## Task Svolti
Si è proceduto in questo modo
* migrazione preliminare VM Hyper-V a VM proxmox QEMU (e fix relativi)
* creazione 3 nodi LXC che saranno il futuro cluster, con storage dedicato a DB montato in /var/lib/mysql nell'LXC
* test di migrazione senza interruzione di servizio
* inserimento layer keepalived (solo sul nodo VM originale) per scambio IP fisico / virtual IP (utilizzato in seguito)
* install prerequisiti sui nodi
* ubuntu 24.04
* mariadb
* galera
* tools vari
* keepalived
* preparazione di un nodo (db03) per i test
* script di migrazione (a caldo e da fare poi a insert bloccati)
* interruzione servizio (blocco virtual IP)
* migrazione vera dei dati (old --> db03)
* fix utenti
* riavvio Virtual IP su nuovo nodo
* avvio del nuovo cluster sul primo nodo
* join dei 2 nodi sul cluster
* dismissione vecchia VM
* fix e correzioni varie (es problema ID utenti proxmox x mount --> subfolder)
## Appunti configurazione
Sono allegati ma si lascia in appunto al conf saliente di MariaDb (50-server.cnf e 60-galera.cnf) impiegata nonché x keepalived
Suddivisione cartelle: configurazioni speciali, gestione utils, gestione migrazione.
Si è provato wssrep con mariadb-backup all'inizio ma non ne ha voluto sapere, usato rsync (da rivalutare in futuro alternativa).
## Applicativi necessari
Elenco applicativi installati
```
apt update && apt upgrade -y
apt install vim mc htop dstat ncdu curl
apt install mariadb-server galera-4 mariadb-backup keepalived
```
## Uso script migrazione DB VM --> LXC
Per eseguire gli helper script di migrazione si sono usati alcuni accorgimenti:
* lo script va eseguito sul nodo donor attuale
* db03.eqn è cablato in /etc/hosts verso nodo receiver
* lo script può essere provato live, ma va eseguito a insert disabilitati per una conversione pulita (per farlo con virtual IP si è spento keepalived)
## Uso script fix DB su nuovo cluster
Per l'impiego deglis cript sql di fix dei dati nel db migrato (in particolare x i campi not nullable senza default e problemi di collation) si è usato questo approccio
```
mysql < scriptDaEseguire.sql > resultFile.sql
```
dove il primo script **scriptDaEseguire.sql** è quello operativo, mentre **resultFile.sql** è il risultato che si può poi impiegare
anzi in particolare per evitare intestazioni colonne ed altri campi non necessari
```
mysql -s -N < scriptDaEseguire.sql > resultFile.sql
```
Gli script sono serviti in particolare per
* sistemare null senza default
* sistemare collation (facendo 10-20 alter alla volta perché il cluster era running e non si voleva impattare eccessivamente)
## Appunti a latere
Per impiegare i 3 noidi in modo bilanciato si è installato haproxy direttametne sugli FE in modo che ogni fe, chiamando localmente sulla prota 3306, venga instradato su uno dei 3 nodi in modo trasparente.
Binary file not shown.
+141
View File
@@ -0,0 +1,141 @@
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
#
# * Basic Settings
#
#user = mysql
pid-file = /run/mysqld/mysqld.pid
basedir = /usr
#datadir = /var/lib/mysql
datadir = /var/lib/mysql/data
#tmpdir = /tmp
# Broken reverse DNS slows down connections considerably and name resolve is
# safe to skip if there are no "host by domain name" access grants
#skip-name-resolve
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 0.0.0.0
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
#key_buffer_size = 128M
#max_allowed_packet = 1G
#thread_stack = 192K
#thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
#myisam_recover_options = BACKUP
#max_connections = 100
#table_cache = 64
max_connections = 500
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 128M
#
# * Logging and Replication
#
# Note: The configured log file or its directory need to be created
# and be writable by the mysql user, e.g.:
# $ sudo mkdir -m 2750 /var/log/mysql
# $ sudo chown mysql /var/log/mysql
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# Recommend only changing this at runtime for short testing periods if needed!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
# When running under systemd, error logging goes via stdout/stderr to journald
# and when running legacy init error logging goes to syslog due to
# /etc/mysql/conf.d/mariadb.conf.d/50-mysqld_safe.cnf
# Enable this if you want to have error logging into a separate file
#log_error = /var/log/mysql/error.log
# Enable the slow query log to see queries with especially long duration
#log_slow_query_file = /var/log/mysql/mariadb-slow.log
#log_slow_query_time = 10
#log_slow_verbosity = query_plan,explain
#log-queries-not-using-indexes
#log_slow_min_examined_row_limit = 1000
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replica, see README.Debian about other
# settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
#expire_logs_days = 10
#max_binlog_size = 100M
binlog_expire_logs_seconds = 432000
#
# * SSL/TLS
#
# For documentation, please read
# https://mariadb.com/kb/en/securing-connections-for-client-and-server/
#ssl-ca = /etc/mysql/cacert.pem
#ssl-cert = /etc/mysql/server-cert.pem
#ssl-key = /etc/mysql/server-key.pem
#require-secure-transport = on
#
# * Character sets
#
# MySQL/MariaDB default is Latin1, but in Debian we rather default to the full
# utf8 4-byte character set. See also client.cnf
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
# Most important is to give InnoDB 80 % of the system RAM for buffer use:
# https://mariadb.com/kb/en/innodb-system-variables/#innodb_buffer_pool_size
#innodb_buffer_pool_size = 8G
# Ottimizzazione per blocchi ZFS (se applicabile)
innodb_page_size = 16k
innodb_doublewrite = 0
innodb_flush_method = O_DIRECT
# Buffer Pool: la regola d'oro è l'80% della RAM se il container fa solo da DB
innodb_buffer_pool_size = 5G # Esempio per container da 6GB
# Log file size (migliora le performance in scrittura)
innodb_log_file_size = 512M
innodb_log_buffer_size = 16M
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.11 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.11]
+41
View File
@@ -0,0 +1,41 @@
#
# * Galera-related settings
#
# See the examples of server wsrep.cnf files in /usr/share/mysql
# and read more at https://mariadb.com/kb/en/galera-cluster/
[galera]
# Mandatory settings
wsrep_on = ON
wsrep_provider =/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name = "MariaDB EQN Galera Cluster"
#wsrep_cluster_address = gcomm://
wsrep_cluster_address = "gcomm://172.16.1.121,172.16.1.122,172.16.1.123"
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_force_primary_key = 1
# Allow server to accept connections on all interfaces.
bind-address = 0.0.0.0
# Optional settings
wsrep_slave_threads = 4
#innodb_flush_log_at_trx_commit = 0
# --- Node Settings ---
wsrep_node_address="172.16.1.121"
wsrep_node_name="db01"
# --- Sync Settings (SST) ---
wsrep_sst_method=rsync
#wsrep_sst_method=mariabackup
wsrep_sst_auth="sstuser:1701D_enterprise"
# --- Proxmox/LXC Optimization ---
# rete locale veloce, riduciamo i timeout per failover rapido
wsrep_retry_autocommit=3
# By default, MariaDB error logs are sent to journald, which can be hard to digest sometimes.
# The following line will save error messages to a plain file.
log_error = /var/log/mysql/error.log
+28
View File
@@ -0,0 +1,28 @@
lobal_defs {
}
vrrp_instance VI_0 {
state BACKUP
interface eth0
virtual_router_id 110
priority 130
advert_int 1
# Gestione unicast x VRack OVH
unicast_src_ip 172.16.1.121
unicast_peer {
172.16.1.119
172.16.1.122
172.16.1.123
}
authentication {
auth_type PASS
auth_pass pappagalliStocastici
}
virtual_ipaddress {
172.16.1.120/21
}
## gestione route definita in netplan
#virtual_rules {
# from 172.16.1.120 table 101
#}
}
+36
View File
@@ -0,0 +1,36 @@
#!/bin/bash
NEW_DB_IP="db03.eqn" # Hostname/IP del nuovo LXC
DB_NAME="eqn_prod"
OLD_PASS="old_db_password"
echo "1. Esportazione Database $DB_NAME..."
# Nota: ho aggiunto --add-drop-database per pulire eventuali test precedenti
mysqldump -u root -p"$OLD_PASS" --single-transaction --routines --triggers --databases $DB_NAME | ssh root@$NEW_DB_IP "mariadb"
echo "2. Generazione file utenti e permessi..."
echo "2. Generazione file utenti e permessi..."
# Estrae l'istruzione completa direttamente come la scrive MariaDB
# Questo metodo evita di ricostruire la stringa manualmente con CONCAT
mysql -u root -p"$OLD_PASS" -B -N -e "SELECT CONCAT('CREATE USER IF NOT EXISTS ', QUOTE(user), '@', QUOTE(host), ' IDENTIFIED VIA mysql_native_password USING ', QUOTE(password), ';') FROM mysql.user WHERE user NOT IN ('root','mysql.sys','mariadb.sys','debian-sys-maint','phpmyadmin');" > users_statements.sql
# Estrae i privilegi (GRANT)
mysql -u root -p"$OLD_PASS" -B -N -e "SELECT CONCAT('SHOW GRANTS FOR ', QUOTE(user), '@', QUOTE(host), ';') FROM mysql.user WHERE user NOT IN ('root','mysql.sys','mariadb.sys','debian-sys-maint','phpmyadmin');" | \
mysql -u root -p"$OLD_PASS" -B -N | \
sed 's/$/;/' | grep -vE "phpmyadmin|eqn_test|wp_eqn" > grants_statements.sql
# Unisce i file
cat users_statements.sql grants_statements.sql > final_users.sql
echo "3. Importazione utenti su $NEW_DB_IP..."
# Copia il file sul nuovo server
scp final_users.sql root@$NEW_DB_IP:/tmp/final_users.sql
# Esegue l'importazione e pulisce i privilegi
ssh root@$NEW_DB_IP "mariadb < /tmp/final_users.sql && mariadb -e 'FLUSH PRIVILEGES;'"
# Pulizia file locali (opzionale)
rm users_statements.sql grants_statements.sql final_users.sql
echo "4. Migrazione completata con successo."
+33
View File
@@ -0,0 +1,33 @@
#!/bin/bash
NEW_DB_IP="db02.eqn" # Hostname/IP del nuovo LXC
DB_NAME="eqn_prod"
OLD_PASS="old_db_password"
echo "2. Generazione file utenti e permessi..."
echo "2. Generazione file utenti e permessi..."
# Estrae l'istruzione completa direttamente come la scrive MariaDB
# Questo metodo evita di ricostruire la stringa manualmente con CONCAT
mysql -u root -p"$OLD_PASS" -B -N -e "SELECT CONCAT('CREATE USER IF NOT EXISTS ', QUOTE(user), '@', QUOTE(host), ' IDENTIFIED VIA mysql_native_password USING ', QUOTE(password), ';') FROM mysql.user WHERE user NOT IN ('root','mysql.sys','mariadb.sys','debian-sys-maint','phpmyadmin');" > users_statements.sql
# Estrae i privilegi (GRANT)
mysql -u root -p"$OLD_PASS" -B -N -e "SELECT CONCAT('SHOW GRANTS FOR ', QUOTE(user), '@', QUOTE(host), ';') FROM mysql.user WHERE user NOT IN ('root','mysql.sys','mariadb.sys','debian-sys-maint','phpmyadmin');" | \
mysql -u root -p"$OLD_PASS" -B -N | \
sed 's/$/;/' | grep -vE "phpmyadmin|eqn_test|wp_eqn" > grants_statements.sql
# Unisce i file
cat users_statements.sql grants_statements.sql > final_users.sql
echo "3. Importazione utenti su $NEW_DB_IP..."
# Copia il file sul nuovo server
scp final_users.sql root@$NEW_DB_IP:/tmp/final_users.sql
# Esegue l'importazione e pulisce i privilegi
ssh root@$NEW_DB_IP "mariadb < /tmp/final_users.sql && mariadb -e 'FLUSH PRIVILEGES;'"
# Pulizia file locali (opzionale)
rm users_statements.sql grants_statements.sql final_users.sql
echo "4. Migrazione completata con successo."
+28
View File
@@ -0,0 +1,28 @@
#!/bin/bash
# prerequisito: apt install mariadb-backup
# Configurazione cablata su utente
BACKUP_BASE_DIR="/home/samuele/backups"
DATE=$(date +%Y%m%d_%H%M%S)
TARGET_DIR="$BACKUP_BASE_DIR/$DATE"
echo "Avvio backup fisico con mariadb-backup..."
# 1. Fase di estrazione (Backup dei file)
mariadb-backup --backup \
--target-dir=$TARGET_DIR \
# --user=root \
# --password='tua_password' # Meglio usare un file .my.cnf per sicurezza
# 2. Fase di 'Prepare' (Rende il backup consistente)
# Questo passaggio è fondamentale: applica i log delle transazioni ai file copiati
echo "Preparazione del backup (apply-log)..."
mariadb-backup --prepare --target-dir=$TARGET_DIR
if [ $? -eq 0 ]; then
echo "Backup creato e preparato con successo in: $TARGET_DIR"
else
echo "Errore durante il backup!"
exit 1
fi
+9 -82
View File
@@ -1,93 +1,20 @@
# MariaDb # MariaDb
Collezione script MariaDB per vari casi ed utilità
## Getting started
To make it easy for you to get started with GitLab, here's a list of recommended next steps. ## Macro Organizzazione
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)! Subfolder per ogni macroprogetto fatto/fgestito, le tematiche variano, di seguito breve recap dei macro progetti
## Add your files ### Eqn
* [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files Contiene gli script e gli appunti usati per la migrazione dal DB MariaDb 10.4 (singola VM, 8C / 8GB Ram) del progetto EQN per il porting su Proxmox con conversione VM --> LXC ed aggiunta cluster Galera x gestione HA, resilenza, scalabilità.
* [Add files using the command line](https://docs.gitlab.com/topics/git/add_files/#add-files-to-a-git-repository) or push an existing Git repository with the following command:
``` ## Versioni
cd existing_repo
git remote add origin https://gitlab.steamware.net/egalware-web/other/mariadb.git
git branch -M main
git push -uf origin main
```
## Integrate with your tools
* [Set up project integrations](https://gitlab.steamware.net/egalware-web/other/mariadb/-/settings/integrations) data | note |
---------|----------
## Collaborate with your team 2026.02.12 | Aggiunga area script x EQN migration
* [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
* [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
* [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
* [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
* [Set auto-merge](https://docs.gitlab.com/user/project/merge_requests/auto_merge/)
## Test and Deploy
Use the built-in continuous integration in GitLab.
* [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/)
* [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
* [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
* [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
* [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
# Editing this README
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
## License
For open source projects, say how it is licensed.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
BIN
View File
Binary file not shown.