BA-translator/README.md

# BA-translator

*A powerful tool for extracting and applying translations from FlatBuffers-based SQLite databases*

## Overview

BA-translator is a comprehensive localization tool designed for games and applications that use FlatBuffers-serialized data stored in SQLite databases. It provides a complete workflow for extracting translatable strings, managing translations through CSV files, and applying them back to the database.

## Key Features

- **Extract translatable strings** from FlatBuffers-based SQLite databases to JSON format
- **Export translations to CSV** format for easy collaboration with translators
- **Import completed translations** from CSV back to JSON
- **Apply translations** to the database with automatic backup creation
- **Validate translation consistency** and detect potential issues
- **Merge multiple CSV files** from different translators
- **Generate FlatBuffers schemas** from C# source files
- **Create repository mappings** for database table relationships

## System Requirements

### Required Dependencies
- **Python 3.7+** (Python 3.8+ recommended)
- **sqlite3** (included with Python)
- **flatbuffers** - for binary data serialization
- **tqdm** - for progress bars
- **unidecode** - for text processing
- **flatc** - FlatBuffers compiler (for generating Python modules from .fbs schemas)

### Required Files
- `ExcelDB.db` - The main SQLite database file
- `repository_map.json` - Generated mapping file (created by `dumpdbschema.py`)
- `types.cs` - C# source file with type definitions (for schema generation)
- `flatc.exe` - FlatBuffers compiler executable (Windows) or `flatc` (Linux/macOS)

## Installation

1. **Clone or download** this repository:
   ```bash
   git clone <repository-url>
   cd BA-translator
   ```

2. **Install Python dependencies**:
   ```bash
   pip install -r requirements.txt
   ```
   
   Or install manually:
   ```bash
   pip install flatbuffers tqdm unidecode
   ```

3. **Prepare required files**:
   - Place `ExcelDB.db` in the project root directory
   - Place `types.cs` (decompiled C# file) in the project root

4. **Generate repository mapping**:
   ```bash
   python dumpdbschema.py
   ```
   This creates `repository_map.json` which maps database tables to their schemas.

5. **Set up FlatBuffers compiler** (if not installed):
   - **Windows**: Run `setup_flatc.bat` to automatically download `flatc.exe`
   - **Linux/macOS**: Install FlatBuffers: `sudo apt install flatbuffers-compiler` or `brew install flatbuffers`

6. **Generate FlatBuffers schemas** (if needed):
   ```bash
   python parser.py
   ```
   This creates `generated_schema.fbs` from C# types.

7. **Generate Python modules from schemas**:
   ```bash
   python generate_flatbuffer_folders.py --auto
   ```
   This automatically finds .fbs files and generates compatible Python modules.

## Quick Start

### Basic Translation Workflow

1. **Extract translatable strings**:
   ```bash
   python BAtranslator.py extract --output translations.json
   ```

2. **Export to CSV for translators**:
   ```bash
   python BAtranslator.py export_csv --input translations.json
   ```
   This creates two files:
   - `translations.csv` - Complete file for merging
   - `translations_for_translators.csv` - Simplified file for translation work

3. **Translate the CSV file** using any CSV editor (LibreOffice Calc, Google Sheets, or any text editor)

4. **Import completed translations**:
   ```bash
   python BAtranslator.py import_csv --input translations_for_translators.csv --output translations_updated.json
   ```

5. **Validate translations** (optional but recommended):
   ```bash
   python BAtranslator.py validate_csv --input translations_for_translators.csv
   ```

6. **Apply translations to database**:
   ```bash
   python BAtranslator.py patch --input translations_updated.json
   ```
   This automatically creates a backup (`ExcelDB.db.bak`) before applying changes.

### Advanced Usage

**Update existing translations with new content**:
```bash
python BAtranslator.py extract --output new_translations.json --update-from old_translations.json
```

**Merge multiple translator CSV files**:
```bash
python BAtranslator.py merge_csv translator1.csv translator2.csv translator3.csv --output merged_translations.csv
```

## Command Reference

### Main Translation Commands

| Command | Description | Usage |
|---------|-------------|-------|
| `extract` | Extract strings from database to JSON | `python BAtranslator.py extract --output file.json` |
| `patch` | Apply translations from JSON to database | `python BAtranslator.py patch --input file.json` |
| `export_csv` | Convert JSON to CSV format | `python BAtranslator.py export_csv --input file.json` |
| `import_csv` | Convert CSV to JSON format | `python BAtranslator.py import_csv --input file.csv --output file.json` |
| `validate_csv` | Validate CSV file consistency | `python BAtranslator.py validate_csv --input file.csv` |
| `merge_csv` | Merge multiple CSV files | `python BAtranslator.py merge_csv file1.csv file2.csv --output merged.csv` |

### FlatBuffers Generation Commands

| Command | Description | Usage |
|---------|-------------|-------|
| `parser.py` | Generate .fbs schema from C# types | `python parser.py` |
| `dumpdbschema.py` | Generate repository mapping | `python dumpdbschema.py` |
| `generate_flatbuffer_folders.py` | Generate Python modules from .fbs | `python generate_flatbuffer_folders.py --auto` |

## File Formats

### JSON Translation Format
```json
{
  "TableName": {
    "row_id": {
      "field_name": {
        "original": "Original text",
        "translation": "Translated text"
      }
    }
  }
}
```

### CSV Format
| Column | Description |
|--------|-------------|
| GroupID | Identifier for grouping identical texts |
| Original | Original text to translate |
| Translation | Translated text (fill this column) |
| SQLTable | Database table name |
| RowID | Row identifier |
| Field | Field name |
| HasCodes | Indicates special characters present |
| Codes | Encoded special characters |

## FlatBuffers Schema Generation

### Quick Setup
Generate Python modules from .fbs schema files:

```bash
# Setup flatc compiler (Windows)
setup_flatc.bat

# Generate Python modules automatically
python generate_flatbuffer_folders.py --auto

# Or from specific files
python generate_flatbuffer_folders.py schema.fbs
```

### Manual FlatBuffers Setup
1. **Install FlatBuffers compiler**:
   - **Windows**: Run `setup_flatc.bat` or download from [FlatBuffers releases](https://github.com/google/flatbuffers/releases)
   - **Linux**: `sudo apt install flatbuffers-compiler`
   - **macOS**: `brew install flatbuffers`

2. **Generate Python modules**:
   ```bash
   # From C# source (if available)
   python parser.py
   
   # Generate Python modules
   python generate_flatbuffer_folders.py --auto --verbose
   ```

3. **Verify generation**:
   ```bash
   # Check integration with BAtranslator
   python -c "from BAtranslator import build_schema_map; build_schema_map(); print('OK')"
   ```

## Troubleshooting

### Common Issues

1. **"Database file 'ExcelDB.db' not found"**
   - Ensure the database file is in the project root directory
   - Check the filename is exactly `ExcelDB.db` (case-sensitive)

2. **"Module not found" errors**
   - Run `python generate_flatbuffer_folders.py --auto` to generate required schema modules
   - Ensure `FlatData/` directory contains the generated Python files
   - Check that `__init__.py` files exist in generated directories

3. **"File 'types.cs' not found"**
   - Place the decompiled C# file in the project root
   - Run `python dumpdbschema.py` to generate the repository mapping

4. **"flatc not found" or "flatc command failed"**
   - **Windows**: Run `setup_flatc.bat` to download flatc.exe automatically
   - **Linux/macOS**: Install FlatBuffers compiler using your package manager
   - Verify installation: `flatc --version`

5. **"Generated modules import errors"**
   - Regenerate modules: `python generate_flatbuffer_folders.py --auto --clean`
   - Check that `__init__.py` files exist in FlatData directory

## License

This project is open source. Please refer to the license file for more details.

## Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

---

**📖 For Russian documentation, see [README_RU.md](README_RU.md)**

**⚠️ This project is tested in older versions of Blue Archive. It may not work correctly in newer versions. Report me if something wrong.**