mirror of
https://github.com/1technophile/OpenMQTTGateway.git
synced 2026-02-19 16:21:44 +01:00
[CI] Refactor GitHub Actions workflows for build, documentation, and linting (#2260)
* Refactor GitHub Actions workflows for build, documentation, and linting - Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows. - Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity. - Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions. - Enhanced `release.yml` to streamline the release process and integrate documentation generation. - Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository. - Simplified manual documentation workflow by leveraging the new reusable documentation workflow. - Improved artifact management and retention policies across workflows. - Updated dependencies and versions in workflows to ensure compatibility and performance. * CI/CD pipeline agnostic of Workflow Engine and integrated on github actions - Implemented ci.sh for orchestrating the complete build pipeline. - Created ci_00_config.sh for centralized configuration of build scripts. - Created ci_build_firmware.sh for building firmware for specified PlatformIO environments. - Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment. - Created ci_set_version.sh for updating version tags in firmware configuration files. - Created ci_build.sh to orchestrate the complete build pipeline. - Created ci_qa.sh for code linting and formatting checks using clang-format. - Created ci_site.sh for building and deploying VuePress documentation with version management. - Implemented checks for required tools and dependencies in the new scripts. - Updated common_wu.py, compressFirmware.py, gen_wu.py, generate_board_docs.py, and prepare_deploy.sh with descriptive comments. Refactor CI/CD scripts for improved modularity and clarity - Consolidated build steps in task-build.yml to utilize ci.sh for version tagging, building, and artifact preparation. - Updated task-lint.yml to use ci.sh for code formatting checks instead of ci_qa.sh. - Enhanced CI_SCRIPTS.md documentation to reflect changes in script usage, command structure, and output organization. - Improved internal scripts for better error handling and logging. - Streamlined the output structure for build artifacts and documentation.
This commit is contained in:
committed by
GitHub
parent
fd433c220e
commit
134c03362c
697
.github/workflows/README.md
vendored
Normal file
697
.github/workflows/README.md
vendored
Normal file
@@ -0,0 +1,697 @@
|
||||
# GitHub Actions Workflows Documentation
|
||||
|
||||
This document provides a overview of all GitHub Actions workflows in the OpenMQTTGateway project.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The workflow system is organized in two layers:
|
||||
|
||||
### **Main Workflows** (User-facing triggers)
|
||||
Entry points triggered by user actions, schedules, or events:
|
||||
- `build.yml` - CI validation on push/PR
|
||||
- `build_and_docs_to_dev.yml` - Daily development builds
|
||||
- `release.yml` - Production releases
|
||||
- `manual_docs.yml` - Documentation deployment
|
||||
- `lint.yml` - Code formatting check
|
||||
- `stale.yml` - Issue management
|
||||
|
||||
### **Task Workflows** (Reusable components)
|
||||
Parameterized building blocks called by main workflows:
|
||||
- `task-build.yml` - Configurable firmware build
|
||||
- `task-docs.yml` - Configurable documentation build
|
||||
- `task-lint.yml` - Configurable code formatting check
|
||||
|
||||
---
|
||||
|
||||
## Workflow Overview Table
|
||||
|
||||
| Workflow | Trigger | Purpose | Artifacts |
|
||||
|----------|---------|---------|-----------|
|
||||
| `build.yml` | Push, Pull Request | CI Build Validation | Firmware binaries (7 days) |
|
||||
| `build_and_docs_to_dev.yml` | Daily Cron, Manual | Development Builds + Docs | Firmware + Docs deployment |
|
||||
| `release.yml` | Release Published | Production Release | Release assets + Docs |
|
||||
| `manual_docs.yml` | Manual, Workflow Call | Documentation Only | GitHub Pages docs |
|
||||
| `lint.yml` | Push, Pull Request | Code Format Check | None |
|
||||
| `stale.yml` | Daily Cron | Issue Management | None |
|
||||
| **`task-build.yml`** | **Workflow Call** | **Reusable Build Logic** | **Configurable** |
|
||||
| **`task-docs.yml`** | **Workflow Call** | **Reusable Docs Logic** | **GitHub Pages** |
|
||||
| **`task-lint.yml`** | **Workflow Call** | **Reusable Lint Logic** | **None** |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Workflow Documentation
|
||||
|
||||
### 1. `build.yml` - Continuous Integration Build
|
||||
|
||||
**Purpose**: Validates that code changes compile successfully across all supported hardware platforms.
|
||||
|
||||
**Triggers**:
|
||||
- **Push**: Every commit pushed to any branch
|
||||
- **Pull Request**: Every PR creation or update
|
||||
|
||||
**What it does**:
|
||||
1. **Build job**: Calls `task-build.yml` with CI parameters
|
||||
- Builds firmware for **83 hardware environments** in parallel
|
||||
2. **Documentation job**: Inline job that validates docs build (doesn't deploy)
|
||||
- Downloads common config from theengs.io
|
||||
- Runs `npm install` and `npm run docs:build`
|
||||
- Uses Node.js 14.x
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-build.yml` only (documentation is inline)
|
||||
- Python version: 3.13 (for build job)
|
||||
- Build strategy: Parallel matrix via task workflow
|
||||
- Artifact retention: 7 days
|
||||
- Development OTA: Disabled (`enable-dev-ota: false`)
|
||||
|
||||
**Outputs**:
|
||||
- Firmware binaries for each environment (83 artifacts)
|
||||
- No documentation deployment (validation only)
|
||||
|
||||
**Use Case**: Ensures no breaking changes before merge. Fast feedback for developers.
|
||||
|
||||
**Execution Context**: Runs for ALL contributors on ALL branches.
|
||||
|
||||
---
|
||||
|
||||
### 2. `build_and_docs_to_dev.yml` - Development Deployment Pipeline
|
||||
|
||||
**Purpose**: Creates nightly development builds and deploys documentation to the `/dev` subdirectory for testing.
|
||||
|
||||
**Triggers**:
|
||||
- **Schedule**: Daily at midnight UTC (`0 0 * * *`)
|
||||
- **Manual**: Via workflow_dispatch button
|
||||
|
||||
**What it does**:
|
||||
1. **Prepare job**: Generates 6-character short SHA
|
||||
2. **Build job**: Calls `task-build.yml` with development parameters
|
||||
- Builds firmware for **83 hardware environments** in parallel
|
||||
- Enables development OTA updates with SHA commit version
|
||||
3. **Deploy job**: Prepares and uploads assets
|
||||
- Creates library dependency zips for each board
|
||||
- Generates source code zip
|
||||
- Removes test environment binaries
|
||||
4. **Documentation job**: Calls `task-docs.yml` with development parameters
|
||||
- Deploys to `/dev` subdirectory
|
||||
- Adds "DEVELOPMENT" watermark
|
||||
- Runs PageSpeed Insights
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-build.yml` + `task-docs.yml`
|
||||
- Python version: 3.13 (build), 3.11 (docs)
|
||||
- Node.js version: 16.x (docs)
|
||||
- Repository restriction: Hardcoded to `1technophile` owner only
|
||||
- Artifact retention: 1 day
|
||||
- Build flag: `enable-dev-ota: true` (passed to task-build.yml)
|
||||
|
||||
**Outputs**:
|
||||
- Firmware binaries with `-firmware.bin` suffix
|
||||
- Bootloader and partition binaries
|
||||
- Library dependency zips per board
|
||||
- Source code zip
|
||||
- Documentation deployed to `docs.openmqttgateway.com/dev/`
|
||||
|
||||
**Version Labeling**:
|
||||
- Git SHA (6 chars) injected into firmware
|
||||
- Docs tagged: "DEVELOPMENT SHA:XXXXXX TEST ONLY"
|
||||
|
||||
**Use Case**: Daily bleeding-edge builds for early adopters and testing. Preview documentation changes.
|
||||
|
||||
**Execution Context**: Only runs on `1technophile` repository owner. Forks will skip this workflow automatically.
|
||||
|
||||
---
|
||||
|
||||
### 3. `release.yml` - Production Release Pipeline
|
||||
|
||||
**Purpose**: Creates official release builds when a new version is published.
|
||||
|
||||
**Triggers**:
|
||||
- **Release**: When a GitHub release is published (tagged)
|
||||
|
||||
**What it does**:
|
||||
1. **Prepare job**: Extracts version tag and release info
|
||||
2. **Build job**: Calls `task-build.yml` with production parameters
|
||||
- Builds firmware for **83 hardware environments** in parallel
|
||||
- Injects release tag version into firmware
|
||||
3. **Deploy job**: Prepares and uploads release assets
|
||||
- Downloads all build artifacts
|
||||
- Reorganizes for `prepare_deploy.sh` script
|
||||
- Creates library zips and source zip
|
||||
- Uploads to GitHub Release
|
||||
4. **Documentation job**: Calls `task-docs.yml` for production docs
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-build.yml` + `task-docs.yml`
|
||||
- Python version: 3.13 (build), 3.11 (docs)
|
||||
- Node.js version: 18.x
|
||||
- Build flag: Standard (no DEVELOPMENTOTA)
|
||||
- Artifact retention: 90 days
|
||||
- Uses `prepare_deploy.sh` script for asset preparation
|
||||
|
||||
**Outputs**:
|
||||
- Production firmware binaries attached to GitHub Release
|
||||
- Library zips per board
|
||||
- Source code zip
|
||||
- Production documentation at `docs.openmqttgateway.com/`
|
||||
|
||||
**Version Labeling**:
|
||||
- Git tag (e.g., `v1.2.3`) injected into firmware and `latest_version.json`
|
||||
|
||||
**Workflow Chain**:
|
||||
```
|
||||
prepare → build (task-build.yml) → deploy → documentation (task-docs.yml)
|
||||
```
|
||||
|
||||
**Use Case**: Official releases for end users. Stable, versioned firmware.
|
||||
|
||||
**Execution Context**: Triggered by repository maintainers creating releases.
|
||||
|
||||
---
|
||||
|
||||
### 4. `manual_docs.yml` - Documentation Deployment
|
||||
|
||||
**Purpose**: Entry point for standalone documentation deployment to GitHub Pages.
|
||||
|
||||
**Triggers**:
|
||||
- **Manual**: Via workflow_dispatch button
|
||||
- **Workflow Call**: Can be called by other workflows (legacy compatibility)
|
||||
|
||||
**What it does**:
|
||||
1. Calls `task-docs.yml` with production parameters
|
||||
2. Deploys to root directory (`/`) of GitHub Pages
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-docs.yml`
|
||||
- Python version: 3.11
|
||||
- Node.js version: 14.x
|
||||
- Version source: Latest GitHub release tag
|
||||
- WebUploader manifest: Enabled
|
||||
|
||||
**Outputs**:
|
||||
- Production documentation at `docs.openmqttgateway.com/`
|
||||
- Custom domain: `docs.openmqttgateway.com` (via CNAME)
|
||||
|
||||
**Use Case**: Standalone documentation updates without full release process.
|
||||
|
||||
**Execution Context**: Manual trigger or legacy workflow calls.
|
||||
|
||||
---
|
||||
|
||||
### 5. `lint.yml` - Code Format Validation
|
||||
|
||||
**Purpose**: Ensures code follows consistent formatting standards.
|
||||
|
||||
**Triggers**:
|
||||
- **Push**: Every commit pushed to any branch
|
||||
- **Pull Request**: Every PR creation or update
|
||||
|
||||
**What it does**:
|
||||
1. Calls `task-lint.yml` with specific parameters
|
||||
2. Checks formatting in `./main` directory only
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-lint.yml`
|
||||
- clang-format version: 9
|
||||
- File extensions: `.h`, `.ino` (not `.cpp`)
|
||||
- Source directory: `main` (single directory)
|
||||
|
||||
**Configuration**:
|
||||
```yaml
|
||||
source: 'main'
|
||||
extensions: 'h,ino'
|
||||
clang-format-version: '9'
|
||||
```
|
||||
|
||||
**Use Case**: Maintains code quality and consistency. Prevents formatting debates in PRs.
|
||||
|
||||
**Execution Context**: Runs for ALL contributors on ALL branches.
|
||||
|
||||
---
|
||||
|
||||
### 6. `stale.yml` - Issue and PR Management
|
||||
|
||||
**Purpose**: Automatically closes inactive issues and pull requests to reduce maintenance burden.
|
||||
|
||||
**Triggers**:
|
||||
- **Schedule**: Daily at 00:30 UTC (`30 0 * * *`)
|
||||
|
||||
**What it does**:
|
||||
1. Marks issues/PRs as stale after 90 days of inactivity
|
||||
2. Closes stale issues/PRs after 14 additional days
|
||||
3. Exempts issues labeled "enhancement"
|
||||
|
||||
**Configuration**:
|
||||
- Stale after: 90 days
|
||||
- Close after: 14 days (104 days total)
|
||||
- Stale label: `stale`
|
||||
- Exempt labels: `enhancement`
|
||||
|
||||
**Messages**:
|
||||
- Stale: "This issue is stale because it has been open for 90 days with no activity."
|
||||
- Close: "This issue was closed because it has been inactive for 14 days since being marked as stale."
|
||||
|
||||
**Use Case**: Housekeeping. Reduces backlog of abandoned issues.
|
||||
|
||||
**Execution Context**: Automated maintenance by GitHub bot.
|
||||
|
||||
---
|
||||
|
||||
## Task Workflows (Reusable Components)
|
||||
|
||||
### 7. `task-build.yml` - Reusable Build Workflow
|
||||
|
||||
**Purpose**: Parameterized firmware build logic used by multiple workflows.
|
||||
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
**Parameters**:
|
||||
- `python-version`: Python version (default: '3.13')
|
||||
- `enable-dev-ota`: Enable development OTA (default: false)
|
||||
- `version-tag`: Version to inject (default: 'unspecified')
|
||||
- `artifact-retention-days`: Artifact retention (default: 7)
|
||||
- `artifact-name-prefix`: Artifact name prefix (default: '')
|
||||
- `prepare-for-deploy`: Prepare for deployment (default: false)
|
||||
|
||||
**What it does**:
|
||||
1. **Load environments**: Reads environment list from `environments.json`
|
||||
2. **Matrix build**: Builds all 83 environments in parallel
|
||||
3. **Build execution**: Calls unified `ci.sh build <environment> [OPTIONS]`:
|
||||
- `<environment>`: Target hardware (e.g., `esp32dev-ble`)
|
||||
- `--version <tag>`: Version to inject (SHA for dev, tag for prod)
|
||||
- `--mode <dev|prod>`: Build mode (enables/disables OTA)
|
||||
- `--deploy-ready`: Prepare artifacts for deployment
|
||||
- `--output <dir>`: Output directory for artifacts (default: `generated/artifacts/`)
|
||||
|
||||
**Command Flow**:
|
||||
```bash
|
||||
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
|
||||
↓
|
||||
├─→ ci_build.sh (orchestrator)
|
||||
│ ├─→ ci_set_version.sh v1.8.0 [--dev]
|
||||
│ ├─→ ci_build_firmware.sh esp32dev-ble [--dev-ota]
|
||||
│ └─→ ci_prepare_artifacts.sh esp32dev-ble [--deploy] → outputs to generated/artifacts/
|
||||
```
|
||||
|
||||
**Technical Details**:
|
||||
- Runs on: Ubuntu latest
|
||||
- PlatformIO version: 6.1.18 (custom fork: `pioarduino/platformio-core`)
|
||||
- Python package manager: `uv` (astral-sh/setup-uv@v6)
|
||||
- Strategy: Matrix with fail-fast: false
|
||||
- Main orchestrator: `ci.sh` → `ci_build.sh` → sub-scripts
|
||||
|
||||
**Callers**:
|
||||
- `build.yml` (CI validation)
|
||||
- `build_and_docs_to_dev.yml` (development builds)
|
||||
- `release.yml` (production releases)
|
||||
|
||||
---
|
||||
|
||||
### 8. `task-docs.yml` - Reusable Documentation Workflow
|
||||
|
||||
**Purpose**: Parameterized documentation build and deployment logic.
|
||||
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
**Parameters**:
|
||||
- `python-version`: Python version (default: '3.11')
|
||||
- `node-version`: Node.js version (default: '14.x')
|
||||
- `version-source`: Version source ('release', 'git-tag', 'custom')
|
||||
- `custom-version`: Custom version string (optional)
|
||||
- `base-path`: Base URL path (default: '/')
|
||||
- `destination-dir`: Deploy directory (default: '.')
|
||||
- `generate-webuploader`: Generate WebUploader manifest (default: true)
|
||||
- `webuploader-args`: WebUploader generation arguments (optional)
|
||||
- `run-pagespeed`: Run PageSpeed Insights (default: false)
|
||||
- `pagespeed-url`: URL for PageSpeed test (optional)
|
||||
|
||||
**What it does**:
|
||||
1. **Build documentation**: Calls unified `ci.sh site [OPTIONS]`:
|
||||
- `--mode <dev|prod>`: Documentation mode
|
||||
- `--custom-version <ver>`: Custom version string
|
||||
- `--version-source <release|custom>`: Version source
|
||||
- `--url-prefix <path>`: Base URL path (e.g., `/dev/`)
|
||||
- `--webuploader-args <args>`: WebUploader options
|
||||
- `--no-webuploader`: Skip manifest generation
|
||||
2. **Deploy**: Publishes to GitHub Pages using `peaceiris/actions-gh-pages@v3`
|
||||
3. **PageSpeed test**: Optionally runs performance audit
|
||||
|
||||
**Command Flow**:
|
||||
```bash
|
||||
./scripts/ci.sh site --mode prod --version-source release --url-prefix /
|
||||
↓
|
||||
└─→ ci_site.sh (orchestrator)
|
||||
├─→ generate_board_docs.py (auto-generate board docs)
|
||||
├─→ npm run docs:build (VuePress compilation)
|
||||
└─→ gen_wu.py (WebUpdater manifest)
|
||||
```
|
||||
|
||||
**Callers**:
|
||||
- `build_and_docs_to_dev.yml` (dev docs to `/dev`)
|
||||
- `release.yml` (production docs to `/`)
|
||||
- `manual_docs.yml` (manual production docs)
|
||||
|
||||
---
|
||||
|
||||
### 9. `task-lint.yml` - Reusable Lint Workflow
|
||||
|
||||
**Purpose**: Parameterized code formatting validation.
|
||||
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
**Parameters**:
|
||||
- `source`: Source directories to lint (default: './lib ./main')
|
||||
- `extensions`: File extensions to check (default: 'h,ino,cpp')
|
||||
- `clang-format-version`: clang-format version (default: '9')
|
||||
- `exclude-pattern`: Pattern to exclude (optional)
|
||||
|
||||
**What it does**:
|
||||
1. Checks out code
|
||||
2. Installs clang-format (specified version)
|
||||
3. Runs unified `ci.sh qa [OPTIONS]`:
|
||||
- `--check`: Validation mode (exit on violations)
|
||||
- `--fix`: Auto-fix formatting issues
|
||||
- `--source <dir>`: Directory to lint
|
||||
- `--extensions <list>`: File extensions (comma-separated)
|
||||
- `--clang-format-version <ver>`: Formatter version
|
||||
4. Fails if formatting violations found
|
||||
|
||||
**Command Flow**:
|
||||
```bash
|
||||
./scripts/ci.sh qa --check --source main --extensions h,ino --clang-format-version 9
|
||||
↓
|
||||
└─→ ci_qa.sh (formatter)
|
||||
└─→ clang-format (checks/fixes code style)
|
||||
```
|
||||
|
||||
**Technical Details**:
|
||||
- Script: `ci_qa.sh` (custom formatting check script)
|
||||
- Install: `clang-format-$version` via apt-get
|
||||
- Default source: `main` (single directory)
|
||||
- Default extensions: `h,ino` (not cpp)
|
||||
|
||||
**Callers**:
|
||||
- `lint.yml` (CI lint check)
|
||||
|
||||
**Default Behavior**: If called without parameters, lints `main` directory for `.h` and `.ino` files.
|
||||
|
||||
---
|
||||
|
||||
## Workflow Dependencies and Call Chain
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
%% Triggers
|
||||
subgraph triggers ["🎯 Triggers"]
|
||||
push["Push / Pull Request"]
|
||||
release["Release Published"]
|
||||
manual["Manual Trigger"]
|
||||
cron1["Cron: Daily 00:00 UTC"]
|
||||
cron2["Cron: Daily 00:30 UTC"]
|
||||
end
|
||||
|
||||
subgraph github_workflows ["📋 GitHub Workflows"]
|
||||
%% Main Workflows
|
||||
subgraph main ["📋 Main Workflows"]
|
||||
lint["lint.yml<br/>Format Check"]
|
||||
|
||||
build["build.yml<br/>CI Build"]
|
||||
|
||||
release_wf["release.yml<br/>Production Release"]
|
||||
manual_docs["manual_docs.yml<br/>Docs Only"]
|
||||
build_dev["build_and_docs_to_dev.yml<br/>Dev Builds"]
|
||||
stale["stale.yml<br/>Issue Management"]
|
||||
end
|
||||
|
||||
%% Task Workflows
|
||||
subgraph tasks ["⚙️ Task Workflows"]
|
||||
task_build["task-build.yml<br/>Build Firmware"]
|
||||
task_docs["task-docs.yml<br/>Build & Deploy Docs"]
|
||||
task_lint["task-lint.yml<br/>Code Format"]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph ci_scripts ["🔧 CI Scripts"]
|
||||
%% CI Scripts Layer
|
||||
subgraph bash ["🔧 Orchestrator"]
|
||||
ci_main["ci.sh<br/>(main dispatcher)"]
|
||||
ci_build_script["ci_build.sh<br/>(build orchestrator)"]
|
||||
ci_site_script["ci_site.sh<br/>(docs orchestrator)"]
|
||||
ci_qa_script["ci_qa.sh<br/>(lint orchestrator)"]
|
||||
end
|
||||
|
||||
%% Sub-Scripts Layer
|
||||
subgraph sub_scripts ["⚙️ Workers"]
|
||||
ci_set_ver["ci_set_version.sh<br/>(version injection)"]
|
||||
ci_build_fw["ci_build_firmware.sh<br/>(PlatformIO build)"]
|
||||
ci_prep_art["ci_prepare_artifacts.sh<br/>(artifact packaging)"]
|
||||
gen_board["generate_board_docs.py<br/>(board docs)"]
|
||||
gen_wu["gen_wu.py<br/>(WebUpdater manifest)"]
|
||||
clang_fmt["clang-format<br/>(code formatter)"]
|
||||
end
|
||||
end
|
||||
|
||||
%% Trigger connections
|
||||
push --> lint
|
||||
push --> build
|
||||
|
||||
release --> release_wf
|
||||
manual --> manual_docs
|
||||
cron1 --> build_dev
|
||||
cron2 --> stale
|
||||
|
||||
%% Main workflow to task workflow connections
|
||||
build -->|calls| task_build
|
||||
lint -->|calls| task_lint
|
||||
build_dev -->|calls| task_build
|
||||
build_dev -->|calls| task_docs
|
||||
release_wf -->|calls| task_build
|
||||
release_wf -->|calls| task_docs
|
||||
manual_docs -->|calls| task_docs
|
||||
|
||||
%% Task workflows to CI scripts
|
||||
task_build -->|"ci.sh build<br/>--version --mode<br/>--deploy-ready"| ci_main
|
||||
task_docs -->|"ci.sh site<br/>--mode --url-prefix<br/>--version-source"| ci_main
|
||||
task_lint -->|"ci.sh qa<br/>--check --source<br/>--extensions"| ci_main
|
||||
|
||||
%% CI main dispatcher to orchestrators
|
||||
ci_main -->|"route: build"| ci_build_script
|
||||
ci_main -->|"route: site"| ci_site_script
|
||||
ci_main -->|"route: qa"| ci_qa_script
|
||||
|
||||
%% Orchestrators to workers
|
||||
ci_build_script --> ci_set_ver
|
||||
ci_build_script --> ci_build_fw
|
||||
ci_build_script --> ci_prep_art
|
||||
|
||||
ci_site_script --> gen_board
|
||||
ci_site_script --> gen_wu
|
||||
|
||||
ci_qa_script --> clang_fmt
|
||||
|
||||
%% Styling
|
||||
classDef triggerStyle fill:#e1f5ff,stroke:#0066cc,stroke-width:2px
|
||||
classDef mainStyle fill:#fff4e6,stroke:#ff9900,stroke-width:2px
|
||||
classDef taskStyle fill:#e6f7e6,stroke:#00aa00,stroke-width:2px
|
||||
classDef ciStyle fill:#ffe6f0,stroke:#cc0066,stroke-width:2px
|
||||
classDef subStyle fill:#f0e6ff,stroke:#9933ff,stroke-width:2px
|
||||
|
||||
class push,release,manual,cron1,cron2 triggerStyle
|
||||
class build,lint,release_wf,manual_docs,build_dev,stale mainStyle
|
||||
class task_build,task_docs,task_lint taskStyle
|
||||
class ci_main,ci_build_script,ci_site_script,ci_qa_script ciStyle
|
||||
class ci_set_ver,ci_build_fw,ci_prep_art,gen_board,gen_wu,clang_fmt subStyle
|
||||
|
||||
|
||||
style github_workflows stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fbfbfc
|
||||
style main stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
|
||||
style tasks stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
|
||||
|
||||
style ci_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fffaf5
|
||||
style bash stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
|
||||
style sub_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
|
||||
|
||||
style triggers fill:none,stroke:none
|
||||
|
||||
```
|
||||
|
||||
### Workflow Relationships
|
||||
|
||||
**Main → Task Mapping**:
|
||||
- `build.yml` → calls `task-build.yml` (also contains inline documentation job)
|
||||
- `lint.yml` → calls `task-lint.yml`
|
||||
- `build_and_docs_to_dev.yml` → calls `task-build.yml` + `task-docs.yml` (with prepare/deploy jobs)
|
||||
- `release.yml` → calls `task-build.yml` + `task-docs.yml` (with prepare/deploy jobs)
|
||||
- `manual_docs.yml` → calls `task-docs.yml`
|
||||
- `stale.yml` → standalone (no dependencies)
|
||||
|
||||
**Task → CI Script Mapping**:
|
||||
- `task-build.yml` → `ci.sh build <env> --version --mode --deploy-ready`
|
||||
- Routes to: `ci_build.sh` → `ci_set_version.sh`, `ci_build_firmware.sh`, `ci_prepare_artifacts.sh`
|
||||
- Output: `generated/artifacts/` (default, can be overridden with `--output`)
|
||||
- `task-docs.yml` → `ci.sh site --mode --version-source --url-prefix --webuploader-args`
|
||||
- Routes to: `ci_site.sh` → `generate_board_docs.py`, `gen_wu.py`, VuePress
|
||||
- Output: `generated/site/`
|
||||
- `task-lint.yml` → `ci.sh qa --check --source --extensions --clang-format-version`
|
||||
- Routes to: `ci_qa.sh` → `clang-format`
|
||||
|
||||
**Job Dependencies**:
|
||||
- `build_and_docs_to_dev.yml`: prepare → build (task) → deploy & documentation (task)
|
||||
- `release.yml`: prepare → build (task) → deploy → documentation (task)
|
||||
|
||||
**Script Execution Flow**:
|
||||
```
|
||||
GitHub Action (task-*.yml)
|
||||
↓
|
||||
./scripts/ci.sh <command> [OPTIONS] ← Main dispatcher
|
||||
↓
|
||||
./scripts/ci_<command>.sh ← Command orchestrator
|
||||
↓
|
||||
./scripts/ci_*.sh / *.py ← Worker scripts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
### Centralized Environment Management
|
||||
|
||||
All build environments are defined in `.github/workflows/environments.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"environments": {
|
||||
"all": [ ...83 environments ],
|
||||
"metadata": {
|
||||
"totalCount": 83,
|
||||
"lastUpdated": "2024-12-25",
|
||||
"categories": {
|
||||
"esp32": 50,
|
||||
"esp8266": 20,
|
||||
"specialized": 13
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- ✅ Single source of truth
|
||||
- ✅ Eliminates duplication across workflows
|
||||
- ✅ Easier to maintain and update
|
||||
- ✅ Consistent builds across CI/dev/release
|
||||
|
||||
### Environment Categories
|
||||
|
||||
**ESP32 Family** (~50 environments):
|
||||
- Standard: `esp32dev-*` variants
|
||||
- ESP32-S3: `esp32s3-*` variants
|
||||
- ESP32-C3: `esp32c3-*` variants
|
||||
- Specialized boards: M5Stack, Heltec, LilyGO, Theengs
|
||||
|
||||
**ESP8266 Family** (~20 environments):
|
||||
- NodeMCU: `nodemcuv2-*` variants
|
||||
- Sonoff: `sonoff-*` variants
|
||||
- Generic: `esp8266-*` variants
|
||||
|
||||
**Specialized Boards** (~13 environments):
|
||||
- Theengs Plug
|
||||
- Theengs Bridge
|
||||
- RF Bridge variants
|
||||
- Custom board configurations
|
||||
|
||||
---
|
||||
|
||||
## Configuration Variables
|
||||
|
||||
### Repository Restrictions
|
||||
|
||||
**Development Builds** (`build_and_docs_to_dev.yml`):
|
||||
- Hardcoded restriction: `github.repository_owner == '1technophile'`
|
||||
- Only runs for the main repository owner
|
||||
- Prevents accidental deployments from forks
|
||||
- No configuration variable needed
|
||||
|
||||
**Release Builds** (`release.yml`):
|
||||
- No repository restrictions
|
||||
- Runs on any fork when a release is published
|
||||
- Deploy step requires proper GitHub token permissions
|
||||
|
||||
**Documentation** (`manual_docs.yml`):
|
||||
- No repository restrictions
|
||||
- Can be triggered manually from any fork
|
||||
- Requires GitHub Pages to be configured
|
||||
|
||||
---
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Environment**: A specific hardware board + gateway combination (e.g., `esp32dev-ble`)
|
||||
- **Matrix Build**: Parallel execution of builds across multiple environments
|
||||
- **Artifact**: Build output stored temporarily for download (firmware binaries)
|
||||
- **workflow_call**: GitHub Actions feature for calling one workflow from another
|
||||
- **workflow_dispatch**: Manual trigger button for workflows
|
||||
- **Task Workflow**: Reusable workflow component with parameterized inputs
|
||||
- **Main Workflow**: Entry point workflow triggered by events or schedules
|
||||
- **CNAME**: Custom domain configuration for GitHub Pages
|
||||
- **OTA**: Over-The-Air firmware update capability
|
||||
- **SHA**: Git commit hash used for version identification in dev builds
|
||||
|
||||
---
|
||||
|
||||
## Maintenance Notes
|
||||
|
||||
### CI/CD Script Architecture
|
||||
|
||||
**Main Entry Point**: `ci.sh` (unified interface)
|
||||
- Commands: `build`, `site`, `qa`, `all`
|
||||
- Routes to specialized orchestrators
|
||||
- Provides consistent CLI across all operations
|
||||
|
||||
**Build System** (`ci.sh build`):
|
||||
- PlatformIO 6.1.18 from custom fork: `pioarduino/platformio-core`
|
||||
- Python package manager: `uv` for fast dependency installation
|
||||
- Orchestrator: `ci_build.sh`
|
||||
- Worker: `ci_set_version.sh` (version injection)
|
||||
- Worker: `ci_build_firmware.sh` (PlatformIO compilation)
|
||||
- Worker: `ci_prepare_artifacts.sh` (artifact packaging)
|
||||
|
||||
**Documentation System** (`ci.sh site`):
|
||||
- Documentation framework: VuePress
|
||||
- Orchestrator: `ci_site.sh`
|
||||
- Worker: `generate_board_docs.py` (auto-generate board pages)
|
||||
- Worker: `gen_wu.py` (WebUpdater manifest)
|
||||
- External: Common config from theengs.io
|
||||
|
||||
**Code Quality** (`ci.sh qa`):
|
||||
- Orchestrator: `ci_qa.sh`
|
||||
- Worker: `clang-format` version 9
|
||||
- Default scope: `main` directory, `.h` and `.ino` files
|
||||
|
||||
**Configuration**:
|
||||
- Environment list: `.github/workflows/environments.json` (83 environments)
|
||||
- Task workflows: `task-*.yml` (reusable GitHub Actions components)
|
||||
- Repository owner restriction: Hardcoded to `1technophile` for dev deployments
|
||||
- All scripts located in: `./scripts/`
|
||||
|
||||
**Local Development**:
|
||||
```bash
|
||||
# Build firmware locally
|
||||
./scripts/ci.sh build esp32dev-ble --mode dev --version test
|
||||
|
||||
# Build documentation locally
|
||||
./scripts/ci.sh site --mode dev --preview
|
||||
|
||||
# Check code format
|
||||
./scripts/ci.sh qa --check
|
||||
|
||||
# Run complete pipeline
|
||||
./scripts/ci.sh all esp32dev-ble --version v1.8.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 2.1
|
||||
**Last Updated**: December 29, 2025
|
||||
**Maintainer**: OpenMQTTGateway Development Team
|
||||
124
.github/workflows/build.yml
vendored
124
.github/workflows/build.yml
vendored
@@ -4,125 +4,17 @@ on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
build:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
environments:
|
||||
- "rfbridge"
|
||||
- "rfbridge-direct"
|
||||
- "esp32dev-all-test"
|
||||
- "esp32dev-rf"
|
||||
- "esp32dev-pilight-cc1101"
|
||||
- "esp32dev-somfy-cc1101"
|
||||
- "esp32dev-pilight-somfy-cc1101"
|
||||
- "esp32dev-weatherstation"
|
||||
- "esp32dev-gf-sun-inverter"
|
||||
- "esp32dev-ir"
|
||||
- "esp32dev-ble"
|
||||
- "esp32dev-ble-mqtt-undecoded"
|
||||
- "esp32dev-ble-aws"
|
||||
- "esp32feather-ble"
|
||||
- "esp32-lolin32lite-ble"
|
||||
- "esp32-olimex-gtw-ble-eth"
|
||||
- "esp32-olimex-gtw-ble-poe"
|
||||
- "esp32-olimex-gtw-ble-poe-iso"
|
||||
- "esp32-wt32-eth01-ble-eth"
|
||||
- "esp32-olimex-gtw-ble-wifi"
|
||||
- "esp32-m5stick-ble"
|
||||
- "esp32-m5stack-ble"
|
||||
- "esp32-m5tough-ble"
|
||||
- "esp32-m5stick-c-ble"
|
||||
- "esp32-m5stick-cp-ble"
|
||||
- "esp32s3-atomS3U"
|
||||
- "esp32-m5atom-matrix"
|
||||
- "esp32-m5atom-lite"
|
||||
- "esp32dev-rtl_433"
|
||||
- "esp32dev-rtl_433-fsk"
|
||||
- "esp32doitv1-aithinker-r01-sx1278"
|
||||
- "heltec-rtl_433"
|
||||
- "heltec-rtl_433-fsk"
|
||||
- "heltec-ble"
|
||||
- "lilygo-rtl_433"
|
||||
- "lilygo-rtl_433-fsk"
|
||||
- "lilygo-ble"
|
||||
- "esp32dev-multi_receiver"
|
||||
- "esp32dev-multi_receiver-pilight"
|
||||
- "tinypico-ble"
|
||||
- "ttgo-lora32-v1"
|
||||
- "ttgo-lora32-v21"
|
||||
- "ttgo-t-beam"
|
||||
- "heltec-wifi-lora-32"
|
||||
- "shelly-plus1"
|
||||
- "nodemcuv2-all-test"
|
||||
- "nodemcuv2-fastled-test"
|
||||
- "nodemcuv2-2g"
|
||||
- "nodemcuv2-ir"
|
||||
- "nodemcuv2-serial"
|
||||
- "avatto-bakeey-ir"
|
||||
- "nodemcuv2-rf"
|
||||
- "nodemcuv2-rf-cc1101"
|
||||
- "nodemcuv2-somfy-cc1101"
|
||||
- "manual-wifi-test"
|
||||
- "rf-wifi-gateway"
|
||||
- "nodemcuv2-rf2"
|
||||
- "nodemcuv2-rf2-cc1101"
|
||||
- "nodemcuv2-pilight"
|
||||
- "nodemcuv2-weatherstation"
|
||||
- "sonoff-basic"
|
||||
- "sonoff-basic-rfr3"
|
||||
- "esp32dev-ble-datatest"
|
||||
- "esp32s3-dev-c1-ble"
|
||||
- "esp32c3-dev-m1-ble"
|
||||
- "airm2m_core_esp32c3"
|
||||
- "esp32c3_lolin_mini"
|
||||
- "esp32c3-m5stamp"
|
||||
- "thingpulse-espgateway"
|
||||
- "theengs-bridge"
|
||||
- "esp32dev-ble-idf"
|
||||
- "theengs-bridge-v11"
|
||||
- "theengs-plug"
|
||||
- "esp32dev-ble-broker"
|
||||
- "esp32s3-m5stack-stamps3"
|
||||
- "esp32c3u-m5stamp"
|
||||
- "lilygo-t3-s3-rtl_433"
|
||||
- "lilygo-t3-s3-rtl_433-fsk"
|
||||
runs-on: ubuntu-latest
|
||||
name: Build with PlatformIO
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
- name: Extract ESP32 platform version from platformio.ini
|
||||
run: |
|
||||
ESP32_VERSION=$(grep 'esp32_platform\s*=' platformio.ini | cut -d'@' -f2 | tr -d '[:space:]')
|
||||
echo "ESP32_PLATFORM_VERSION=${ESP32_VERSION}" >> $GITHUB_ENV
|
||||
- name: Run PlatformIO
|
||||
env:
|
||||
PYTHONIOENCODING: utf-8
|
||||
PYTHONUTF8: '1'
|
||||
run: platformio run -e ${{ matrix.environments }}
|
||||
- name: Upload Assets
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ matrix.environments }}
|
||||
path: |
|
||||
.pio/build/*/firmware.bin
|
||||
.pio/build/*/partitions.bin
|
||||
retention-days: 7
|
||||
name: Build firmware
|
||||
uses: ./.github/workflows/task-build.yml
|
||||
with:
|
||||
python-version: '3.13'
|
||||
enable-dev-ota: false
|
||||
artifact-retention-days: 7
|
||||
prepare-for-deploy: false
|
||||
|
||||
documentation:
|
||||
name: Build documentation
|
||||
runs-on: ubuntu-latest
|
||||
name: Create the documentation
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Node.js
|
||||
|
||||
204
.github/workflows/build_and_docs_to_dev.yml
vendored
204
.github/workflows/build_and_docs_to_dev.yml
vendored
@@ -1,173 +1,65 @@
|
||||
name: Build binaries, docs and publish to dev folder
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: '0 0 * * *'
|
||||
|
||||
jobs:
|
||||
build:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
environments:
|
||||
- "rfbridge"
|
||||
- "rfbridge-direct"
|
||||
- "theengs-bridge"
|
||||
- "theengs-bridge-v11"
|
||||
- "theengs-plug"
|
||||
- "esp32dev-all-test"
|
||||
- "esp32dev-rf"
|
||||
- "esp32dev-pilight"
|
||||
- "esp32dev-pilight-cc1101"
|
||||
- "esp32dev-somfy-cc1101"
|
||||
- "esp32dev-pilight-somfy-cc1101"
|
||||
- "esp32dev-weatherstation"
|
||||
- "esp32dev-gf-sun-inverter"
|
||||
- "esp32dev-ir"
|
||||
- "esp32dev-ble"
|
||||
- "esp32dev-ble-broker"
|
||||
- "esp32dev-ble-mqtt-undecoded"
|
||||
- "esp32dev-ble-aws"
|
||||
- "esp32feather-ble"
|
||||
- "esp32-lolin32lite-ble"
|
||||
- "esp32-olimex-gtw-ble-eth"
|
||||
- "esp32-olimex-gtw-ble-poe"
|
||||
- "esp32-olimex-gtw-ble-poe-iso"
|
||||
- "esp32-wt32-eth01-ble-eth"
|
||||
- "esp32-olimex-gtw-ble-wifi"
|
||||
- "esp32-m5stick-ble"
|
||||
- "esp32-m5stack-ble"
|
||||
- "esp32-m5tough-ble"
|
||||
- "esp32-m5stick-c-ble"
|
||||
- "esp32-m5stick-cp-ble"
|
||||
- "esp32-m5atom-matrix"
|
||||
- "esp32-m5atom-lite"
|
||||
- "esp32doitv1-aithinker-r01-sx1278"
|
||||
- "esp32dev-rtl_433"
|
||||
- "esp32dev-rtl_433-fsk"
|
||||
- "heltec-rtl_433"
|
||||
- "heltec-rtl_433-fsk"
|
||||
- "heltec-ble"
|
||||
- "lilygo-rtl_433"
|
||||
- "lilygo-rtl_433-fsk"
|
||||
- "lilygo-t3-s3-rtl_433"
|
||||
- "lilygo-t3-s3-rtl_433-fsk"
|
||||
- "lilygo-ble"
|
||||
- "esp32dev-multi_receiver"
|
||||
- "esp32dev-multi_receiver-pilight"
|
||||
- "tinypico-ble"
|
||||
- "ttgo-lora32-v1"
|
||||
- "ttgo-lora32-v21"
|
||||
- "ttgo-t-beam"
|
||||
- "heltec-wifi-lora-32"
|
||||
- "shelly-plus1"
|
||||
- "nodemcuv2-all-test"
|
||||
- "nodemcuv2-fastled-test"
|
||||
- "nodemcuv2-2g"
|
||||
- "nodemcuv2-ir"
|
||||
- "nodemcuv2-serial"
|
||||
- "avatto-bakeey-ir"
|
||||
- "nodemcuv2-rf"
|
||||
- "nodemcuv2-rf-cc1101"
|
||||
- "nodemcuv2-somfy-cc1101"
|
||||
- "manual-wifi-test"
|
||||
- "rf-wifi-gateway"
|
||||
- "nodemcuv2-rf2"
|
||||
- "nodemcuv2-rf2-cc1101"
|
||||
- "nodemcuv2-pilight"
|
||||
- "nodemcuv2-weatherstation"
|
||||
- "sonoff-basic"
|
||||
- "sonoff-basic-rfr3"
|
||||
- "esp32dev-ble-datatest"
|
||||
- "esp32s3-dev-c1-ble"
|
||||
- "esp32s3-m5stack-stamps3"
|
||||
- "esp32s3-atomS3U"
|
||||
- "esp32c3-dev-m1-ble"
|
||||
- "airm2m_core_esp32c3"
|
||||
- "esp32c3-dev-c2-ble"
|
||||
- "esp32c3-dev-c2-ble-no-serial"
|
||||
- "esp32c3_lolin_mini"
|
||||
- "esp32c3_lolin_mini_with_serial"
|
||||
- "esp32c3-m5stamp"
|
||||
- "esp32c3u-m5stamp"
|
||||
- "thingpulse-espgateway"
|
||||
- "esp32dev-ble-idf"
|
||||
prepare:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository_owner == '1technophile'
|
||||
name: Build ${{ matrix.environments }}
|
||||
outputs:
|
||||
short-sha: ${{ steps.short-sha.outputs.sha }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: benjlevesque/short-sha@v2.1
|
||||
id: short-sha
|
||||
with:
|
||||
length: 6
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
- name: Set sha tag
|
||||
run: |
|
||||
sed -i "s/version_tag/${{ steps.short-sha.outputs.sha }}/g" main/User_config.h scripts/latest_version_dev.json
|
||||
- name: Run PlatformIO
|
||||
env:
|
||||
PYTHONIOENCODING: utf-8
|
||||
PYTHONUTF8: '1'
|
||||
run: |
|
||||
export PLATFORMIO_BUILD_FLAGS="'-DDEVELOPMENTOTA=true'"
|
||||
platformio run -e ${{ matrix.environments }}
|
||||
- name: Prepare firmware artifacts
|
||||
run: |
|
||||
mkdir -p firmware
|
||||
cp .pio/build/${{ matrix.environments }}/firmware.bin firmware/${{ matrix.environments }}-firmware.bin
|
||||
if [ -f .pio/build/${{ matrix.environments }}/partitions.bin ]; then
|
||||
cp .pio/build/${{ matrix.environments }}/partitions.bin firmware/${{ matrix.environments }}-partitions.bin
|
||||
fi
|
||||
if [ -f .pio/build/${{ matrix.environments }}/bootloader.bin ]; then
|
||||
cp .pio/build/${{ matrix.environments }}/bootloader.bin firmware/${{ matrix.environments }}-bootloader.bin
|
||||
fi
|
||||
- name: Upload firmware
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: firmware-${{ matrix.environments }}
|
||||
path: firmware/
|
||||
retention-days: 1
|
||||
|
||||
build:
|
||||
needs: prepare
|
||||
name: Build development firmware
|
||||
uses: ./.github/workflows/task-build.yml
|
||||
with:
|
||||
python-version: '3.13'
|
||||
enable-dev-ota: true
|
||||
version-tag: ${{ needs.prepare.outputs.short-sha }}
|
||||
artifact-retention-days: 1
|
||||
artifact-name-prefix: 'firmware-'
|
||||
prepare-for-deploy: true
|
||||
|
||||
deploy:
|
||||
needs: build
|
||||
needs: [prepare, build]
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository_owner == '1technophile'
|
||||
name: Deploy binaries and docs
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: benjlevesque/short-sha@v2.1
|
||||
id: short-sha
|
||||
with:
|
||||
length: 6
|
||||
|
||||
- name: Download all firmware artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: firmware-*
|
||||
path: toDeploy
|
||||
merge-multiple: true
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
|
||||
- name: Create library zips
|
||||
run: |
|
||||
# Install libraries for a representative environment to get libdeps
|
||||
@@ -180,6 +72,7 @@ jobs:
|
||||
zip -r "${i%/}-libraries.zip" "$i"
|
||||
done
|
||||
mv *.zip ../../toDeploy/
|
||||
|
||||
- name: Prepare additional assets
|
||||
run: |
|
||||
cd toDeploy
|
||||
@@ -189,33 +82,22 @@ jobs:
|
||||
# Zip source code
|
||||
zip -r toDeploy/OpenMQTTGateway_sources.zip main LICENSE.txt
|
||||
ls -lA toDeploy/
|
||||
- name: Set sha tag for docs
|
||||
run: |
|
||||
sed -i "s/version_tag/DEVELOPMENT SHA:${{ steps.short-sha.outputs.sha }} TEST ONLY/g" docs/.vuepress/config.js
|
||||
sed -i "s|base: '/'|base: '/dev/'|g" docs/.vuepress/config.js
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "16.x"
|
||||
- name: Download Common Config
|
||||
run: |
|
||||
curl -o docs/.vuepress/public/commonConfig.js https://www.theengs.io/commonConfig.js
|
||||
- name: Build documentation
|
||||
run: |
|
||||
python ./scripts/gen_wu.py --dev
|
||||
npm install
|
||||
npm run docs:build
|
||||
- name: Deploy to GitHub Pages
|
||||
uses: peaceiris/actions-gh-pages@v3
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
publish_dir: ./docs/.vuepress/dist
|
||||
destination_dir: dev
|
||||
cname: docs.openmqttgateway.com
|
||||
- name: Running Page Speed Insights
|
||||
uses: jakepartusch/psi-action@v1.3
|
||||
id: psi
|
||||
with:
|
||||
url: "https://docs.openmqttgateway.com/dev/"
|
||||
threshold: 60
|
||||
key: ${{ secrets.APIKEY }}
|
||||
|
||||
documentation:
|
||||
needs: [prepare, build]
|
||||
name: Build and deploy development documentation
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
python-version: '3.11'
|
||||
node-version: '16.x'
|
||||
version-source: 'custom'
|
||||
custom-version: 'DEVELOPMENT SHA:${{ needs.prepare.outputs.short-sha }} TEST ONLY'
|
||||
base-path: '/dev/'
|
||||
destination-dir: 'dev'
|
||||
generate-webuploader: true
|
||||
webuploader-args: '--dev'
|
||||
run-pagespeed: true
|
||||
pagespeed-url: 'https://docs.openmqttgateway.com/dev/'
|
||||
secrets:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
APIKEY: ${{ secrets.APIKEY }}
|
||||
|
||||
106
.github/workflows/environments.json
vendored
Normal file
106
.github/workflows/environments.json
vendored
Normal file
@@ -0,0 +1,106 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft-07/schema",
|
||||
"description": "Centralized list of all PlatformIO build environments for OpenMQTTGateway workflows",
|
||||
"version": "1.0.0",
|
||||
"lastUpdated": "2025-12-24",
|
||||
"environments": {
|
||||
"all": [
|
||||
"rfbridge",
|
||||
"rfbridge-direct",
|
||||
"theengs-bridge",
|
||||
"theengs-bridge-v11",
|
||||
"theengs-plug",
|
||||
"esp32dev-all-test",
|
||||
"esp32dev-rf",
|
||||
"esp32dev-pilight",
|
||||
"esp32dev-pilight-cc1101",
|
||||
"esp32dev-somfy-cc1101",
|
||||
"esp32dev-pilight-somfy-cc1101",
|
||||
"esp32dev-weatherstation",
|
||||
"esp32dev-gf-sun-inverter",
|
||||
"esp32dev-ir",
|
||||
"esp32dev-ble",
|
||||
"esp32dev-ble-broker",
|
||||
"esp32dev-ble-mqtt-undecoded",
|
||||
"esp32dev-ble-aws",
|
||||
"esp32dev-ble-datatest",
|
||||
"esp32dev-ble-idf",
|
||||
"esp32dev-rtl_433",
|
||||
"esp32dev-rtl_433-fsk",
|
||||
"esp32dev-multi_receiver",
|
||||
"esp32dev-multi_receiver-pilight",
|
||||
"esp32feather-ble",
|
||||
"esp32-lolin32lite-ble",
|
||||
"esp32-olimex-gtw-ble-eth",
|
||||
"esp32-olimex-gtw-ble-poe",
|
||||
"esp32-olimex-gtw-ble-poe-iso",
|
||||
"esp32-wt32-eth01-ble-eth",
|
||||
"esp32-olimex-gtw-ble-wifi",
|
||||
"esp32-m5stick-ble",
|
||||
"esp32-m5stack-ble",
|
||||
"esp32-m5tough-ble",
|
||||
"esp32-m5stick-c-ble",
|
||||
"esp32-m5stick-cp-ble",
|
||||
"esp32-m5atom-matrix",
|
||||
"esp32-m5atom-lite",
|
||||
"esp32doitv1-aithinker-r01-sx1278",
|
||||
"esp32s3-dev-c1-ble",
|
||||
"esp32s3-m5stack-stamps3",
|
||||
"esp32s3-atomS3U",
|
||||
"esp32c3-dev-m1-ble",
|
||||
"esp32c3-dev-c2-ble",
|
||||
"esp32c3-dev-c2-ble-no-serial",
|
||||
"esp32c3_lolin_mini",
|
||||
"esp32c3_lolin_mini_with_serial",
|
||||
"esp32c3-m5stamp",
|
||||
"esp32c3u-m5stamp",
|
||||
"airm2m_core_esp32c3",
|
||||
"heltec-rtl_433",
|
||||
"heltec-rtl_433-fsk",
|
||||
"heltec-ble",
|
||||
"heltec-wifi-lora-32",
|
||||
"lilygo-rtl_433",
|
||||
"lilygo-rtl_433-fsk",
|
||||
"lilygo-t3-s3-rtl_433",
|
||||
"lilygo-t3-s3-rtl_433-fsk",
|
||||
"lilygo-ble",
|
||||
"tinypico-ble",
|
||||
"ttgo-lora32-v1",
|
||||
"ttgo-lora32-v21",
|
||||
"ttgo-t-beam",
|
||||
"thingpulse-espgateway",
|
||||
"shelly-plus1",
|
||||
"nodemcuv2-all-test",
|
||||
"nodemcuv2-fastled-test",
|
||||
"nodemcuv2-2g",
|
||||
"nodemcuv2-ir",
|
||||
"nodemcuv2-serial",
|
||||
"nodemcuv2-rf",
|
||||
"nodemcuv2-rf-cc1101",
|
||||
"nodemcuv2-somfy-cc1101",
|
||||
"nodemcuv2-rf2",
|
||||
"nodemcuv2-rf2-cc1101",
|
||||
"nodemcuv2-pilight",
|
||||
"nodemcuv2-weatherstation",
|
||||
"avatto-bakeey-ir",
|
||||
"manual-wifi-test",
|
||||
"rf-wifi-gateway",
|
||||
"sonoff-basic",
|
||||
"sonoff-basic-rfr3"
|
||||
],
|
||||
"metadata": {
|
||||
"totalCount": 83,
|
||||
"categories": {
|
||||
"esp32": 42,
|
||||
"esp8266": 17,
|
||||
"specialized": 24
|
||||
},
|
||||
"notes": [
|
||||
"This unified list contains all environments from both build.yml and build_and_docs_to_dev.yml",
|
||||
"Previously build.yml had 85 environments and build_and_docs_to_dev.yml had 88",
|
||||
"After deduplication and proper ordering, the total is 83 unique environments",
|
||||
"Environments ending in -test or -all-test are typically excluded from production releases"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
16
.github/workflows/lint.yml
vendored
16
.github/workflows/lint.yml
vendored
@@ -4,13 +4,9 @@ on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Check main format
|
||||
uses: DoozyX/clang-format-lint-action@v0.6
|
||||
with:
|
||||
source: "./main"
|
||||
extensions: "h,ino"
|
||||
clangFormatVersion: 9
|
||||
name: Lint code format
|
||||
uses: ./.github/workflows/task-lint.yml
|
||||
with:
|
||||
source: 'main'
|
||||
extensions: 'h,ino'
|
||||
clang-format-version: '9'
|
||||
|
||||
54
.github/workflows/manual_docs.yml
vendored
54
.github/workflows/manual_docs.yml
vendored
@@ -1,45 +1,21 @@
|
||||
name: Create and publish documentation
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
documentation:
|
||||
runs-on: ubuntu-latest
|
||||
name: Create the documentation and deploy it to GitHub Pages
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "14.x"
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install requests pandas markdown pytablereader tabulate
|
||||
npm install
|
||||
- name: Download Common Config
|
||||
run: |
|
||||
curl -o docs/.vuepress/public/commonConfig.js https://www.theengs.io/commonConfig.js
|
||||
- name: get lastest release tag
|
||||
id: last_release
|
||||
uses: InsonusK/get-latest-release@v1.0.1
|
||||
with:
|
||||
myToken: ${{ github.token }}
|
||||
view_top: 1
|
||||
- name: Set version tag from git
|
||||
run: sed -i "s/version_tag/${{steps.last_release.outputs.tag_name}}/g" docs/.vuepress/config.js scripts/latest_version.json
|
||||
- name: Build documentation
|
||||
run: |
|
||||
python ./scripts/generate_board_docs.py
|
||||
python ./scripts/gen_wu.py ${GITHUB_REPOSITORY}
|
||||
npm run docs:build
|
||||
- name: Deploy to GitHub Pages
|
||||
uses: peaceiris/actions-gh-pages@v3
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
publish_dir: ./docs/.vuepress/dist
|
||||
cname: docs.openmqttgateway.com
|
||||
name: Build and deploy production documentation
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
python-version: '3.11'
|
||||
node-version: '14.x'
|
||||
version-source: 'release'
|
||||
base-path: '/'
|
||||
destination-dir: '.'
|
||||
generate-webuploader: true
|
||||
webuploader-args: '${GITHUB_REPOSITORY}'
|
||||
run-pagespeed: false
|
||||
secrets:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
96
.github/workflows/release.yml
vendored
96
.github/workflows/release.yml
vendored
@@ -5,47 +5,107 @@ on:
|
||||
types: [published]
|
||||
|
||||
jobs:
|
||||
build-upload:
|
||||
prepare:
|
||||
runs-on: ubuntu-latest
|
||||
name: Build and upload Assets to Release
|
||||
outputs:
|
||||
version-tag: ${{ steps.extract-tag.outputs.version }}
|
||||
release-id: ${{ steps.extract-release.outputs.id }}
|
||||
upload-url: ${{ steps.extract-release.outputs.upload_url }}
|
||||
steps:
|
||||
- name: Extract version tag
|
||||
id: extract-tag
|
||||
run: |
|
||||
VERSION_TAG=${GITHUB_REF#refs/tags/}
|
||||
echo "version=${VERSION_TAG}" >> $GITHUB_OUTPUT
|
||||
echo "Extracted version: ${VERSION_TAG}"
|
||||
|
||||
- name: Extract release info
|
||||
id: extract-release
|
||||
run: |
|
||||
RELEASE_ID=$(jq --raw-output '.release.id' $GITHUB_EVENT_PATH)
|
||||
UPLOAD_URL="https://uploads.github.com/repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}/assets{?name,label}"
|
||||
echo "id=${RELEASE_ID}" >> $GITHUB_OUTPUT
|
||||
echo "upload_url=${UPLOAD_URL}" >> $GITHUB_OUTPUT
|
||||
echo "Release ID: ${RELEASE_ID}"
|
||||
|
||||
build:
|
||||
needs: prepare
|
||||
name: Build release firmware
|
||||
uses: ./.github/workflows/task-build.yml
|
||||
with:
|
||||
python-version: '3.13'
|
||||
enable-dev-ota: false
|
||||
version-tag: ${{ needs.prepare.outputs.version-tag }}
|
||||
artifact-retention-days: 90
|
||||
artifact-name-prefix: 'firmware-'
|
||||
prepare-for-deploy: true
|
||||
|
||||
deploy:
|
||||
needs: [prepare, build]
|
||||
runs-on: ubuntu-latest
|
||||
name: Deploy release assets
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Download all firmware artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: firmware-*
|
||||
path: .pio/build
|
||||
merge-multiple: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
|
||||
- name: Install platformio
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
- name: Set version tag from git
|
||||
run: sed -i "s/version_tag/${GITHUB_REF#refs/tags/}/g" main/User_config.h scripts/latest_version.json
|
||||
- name: Extract ESP32 platform version from platformio.ini
|
||||
|
||||
- name: Reorganize artifacts for prepare_deploy.sh
|
||||
run: |
|
||||
ESP32_VERSION=$(grep 'esp32_platform\s*=' platformio.ini | cut -d'@' -f2 | tr -d '[:space:]')
|
||||
echo "ESP32_PLATFORM_VERSION=${ESP32_VERSION}" >> $GITHUB_ENV
|
||||
- name: Run PlatformIO
|
||||
run: platformio run
|
||||
# Move artifacts from download structure to expected .pio/build structure
|
||||
for env_dir in .pio/build/firmware-*; do
|
||||
if [ -d "$env_dir" ]; then
|
||||
env_name=$(basename "$env_dir" | sed 's/^firmware-//')
|
||||
mkdir -p ".pio/build/${env_name}"
|
||||
mv "$env_dir"/*.bin ".pio/build/${env_name}/" 2>/dev/null || true
|
||||
rm -rf "$env_dir"
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Prepare Release Assets
|
||||
run: |
|
||||
sudo apt install rename
|
||||
./scripts/prepare_deploy.sh
|
||||
- name: Get upload url
|
||||
id: release-id
|
||||
run: |
|
||||
RELEASE_ID=$(jq --raw-output '.release.id' $GITHUB_EVENT_PATH)
|
||||
echo "::set-output name=upload_url::https://uploads.github.com/repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}/assets{?name,label}"
|
||||
|
||||
- name: Upload Release Assets
|
||||
uses: bgpat/release-asset-action@03b0c30db1c4031ce3474740b0e4275cd7e126a3
|
||||
with:
|
||||
pattern: "toDeploy/*"
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
release-url: ${{ steps.release-id.outputs.upload_url }}
|
||||
release-url: ${{ needs.prepare.outputs.upload-url }}
|
||||
allow-overwrite: true
|
||||
call-workflow-passing-data:
|
||||
needs: build-upload
|
||||
uses: ./.github/workflows/manual_docs.yml
|
||||
|
||||
documentation:
|
||||
needs: [prepare, deploy]
|
||||
name: Build and deploy release documentation
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
python-version: '3.11'
|
||||
node-version: '18.x'
|
||||
version-source: 'git-tag'
|
||||
base-path: '/'
|
||||
destination-dir: '.'
|
||||
generate-webuploader: true
|
||||
run-pagespeed: false
|
||||
secrets:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
APIKEY: ${{ secrets.APIKEY }}
|
||||
|
||||
104
.github/workflows/task-build.yml
vendored
Normal file
104
.github/workflows/task-build.yml
vendored
Normal file
@@ -0,0 +1,104 @@
|
||||
name: Reusable Build Workflow
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
python-version:
|
||||
description: 'Python version to use'
|
||||
required: false
|
||||
type: string
|
||||
default: '3.13'
|
||||
enable-dev-ota:
|
||||
description: 'Enable development OTA builds'
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
version-tag:
|
||||
description: 'Version tag to inject into firmware (e.g., SHA or release tag)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'unspecified'
|
||||
artifact-retention-days:
|
||||
description: 'Number of days to retain build artifacts'
|
||||
required: false
|
||||
type: number
|
||||
default: 7
|
||||
artifact-name-prefix:
|
||||
description: 'Prefix for artifact names (e.g., "firmware-" for dev builds)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
prepare-for-deploy:
|
||||
description: 'Prepare firmware files for deployment with specific naming'
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
jobs:
|
||||
load-environments:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
matrix: ${{ steps.set-matrix.outputs.matrix }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- id: set-matrix
|
||||
run: |
|
||||
ENVIRONMENTS=$(jq -c '.environments.all' .github/workflows/environments.json)
|
||||
echo "matrix=${ENVIRONMENTS}" >> $GITHUB_OUTPUT
|
||||
|
||||
build:
|
||||
needs: load-environments
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
environments: ${{ fromJson(needs.load-environments.outputs.matrix) }}
|
||||
name: Build ${{ matrix.environments }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ inputs.python-version }}
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
|
||||
- name: Install PlatformIO dependencies
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
|
||||
- name: Build firmware using ci.sh
|
||||
run: |
|
||||
BUILD_ARGS="${{ matrix.environments }}"
|
||||
|
||||
# Add version tag if specified
|
||||
if [ "${{ inputs.version-tag }}" != "unspecified" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --version ${{ inputs.version-tag }}"
|
||||
fi
|
||||
|
||||
# Add mode flag (dev/prod)
|
||||
if [ "${{ inputs.enable-dev-ota }}" = "true" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --mode dev"
|
||||
else
|
||||
BUILD_ARGS="$BUILD_ARGS --mode prod"
|
||||
fi
|
||||
|
||||
# Add deploy flag if needed
|
||||
if [ "${{ inputs.prepare-for-deploy }}" = "true" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --deploy-ready"
|
||||
fi
|
||||
|
||||
# Execute build (uses default generated/artifacts/ directory)
|
||||
./scripts/ci.sh build $BUILD_ARGS
|
||||
|
||||
- name: Upload firmware artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ inputs.artifact-name-prefix }}${{ matrix.environments }}
|
||||
path: generated/artifacts/
|
||||
retention-days: ${{ inputs.artifact-retention-days }}
|
||||
127
.github/workflows/task-docs.yml
vendored
Normal file
127
.github/workflows/task-docs.yml
vendored
Normal file
@@ -0,0 +1,127 @@
|
||||
name: Reusable Documentation Workflow
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
python-version:
|
||||
description: 'Python version to use'
|
||||
required: false
|
||||
type: string
|
||||
default: '3.11'
|
||||
node-version:
|
||||
description: 'Node.js version to use'
|
||||
required: false
|
||||
type: string
|
||||
default: '14.x'
|
||||
version-source:
|
||||
description: 'Source of version: "release" (from GitHub release) or "custom" (provided)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'release'
|
||||
custom-version:
|
||||
description: 'Custom version string when version-source is "custom"'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
base-path:
|
||||
description: 'Base path for documentation (e.g., "/" for prod, "/dev/" for dev)'
|
||||
required: false
|
||||
type: string
|
||||
default: '/'
|
||||
destination-dir:
|
||||
description: 'Destination directory for GitHub Pages deployment'
|
||||
required: false
|
||||
type: string
|
||||
default: '.'
|
||||
generate-webuploader:
|
||||
description: 'Generate WebUploader manifest'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
webuploader-args:
|
||||
description: 'Additional arguments for gen_wu.py script'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
run-pagespeed:
|
||||
description: 'Run PageSpeed Insights after deployment'
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
pagespeed-url:
|
||||
description: 'URL to test with PageSpeed Insights'
|
||||
required: false
|
||||
type: string
|
||||
default: 'https://docs.openmqttgateway.com/'
|
||||
secrets:
|
||||
GITHUB_TOKEN:
|
||||
required: true
|
||||
APIKEY:
|
||||
required: false
|
||||
|
||||
jobs:
|
||||
documentation:
|
||||
runs-on: ubuntu-latest
|
||||
name: Create and deploy documentation
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ inputs.node-version }}
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ inputs.python-version }}
|
||||
|
||||
- name: Build documentation site
|
||||
run: |
|
||||
# Build script arguments based on inputs
|
||||
ARGS=""
|
||||
|
||||
# Set version source and custom version
|
||||
if [ "${{ inputs.version-source }}" = "custom" ] && [ -n "${{ inputs.custom-version }}" ]; then
|
||||
ARGS="$ARGS --custom-version '${{ inputs.custom-version }}' --version-source custom"
|
||||
elif [ "${{ inputs.version-source }}" = "release" ]; then
|
||||
# Get latest release tag
|
||||
if command -v git >/dev/null 2>&1; then
|
||||
LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
|
||||
ARGS="$ARGS --custom-version '${LATEST_TAG}' --version-source release"
|
||||
else
|
||||
ARGS="$ARGS --version-source release"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Set base path/URL prefix
|
||||
if [ "${{ inputs.base-path }}" != "/" ]; then
|
||||
ARGS="$ARGS --url-prefix '${{ inputs.base-path }}'"
|
||||
fi
|
||||
|
||||
# WebUploader generation
|
||||
if [ "${{ inputs.generate-webuploader }}" != "true" ]; then
|
||||
ARGS="$ARGS --no-webuploader"
|
||||
elif [ -n "${{ inputs.webuploader-args }}" ]; then
|
||||
ARGS="$ARGS --webuploader-args '${{ inputs.webuploader-args }}'"
|
||||
fi
|
||||
|
||||
# Run the build script
|
||||
eval "./scripts/ci_site.sh ${ARGS}"
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
uses: peaceiris/actions-gh-pages@v3
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
publish_dir: ./generated/site
|
||||
destination_dir: ${{ inputs.destination-dir }}
|
||||
cname: docs.openmqttgateway.com
|
||||
|
||||
- name: Run PageSpeed Insights
|
||||
if: inputs.run-pagespeed
|
||||
uses: jakepartusch/psi-action@v1.3
|
||||
id: psi
|
||||
with:
|
||||
url: ${{ inputs.pagespeed-url }}
|
||||
threshold: 60
|
||||
key: ${{ secrets.APIKEY }}
|
||||
40
.github/workflows/task-lint.yml
vendored
Normal file
40
.github/workflows/task-lint.yml
vendored
Normal file
@@ -0,0 +1,40 @@
|
||||
name: Reusable Lint Workflow
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
source:
|
||||
description: 'Source directory to lint (single directory)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'main'
|
||||
extensions:
|
||||
description: 'File extensions to check (comma-separated)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'h,ino,cpp'
|
||||
clang-format-version:
|
||||
description: 'clang-format version to use'
|
||||
required: false
|
||||
type: string
|
||||
default: '9'
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
name: Check code format
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install clang-format
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y clang-format
|
||||
|
||||
- name: Check code format with ci.sh qa
|
||||
run: |
|
||||
./scripts/ci.sh qa \
|
||||
--check \
|
||||
--source "${{ inputs.source }}" \
|
||||
--extensions "${{ inputs.extensions }}" \
|
||||
--clang-format-version "${{ inputs.clang-format-version }}"
|
||||
13
.gitignore
vendored
13
.gitignore
vendored
@@ -21,4 +21,15 @@ managed_components
|
||||
|
||||
.github/chatmodes
|
||||
.github/prompts
|
||||
.github/*instructions.md
|
||||
.github/*instructions.md
|
||||
.github/workflows/_docs
|
||||
# CI/CD Generated outputs
|
||||
docs/.vuepress/dist/
|
||||
docs/.vuepress/components/
|
||||
docs/.vuepress/public/
|
||||
^docs/.vuepress/public/img/
|
||||
^docs/.vuepress/public/*.png
|
||||
^docs/.vuepress/public/*.txt
|
||||
|
||||
generated/
|
||||
|
||||
|
||||
575
scripts/CI_SCRIPTS.md
Normal file
575
scripts/CI_SCRIPTS.md
Normal file
@@ -0,0 +1,575 @@
|
||||
# CI/CD Scripts Documentation
|
||||
|
||||
This documentation describes the CI/CD scripts used to build OpenMQTTGateway firmware and documentation. These scripts work in GitHub Actions, locally, and in any CI/CD environment.
|
||||
|
||||
- [Overview](#cicd-scripts-documentation)
|
||||
- [Quick Reference](#quick-reference)
|
||||
- [Script Hierarchy](#script-hierarchy)
|
||||
- [Script Description](#script-description)
|
||||
- [Output Structure](#output-structure)
|
||||
- [Commands](#commands)
|
||||
- [ci.sh - Main Entry Point](#commands)
|
||||
- [ci.sh build - Build Firmware](#cish-build---build-firmware)
|
||||
- [ci.sh site - Build Documentation](#cish-site---build-documentation)
|
||||
- [ci.sh qa - Code Formatting Check](#cish-qa---code-formatting-check)
|
||||
- [Internal Scripts](#internal-scripts)
|
||||
- [ci_set_version.sh](#ci_set_versionsh)
|
||||
- [ci_build_firmware.sh](#ci_build_firmwaresh)
|
||||
- [ci_prepare_artifacts.sh](#ci_prepare_artifactssh)
|
||||
- [ci_00_config.sh](#ci_00_configsh)
|
||||
- [Python Helper Scripts](#python-helper-scripts)
|
||||
- [generate_board_docs.py](#generate_board_docspy)
|
||||
- [gen_wu.py](#gen_wupy)
|
||||
- [Environment Variables](#environment-variables)
|
||||
- [Exit Codes](#exit-codes)
|
||||
- [Environment Detection](#environment-detection)
|
||||
|
||||
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Script Hierarchy
|
||||
|
||||
```
|
||||
ci.sh (dispatcher)
|
||||
├── build → ci_build.sh → ci_set_version.sh
|
||||
│ → ci_build_firmware.sh
|
||||
│ → ci_prepare_artifacts.sh
|
||||
├── site → ci_site.sh → generate_board_docs.py
|
||||
│ → gen_wu.py
|
||||
├── qa → ci_qa.sh → clang-format
|
||||
└── all → qa + build + site (sequential)
|
||||
```
|
||||
|
||||
### Script Description
|
||||
|
||||
| Script | Purpose | Called By |
|
||||
|--------|---------|-----------|
|
||||
| `ci.sh` | Main command dispatcher | User/GitHub Actions |
|
||||
| `ci_build.sh` | Build firmware orchestrator | ci.sh build |
|
||||
| `ci_site.sh` | Documentation build orchestrator | ci.sh site |
|
||||
| `ci_qa.sh` | Code formatting checker | ci.sh qa |
|
||||
| `ci_set_version.sh` | Version injection in firmware | ci_build.sh |
|
||||
| `ci_build_firmware.sh` | PlatformIO build execution | ci_build.sh |
|
||||
| `ci_prepare_artifacts.sh` | Artifact packaging | ci_build.sh |
|
||||
| `ci_00_config.sh` | Shared configuration and functions | All scripts |
|
||||
|
||||
|
||||
### Output Structure
|
||||
|
||||
Build outputs are organized in the project root:
|
||||
|
||||
```
|
||||
.pio/build/<environment>/ # PlatformIO build outputs
|
||||
├── firmware.bin # Main firmware binary
|
||||
├── bootloader.bin # ESP32 bootloader
|
||||
└── partitions.bin # ESP32 partition table
|
||||
|
||||
generated/
|
||||
├── artifacts/ # Packaged firmware artifacts
|
||||
└── site/ # Built documentation (VuePress output)
|
||||
|
||||
scripts/
|
||||
├── latest_version.json # Production version metadata
|
||||
└── latest_version_dev.json # Development version metadata
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
`ci.sh` is the main Entry Point. Command dispatcher that routes to specialized scripts.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh <command> [OPTIONS]
|
||||
```
|
||||
|
||||
**Commands:**
|
||||
- `build` - Build firmware for specified environment
|
||||
- `site` or `docs` - Build documentation website
|
||||
- `qa` or `lint` - Run code formatting checks
|
||||
- `all` or `pipeline` - Run complete pipeline (qa + build + site)
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Get Help
|
||||
./scripts/ci.sh build --help
|
||||
./scripts/ci.sh qa --help
|
||||
./scripts/ci.sh site --help
|
||||
|
||||
# Build firmware
|
||||
./scripts/ci.sh build esp32dev-ble --mode dev
|
||||
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
|
||||
|
||||
# Build documentation
|
||||
./scripts/ci.sh site --mode prod
|
||||
./scripts/ci.sh site --mode dev --url-prefix /dev/
|
||||
|
||||
# Check code formatting
|
||||
./scripts/ci.sh qa --check
|
||||
./scripts/ci.sh qa --fix
|
||||
|
||||
# Run complete pipeline
|
||||
./scripts/ci.sh all esp32dev-ble --version v1.8.0
|
||||
./scripts/ci.sh all esp32dev-ble --no-site
|
||||
```
|
||||
|
||||
**Options for `all` command:**
|
||||
- `--no-site` - Skip documentation build step
|
||||
- All options from `build` command are passed through
|
||||
|
||||
---
|
||||
|
||||
### ci.sh build - Build Firmware
|
||||
|
||||
Orchestrates complete firmware build: version injection, compilation, artifact packaging.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh build <environment> [OPTIONS]
|
||||
```
|
||||
|
||||
**Required Arguments:**
|
||||
- `<environment>` - PlatformIO environment name (e.g., esp32dev-ble, nodemcuv2-rf)
|
||||
|
||||
**Options:**
|
||||
- `--version <tag>` - Version string to inject (default: auto-generated from git)
|
||||
- `--mode <dev|prod>` - Build mode (default: prod)
|
||||
- `dev` - Enables development OTA, sets DEVELOPMENTOTA=true
|
||||
- `prod` - Standard production build
|
||||
- `--deploy-ready` - Package artifacts with deployment naming (env-firmware.bin)
|
||||
- `--output <dir>` - Output directory for artifacts (default: generated/artifacts)
|
||||
- `--skip-verification` - Skip build tools verification
|
||||
- `--clean` - Clean previous build before starting
|
||||
- `--verbose` - Enable verbose PlatformIO output
|
||||
- `--list-envs` - List all available PlatformIO environments
|
||||
- `--help` - Show help message
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
|
||||
│
|
||||
├─> ci_build.sh (orchestrator)
|
||||
│ ├─> verify_build_tools() - Check python3, platformio, git
|
||||
│ ├─> ci_set_version.sh v1.8.0 - Inject version in User_config.h
|
||||
│ ├─> ci_build_firmware.sh esp32dev-ble - Execute PlatformIO build
|
||||
│ └─> ci_prepare_artifacts.sh esp32dev-ble --deploy - Package binaries
|
||||
│
|
||||
└─> Outputs in generated/artifacts/
|
||||
├─ esp32dev-ble-firmware.bin
|
||||
├─ esp32dev-ble-bootloader.bin
|
||||
└─ esp32dev-ble-partitions.bin
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Development build
|
||||
./scripts/ci.sh build esp32dev-ble --mode dev
|
||||
|
||||
# Production build with version
|
||||
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod
|
||||
|
||||
# Deploy-ready build
|
||||
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
|
||||
|
||||
# Clean build with verbose output
|
||||
./scripts/ci.sh build nodemcuv2-rf --clean --verbose
|
||||
|
||||
# List available environments
|
||||
./scripts/ci.sh build --list-envs
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
- `CI` - Set to 'true' in CI/CD environments
|
||||
- `BUILD_NUMBER` - Build number from CI/CD system
|
||||
- `GIT_COMMIT` - Git commit hash for auto-versioning
|
||||
- `PLATFORMIO_BUILD_FLAGS` - Additional PlatformIO flags (set by script when --mode dev)
|
||||
|
||||
**Output Files:**
|
||||
- Standard mode: `firmware.bin`, `partitions.bin` in generated/artifacts/
|
||||
- Deploy mode: `<env>-firmware.bin`, `<env>-bootloader.bin`, `<env>-partitions.bin`
|
||||
|
||||
---
|
||||
|
||||
### ci.sh site - Build Documentation
|
||||
|
||||
Builds VuePress documentation website with version management and WebUploader manifest generation.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh site [OPTIONS]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--mode <dev|prod>` - Documentation mode (default: prod)
|
||||
- `dev` - Development documentation with watermark
|
||||
- `prod` - Production documentation
|
||||
- `--version-source <release|custom>` - Version source (default: release)
|
||||
- `release` - Use git tag as version
|
||||
- `custom` - Use custom version string
|
||||
- `--custom-version <version>` - Custom version string (requires --version-source custom)
|
||||
- `--url-prefix <path>` - Base URL path (default: /)
|
||||
- Example: `/dev/` for development subdirectory
|
||||
- `--no-webuploader` - Skip WebUploader manifest generation
|
||||
- `--webuploader-args <args>` - Additional arguments for gen_wu.py
|
||||
- `--preview` - Open browser after build (local development)
|
||||
- `--help` - Show help message
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
ci.sh site --mode prod --version-source release
|
||||
│
|
||||
├─> ci_site.sh (orchestrator)
|
||||
│ ├─> check_requirements() - Verify node, npm, python3, pip3
|
||||
│ ├─> install_dependencies() - npm install, pip3 install packages
|
||||
│ ├─> download_common_config() - Fetch from theengs.io
|
||||
│ ├─> get_version() - Extract from git tag or use custom
|
||||
│ ├─> set_version() - Update VuePress config and JSON files
|
||||
│ ├─> set_url_prefix() - Set base path in config
|
||||
│ ├─> generate_board_docs.py - Auto-generate board documentation
|
||||
│ ├─> npm run docs:build - Build VuePress site
|
||||
│ └─> gen_wu.py - Generate WebUploader manifest
|
||||
│
|
||||
└─> Outputs in generated/site/
|
||||
├─ index.html
|
||||
├─ assets/
|
||||
└─ [board documentation pages]
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Production documentation
|
||||
./scripts/ci.sh site --mode prod
|
||||
|
||||
# Development documentation with custom version
|
||||
./scripts/ci.sh site --mode dev --version-source custom --custom-version "DEVELOPMENT SHA:abc123"
|
||||
|
||||
# Documentation for /dev/ subdirectory
|
||||
./scripts/ci.sh site --mode dev --url-prefix /dev/
|
||||
|
||||
# Skip WebUploader manifest
|
||||
./scripts/ci.sh site --no-webuploader
|
||||
|
||||
# Local preview
|
||||
./scripts/ci.sh site --preview
|
||||
```
|
||||
|
||||
**Required Tools:**
|
||||
- Node.js (for VuePress)
|
||||
- npm (for package management)
|
||||
- Python 3 (for board docs generator)
|
||||
- pip3 (for Python dependencies: requests, pandas, markdown, pytablereader, tabulate)
|
||||
|
||||
**Output Files:**
|
||||
- `generated/site/` - Complete static website
|
||||
- `scripts/latest_version.json` - Production version metadata (updated)
|
||||
- `scripts/latest_version_dev.json` - Development version metadata (updated)
|
||||
|
||||
---
|
||||
|
||||
### ci.sh qa - Code Formatting Check
|
||||
|
||||
Checks and fixes code formatting using clang-format.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh qa [OPTIONS]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--check` - Check formatting without modifying files (default)
|
||||
- `--fix` - Automatically fix formatting issues
|
||||
- `--source <dir>` - Source directory to check (default: main)
|
||||
- `--extensions <list>` - File extensions to check, comma-separated (default: h,ino,cpp)
|
||||
- `--clang-format-version <ver>` - clang-format version to use (default: 9)
|
||||
- `--verbose` - Show detailed output for each file
|
||||
- `--help` - Show help message
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
ci.sh qa --check --source main --extensions h,ino
|
||||
│
|
||||
├─> ci_qa.sh (orchestrator)
|
||||
│ ├─> check_clang_format() - Find clang-format-9 or clang-format
|
||||
│ ├─> find_files() - Locate files matching extensions in source dir
|
||||
│ └─> check_formatting() - Run clang-format --dry-run --Werror
|
||||
│ └─> Report files with formatting issues
|
||||
│
|
||||
└─> Exit code: 0 (pass) or 1 (formatting issues found)
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Check formatting (CI mode)
|
||||
./scripts/ci.sh qa --check
|
||||
|
||||
# Fix formatting automatically
|
||||
./scripts/ci.sh qa --fix
|
||||
|
||||
# Check specific directory
|
||||
./scripts/ci.sh qa --check --source lib
|
||||
|
||||
# Check only .h and .ino files
|
||||
./scripts/ci.sh qa --check --extensions h,ino
|
||||
|
||||
# Check with verbose output
|
||||
./scripts/ci.sh qa --check --verbose
|
||||
|
||||
# Use different clang-format version
|
||||
./scripts/ci.sh qa --check --clang-format-version 11
|
||||
```
|
||||
|
||||
**Required Tools:**
|
||||
- clang-format (version specified, default: 9)
|
||||
- Install: `sudo apt-get install clang-format-9`
|
||||
|
||||
**Output:**
|
||||
- Check mode: Lists files with formatting issues and shows diffs
|
||||
- Fix mode: Modifies files in-place and reports changes
|
||||
- Exit code 0: All files properly formatted
|
||||
- Exit code 1: Formatting issues found (in check mode)
|
||||
|
||||
---
|
||||
|
||||
## Internal Scripts
|
||||
|
||||
These scripts are called by the main orchestrators. It can be invoked directly but is not raccomanded.
|
||||
|
||||
### ci_set_version.sh
|
||||
|
||||
Injects version string into firmware configuration files.
|
||||
|
||||
**Called By:** `ci_build.sh`
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci_set_version.sh <version> [--dev]
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `<version>` - Version string to inject (e.g., v1.8.0 or abc123)
|
||||
- `--dev` - Development mode (updates latest_version_dev.json)
|
||||
|
||||
**Files Modified:**
|
||||
- `main/User_config.h` - Replaces "version_tag" with actual version
|
||||
- `scripts/latest_version.json` - Production version metadata
|
||||
- `scripts/latest_version_dev.json` - Development version metadata (--dev mode)
|
||||
|
||||
**Behavior:**
|
||||
- Creates .bak backup files before modification
|
||||
- Replaces all occurrences of "version_tag" string
|
||||
- Validates version string (must not be empty or "version_tag")
|
||||
- Cleans up backup files on success
|
||||
|
||||
---
|
||||
|
||||
### ci_build_firmware.sh
|
||||
|
||||
Executes PlatformIO build for specified environment.
|
||||
|
||||
**Called By:** `ci_build.sh`
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci_build_firmware.sh <environment> [OPTIONS]
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `<environment>` - PlatformIO environment name
|
||||
|
||||
**Options:**
|
||||
- `--dev-ota` - Enable development OTA (sets PLATFORMIO_BUILD_FLAGS)
|
||||
- `--clean` - Clean before build
|
||||
- `--verbose` - Verbose PlatformIO output
|
||||
|
||||
**Environment Variables Set:**
|
||||
- `PYTHONIOENCODING=utf-8`
|
||||
- `PYTHONUTF8=1`
|
||||
- `PLATFORMIO_BUILD_FLAGS="-DDEVELOPMENTOTA=true"` (when --dev-ota)
|
||||
|
||||
**PlatformIO Command:**
|
||||
```bash
|
||||
platformio run -e <environment> [--verbose]
|
||||
```
|
||||
|
||||
**Output Location:**
|
||||
- `.pio/build/<environment>/firmware.bin`
|
||||
- `.pio/build/<environment>/bootloader.bin` (ESP32 only)
|
||||
- `.pio/build/<environment>/partitions.bin` (ESP32 only)
|
||||
|
||||
---
|
||||
|
||||
### ci_prepare_artifacts.sh
|
||||
|
||||
Packages firmware binaries from PlatformIO build directory.
|
||||
|
||||
**Called By:** `ci_build.sh`
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci_prepare_artifacts.sh <environment> [OPTIONS]
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `<environment>` - PlatformIO environment name
|
||||
|
||||
**Options:**
|
||||
- `--deploy` - Use deployment naming (prefix with environment name)
|
||||
- `--output <dir>` - Output directory (default: generated/artifacts)
|
||||
|
||||
**Behavior:**
|
||||
|
||||
Standard mode (no --deploy):
|
||||
- Copies: `firmware.bin`, `partitions.bin`
|
||||
- Does NOT copy: `bootloader.bin`
|
||||
|
||||
Deploy mode (with --deploy):
|
||||
- Copies and renames:
|
||||
- `firmware.bin` → `<env>-firmware.bin`
|
||||
- `bootloader.bin` → `<env>-bootloader.bin`
|
||||
- `partitions.bin` → `<env>-partitions.bin`
|
||||
|
||||
**Source Location:**
|
||||
- `.pio/build/<environment>/`
|
||||
|
||||
**Output Location:**
|
||||
- Specified by `--output` or default `generated/artifacts/`
|
||||
|
||||
---
|
||||
|
||||
### ci_00_config.sh
|
||||
|
||||
Shared configuration and helper functions for all CI scripts.
|
||||
|
||||
**Sourced By:** All ci_*.sh scripts
|
||||
|
||||
**Provides:**
|
||||
- Color codes for terminal output (BLUE, GREEN, RED, YELLOW, NC)
|
||||
- Logging functions: `log_info()`, `log_warn()`, `log_error()`, `log_success()`
|
||||
- Path constants: `BUILD_DIR`, `ARTIFACTS_DIR`, `SITE_DIR`
|
||||
- Common utility functions
|
||||
|
||||
**Constants Defined:**
|
||||
- `BUILD_DIR=".pio/build"` - PlatformIO build directory
|
||||
- `ARTIFACTS_DIR="generated/artifacts"` - Artifact output directory
|
||||
- `SITE_DIR="generated/site"` - Documentation output directory
|
||||
|
||||
**Logging Functions:**
|
||||
```bash
|
||||
log_info "message" # Blue [INFO] prefix
|
||||
log_warn "message" # Yellow [WARN] prefix
|
||||
log_error "message" # Red [ERROR] prefix
|
||||
log_success "message" # Green [SUCCESS] prefix
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python Helper Scripts
|
||||
|
||||
Other scripts are present and used as internal scripts and it's used as retrocompatibility. Below the lists:
|
||||
- `generate_board_docs.py`
|
||||
- `gen_wu.py`
|
||||
|
||||
### generate_board_docs.py
|
||||
|
||||
Auto-generates board-specific documentation pages from platformio.ini.
|
||||
|
||||
**Called By:** `ci_site.sh`
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 ./scripts/generate_board_docs.py
|
||||
```
|
||||
|
||||
**Input:**
|
||||
- `platformio.ini` - Board configurations
|
||||
- `environments.ini` - Additional environments
|
||||
|
||||
**Output:**
|
||||
- Markdown files in `docs/` directory for each board configuration
|
||||
|
||||
**Purpose:**
|
||||
- Creates documentation pages for each hardware board
|
||||
- Extracts configuration details from PlatformIO environment definitions
|
||||
- Formats technical specifications and pin mappings
|
||||
|
||||
---
|
||||
|
||||
### gen_wu.py
|
||||
|
||||
Generates WebUploader manifest for OTA firmware updates.
|
||||
|
||||
**Called By:** `ci_site.sh`
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 ./scripts/gen_wu.py [--dev] [repository]
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `--dev` - Generate development manifest
|
||||
- `repository` - GitHub repository name (e.g., 1technophile/OpenMQTTGateway)
|
||||
|
||||
**Input:**
|
||||
- `.pio/build/<env>/firmware.bin` - Compiled firmware files
|
||||
- `scripts/latest_version.json` or `scripts/latest_version_dev.json`
|
||||
|
||||
**Output:**
|
||||
- WebUploader manifest JSON file in `docs/.vuepress/public/`
|
||||
|
||||
**Purpose:**
|
||||
- Creates manifest for web-based firmware updater
|
||||
- Lists available firmware files with metadata
|
||||
- Used by documentation site for OTA updates
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Scripts respect these environment variables:
|
||||
|
||||
- `PYTHONIOENCODING=utf-8`: Python encoding
|
||||
- `PYTHONUTF8=1`: UTF-8 mode
|
||||
- `PLATFORMIO_BUILD_FLAGS`: Custom build flags
|
||||
- `ESP32_PLATFORM_VERSION`: Extracted automatically
|
||||
|
||||
---
|
||||
|
||||
## Exit Codes
|
||||
|
||||
All scripts use standard exit codes:
|
||||
|
||||
- `0` - Success
|
||||
- `1` - General error or failure
|
||||
- `2` - Missing required tools or dependencies
|
||||
|
||||
Scripts use `set -euo pipefail` for strict error handling:
|
||||
- `-e` - Exit on error
|
||||
- `-u` - Exit on undefined variable
|
||||
- `-o pipefail` - Exit on pipe failure
|
||||
|
||||
---
|
||||
|
||||
## Environment Detection
|
||||
|
||||
Scripts automatically detect if running in CI/CD:
|
||||
|
||||
```bash
|
||||
if [[ "${CI:-false}" == "true" ]]; then
|
||||
# Running in CI/CD
|
||||
# Disable interactive prompts
|
||||
# Use different output formatting
|
||||
fi
|
||||
```
|
||||
|
||||
CI/CD environments typically set:
|
||||
- `CI=true`
|
||||
- `GITHUB_ACTIONS=true` (GitHub Actions)
|
||||
- `BUILD_NUMBER` (build number)
|
||||
- `GIT_COMMIT` (commit hash)
|
||||
|
||||
|
||||
---
|
||||
|
||||
This documentation reflects the current implementation of CI/CD scripts. All scripts are located in `./scripts/` directory.
|
||||
|
||||
For GitHub Actions workflow documentation, see `.github/workflows/README.md`.
|
||||
@@ -1,3 +1,5 @@
|
||||
# Adds compiler flags to suppress warnings during PlatformIO build
|
||||
# Used by: PlatformIO environments (esp32dev-pilight*, esp32-m5stick-c*)
|
||||
Import("env")
|
||||
|
||||
|
||||
|
||||
174
scripts/ci.sh
Executable file
174
scripts/ci.sh
Executable file
@@ -0,0 +1,174 @@
|
||||
#!/bin/bash
|
||||
# CI/CD Main Entry Point - Command Dispatcher
|
||||
# Routes commands to specialized scripts for build, site, qa, and all
|
||||
# Usage: ./scripts/ci.sh <command> [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to print banner
|
||||
print_banner() {
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ OpenMQTTGateway CI/CD Pipeline ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <command> [OPTIONS]
|
||||
|
||||
OpenMQTTGateway CI/CD Pipeline - Main Entry Point
|
||||
|
||||
Commands:
|
||||
build Build firmware for specified environment
|
||||
site Build and deploy documentation/website
|
||||
qa Run quality assurance checks (linting, formatting)
|
||||
all Run complete pipeline (qa + build + site)
|
||||
|
||||
Examples:
|
||||
# Build firmware
|
||||
$0 build esp32dev-all-test --mode dev
|
||||
$0 build esp32dev-bt --version v1.8.0 --deploy-ready
|
||||
|
||||
# Build and deploy documentation
|
||||
$0 site --mode prod --deploy
|
||||
$0 site --mode dev --preview
|
||||
|
||||
# Run quality checks
|
||||
$0 qa --check
|
||||
$0 qa --fix
|
||||
|
||||
# Run complete pipeline
|
||||
$0 all esp32dev-bt --version v1.8.0
|
||||
|
||||
Get help for specific commands:
|
||||
$0 build --help
|
||||
$0 site --help
|
||||
$0 qa --help
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Function to run build pipeline
|
||||
run_build_pipeline() {
|
||||
log_info "Executing build pipeline..."
|
||||
"${SCRIPT_DIR}/ci_build.sh" "$@"
|
||||
}
|
||||
|
||||
# Function to run site pipeline
|
||||
run_site_pipeline() {
|
||||
log_info "Executing site pipeline..."
|
||||
"${SCRIPT_DIR}/ci_site.sh" "$@"
|
||||
}
|
||||
|
||||
# Function to run QA pipeline
|
||||
run_qa_pipeline() {
|
||||
log_info "Executing QA pipeline..."
|
||||
"${SCRIPT_DIR}/ci_qa.sh" "$@"
|
||||
}
|
||||
|
||||
# Function to run complete pipeline
|
||||
run_all_pipeline() {
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
log_info "Starting complete CI/CD pipeline..."
|
||||
echo ""
|
||||
|
||||
# Step 1: Quality Assurance
|
||||
log_info "═══ Step 1/3: Quality Assurance ═══"
|
||||
run_qa_pipeline --check || {
|
||||
log_error "QA checks failed. Pipeline aborted."
|
||||
return 1
|
||||
}
|
||||
echo ""
|
||||
|
||||
# Step 2: Build Firmware
|
||||
log_info "═══ Step 2/3: Build Firmware ═══"
|
||||
run_build_pipeline "$@" || {
|
||||
log_error "Build failed. Pipeline aborted."
|
||||
return 1
|
||||
}
|
||||
echo ""
|
||||
|
||||
# Step 3: Build Site (only if not in --no-site mode)
|
||||
if [[ ! " $* " =~ " --no-site " ]]; then
|
||||
log_info "═══ Step 3/3: Build Documentation ═══"
|
||||
run_site_pipeline --mode prod || {
|
||||
log_warn "Site build failed, but continuing..."
|
||||
}
|
||||
else
|
||||
log_info "Skipping site build (--no-site flag)"
|
||||
fi
|
||||
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
local duration=$((end_time - start_time))
|
||||
|
||||
echo ""
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ Complete Pipeline Summary ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo " Total Duration: ${duration}s"
|
||||
echo " Status: SUCCESS ✓"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
# Check if no arguments provided
|
||||
if [[ $# -eq 0 ]]; then
|
||||
print_banner
|
||||
usage
|
||||
fi
|
||||
|
||||
# Get command
|
||||
local command="$1"
|
||||
shift || true
|
||||
|
||||
# Handle help flags
|
||||
if [[ "$command" == "--help" || "$command" == "-h" ]]; then
|
||||
print_banner
|
||||
usage
|
||||
fi
|
||||
|
||||
print_banner
|
||||
|
||||
# Route to appropriate pipeline
|
||||
case "$command" in
|
||||
build)
|
||||
run_build_pipeline "$@"
|
||||
;;
|
||||
site|docs)
|
||||
run_site_pipeline "$@"
|
||||
;;
|
||||
qa|lint)
|
||||
run_qa_pipeline "$@"
|
||||
;;
|
||||
all|pipeline)
|
||||
run_all_pipeline "$@"
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown command: $command"
|
||||
echo ""
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Execute main function
|
||||
main "$@"
|
||||
58
scripts/ci_00_config.sh
Executable file
58
scripts/ci_00_config.sh
Executable file
@@ -0,0 +1,58 @@
|
||||
# Build Scripts Configuration
|
||||
# Used by: All build scripts for centralized configuration
|
||||
|
||||
# Python Configuration
|
||||
PYTHON_VERSION="3.13"
|
||||
PLATFORMIO_VERSION="https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip"
|
||||
|
||||
# Centralized Output Directory Structure
|
||||
# All CI/CD generated files go under generated/
|
||||
GENERATED_BASE_DIR="generated"
|
||||
ARTIFACTS_DIR="${GENERATED_BASE_DIR}/artifacts"
|
||||
SITE_DIR="${GENERATED_BASE_DIR}/site"
|
||||
REPORTS_DIR="${GENERATED_BASE_DIR}/reports"
|
||||
|
||||
# PlatformIO Directory Configuration
|
||||
BUILD_DIR=".pio/build"
|
||||
SCRIPTS_DIR="scripts"
|
||||
|
||||
# Build Configuration
|
||||
DEFAULT_ENVIRONMENT="esp32dev-all-test"
|
||||
ENABLE_VERBOSE_BUILD="false"
|
||||
ENABLE_BUILD_CACHE="true"
|
||||
|
||||
# Artifact Configuration
|
||||
ARTIFACT_RETENTION_DAYS="7"
|
||||
CREATE_MANIFEST="true"
|
||||
COMPRESS_ARTIFACTS="false"
|
||||
|
||||
# Version Configuration
|
||||
VERSION_FILE_PROD="scripts/latest_version.json"
|
||||
VERSION_FILE_DEV="scripts/latest_version_dev.json"
|
||||
USER_CONFIG_FILE="main/User_config.h"
|
||||
|
||||
# Logging Configuration
|
||||
ENABLE_COLOR_OUTPUT="true"
|
||||
LOG_LEVEL="INFO" # DEBUG, INFO, WARN, ERROR
|
||||
|
||||
# ============================================================================
|
||||
# Colors - ANSI color codes for terminal output
|
||||
# ============================================================================
|
||||
readonly RED='\033[0;31m'
|
||||
readonly GREEN='\033[0;32m'
|
||||
readonly YELLOW='\033[1;33m'
|
||||
readonly BLUE='\033[0;34m'
|
||||
readonly NC='\033[0m' # No Color
|
||||
|
||||
# ============================================================================
|
||||
# Logging Functions - Standardized logging across all build scripts
|
||||
# ============================================================================
|
||||
log_info() { echo -e "${GREEN}[INFO]${NC} $*" >&2; }
|
||||
log_warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; }
|
||||
log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
|
||||
log_step() { echo -e "${BLUE}[STEP]${NC} $*" >&2; }
|
||||
|
||||
# Advanced Options
|
||||
ENABLE_CCACHE="false"
|
||||
CCACHE_DIR=".ccache"
|
||||
MAX_BUILD_JOBS="4"
|
||||
384
scripts/ci_build.sh
Executable file
384
scripts/ci_build.sh
Executable file
@@ -0,0 +1,384 @@
|
||||
#!/bin/bash
|
||||
# CI/CD agnostic wrapper for complete build pipeline
|
||||
# Orchestrates all build scripts with a single command
|
||||
# Usage: ./scripts/ci.sh <environment> [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to print banner
|
||||
print_banner() {
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ OpenMQTTGateway CI/CD Build ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# Function to print summary
|
||||
print_summary() {
|
||||
local env="$1"
|
||||
local version="$2"
|
||||
local start_time="$3"
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
local duration=$((end_time - start_time))
|
||||
|
||||
echo ""
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ Build Summary ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo " Environment: $env"
|
||||
echo " Version: $version"
|
||||
echo " Duration: ${duration}s"
|
||||
echo " Status: SUCCESS ✓"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
}
|
||||
|
||||
# Function to check if command exists
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# Function to get command version
|
||||
get_command_version() {
|
||||
local cmd="$1"
|
||||
case "$cmd" in
|
||||
platformio)
|
||||
platformio --version 2>&1 | head -n1 | grep -oP '\d+\.\d+\.\d+' || echo "unknown"
|
||||
;;
|
||||
python|python3)
|
||||
python3 --version 2>&1 | grep -oP '\d+\.\d+' || echo "unknown"
|
||||
;;
|
||||
*)
|
||||
echo "unknown"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Function to verify required tools
|
||||
verify_build_tools() {
|
||||
log_info "Verifying required build tools..."
|
||||
|
||||
local missing_tools=()
|
||||
local version_mismatch=()
|
||||
|
||||
# Check Python
|
||||
if ! command_exists python3; then
|
||||
missing_tools+=("python3")
|
||||
else
|
||||
local python_version
|
||||
python_version=$(get_command_version python3)
|
||||
log_info "✓ Python ${python_version} found"
|
||||
fi
|
||||
|
||||
# Check PlatformIO
|
||||
if ! command_exists platformio; then
|
||||
missing_tools+=("platformio")
|
||||
else
|
||||
local pio_version
|
||||
pio_version=$(get_command_version platformio)
|
||||
log_info "✓ PlatformIO ${pio_version} found"
|
||||
fi
|
||||
|
||||
# Check git (for version auto-generation)
|
||||
if ! command_exists git; then
|
||||
log_warn "git not found (optional, but recommended)"
|
||||
else
|
||||
log_info "✓ git found"
|
||||
fi
|
||||
|
||||
# Report missing tools
|
||||
if [[ ${#missing_tools[@]} -gt 0 ]]; then
|
||||
log_error "Missing required tools: ${missing_tools[*]}"
|
||||
log_error ""
|
||||
log_error "Please install missing tools:"
|
||||
for tool in "${missing_tools[@]}"; do
|
||||
case "$tool" in
|
||||
python3)
|
||||
log_error " - Python 3: https://www.python.org/downloads/"
|
||||
;;
|
||||
platformio)
|
||||
log_error " - PlatformIO: pip3 install platformio"
|
||||
log_error " or: pip3 install ${PLATFORMIO_VERSION:-platformio}"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
log_error ""
|
||||
log_error "Or skip this check with: --skip-verification"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "All required tools are available"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to cleanup on error
|
||||
cleanup_on_error() {
|
||||
log_error "Build failed, cleaning up..."
|
||||
# Restore any backups
|
||||
find . -name "*.bak" -type f -exec bash -c 'mv "$1" "${1%.bak}"' _ {} \; 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Function to list available environments
|
||||
list_environments() {
|
||||
log_info "Available PlatformIO environments:"
|
||||
echo ""
|
||||
|
||||
local env_files=("${PROJECT_ROOT}/platformio.ini" "${PROJECT_ROOT}/environments.ini")
|
||||
local envs=()
|
||||
|
||||
for file in "${env_files[@]}"; do
|
||||
if [[ -f "$file" ]]; then
|
||||
while IFS= read -r line; do
|
||||
if [[ "$line" =~ ^\[env:([^\]]+)\] ]]; then
|
||||
local env_name="${BASH_REMATCH[1]}"
|
||||
# Skip test environments
|
||||
if [[ ! "$env_name" =~ -test$ && ! "$env_name" =~ -all- ]]; then
|
||||
envs+=("$env_name")
|
||||
fi
|
||||
fi
|
||||
done < "$file"
|
||||
fi
|
||||
done
|
||||
|
||||
# Sort and display unique environments
|
||||
if [[ ${#envs[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
|
||||
else
|
||||
log_warn "No environments found in configuration files"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <environment> [OPTIONS]
|
||||
|
||||
Complete CI/CD build pipeline wrapper.
|
||||
|
||||
Arguments:
|
||||
environment PlatformIO environment name
|
||||
|
||||
Options:
|
||||
--version [TAG] Set version tag (if TAG omitted, auto-generated)
|
||||
--mode MODE Build mode: 'prod' or 'dev' [default: prod]
|
||||
'dev' enables OTA and development version
|
||||
--deploy-ready Prepare for deployment (renamed artifacts)
|
||||
--output DIR Output directory for artifacts [default: generated/artifacts/]
|
||||
--skip-verification Skip build tools verification
|
||||
--clean Clean build before starting
|
||||
--verbose Enable verbose output
|
||||
--list-envs List all available PlatformIO environments
|
||||
--help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
CI Set to 'true' in CI/CD environments
|
||||
BUILD_NUMBER Build number from CI/CD
|
||||
GIT_COMMIT Git commit hash for versioning
|
||||
|
||||
Examples:
|
||||
# List available environments
|
||||
$0 --list-envs
|
||||
|
||||
# Local development build
|
||||
$0 esp32dev-all-test --mode dev
|
||||
|
||||
# Production release build
|
||||
$0 esp32dev-bt --version v1.7.0 --mode prod --deploy-ready
|
||||
|
||||
# CI/CD build (auto-detects version)
|
||||
$0 theengs-bridge --version --mode dev
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Main pipeline
|
||||
main() {
|
||||
local environment=""
|
||||
local version=""
|
||||
local set_version=false
|
||||
local mode=""
|
||||
local deploy=false
|
||||
local output_dir=""
|
||||
local skip_verification=false
|
||||
local clean=false
|
||||
local verbose=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--version)
|
||||
set_version=true
|
||||
# Check if next argument is a version tag or another option
|
||||
if [[ $# -gt 1 && ! "$2" =~ ^-- ]]; then
|
||||
version="$2"
|
||||
shift 2
|
||||
else
|
||||
shift
|
||||
fi
|
||||
;;
|
||||
--mode)
|
||||
if [[ $# -lt 2 ]]; then
|
||||
log_error "--mode requires an argument: 'prod' or 'dev'"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$2" != "prod" && "$2" != "dev" ]]; then
|
||||
log_error "Invalid mode: $2. Must be 'prod' or 'dev'"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
mode="$2"
|
||||
shift 2
|
||||
;;
|
||||
--deploy-ready)
|
||||
deploy=true
|
||||
shift
|
||||
;;
|
||||
--output)
|
||||
if [[ $# -lt 2 ]]; then
|
||||
log_error "--output requires a directory argument"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
output_dir="$2"
|
||||
shift 2
|
||||
;;
|
||||
--skip-verification)
|
||||
skip_verification=true
|
||||
shift
|
||||
;;
|
||||
--clean)
|
||||
clean=true
|
||||
shift
|
||||
;;
|
||||
--verbose)
|
||||
verbose=true
|
||||
shift
|
||||
;;
|
||||
--list-envs)
|
||||
list_environments
|
||||
exit 0
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
environment="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Set default mode if not specified
|
||||
if [[ -z "$mode" ]]; then
|
||||
mode="prod"
|
||||
log_info "Mode not specified, defaulting to production"
|
||||
fi
|
||||
|
||||
# Validate environment
|
||||
if [[ -z "$environment" ]]; then
|
||||
log_error "Environment name is required"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Auto-generate version if --version flag is set but no tag provided
|
||||
if [[ "$set_version" == "true" && -z "$version" ]]; then
|
||||
if [[ "${CI:-false}" == "true" ]]; then
|
||||
# CI/CD environment
|
||||
version="${BUILD_NUMBER:-${GIT_COMMIT:-unknown}}"
|
||||
else
|
||||
# Local development
|
||||
version="local-$(date +%Y%m%d-%H%M%S)"
|
||||
fi
|
||||
log_info "Auto-generated version: $version"
|
||||
fi
|
||||
|
||||
# Setup error handling
|
||||
trap cleanup_on_error ERR
|
||||
|
||||
# Change to project root
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
# Start timer
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
# Print banner
|
||||
print_banner
|
||||
|
||||
# Step 1: Verify build tools
|
||||
if [[ "$skip_verification" == "false" ]]; then
|
||||
log_step "1/4 Verifying build tools..."
|
||||
verify_build_tools || exit 1
|
||||
echo ""
|
||||
else
|
||||
log_warn "Skipping build tools verification (--skip-verification)"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Step 2: Set version (only if --version flag was provided)
|
||||
if [[ "$set_version" == "true" ]]; then
|
||||
log_step "2/4 Setting version: $version"
|
||||
if [[ "$mode" == "dev" ]]; then
|
||||
"${SCRIPT_DIR}/ci_set_version.sh" "$version" --dev || exit 1
|
||||
else
|
||||
"${SCRIPT_DIR}/ci_set_version.sh" "$version" --prod || exit 1
|
||||
fi
|
||||
echo ""
|
||||
else
|
||||
log_info "Skipping version setting (use --version to set version)"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Step 3: Build firmware
|
||||
log_step "3/4 Building firmware for: $environment"
|
||||
local build_opts=()
|
||||
[[ "$mode" == "dev" ]] && build_opts+=(--dev-ota)
|
||||
[[ "$clean" == "true" ]] && build_opts+=(--clean)
|
||||
[[ "$verbose" == "true" ]] && build_opts+=(--verbose)
|
||||
|
||||
"${SCRIPT_DIR}/ci_build_firmware.sh" "$environment" "${build_opts[@]}" || exit 1
|
||||
echo ""
|
||||
|
||||
# Step 4: Prepare artifacts
|
||||
log_step "4/4 Preparing artifacts..."
|
||||
local artifact_opts=()
|
||||
[[ "$deploy" == "true" ]] && artifact_opts+=(--deploy) || artifact_opts+=(--standard)
|
||||
artifact_opts+=(--manifest)
|
||||
[[ -n "$output_dir" ]] && artifact_opts+=(--output "$output_dir")
|
||||
|
||||
"${SCRIPT_DIR}/ci_prepare_artifacts.sh" "$environment" "${artifact_opts[@]}" || exit 1
|
||||
echo ""
|
||||
|
||||
# Print summary
|
||||
print_summary "$environment" "$version" "$start_time"
|
||||
|
||||
log_info "✓ Complete build pipeline finished successfully"
|
||||
}
|
||||
|
||||
# Run main if executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
281
scripts/ci_build_firmware.sh
Executable file
281
scripts/ci_build_firmware.sh
Executable file
@@ -0,0 +1,281 @@
|
||||
#!/bin/bash
|
||||
# Builds firmware for specified PlatformIO environment
|
||||
# Used by: CI/CD pipelines and local development
|
||||
# Usage: ./build_firmware.sh <environment> [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Set absolute path for BUILD_DIR
|
||||
BUILD_DIR="${PROJECT_ROOT}/${BUILD_DIR}"
|
||||
|
||||
# Script-specific logging function
|
||||
log_build() { echo -e "${BLUE}[BUILD]${NC} $*"; }
|
||||
|
||||
# Function to validate environment name
|
||||
validate_environment() {
|
||||
local env="$1"
|
||||
|
||||
if [[ -z "$env" ]]; then
|
||||
log_error "Environment name is required"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check if environment exists in platformio.ini or environments.ini
|
||||
if ! grep -q "^\[env:${env}\]" "${PROJECT_ROOT}/platformio.ini" "${PROJECT_ROOT}/environments.ini" 2>/dev/null; then
|
||||
log_warn "Environment '${env}' not found in configuration files"
|
||||
log_warn "Proceeding anyway (PlatformIO will validate)"
|
||||
fi
|
||||
|
||||
log_info "Building environment: $env"
|
||||
}
|
||||
|
||||
# Function to setup build environment variables
|
||||
setup_build_env() {
|
||||
local enable_dev_ota="${1:-false}"
|
||||
|
||||
export PYTHONIOENCODING=utf-8
|
||||
export PYTHONUTF8=1
|
||||
|
||||
if [[ "$enable_dev_ota" == "true" ]]; then
|
||||
export PLATFORMIO_BUILD_FLAGS='"-DDEVELOPMENTOTA=true"'
|
||||
log_info "Development OTA enabled"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to check PlatformIO availability
|
||||
check_platformio() {
|
||||
if ! command -v platformio >/dev/null 2>&1; then
|
||||
log_error "PlatformIO not found. Run setup_build_env.sh first"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to clean build artifacts
|
||||
clean_build() {
|
||||
local env="$1"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
if [[ -d "$env_dir" ]]; then
|
||||
log_info "Cleaning previous build artifacts for: $env"
|
||||
rm -rf "$env_dir"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to run PlatformIO build
|
||||
run_build() {
|
||||
local env="$1"
|
||||
local clean="${2:-false}"
|
||||
local verbose="${3:-false}"
|
||||
|
||||
log_build "Starting build for environment: $env"
|
||||
|
||||
local build_cmd="platformio run -e $env"
|
||||
|
||||
if [[ "$clean" == "true" ]]; then
|
||||
build_cmd="platformio run -e $env --target clean && $build_cmd"
|
||||
fi
|
||||
|
||||
if [[ "$verbose" == "true" ]]; then
|
||||
build_cmd="$build_cmd --verbose"
|
||||
fi
|
||||
|
||||
# Execute build with timing
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
if eval "$build_cmd"; then
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
local duration=$((end_time - start_time))
|
||||
|
||||
log_build "Build completed successfully in ${duration}s"
|
||||
return 0
|
||||
else
|
||||
log_error "Build failed for environment: $env"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to verify build artifacts
|
||||
verify_artifacts() {
|
||||
local env="$1"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
log_info "Verifying build artifacts..."
|
||||
|
||||
local artifacts_found=0
|
||||
local firmware="${env_dir}/firmware.bin"
|
||||
local partitions="${env_dir}/partitions.bin"
|
||||
local bootloader="${env_dir}/bootloader.bin"
|
||||
|
||||
if [[ -f "$firmware" ]]; then
|
||||
local size
|
||||
size=$(stat -f%z "$firmware" 2>/dev/null || stat -c%s "$firmware" 2>/dev/null)
|
||||
log_info "✓ firmware.bin (${size} bytes)"
|
||||
((artifacts_found++))
|
||||
else
|
||||
log_warn "✗ firmware.bin not found"
|
||||
fi
|
||||
|
||||
if [[ -f "$partitions" ]]; then
|
||||
log_info "✓ partitions.bin"
|
||||
((artifacts_found++))
|
||||
fi
|
||||
|
||||
if [[ -f "$bootloader" ]]; then
|
||||
log_info "✓ bootloader.bin"
|
||||
((artifacts_found++))
|
||||
fi
|
||||
|
||||
if [[ $artifacts_found -eq 0 ]]; then
|
||||
log_error "No build artifacts found"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Found ${artifacts_found} artifact(s)"
|
||||
}
|
||||
|
||||
# Function to show build summary
|
||||
show_build_summary() {
|
||||
local env="$1"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
echo ""
|
||||
echo "═══════════════════════════════════════"
|
||||
echo " Build Summary: $env"
|
||||
echo "═══════════════════════════════════════"
|
||||
|
||||
if [[ -d "$env_dir" ]]; then
|
||||
find "$env_dir" -name "*.bin" -o -name "*.elf" | while read -r file; do
|
||||
local size
|
||||
size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
echo " $(basename "$file"): ${size_kb} KB"
|
||||
done
|
||||
fi
|
||||
|
||||
echo "═══════════════════════════════════════"
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <environment> [OPTIONS]
|
||||
|
||||
Build firmware for a specific PlatformIO environment.
|
||||
|
||||
Arguments:
|
||||
environment PlatformIO environment name (e.g., esp32dev-all-test)
|
||||
|
||||
Options:
|
||||
--dev-ota Enable development OTA build flags
|
||||
--clean Clean build artifacts before building
|
||||
--verbose Enable verbose build output
|
||||
--no-verify Skip artifact verification
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
$0 esp32dev-all-test
|
||||
$0 esp32dev-bt --dev-ota
|
||||
$0 theengs-bridge --clean --verbose
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local environment=""
|
||||
local enable_dev_ota=false
|
||||
local clean_build_flag=false
|
||||
local verbose=false
|
||||
local verify=true
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dev-ota)
|
||||
enable_dev_ota=true
|
||||
shift
|
||||
;;
|
||||
--clean)
|
||||
clean_build_flag=true
|
||||
shift
|
||||
;;
|
||||
--verbose)
|
||||
verbose=true
|
||||
shift
|
||||
;;
|
||||
--no-verify)
|
||||
verify=false
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
environment="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate inputs
|
||||
if [[ -z "$environment" ]]; then
|
||||
log_error "Environment name is required"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Change to project root
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
# Check prerequisites
|
||||
check_platformio || exit 1
|
||||
|
||||
# Validate environment
|
||||
validate_environment "$environment" || exit 1
|
||||
|
||||
# Setup build environment
|
||||
setup_build_env "$enable_dev_ota"
|
||||
|
||||
# Clean if requested
|
||||
if [[ "$clean_build_flag" == "true" ]]; then
|
||||
clean_build "$environment"
|
||||
fi
|
||||
|
||||
# Run build
|
||||
run_build "$environment" "$clean_build_flag" "$verbose" || exit 1
|
||||
|
||||
# Verify artifacts
|
||||
if [[ "$verify" == "true" ]]; then
|
||||
verify_artifacts "$environment" || exit 1
|
||||
fi
|
||||
|
||||
# Show summary
|
||||
show_build_summary "$environment"
|
||||
|
||||
log_info "Build process completed successfully"
|
||||
}
|
||||
|
||||
# Run main if executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
318
scripts/ci_prepare_artifacts.sh
Executable file
318
scripts/ci_prepare_artifacts.sh
Executable file
@@ -0,0 +1,318 @@
|
||||
#!/bin/bash
|
||||
# Prepares firmware artifacts for upload or deployment
|
||||
# Used by: CI/CD pipelines for artifact packaging
|
||||
# Usage: ./prepare_artifacts.sh <environment> [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Set absolute paths
|
||||
BUILD_DIR="${PROJECT_ROOT}/${BUILD_DIR}"
|
||||
DEFAULT_OUTPUT_DIR="${PROJECT_ROOT}/${ARTIFACTS_DIR}"
|
||||
|
||||
# Function to create output directory
|
||||
create_output_dir() {
|
||||
local output_dir="$1"
|
||||
|
||||
if [[ -d "$output_dir" ]]; then
|
||||
log_warn "Output directory already exists: $output_dir"
|
||||
log_info "Cleaning existing artifacts..."
|
||||
rm -rf "$output_dir"
|
||||
fi
|
||||
|
||||
mkdir -p "$output_dir"
|
||||
log_info "Created output directory: $output_dir"
|
||||
}
|
||||
|
||||
# Function to copy artifact with optional renaming
|
||||
copy_artifact() {
|
||||
local source="$1"
|
||||
local dest="$2"
|
||||
local artifact_type="$3"
|
||||
|
||||
if [[ ! -f "$source" ]]; then
|
||||
log_warn "${artifact_type} not found: $source"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if cp "$source" "$dest"; then
|
||||
local size
|
||||
size=$(stat -f%z "$dest" 2>/dev/null || stat -c%s "$dest" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
log_info "✓ Copied ${artifact_type}: $(basename "$dest") (${size_kb} KB)"
|
||||
return 0
|
||||
else
|
||||
log_error "Failed to copy ${artifact_type}: $source"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to prepare standard artifacts (no renaming)
|
||||
prepare_standard_artifacts() {
|
||||
local env="$1"
|
||||
local output_dir="$2"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
log_info "Preparing STANDARD artifacts for: $env"
|
||||
|
||||
local copied=0
|
||||
|
||||
# Copy firmware.bin (required)
|
||||
if copy_artifact "${env_dir}/firmware.bin" "${output_dir}/firmware.bin" "firmware"; then
|
||||
((copied++))
|
||||
fi
|
||||
|
||||
# Copy partitions.bin (optional)
|
||||
copy_artifact "${env_dir}/partitions.bin" "${output_dir}/partitions.bin" "partitions" && ((copied++)) || true
|
||||
|
||||
# Note: bootloader.bin is NOT copied in standard mode (only needed for deployment)
|
||||
|
||||
if [[ $copied -eq 0 ]]; then
|
||||
log_error "No artifacts were copied"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Copied ${copied} artifact(s) in standard mode"
|
||||
}
|
||||
|
||||
# Function to prepare deployment artifacts (with renaming)
|
||||
prepare_deployment_artifacts() {
|
||||
local env="$1"
|
||||
local output_dir="$2"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
log_info "Preparing DEPLOYMENT artifacts for: $env"
|
||||
|
||||
local copied=0
|
||||
|
||||
# Copy and rename firmware.bin
|
||||
if copy_artifact "${env_dir}/firmware.bin" "${output_dir}/${env}-firmware.bin" "firmware"; then
|
||||
((copied++))
|
||||
fi
|
||||
|
||||
# Copy and rename partitions.bin (optional)
|
||||
copy_artifact "${env_dir}/partitions.bin" "${output_dir}/${env}-partitions.bin" "partitions" && ((copied++)) || true
|
||||
|
||||
# Copy and rename bootloader.bin (optional)
|
||||
copy_artifact "${env_dir}/bootloader.bin" "${output_dir}/${env}-bootloader.bin" "bootloader" && ((copied++)) || true
|
||||
|
||||
# Copy boot_app0.bin if exists (ESP32 specific)
|
||||
copy_artifact "${env_dir}/boot_app0.bin" "${output_dir}/${env}-boot_app0.bin" "boot_app0" && ((copied++)) || true
|
||||
|
||||
if [[ $copied -eq 0 ]]; then
|
||||
log_error "No artifacts were copied"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Copied ${copied} artifact(s) in deployment mode"
|
||||
}
|
||||
|
||||
# Function to create manifest file
|
||||
create_manifest() {
|
||||
local env="$1"
|
||||
local output_dir="$2"
|
||||
local manifest="${output_dir}/manifest.txt"
|
||||
|
||||
log_info "Creating artifact manifest..."
|
||||
|
||||
{
|
||||
echo "Environment: $env"
|
||||
echo "Build Date: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
|
||||
echo "Build Host: $(hostname)"
|
||||
echo ""
|
||||
echo "Artifacts:"
|
||||
|
||||
find "$output_dir" -type f -name "*.bin" | sort | while read -r file; do
|
||||
local size
|
||||
size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
local md5sum_val
|
||||
md5sum_val=$(md5sum "$file" 2>/dev/null | cut -d' ' -f1 || md5 -q "$file" 2>/dev/null)
|
||||
echo " - $(basename "$file"): ${size_kb} KB (MD5: ${md5sum_val})"
|
||||
done
|
||||
} > "$manifest"
|
||||
|
||||
log_info "Manifest created: $manifest"
|
||||
}
|
||||
|
||||
# Function to compress artifacts (optional)
|
||||
compress_artifacts() {
|
||||
local output_dir="$1"
|
||||
local archive_name="$2"
|
||||
|
||||
log_info "Compressing artifacts..."
|
||||
|
||||
local archive="${output_dir}/${archive_name}.tar.gz"
|
||||
|
||||
if tar -czf "$archive" -C "$output_dir" .; then
|
||||
local size
|
||||
size=$(stat -f%z "$archive" 2>/dev/null || stat -c%s "$archive" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
log_info "Archive created: ${archive_name}.tar.gz (${size_kb} KB)"
|
||||
else
|
||||
log_error "Failed to create archive"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to list artifacts
|
||||
list_artifacts() {
|
||||
local output_dir="$1"
|
||||
|
||||
echo ""
|
||||
echo "═══════════════════════════════════════"
|
||||
echo " Prepared Artifacts"
|
||||
echo "═══════════════════════════════════════"
|
||||
|
||||
if [[ -d "$output_dir" ]]; then
|
||||
find "$output_dir" -type f | sort | while read -r file; do
|
||||
local size
|
||||
size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
echo " $(basename "$file"): ${size_kb} KB"
|
||||
done
|
||||
else
|
||||
echo " No artifacts found"
|
||||
fi
|
||||
|
||||
echo "═══════════════════════════════════════"
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <environment> [OPTIONS]
|
||||
|
||||
Prepare firmware artifacts for upload or deployment.
|
||||
|
||||
Arguments:
|
||||
environment PlatformIO environment name
|
||||
|
||||
Options:
|
||||
--deploy Prepare for deployment (rename with environment prefix)
|
||||
--standard Prepare standard artifacts (no renaming) [default]
|
||||
--output DIR Output directory [default: generated/artifacts/]
|
||||
--manifest Create manifest file with artifact metadata
|
||||
--compress Compress artifacts into tar.gz archive
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
$0 esp32dev-all-test
|
||||
$0 esp32dev-bt --deploy --manifest
|
||||
$0 theengs-bridge --output build/artifacts --compress
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local environment=""
|
||||
local mode="standard"
|
||||
local output_dir="$DEFAULT_OUTPUT_DIR"
|
||||
local create_manifest_flag=false
|
||||
local compress_flag=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--deploy)
|
||||
mode="deploy"
|
||||
shift
|
||||
;;
|
||||
--standard)
|
||||
mode="standard"
|
||||
shift
|
||||
;;
|
||||
--output)
|
||||
output_dir="$2"
|
||||
shift 2
|
||||
;;
|
||||
--manifest)
|
||||
create_manifest_flag=true
|
||||
shift
|
||||
;;
|
||||
--compress)
|
||||
compress_flag=true
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
environment="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate inputs
|
||||
if [[ -z "$environment" ]]; then
|
||||
log_error "Environment name is required"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Change to project root
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
# Check if build directory exists
|
||||
if [[ ! -d "${BUILD_DIR}/${environment}" ]]; then
|
||||
log_error "Build directory not found for environment: $environment"
|
||||
log_error "Run build_firmware.sh first"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create output directory
|
||||
create_output_dir "$output_dir"
|
||||
|
||||
# Prepare artifacts based on mode
|
||||
case "$mode" in
|
||||
standard)
|
||||
prepare_standard_artifacts "$environment" "$output_dir" || exit 1
|
||||
;;
|
||||
deploy)
|
||||
prepare_deployment_artifacts "$environment" "$output_dir" || exit 1
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown mode: $mode"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Create manifest if requested
|
||||
if [[ "$create_manifest_flag" == "true" ]]; then
|
||||
create_manifest "$environment" "$output_dir"
|
||||
fi
|
||||
|
||||
# Compress if requested
|
||||
if [[ "$compress_flag" == "true" ]]; then
|
||||
compress_artifacts "$output_dir" "$environment"
|
||||
fi
|
||||
|
||||
# Show summary
|
||||
list_artifacts "$output_dir"
|
||||
|
||||
log_info "Artifact preparation completed successfully"
|
||||
}
|
||||
|
||||
# Run main if executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
409
scripts/ci_qa.sh
Executable file
409
scripts/ci_qa.sh
Executable file
@@ -0,0 +1,409 @@
|
||||
#!/bin/bash
|
||||
# CI/CD Quality Assurance (QA) - Code Linting and Formatting
|
||||
# Checks and fixes code formatting using clang-format
|
||||
# Usage: ./scripts/ci_qa.sh [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
|
||||
# Load shared configuration
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Default values
|
||||
CHECK_MODE=true
|
||||
FIX_MODE=false
|
||||
FORMAT_ONLY=false
|
||||
SOURCE_DIR="main"
|
||||
EXTENSIONS="h,ino,cpp"
|
||||
CLANG_FORMAT_VERSION="9"
|
||||
VERBOSE=false
|
||||
|
||||
# Function to check if clang-format is available
|
||||
check_clang_format() {
|
||||
local version="$1"
|
||||
local cmd="clang-format-${version}"
|
||||
|
||||
if command -v "$cmd" >/dev/null 2>&1; then
|
||||
echo "$cmd"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Try without version suffix
|
||||
if command -v clang-format >/dev/null 2>&1; then
|
||||
echo "clang-format"
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
# Function to find files to check
|
||||
find_files() {
|
||||
local source="$1"
|
||||
local extensions="$2"
|
||||
|
||||
if [[ ! -d "${PROJECT_ROOT}/${source}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
local find_patterns=()
|
||||
IFS=',' read -ra exts <<< "$extensions"
|
||||
for ext in "${exts[@]}"; do
|
||||
find_patterns+=(-name "*.${ext}" -o)
|
||||
done
|
||||
# Remove last -o
|
||||
unset 'find_patterns[-1]'
|
||||
|
||||
local files
|
||||
files=$(find "${PROJECT_ROOT}/${source}" -type f \( "${find_patterns[@]}" \) 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$files" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "$files"
|
||||
}
|
||||
|
||||
# Function to check formatting
|
||||
check_formatting() {
|
||||
local clang_format_cmd="$1"
|
||||
local files="$2"
|
||||
|
||||
log_info "Checking code formatting..."
|
||||
|
||||
local failed_files=()
|
||||
local checked_count=0
|
||||
local has_issues=false
|
||||
|
||||
while IFS= read -r file; do
|
||||
if [[ -z "$file" ]]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
checked_count=$((checked_count + 1))
|
||||
|
||||
if [[ "$VERBOSE" == true ]]; then
|
||||
log_info "Checking: $file"
|
||||
fi
|
||||
|
||||
# Check if file needs formatting and capture diff
|
||||
local diff_output
|
||||
diff_output=$("$clang_format_cmd" --dry-run --Werror "$file" 2>&1)
|
||||
local format_result=$?
|
||||
|
||||
if [[ $format_result -ne 0 ]]; then
|
||||
failed_files+=("$file")
|
||||
has_issues=true
|
||||
|
||||
# Show the actual formatting differences
|
||||
echo ""
|
||||
log_warn "⚠ Formatting issues in: $file"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Generate and show diff with colors
|
||||
local actual_diff
|
||||
actual_diff=$(diff -u "$file" <("$clang_format_cmd" "$file") 2>/dev/null || true)
|
||||
|
||||
if [[ -n "$actual_diff" ]]; then
|
||||
echo "$actual_diff" | head -50 | while IFS= read -r line; do
|
||||
if [[ "$line" =~ ^-[^-] ]]; then
|
||||
echo -e "\033[31m$line\033[0m" # Red for removed lines
|
||||
elif [[ "$line" =~ ^+[^+] ]]; then
|
||||
echo -e "\033[32m$line\033[0m" # Green for added lines
|
||||
elif [[ "$line" =~ ^@@ ]]; then
|
||||
echo -e "\033[36m$line\033[0m" # Cyan for line numbers
|
||||
else
|
||||
echo "$line"
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo "$diff_output"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
if [[ "$VERBOSE" == true ]]; then
|
||||
log_info " ✓ OK"
|
||||
fi
|
||||
fi
|
||||
done <<< "$files"
|
||||
|
||||
echo ""
|
||||
log_info "Checked ${checked_count} files"
|
||||
|
||||
if [[ $has_issues == true ]]; then
|
||||
echo ""
|
||||
log_error "Found ${#failed_files[@]} files with formatting issues:"
|
||||
for file in "${failed_files[@]}"; do
|
||||
log_error " - $file"
|
||||
done
|
||||
echo ""
|
||||
log_error "To fix these issues automatically, run:"
|
||||
log_error " $0 --fix"
|
||||
echo ""
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "✓ All files are properly formatted"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to fix formatting
|
||||
fix_formatting() {
|
||||
local clang_format_cmd="$1"
|
||||
local files="$2"
|
||||
|
||||
log_info "Fixing code formatting..."
|
||||
|
||||
local fixed_count=0
|
||||
local total_count=0
|
||||
|
||||
while IFS= read -r file; do
|
||||
if [[ -z "$file" ]]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
total_count=$((total_count + 1))
|
||||
|
||||
if [[ "$VERBOSE" == true ]]; then
|
||||
log_info "Processing: $file"
|
||||
fi
|
||||
|
||||
# Apply formatting in-place
|
||||
if "$clang_format_cmd" -i "$file" 2>/dev/null; then
|
||||
fixed_count=$((fixed_count + 1))
|
||||
if [[ "$VERBOSE" == true ]]; then
|
||||
log_info " ✓ Formatted"
|
||||
fi
|
||||
else
|
||||
if [[ "$VERBOSE" == true ]]; then
|
||||
log_warn " ✗ Failed to format"
|
||||
fi
|
||||
fi
|
||||
done <<< "$files"
|
||||
|
||||
echo ""
|
||||
log_info "Processed ${total_count} files"
|
||||
log_info "✓ Formatting applied to ${fixed_count} files"
|
||||
|
||||
if [[ $fixed_count -gt 0 ]]; then
|
||||
log_warn ""
|
||||
log_warn "Files have been modified. Please review and commit the changes:"
|
||||
log_warn " git diff"
|
||||
log_warn " git add -u"
|
||||
log_warn " git commit -m 'style: apply clang-format'"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to run all QA checks
|
||||
run_all_checks() {
|
||||
log_info "Running all quality assurance checks..."
|
||||
|
||||
local all_passed=true
|
||||
|
||||
# Format check
|
||||
log_info "═══ Code Formatting ═══"
|
||||
if ! run_format_check; then
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Future: Add more checks here
|
||||
# - cppcheck static analysis
|
||||
# - code complexity metrics
|
||||
# - TODO/FIXME detection
|
||||
# - license header validation
|
||||
|
||||
if [[ "$all_passed" == false ]]; then
|
||||
log_error "Some QA checks failed"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "✓ All QA checks passed"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to run format check
|
||||
run_format_check() {
|
||||
log_info "Checking for clang-format version ${CLANG_FORMAT_VERSION}..."
|
||||
|
||||
local clang_format_cmd
|
||||
clang_format_cmd=$(check_clang_format "$CLANG_FORMAT_VERSION")
|
||||
|
||||
if [[ $? -ne 0 ]] || [[ -z "$clang_format_cmd" ]]; then
|
||||
log_error "clang-format not found"
|
||||
log_error "Please install clang-format:"
|
||||
log_error " Ubuntu/Debian: sudo apt-get install clang-format-${CLANG_FORMAT_VERSION}"
|
||||
log_error " macOS: brew install clang-format"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "$clang_format_cmd" == "clang-format-${CLANG_FORMAT_VERSION}" ]]; then
|
||||
log_info "✓ clang-format-${CLANG_FORMAT_VERSION} found"
|
||||
else
|
||||
local installed_version
|
||||
installed_version=$(clang-format --version | grep -oP '\d+\.\d+' | head -1 || echo "unknown")
|
||||
log_warn "clang-format-${CLANG_FORMAT_VERSION} not found, using clang-format (version ${installed_version})"
|
||||
fi
|
||||
|
||||
log_info "Finding files in '${SOURCE_DIR}' with extensions: ${EXTENSIONS}"
|
||||
|
||||
local files
|
||||
files=$(find_files "$SOURCE_DIR" "$EXTENSIONS")
|
||||
|
||||
if [[ $? -ne 0 ]] || [[ -z "$files" ]]; then
|
||||
log_error "Source directory not found: ${PROJECT_ROOT}/${SOURCE_DIR}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local file_count
|
||||
file_count=$(echo "$files" | wc -l)
|
||||
log_info "Found ${file_count} files to check"
|
||||
|
||||
if [[ "$FIX_MODE" == true ]]; then
|
||||
fix_formatting "$clang_format_cmd" "$files"
|
||||
else
|
||||
check_formatting "$clang_format_cmd" "$files"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 [OPTIONS]
|
||||
|
||||
Run quality assurance checks on OpenMQTTGateway code.
|
||||
|
||||
Options:
|
||||
--check Check formatting only (CI mode) [default]
|
||||
--fix Fix formatting issues automatically
|
||||
--format Run only format checks
|
||||
--all Run all QA checks [default]
|
||||
--source DIR Source directory to check [default: main]
|
||||
--extensions EXTS File extensions (comma-separated) [default: h,ino,cpp]
|
||||
--clang-format-version V clang-format version [default: 9]
|
||||
--verbose Enable verbose output
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
# Check formatting (CI mode)
|
||||
$0 --check
|
||||
|
||||
# Fix formatting issues
|
||||
$0 --fix
|
||||
|
||||
# Check specific directory
|
||||
$0 --check --source lib/LEDManager
|
||||
|
||||
# Check with custom extensions
|
||||
$0 --check --extensions h,cpp
|
||||
|
||||
# Verbose output
|
||||
$0 --check --verbose
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Parse command line arguments
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--check)
|
||||
CHECK_MODE=true
|
||||
FIX_MODE=false
|
||||
shift
|
||||
;;
|
||||
--fix)
|
||||
FIX_MODE=true
|
||||
CHECK_MODE=false
|
||||
shift
|
||||
;;
|
||||
--format)
|
||||
FORMAT_ONLY=true
|
||||
shift
|
||||
;;
|
||||
--all)
|
||||
FORMAT_ONLY=false
|
||||
shift
|
||||
;;
|
||||
--source)
|
||||
SOURCE_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--extensions)
|
||||
EXTENSIONS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--clang-format-version)
|
||||
CLANG_FORMAT_VERSION="$2"
|
||||
shift 2
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE=true
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
log_info "Starting QA pipeline..."
|
||||
|
||||
if [[ "$FIX_MODE" == true ]]; then
|
||||
log_info "Mode: FIX (will modify files)"
|
||||
else
|
||||
log_info "Mode: CHECK (read-only)"
|
||||
fi
|
||||
|
||||
# Run checks
|
||||
local result=0
|
||||
if [[ "$FORMAT_ONLY" == true ]]; then
|
||||
run_format_check || result=$?
|
||||
else
|
||||
run_all_checks || result=$?
|
||||
fi
|
||||
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
local duration=$((end_time - start_time))
|
||||
|
||||
echo ""
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ QA Pipeline Summary ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo " Duration: ${duration}s"
|
||||
|
||||
if [[ $result -eq 0 ]]; then
|
||||
echo " Status: SUCCESS ✓"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
return 0
|
||||
else
|
||||
echo " Status: FAILED ✗"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Execute main function
|
||||
main "$@"
|
||||
197
scripts/ci_set_version.sh
Executable file
197
scripts/ci_set_version.sh
Executable file
@@ -0,0 +1,197 @@
|
||||
#!/bin/bash
|
||||
# Updates version tags in firmware configuration and JSON files
|
||||
# Used by: CI/CD pipelines for versioning builds
|
||||
# Usage: ./set_version.sh <version_tag> [--dev]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly USER_CONFIG="${PROJECT_ROOT}/main/User_config.h"
|
||||
readonly LATEST_VERSION_PROD="${SCRIPT_DIR}/latest_version.json"
|
||||
readonly LATEST_VERSION_DEV="${SCRIPT_DIR}/latest_version_dev.json"
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to validate version tag
|
||||
validate_version() {
|
||||
local version="$1"
|
||||
|
||||
if [[ -z "$version" ]] || [[ "$version" == "version_tag" ]]; then
|
||||
log_error "Invalid version tag: '$version'"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Version tag validated: $version"
|
||||
}
|
||||
|
||||
# Function to backup files
|
||||
backup_file() {
|
||||
local file="$1"
|
||||
|
||||
if [[ -f "$file" ]]; then
|
||||
cp "$file" "${file}.bak"
|
||||
log_info "Backed up: $file"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to replace version in file
|
||||
replace_version() {
|
||||
local file="$1"
|
||||
local version="$2"
|
||||
|
||||
if [[ ! -f "$file" ]]; then
|
||||
log_warn "File not found: $file (skipping)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Backup before modification
|
||||
backup_file "$file"
|
||||
|
||||
# Replace version_tag placeholder
|
||||
if sed -i "s/version_tag/${version}/g" "$file"; then
|
||||
log_info "Updated version in: $file"
|
||||
else
|
||||
log_error "Failed to update version in: $file"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to set production version
|
||||
set_production_version() {
|
||||
local version="$1"
|
||||
|
||||
log_info "Setting PRODUCTION version: $version"
|
||||
|
||||
replace_version "$USER_CONFIG" "$version" || return 1
|
||||
replace_version "$VERSION_JSON" "$version" || return 1
|
||||
}
|
||||
|
||||
# Function to set development version
|
||||
set_development_version() {
|
||||
local version="$1"
|
||||
|
||||
log_info "Setting DEVELOPMENT version: $version"
|
||||
|
||||
replace_version "$USER_CONFIG" "$version" || return 1
|
||||
replace_version "$VERSION_DEV_JSON" "$version" || return 1
|
||||
}
|
||||
|
||||
# Function to restore backups
|
||||
restore_backups() {
|
||||
log_warn "Restoring backups..."
|
||||
|
||||
for file in "$USER_CONFIG" "$VERSION_JSON" "$VERSION_DEV_JSON"; do
|
||||
if [[ -f "${file}.bak" ]]; then
|
||||
mv "${file}.bak" "$file"
|
||||
log_info "Restored: $file"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Function to clean backups
|
||||
clean_backups() {
|
||||
for file in "$USER_CONFIG" "$VERSION_JSON" "$VERSION_DEV_JSON"; do
|
||||
if [[ -f "${file}.bak" ]]; then
|
||||
rm "${file}.bak"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <version_tag> [OPTIONS]
|
||||
|
||||
Update version tags in firmware configuration files.
|
||||
|
||||
Arguments:
|
||||
version_tag Version string to inject (e.g., v1.2.3, abc123, dev-20230101)
|
||||
|
||||
Options:
|
||||
--dev Use development version files (latest_version_dev.json)
|
||||
--prod Use production version files (latest_version.json) [default]
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
$0 v1.2.3 # Production release
|
||||
$0 abc123 --dev # Development build
|
||||
$0 \${{ github.sha }} --dev # CI/CD with commit SHA
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local version=""
|
||||
local is_dev=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dev)
|
||||
is_dev=true
|
||||
shift
|
||||
;;
|
||||
--prod)
|
||||
is_dev=false
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
version="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate version
|
||||
if [[ -z "$version" ]]; then
|
||||
log_error "Version tag is required"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
validate_version "$version" || exit 1
|
||||
|
||||
# Change to project root
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
# Set version with error handling
|
||||
if [[ "$is_dev" == true ]]; then
|
||||
set_development_version "$version" || {
|
||||
restore_backups
|
||||
exit 1
|
||||
}
|
||||
else
|
||||
set_production_version "$version" || {
|
||||
restore_backups
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
|
||||
# Clean up backups on success
|
||||
clean_backups
|
||||
|
||||
log_info "Version update completed successfully"
|
||||
}
|
||||
|
||||
# Run main if executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
505
scripts/ci_site.sh
Executable file
505
scripts/ci_site.sh
Executable file
@@ -0,0 +1,505 @@
|
||||
#!/bin/bash
|
||||
# CI/CD Site/Documentation Builder
|
||||
# Builds and deploys VuePress documentation with version management
|
||||
# Usage: ./scripts/ci_site.sh [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly DOCS_DIR="${PROJECT_ROOT}/docs"
|
||||
readonly VUEPRESS_CONFIG="${DOCS_DIR}/.vuepress/config.js"
|
||||
readonly VUEPRESS_BUILD_DIR="${DOCS_DIR}/.vuepress/dist"
|
||||
|
||||
# Load shared configuration
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Final output directory for site
|
||||
readonly SITE_OUTPUT_DIR="${PROJECT_ROOT}/${SITE_DIR}"
|
||||
|
||||
# Default values
|
||||
MODE="prod"
|
||||
URL_PREFIX="/"
|
||||
CUSTOM_VERSION=""
|
||||
VERSION_SOURCE="release"
|
||||
PREVIEW=false
|
||||
GENERATE_WEBUPLOADER=true
|
||||
WEBUPLOADER_ARGS=""
|
||||
DEPLOY_DIR="."
|
||||
RUN_PAGESPEED=false
|
||||
PAGESPEED_URL="https://docs.openmqttgateway.com/"
|
||||
|
||||
# Function to check required tools
|
||||
check_requirements() {
|
||||
log_info "Checking required tools..."
|
||||
|
||||
local missing_tools=()
|
||||
|
||||
if ! command -v node >/dev/null 2>&1; then
|
||||
missing_tools+=("node")
|
||||
else
|
||||
local node_version
|
||||
node_version=$(node --version)
|
||||
log_info "✓ Node.js ${node_version} found"
|
||||
fi
|
||||
|
||||
if ! command -v npm >/dev/null 2>&1; then
|
||||
missing_tools+=("npm")
|
||||
else
|
||||
local npm_version
|
||||
npm_version=$(npm --version)
|
||||
log_info "✓ npm ${npm_version} found"
|
||||
fi
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
missing_tools+=("python3")
|
||||
else
|
||||
local python_version
|
||||
python_version=$(python3 --version | grep -oP '\d+\.\d+' || echo "unknown")
|
||||
log_info "✓ Python ${python_version} found"
|
||||
fi
|
||||
|
||||
if ! command -v pip3 >/dev/null 2>&1; then
|
||||
missing_tools+=("pip3")
|
||||
else
|
||||
local pip_version
|
||||
pip_version=$(pip3 --version | grep -oP '\d+\.\d+' || echo "unknown")
|
||||
log_info "✓ pip ${pip_version} found"
|
||||
fi
|
||||
|
||||
if [[ ${#missing_tools[@]} -gt 0 ]]; then
|
||||
log_error "Missing required tools: ${missing_tools[*]}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "All required tools are available"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to install dependencies
|
||||
install_dependencies() {
|
||||
log_info "Installing dependencies..."
|
||||
|
||||
# Upgrade pip first (if pip module is available)
|
||||
if python3 -m pip --version >/dev/null 2>&1; then
|
||||
log_info "Upgrading pip..."
|
||||
python3 -m pip install --upgrade pip --quiet || {
|
||||
log_warn "Failed to upgrade pip, continuing with existing version..."
|
||||
}
|
||||
fi
|
||||
|
||||
# Install Python dependencies (without --user if in virtualenv)
|
||||
log_info "Installing Python dependencies..."
|
||||
pip3 install requests pandas markdown pytablereader tabulate || {
|
||||
log_error "Failed to install Python dependencies"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Install Node dependencies
|
||||
log_info "Installing Node.js dependencies..."
|
||||
cd "${PROJECT_ROOT}"
|
||||
npm install --quiet || {
|
||||
log_error "Failed to install Node.js dependencies"
|
||||
return 1
|
||||
}
|
||||
|
||||
log_info "Dependencies installed successfully"
|
||||
}
|
||||
|
||||
# Function to download common config
|
||||
download_common_config() {
|
||||
log_info "Downloading common configuration..."
|
||||
|
||||
local config_url="https://www.theengs.io/commonConfig.js"
|
||||
local config_dest="${DOCS_DIR}/.vuepress/public/commonConfig.js"
|
||||
|
||||
mkdir -p "$(dirname "$config_dest")"
|
||||
|
||||
if curl -sSf -o "$config_dest" "$config_url"; then
|
||||
log_info "✓ Common config downloaded"
|
||||
else
|
||||
log_warn "Failed to download common config, continuing anyway..."
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to get version
|
||||
get_version() {
|
||||
local version=""
|
||||
|
||||
if [[ "$VERSION_SOURCE" == "custom" && -n "$CUSTOM_VERSION" ]]; then
|
||||
version="$CUSTOM_VERSION"
|
||||
log_info "Using custom version: $version"
|
||||
elif [[ "$VERSION_SOURCE" == "release" ]]; then
|
||||
# Try to get latest git tag (simulating GitHub release)
|
||||
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
|
||||
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
|
||||
log_info "Using release version from git: $version"
|
||||
else
|
||||
version="development"
|
||||
log_warn "Git not available, using version: $version"
|
||||
fi
|
||||
else
|
||||
# Auto-detect from git
|
||||
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
|
||||
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
|
||||
log_info "Using git version: $version"
|
||||
else
|
||||
version="development"
|
||||
log_warn "Git not available, using version: $version"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "$version"
|
||||
}
|
||||
|
||||
# Function to set version in config files
|
||||
set_version() {
|
||||
local version="$1"
|
||||
|
||||
log_info "Setting version: $version"
|
||||
|
||||
# Update VuePress config
|
||||
if [[ -f "$VUEPRESS_CONFIG" ]]; then
|
||||
sed -i "s|version_tag|${version}|g" "$VUEPRESS_CONFIG"
|
||||
fi
|
||||
|
||||
# Update version JSON file based on version source
|
||||
if [[ "$VERSION_SOURCE" == "custom" ]]; then
|
||||
# Custom version updates dev file
|
||||
local version_file="${SCRIPT_DIR}/latest_version_dev.json"
|
||||
if [[ -f "$version_file" ]]; then
|
||||
sed -i "s|version_tag|${version}|g" "$version_file"
|
||||
fi
|
||||
else
|
||||
# Release version updates production file
|
||||
local version_file="${SCRIPT_DIR}/latest_version.json"
|
||||
if [[ -f "$version_file" ]]; then
|
||||
sed -i "s|version_tag|${version}|g" "$version_file"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to set URL prefix (base path)
|
||||
set_url_prefix() {
|
||||
local url_prefix="$1"
|
||||
|
||||
if [[ "$url_prefix" != "/" ]]; then
|
||||
log_info "Setting URL prefix: $url_prefix"
|
||||
sed -i "s|base: '/'|base: '${url_prefix}'|g" "$VUEPRESS_CONFIG"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to generate board documentation
|
||||
generate_board_docs() {
|
||||
log_info "Generating board documentation..."
|
||||
|
||||
local generator="${SCRIPT_DIR}/generate_board_docs.py"
|
||||
|
||||
if [[ -f "$generator" ]]; then
|
||||
cd "${PROJECT_ROOT}"
|
||||
python3 "$generator" || {
|
||||
log_warn "Board documentation generation failed, continuing..."
|
||||
}
|
||||
else
|
||||
log_warn "Board documentation generator not found, skipping..."
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to generate WebUploader manifest
|
||||
generate_webuploader() {
|
||||
if [[ "$GENERATE_WEBUPLOADER" != true ]]; then
|
||||
log_info "Skipping WebUploader generation"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_info "Generating WebUploader manifest..."
|
||||
|
||||
local generator="${SCRIPT_DIR}/gen_wu.py"
|
||||
|
||||
if [[ -f "$generator" ]]; then
|
||||
cd "${PROJECT_ROOT}"
|
||||
python3 "$generator" $WEBUPLOADER_ARGS || {
|
||||
log_warn "WebUploader generation failed, continuing..."
|
||||
}
|
||||
else
|
||||
log_warn "WebUploader generator not found, skipping..."
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to build documentation
|
||||
build_docs() {
|
||||
log_info "Building documentation..."
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Set Node options for compatibility with newer Node.js versions
|
||||
export NODE_OPTIONS="--openssl-legacy-provider"
|
||||
|
||||
npm run docs:build || {
|
||||
log_error "Documentation build failed"
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ -d "$VUEPRESS_BUILD_DIR" ]]; then
|
||||
log_info "✓ Documentation built successfully"
|
||||
|
||||
# Copy to centralized output directory
|
||||
log_info "Copying site to: $SITE_OUTPUT_DIR"
|
||||
mkdir -p "$SITE_OUTPUT_DIR"
|
||||
rm -rf "${SITE_OUTPUT_DIR}"/*
|
||||
cp -r "${VUEPRESS_BUILD_DIR}"/* "${SITE_OUTPUT_DIR}/"
|
||||
|
||||
log_info " VuePress output: $VUEPRESS_BUILD_DIR"
|
||||
log_info " Final output: $SITE_OUTPUT_DIR"
|
||||
else
|
||||
log_error "Build output directory not found: $VUEPRESS_BUILD_DIR"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to preview documentation
|
||||
preview_docs() {
|
||||
log_info "Starting documentation preview..."
|
||||
|
||||
if [[ ! -d "$SITE_OUTPUT_DIR" ]]; then
|
||||
log_error "Build output not found. Run build first."
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Preview server starting at: http://localhost:8080"
|
||||
log_info "Press Ctrl+C to stop"
|
||||
|
||||
cd "$SITE_OUTPUT_DIR"
|
||||
python3 -m http.server 8080
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 [OPTIONS]
|
||||
|
||||
Build and deploy OpenMQTTGateway documentation site using VuePress.
|
||||
|
||||
This script handles the complete documentation build pipeline including:
|
||||
- Installing Node.js and Python dependencies
|
||||
- Downloading shared configuration from theengs.io
|
||||
- Version management (from git tags or custom version)
|
||||
- Generating board documentation from environments
|
||||
- Creating WebUploader manifest for firmware updates
|
||||
- Building VuePress static site
|
||||
- Optional local preview or deployment to GitHub Pages
|
||||
|
||||
OPTIONS:
|
||||
--mode MODE
|
||||
Build mode: 'prod' or 'dev' [default: prod]
|
||||
- prod: Production documentation with release version
|
||||
- dev: Development documentation with custom version tag
|
||||
|
||||
--url-prefix PATH
|
||||
URL prefix for documentation routing (VuePress base path) [default: /]
|
||||
Controls how URLs are generated in the built site.
|
||||
Examples:
|
||||
--url-prefix / # Production site at root URL
|
||||
--url-prefix /dev/ # Development site at /dev/ URL path
|
||||
Note: Should match --deploy-dir for correct link generation
|
||||
|
||||
--custom-version TAG
|
||||
Override version tag displayed in documentation
|
||||
Example: --custom-version "v1.8.0-beta"
|
||||
Note: Automatically sets --version-source to 'custom'
|
||||
|
||||
--version-source SOURCE
|
||||
Source for version information: 'release' or 'custom' [default: release]
|
||||
- release: Use latest git tag from repository
|
||||
- custom: Use version from --custom-version parameter
|
||||
|
||||
--preview
|
||||
Start local HTTP server to preview built documentation
|
||||
Server will run at http://localhost:8080
|
||||
Press Ctrl+C to stop the preview server
|
||||
|
||||
--deploy-dir DIR
|
||||
Deployment directory on GitHub Pages [default: .]
|
||||
Controls where files are copied in the gh-pages branch.
|
||||
Examples:
|
||||
--deploy-dir . # Deploy to root of gh-pages branch
|
||||
--deploy-dir dev # Deploy to dev/ folder in gh-pages
|
||||
Note: Should match --url-prefix for correct site structure
|
||||
|
||||
--no-webuploader
|
||||
Skip WebUploader manifest generation
|
||||
By default, generates manifest for web-based firmware updates
|
||||
|
||||
--webuploader-args ARGS
|
||||
Additional arguments passed to gen_wu.py script
|
||||
Example: --webuploader-args "--dev"
|
||||
|
||||
--run-pagespeed
|
||||
Run Google PageSpeed Insights after deployment
|
||||
Requires APIKEY to be configured in workflow
|
||||
|
||||
--pagespeed-url URL
|
||||
URL to test with PageSpeed Insights
|
||||
Default: https://docs.openmqttgateway.com/
|
||||
|
||||
--help
|
||||
Show this help message and exit
|
||||
|
||||
EXAMPLES:
|
||||
|
||||
# Build production documentation with latest release version
|
||||
$0 --mode prod
|
||||
|
||||
# Build and preview development documentation locally
|
||||
$0 --mode dev --url-prefix /dev/ --preview
|
||||
|
||||
# Build with custom version tag
|
||||
$0 --custom-version "v1.8.0-beta" --version-source custom
|
||||
|
||||
# Build without WebUploader manifest
|
||||
$0 --mode prod --no-webuploader
|
||||
|
||||
# Build for dev environment with custom WebUploader args
|
||||
$0 --mode dev --url-prefix /dev/ --deploy-dir dev --webuploader-args "--dev"
|
||||
|
||||
WORKFLOW:
|
||||
1. Check requirements (Node.js, npm, Python)
|
||||
2. Install dependencies (npm packages, Python libraries)
|
||||
3. Download common configuration from theengs.io
|
||||
4. Determine version (from git tags or custom)
|
||||
5. Generate board documentation from PlatformIO environments
|
||||
6. Generate WebUploader manifest (optional)
|
||||
7. Build VuePress static site
|
||||
8. Preview (optional)
|
||||
|
||||
OUTPUT:
|
||||
Built documentation will be in: generated/site/
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Parse command line arguments
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--mode)
|
||||
MODE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--url-prefix|--base-path)
|
||||
URL_PREFIX="$2"
|
||||
shift 2
|
||||
;;
|
||||
--deploy-dir|--destination-dir)
|
||||
DEPLOY_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--custom-version)
|
||||
CUSTOM_VERSION="$2"
|
||||
VERSION_SOURCE="custom"
|
||||
shift 2
|
||||
;;
|
||||
--version-source)
|
||||
VERSION_SOURCE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--preview)
|
||||
PREVIEW=true
|
||||
shift
|
||||
;;
|
||||
|
||||
--no-webuploader)
|
||||
GENERATE_WEBUPLOADER=false
|
||||
shift
|
||||
;;
|
||||
--webuploader-args)
|
||||
WEBUPLOADER_ARGS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--run-pagespeed)
|
||||
RUN_PAGESPEED=true
|
||||
shift
|
||||
;;
|
||||
--pagespeed-url)
|
||||
PAGESPEED_URL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
log_info "Starting site build pipeline..."
|
||||
log_info "Mode: $MODE"
|
||||
log_info "URL prefix: $URL_PREFIX"
|
||||
log_info "Deploy directory: $DEPLOY_DIR"
|
||||
|
||||
# Check requirements
|
||||
check_requirements || exit 1
|
||||
|
||||
# Install dependencies
|
||||
install_dependencies || exit 1
|
||||
|
||||
# Download common config
|
||||
download_common_config
|
||||
|
||||
# Get and set version
|
||||
local version
|
||||
version=$(get_version)
|
||||
set_version "$version"
|
||||
|
||||
# Set URL prefix
|
||||
set_url_prefix "$URL_PREFIX"
|
||||
|
||||
# Generate board documentation
|
||||
generate_board_docs
|
||||
|
||||
# Generate WebUploader manifest
|
||||
generate_webuploader
|
||||
|
||||
# Build documentation
|
||||
build_docs || exit 1
|
||||
|
||||
# Preview if requested
|
||||
if [[ "$PREVIEW" == true ]]; then
|
||||
preview_docs
|
||||
fi
|
||||
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
local duration=$((end_time - start_time))
|
||||
|
||||
echo ""
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ Site Build Summary ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo " Mode: $MODE"
|
||||
echo " Version: $version"
|
||||
echo " Duration: ${duration}s"
|
||||
echo " Output: $SITE_OUTPUT_DIR"
|
||||
echo " Status: SUCCESS ✓"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
}
|
||||
|
||||
# Execute main function
|
||||
main "$@"
|
||||
@@ -1,3 +1,5 @@
|
||||
# Common templates and constants for web installer manifest generation
|
||||
# Used by: scripts/gen_wu.py
|
||||
import string
|
||||
|
||||
mf_temp32 = string.Template('''{
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# Compresses firmware binaries with gzip for OTA updates during build
|
||||
# Used by: PlatformIO environments (optional, commented in environments.ini)
|
||||
import gzip
|
||||
import shutil
|
||||
import os
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# Creates web installer manifests for ESP Web Tools firmware installation
|
||||
# Used by: .github/workflows/task-docs.yml
|
||||
import os
|
||||
import requests
|
||||
import json
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# Generates board documentation table from platformio.ini environments
|
||||
# Used by: .github/workflows/task-docs.yml
|
||||
import pytablereader as ptr
|
||||
import pandas as pd
|
||||
import os
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
#!/bin/bash
|
||||
# Prepares firmware binaries and libraries for GitHub release deployment
|
||||
# Used by: .github/workflows/release.yml
|
||||
set -e
|
||||
echo "renaming bin files with the environment name"
|
||||
rename -v 's:/:-:g' .pio/build/*/*.bin
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
|
||||
# Replaces BLE library with custom version during PlatformIO build
|
||||
# Used by: Currently unused (utility script for BLE library replacement)
|
||||
import shutil
|
||||
import os
|
||||
import hashlib
|
||||
|
||||
Reference in New Issue
Block a user