Files
OpenMQTTGateway/scripts/ci_build.sh
Alessandro Staniscia 98481c5145 [SITE] Renew the web board presentation and the ESP32 web upload + [SYS] Security checks (#2277)
* Refactor GitHub Actions workflows for build, documentation, and linting

- Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows.
- Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity.
- Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions.
- Enhanced `release.yml` to streamline the release process and integrate documentation generation.
- Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository.
- Simplified manual documentation workflow by leveraging the new reusable documentation workflow.
- Improved artifact management and retention policies across workflows.
- Updated dependencies and versions in workflows to ensure compatibility and performance.

CI/CD pipeline agnostic of Workflow Engine and integrated on github actions

- Implemented ci.sh for orchestrating the complete build pipeline.
- Created ci_00_config.sh for centralized configuration of build scripts.
- Created ci_build_firmware.sh for building firmware for specified PlatformIO environments.
- Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment.
- Created ci_set_version.sh for updating version tags in firmware configuration files.
- Created ci_build.sh to orchestrate the complete build pipeline.
- Created ci_qa.sh for code linting and formatting checks using clang-format.
- Created ci_site.sh for building and deploying VuePress documentation with version management.
- Implemented checks for required tools and dependencies in the new scripts.
- Improved internal scripts for better error handling and logging.

UPDATE the web installer manifest generation and update documentation structure
- Enhanced ci_list-env.sh to list environments from a JSON file.
- Replaced  common_wu.py and gen_wu.py scripts with new npm scripts for site generation and previewing on docsgen/gen_wu.js
- Replaced  generate_board_docs.py with docsgen/generated_board_docs.js
- Added new npm scripts for integration of site generation on build phase.
- Created preview_site.js to serve locally generated site over HTTPS with improved error handling.
- Added new CI environments for CI builds in environments.json.
- Deleted lint.yml as part of workflow cleanup.
- Enhanced task-build.yml to include linting as a job and added support for specifying PlatformIO version.
- Improved task-docs.yml to handle versioning more effectively and added clean option.

Enhance documentation
- ADD CLEAR Mark of development version of site
- Updated README.md to include detailed workflow dependencies and relationships using mermaid diagrams.
- Improved development.md with a quick checklist for contributors and clarified the code style guide.
- Enhanced quick_start.md with tips for contributors and streamlined the workflow explanation.

LINT FIX
- Refined User_config.h for better formatting consistency.
- Adjusted blufi.cpp and gatewayBT.cpp for improved code readability and consistency in formatting.
- Updated gatewaySERIAL.cpp and mqttDiscovery.cpp to enhance logging error messages.
- Improved sensorDS1820.cpp for better logging of device information.

Add security scan workflows for vulnerability detection

Add SBOM generation and upload to release workflow; update security scan summary handling

Add shellcheck suppor + FIX shellcheck warning

Enhance documentation for CI/CD scripts and workflows, adding details for security scanning and SBOM generation processes

Fix formatting and alignment in BLE connection handling

Reviewed the full web board presentation and the ESP32 web upload. The project uses a modern pattern where data is divided from the presentation layer.

- Removed the `generate_board_docs` script.
- Updated the `gen_wu` script in order to generate `boards-info.json`: the fail that containe all information about the configuration
- Created and isolate the file `boards-info.js` to streamline the parsing of PlatformIO dependencies, modules, environments and improve the handling of library information.
- Introduced vuepress component `BoardEnvironmentTable.vue` that render `boards-info.json` as UI card component
- Introduced vuepress component `FlashEnvironmentSelector.vue` that render a selectred environment from  `boards-info.json` and provide esp-web-upload feature on it
- Introduced a new board page `board-selector.md` for improved firmware selection.
- Updated `web-install.md` to enhance the firmware upload process, including a new board environment table.
- Enhanced custom descriptions in `environments.ini` to include HTML links for better user guidance and board image link

Add CC1101 initialization improvements and logging enhancements
Add installation step for PlatformIO dependencies in documentation workflow

Remove ci_set_version.sh script and associated versioning functionality

* Fix comment provisined

Fix PlatformIO version input reference in documentation workflow

Remove outdated Squeezelite-ESP32 installer documentation
2026-03-09 07:47:30 -05:00

360 lines
11 KiB
Bash
Executable File

#!/bin/bash
# CI/CD agnostic wrapper for complete build pipeline
# Orchestrates all build scripts with a single command
# Usage: ./scripts/ci.sh <environment> [OPTIONS]
set -euo pipefail
# Constants
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
source "${SCRIPT_DIR}/ci_00_config.sh"
else
echo "ERROR: ci_00_config.sh not found" >&2
exit 1
fi
# Function to print banner
print_banner() {
echo "╔════════════════════════════════════════╗"
echo "║ OpenMQTTGateway CI/CD Build ║"
echo "╚════════════════════════════════════════╝"
echo ""
}
# Function to print summary
print_summary() {
local env="$1"
local version="$2"
local start_time="$3"
local end_time
end_time=$(date +%s)
local duration=$((end_time - start_time))
echo ""
echo "╔════════════════════════════════════════╗"
echo "║ Build Summary ║"
echo "╚════════════════════════════════════════╝"
echo " Environment: $env"
echo " Version: $version"
echo " Duration: ${duration}s"
echo " Status: SUCCESS ✓"
echo "╚════════════════════════════════════════╝"
}
# Function to check if command exists
command_exists() {
command -v "$1" >/dev/null 2>&1
}
# Function to get command version
get_command_version() {
local cmd="$1"
case "$cmd" in
platformio)
platformio --version 2>&1 | head -n1 | grep -oP '\d+\.\d+\.\d+' || echo "unknown"
;;
python|python3)
python3 --version 2>&1 | grep -oP '\d+\.\d+' || echo "unknown"
;;
*)
echo "unknown"
;;
esac
}
# Function to verify required tools
verify_build_tools() {
log_info "Verifying required build tools..."
local missing_tools=()
# Check Python
if ! command_exists python3; then
missing_tools+=("python3")
else
local python_version
python_version=$(get_command_version python3)
log_info "✓ Python ${python_version} found"
fi
# Check PlatformIO
if ! command_exists platformio; then
missing_tools+=("platformio")
else
local pio_version
pio_version=$(get_command_version platformio)
log_info "✓ PlatformIO ${pio_version} found"
fi
# Check git (for version auto-generation)
if ! command_exists git; then
log_warn "git not found (optional, but recommended)"
else
log_info "✓ git found"
fi
# Report missing tools
if [[ ${#missing_tools[@]} -gt 0 ]]; then
log_error "Missing required tools: ${missing_tools[*]}"
log_error ""
log_error "Please install missing tools:"
for tool in "${missing_tools[@]}"; do
case "$tool" in
python3)
log_error " - Python 3: https://www.python.org/downloads/"
;;
platformio)
log_error " - PlatformIO: pip3 install platformio"
log_error " or: pip3 install ${PLATFORMIO_VERSION:-platformio}"
;;
esac
done
log_error ""
log_error "Or skip this check with: --skip-verification"
return 1
fi
log_info "All required tools are available"
return 0
}
# Function to cleanup on error
cleanup_on_error() {
log_error "Build failed, cleaning up..."
# Restore any backups
find . -name "*.bak" -type f -exec bash -c 'mv "$1" "${1%.bak}"' _ {} \; 2>/dev/null || true
}
# Show usage
usage() {
cat << EOF
Usage: $0 <environment> [OPTIONS]
Complete CI/CD build pipeline wrapper.
Arguments:
environment PlatformIO environment name
Options:
--mode MODE Build mode: 'prod' or 'dev' [default: prod]
'dev' enables OTA and development features
--deploy-ready Prepare for deployment (renamed artifacts)
--version [TAG] Set version tag (if TAG omitted, auto-generated)
--output DIR Output directory for artifacts [default: generated/artifacts/]
--skip-verification Skip build tools verification
--clean Clean build before starting
--verbose Enable verbose output
--help Show this help message
Environment Variables:
CI Set to 'true' in CI/CD environments
BUILD_NUMBER Build number from CI/CD
GIT_COMMIT Git commit hash for versioning
Examples:
# Local development build
$0 esp32dev-all-test --mode dev
# Production release build
$0 esp32dev-bt --version v1.7.0 --mode prod --deploy-ready
# CI/CD build (auto-detects version)
$0 theengs-bridge --version --mode dev
EOF
}
# Main pipeline
main() {
local environment=""
local version=""
local set_version=false
local mode=""
local prepare_for_deploy=false
local output_dir=""
local skip_verification=false
local clean=false
local verbose=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--version)
set_version=true
# Check if next argument is a version tag or another option
if [[ $# -gt 1 && ! "$2" =~ ^-- ]]; then
version="$2"
shift 2
else
shift
fi
;;
--mode)
if [[ $# -lt 2 ]]; then
log_error "--mode requires an argument: 'prod' or 'dev'"
usage
exit 1
fi
if [[ "$2" != "prod" && "$2" != "dev" ]]; then
log_error "Invalid mode: $2. Must be 'prod' or 'dev'"
usage
exit 1
fi
mode="$2"
shift 2
;;
--deploy-ready)
prepare_for_deploy=true
shift
;;
--output)
if [[ $# -lt 2 ]]; then
log_error "--output requires a directory argument"
usage
exit 1
fi
output_dir="$2"
shift 2
;;
--skip-verification)
skip_verification=true
shift
;;
--clean)
clean=true
shift
;;
--verbose)
verbose=true
shift
;;
--help|-h)
usage
exit 0
;;
-*)
log_error "Unknown option: $1"
usage
exit 1
;;
*)
environment="$1"
shift
;;
esac
done
# Validate environment
if [[ -z "$environment" ]]; then
log_error "Environment name is required"
usage
exit 1
fi
# Set default mode if not specified
if [[ -z "$mode" ]]; then
mode="prod"
log_info "Mode not specified, defaulting to production"
fi
# Auto-generate version if --version flag is set but no tag provided
if [[ "$set_version" == "true" && -z "$version" ]]; then
if [[ "${CI:-false}" == "true" ]]; then
# CI/CD environment
version="${BUILD_NUMBER:-${GIT_COMMIT:-unknown}}"
else
# Local development
version="local-$(date +%Y%m%d-%H%M%S)"
fi
log_info "Auto-generated version: $version"
fi
# Setup error handling
trap cleanup_on_error ERR
# Change to project root
cd "$PROJECT_ROOT"
# Start timer
local start_time
start_time=$(date +%s)
# Print banner
print_banner
# Step 1: Verify build tools
if [[ "$skip_verification" == "false" ]]; then
log_step "1/4 Verifying build tools..."
verify_build_tools || exit 1
echo ""
else
log_warn "Skipping build tools verification (--skip-verification)"
echo ""
fi
# Step 2: Set version
# not required
# Step 3: Build firmware
log_step "3/4 Building firmware for: $environment"
local build_opts=()
[[ "$mode" == "dev" ]] && build_opts+=(--dev-ota)
[[ "$clean" == "true" ]] && build_opts+=(--clean)
[[ "$verbose" == "true" ]] && build_opts+=(--verbose)
[[ "$set_version" == "true" ]] && build_opts+=(--version "$version")
"${SCRIPT_DIR}/ci_build_firmware.sh" "$environment" "${build_opts[@]}" || exit 1
echo ""
# Step 4: Prepare artifacts
log_step "4/4 Preparing artifacts..."
if [[ "$prepare_for_deploy" == "true" ]]; then
log_info "Preparing artifacts for deployment"
local artifact_opts=()
[[ "$clean" == "true" ]] && artifact_opts+=(--clean)
[[ -n "$output_dir" ]] && artifact_opts+=(--output "$output_dir")
[[ "$set_version" == "true" ]] && artifact_opts+=(--version "$version")
"${SCRIPT_DIR}/ci_prepare_artifacts.sh" "$environment" "${artifact_opts[@]}" || exit 1
echo ""
# Check if site folder exists and copy built files to avoid rebuilding the site
if [[ "$mode" == "dev" ]]; then
local site_dir="${PROJECT_ROOT}/generated/site/dev"
local artifacts_dir="${output_dir:-${PROJECT_ROOT}/generated/artifacts/firmware_build}"
if [[ -d "$site_dir" ]]; then
log_info "Site folder exists, copying built firmware files to site/dev..."
# Copy firmware files for the current environment
for file in "${artifacts_dir}/${environment}"-*.bin "${artifacts_dir}/${environment}"-*.tgz; do
if [[ -f "$file" ]]; then
cp -v "$file" "$site_dir/" || log_warn "Failed to copy $(basename "$file")"
fi
done
log_info "✓ Firmware files copied to site/dev (no site rebuild needed)"
else
log_warn "Site folder not found at: $site_dir"
log_info "Run 'ci.sh site ' to generate it"
fi
fi
fi
# Print summary
print_summary "$environment" "$version" "$start_time"
log_info "✓ Complete build pipeline finished successfully"
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi