#devops

Tempo 3.0 release: a new architecture for scale and lower TCO, TraceQL metrics GA, and more

Tempo 3.0, phiên bản mới của hệ thống truy vết phân tán mã nguồn mở, giới thiệu kiến trúc tương thích Kafka cho microservices, tách biệt đường đọc-ghi, giảm yêu cầu sao chép RF3 xuống RF1, và thay thế ingesters/compactors bằng block-builders, live-stores cùng scheduler. Tính năng TraceQL metrics giờ đã sẵn sàng, hỗ trợ truy vấn metric trực tiếp từ trace data cùng toán tử so sánh mới, cùng nhiều cải tiến khác như giới hạn cardinality theo label, tối ưu truy vấn TraceQL AST, và công cụ di chuyển từ phiên bản 2.x.

Lập trình viên phát triển ứng dụng microservices nên đọc vì Tempo 3.0 mang đến kiến trúc Kafka-compatible cải tiến, giúp tối ưu hóa quy mô, giảm chi phí vận hành và cung cấp công cụ TraceQL mạnh mẽ để phân tích hiệu suất trực tiếp từ dữ liệu theo dõi phân tán.

The inside scoop on alerting changes in Kubernetes Monitoring

Grafana Cloud's Kubernetes Monitoring có hai hệ thống cảnh báo riêng biệt: cảnh báo quản lý bởi data source (Mimir/Prometheus) và cảnh báo quản lý bởi Grafana. Việc cài đặt lại app sẽ tự động chuyển quy tắc cảnh báo sang hệ thống Grafana, có thể làm gián đoạn các tuyến thông báo đã cấu hình trong Alertmanager. Bài viết hướng dẫn cách nhận diện hệ thống cảnh báo đang sử dụng, nguyên nhân ngừng hoạt động sau khi cài đặt lại, và các phương pháp tốt nhất như sử dụng nút Update thay vì cài đặt lại, sao lưu quy tắc tùy chỉnh trước khi nâng cấp, và lưu ý rằng cảnh báo quản lý bởi data source (Prometheus/Loki) sẽ ngừng hoạt động từ tháng 4/2026.

Lập trình viên cần đọc bài này để tránh mất hiệu suất cảnh báo trong Kubernetes khi tái cài đặt Grafana Cloud, vì nó có thể phá hủy cấu hình thông báo hiện có và cảnh báo cũ sẽ chuyển sang hệ thống quản lý mới, gây mất liên lạc với các hệ thống cảnh báo bên ngoài.

A revamped way to create and manage alerts across all your telemetry

Last9 đã tái thiết hệ thống cảnh báo từ đầu, với trình chỉnh sửa mới hợp nhất việc tạo cảnh báo cho metrics và logs, hỗ trợ PromQL/LogQL cùng Builder không code, cùng chế độ xem trước trực tiếp. Bản cập nhật bổ sung trang Rules mới, Alert Monitor, cài đặt cảnh báo theo nhóm, khả năng kiểm tra kênh thông báo trước khi triển khai, cùng cải tiến dashboard (tải CSV, panel markdown, hỗ trợ Terraform) và sửa lỗi trên mobile SDK.

Lập trình viên nên đọc bài này vì giải pháp mới của Last9 giúp tối ưu hóa quản lý cảnh báo từ dữ liệu telemetry một cách hiệu quả hơn, với công cụ trực quan và tích hợp no-code, giúp phát triển và kiểm soát quy trình theo dõi lỗi và cảnh báo nhanh chóng hơn.

Mastering Secure CI/CD for ECS with GitHub Actions

Bài viết hướng dẫn xây dựng quy trình CI/CD an toàn cho ECS trên Fargate bằng GitHub Actions, cải tiến vận hành như hợp nhất workflow, dùng commit hash làm tag ảnh container, quản lý định nghĩa task và service với ecspresso. Ngoài ra, bài cũng đề cập đến biện pháp bảo mật như AssumeRole không dùng key với OpenID Connect, quét lỗ hổng bằng Trivy và Dockle, cùng multi-stage build để giảm bề mặt tấn công.

Lập trình viên cần đọc bài này để tìm hiểu cách xây dựng và tối ưu hóa một chu trình CI/CD an toàn trên AWS ECS Fargate bằng GitHub Actions, từ việc quản lý mã nguồn đến bảo mật container và giảm diện tích tấn công.

June 23, 2026

A brief explanation of the Microsoft internal term 'escrow' as it relates to software releases, noting that it is essentially the same as a release candidate but the different name serves a psychological purpose.

Automatically enrich security logs with MITRE ATT&CK context before they reach your SIEM

Datadog's Observability Pipelines now includes MITRE ATT&CK Enrichment Packs — preconfigured mappings that automatically tag security logs with relevant ATT&CK tactics and techniques before they reach a SIEM, data lake, or archive. The initial release covers six sources: Okta, Palo Alto, FortiGate, AWS WAF, CloudTrail, and Windows endpoints. By enriching logs in the pipeline rather than at query time, security teams can write detection rules against standardized MITRE fields instead of vendor-specific event names, investigate incidents across sources with a shared taxonomy, and route consistently tagged events to any destination including Splunk, Microsoft Sentinel, Datadog Cloud SIEM, or ClickHouse. The packs are included with Observability Pipelines at no extra cost.

Undo Enables AI Agents to Diagnose Root Cause of Application Issues

Undo has launched an MCP server integration that lets AI agents access its application recording platform to diagnose root causes of bugs. The platform captures complete program execution — every instruction, variable, thread event, and system call — giving AI agents ground-truth runtime evidence rather than relying solely on logs or static code analysis. This enables agents to resolve intermittent failures and state-dependent bugs in complex multithreaded systems that would otherwise take weeks to debug. The capability is positioned as critical as AI-generated code volumes outpace human review capacity, and as adversaries increasingly use AI to find and exploit vulnerabilities.

Log management for AI workloads: How to bring your logs and telemetry plan into the AI-first century

AI workloads are overwhelming traditional log management systems, exposing limits in how teams capture, store, and correlate telemetry. A five-point action plan covers: unifying all telemetry signals on a single platform, correlating logs with traces for causation, controlling costs by eliminating rigid schemas and rehydration overhead, standardizing instrumentation at ingest, and enabling preventive operations through real-time contextual analytics. Key stats from a 2026 survey of 450 IT leaders highlight that 85% of organizations struggle to ingest logs at AI scale, 74% cite indexing and rehydration costs as barriers, and 84% say customer trust in AI depends on predictive log analytics.

How to migrate feature flags without breaking production

Feature flag migrations often stall due to three core challenges: evaluation logic parity between systems, configuration synchronization during the migration window, and confirming cutover safety. Running two systems in parallel is safe and enables incremental migration. The recommended approach starts with auditing flags into zombie, short-lived, and long-lived categories, then redirecting all new flags to the new system, letting short-lived flags expire naturally, and carefully cutting over long-lived flags. To verify parity before cutover, teams can use a wrapper function that evaluates both systems asynchronously and logs discrepancies, or use OpenFeature's ComparisonStrategy for shadow mode evaluation. Mismatch metrics should be tracked on a dashboard showing evaluation counts, mismatch rate trends, and timestamps. A brief configuration freeze of 24–48 hours before cutover ensures the validated state matches production.

How Enterprise Teams Manage Infrastructure at Scale with Terraform

Large-scale Terraform usage introduces challenges that solo tutorials never cover. This deep-dive explains how enterprise teams handle state corruption (concurrent applies, interrupted pipelines, wrong-environment commands, dual ownership), why state files are treated like production databases containing plaintext secrets, and how repositories and state files are structured around team ownership boundaries. It covers module versioning with semantic versioning to prevent company-wide pipeline breakages, sharing infrastructure data between state files via remote state vs. cloud data sources, the full PR-to-production pipeline workflow including CODEOWNERS enforcement, scheduled drift detection, and a step-by-step state recovery playbook including backups, S3 versioning restores, force-unlock, and re-import.

It’s Only When You Look Back

A developer reflects on 40 years of computing and 25 years of maintaining a personal website, tracing the evolution from ZX Spectrum and BBS systems through early Linux, LAMP stacks, Solaris, virtualization, Docker, Kubernetes, and modern cloud-native infrastructure. The post serves as a personal time capsule covering tech trends, tools, and communities that have come and gone, from hand-crafted HTML and Dreamweaver to GitOps and Terraform pipelines.

Query logs and traces with SQL in Observability Analytics

Google Cloud has rebranded Log Analytics as Observability Analytics, adding general availability for trace data querying and a new Observability API. The platform lets developers and SREs run SQL queries that JOIN logs and distributed trace spans in a single place, powered by BigQuery under the hood. Key use cases include analyzing AI agent tool failure rates and P95 latency across millions of spans, correlating trace data with application logs to identify which customers experience the worst performance, and joining observability data with business metrics in BigQuery. The Observability API enables programmatic management of observability buckets and allows AI agents or analytical workloads to query telemetry directly via BigQuery APIs. Concrete SQL query examples are provided for both AI agent optimization and customer latency analysis scenarios.

AWS hypes continuous agentic DevOps, puts Kiro in your pocket

AWS announced several new and enhanced AI-driven DevOps and security tools at its New York Summit. AWS Continuum, now in closed preview, provides continuous vulnerability scanning and penetration testing for AWS environments, with AI-generated fix suggestions. The DevOps Agent gains release management capabilities and now exposes its own MCP endpoint plus Agent2Agent (A2A) protocol support, enabling integration with observability tools like Datadog, Dynatrace, and New Relic. Kiro, the specification-driven AI coding IDE, is getting a native iOS mobile app for managing remote sessions. AWS Transform adds continuous modernization for Java and .NET workloads. Amazon Bedrock AgentCore gains managed knowledge bases, web search, and policy/guardrail features to improve agent reliability and trust. Pricing for these agentic services is noted as opaque and potentially costly, with per-second usage billing and additional charges for consumed AWS services.

Using Evaluation Frameworks with Agent Observability

Datadog Agent Observability now supports running DeepEval and Pydantic Evals evaluation frameworks natively within Datadog Experiments, eliminating the need to rewrite existing evaluators or adopt proprietary metric definitions. The integration lets teams define datasets, configure existing evaluators without modification, and run experiments that automatically link eval scores to production traces, token usage, and latency data. Teams can also run evaluations continuously on sampled production traffic rather than only as a pre-deployment CI gate, enabling real-time quality regression detection across the full development and deployment lifecycle.

Tsuga raises $35m to keep AI-era observability inside the customer’s own cloud

Paris-based startup Tsuga has raised a $35m Series A led by Singular, bringing its total funding to ~$45m just six months after emerging from stealth. Founded by two ex-Datadog engineers, Tsuga challenges the traditional observability model by deploying inside the customer's own cloud rather than ingesting telemetry into a vendor-controlled environment. This eliminates per-byte pricing — a growing pain as AI agent workloads generate massive telemetry volumes — and keeps sensitive data within the customer's security and regulatory perimeter. The platform offers automated root-cause analysis on unsampled data, an MCP server, and a CLI for building custom agents. A Databricks partnership allows routing observability data directly into Databricks for further analysis. Early customers include Le Monde, Camunda, and Black Forest Labs, with several million dollars in ARR and six-figure average contract values reported.

Building Jaeger’s ClickHouse backend: 8.6× compression on 10 million spans

Jaeger v2.18.0 introduces ClickHouse as a storage backend for distributed traces. A Jaeger maintainer explains the architectural decisions behind the implementation, including schema design choices for the primary key (sorted by service_name, name, start_time rather than trace_id), typed attribute storage using ClickHouse Nested columns, and materialized views for fast service/operation lookups. Benchmarks on 10 million spans across a single-node deployment show 8.6× compression (reducing ~6 GiB to ~722 MiB), ingestion throughput above 50k spans/sec, trace retrieval averaging ~100ms, and most search queries completing under 50ms. The feature is available in alpha and leverages ClickHouse's columnar OLAP architecture to handle high-throughput append-only writes and fast analytical aggregations without a separate metrics pipeline.

Deprecating OpenCensus compatibility requirements

OpenTelemetry has officially deprecated OpenCensus compatibility requirements in its specification, effective June 2026, following the archival of OpenCensus in July 2023. New SDKs and implementations are no longer required to implement OpenCensus compatibility. Existing shims will continue to receive maintenance for at least one year, with the earliest possible specification removal set for June 2027. Users still relying on OpenCensus shims are advised to begin planning migration to native OpenTelemetry APIs and OTLP-based workflows. Language-specific migration resources are available for Go, Java, JavaScript, and Python.